I did something I usually try hard not to do in a field test. I changed the design after it had already started.
Halfway through validating AdversarialDebate, I realized the model set was too narrow to answer the most important question in the project. So I added a fourth model in the middle of the run.
That was messy. It wasted work. It made the corpus inconsistent for a while. It also turned out to be one of the best decisions in the whole release.
This post is about a lesson I trust far more now than I did before building this project: a field test is not just there to produce numbers. It is there to reveal whether your experiment can actually answer the question you think it is answering.
I began with three models: GPT-4o-mini, Gemini 2.5 Flash, and DeepSeek-V3. That gave me three useful pairings — GPT + Gemini, Gemini + DeepSeek, and GPT + GPT as a homogeneous control. Three labs, two regions, one same-model control. Reasonable spread.
| Pair | Small-corpus score | Verdict rate | |---|---:|---:| | Gemini + DeepSeek | 0.835 | 33% | | GPT + GPT | 0.667 | 33% | | GPT + Gemini | 0.148 | 0% |
The diverse pair was ahead. The weak pair was struggling. The homogeneous control was doing something interesting. If I had stopped there, I would have told a clean story — and it would have been the wrong one.
The issue was not that the first three models were bad. The issue was that the experiment could only see part of the diversity spectrum. With those three models, the farthest useful pairing I had was US + China. I did not have a genuinely cross-continent pair that could show what happened at the far end of diversity.
The test could suggest whether diversity helped. It could not show whether maximum diversity behaved differently from moderate diversity. That is a major blind spot when the whole thesis is about pairing behavior.
I needed a fourth model — not because more models always make a benchmark better, but because this specific thesis needed one more corner of the matrix.
I added Mistral Small 3.2 because it gave me three new pairings instantly: GPT + Mistral, Gemini + Mistral, and DeepSeek + Mistral. That last one was the important one. DeepSeek + Mistral created the strongest diversity pairing in the run: China + EU.
Once that pairing existed, the experiment could finally observe a fuller spectrum: homogeneous, weak diversity, moderate diversity, strong diversity. That made the field test qualitatively better, not just quantitatively bigger.
| Metric | 3-model setup | 4-model setup | |---|---|---| | Pairs | 3 | 6 | | Corpus signal | partial | broad enough to compare the full spectrum | | Best pair | Gemini + DeepSeek (0.835) | DeepSeek + Mistral (0.982) | | Cross-continent pair | none | DeepSeek + Mistral | | Capitulation visible | barely | clearly visible | | Weak-diversity valley visible | not really | yes |
Without Mistral, the story would have been: diversity helps, US + China looks strongest, homogeneous is interesting but secondary.
With Mistral, the actual story became: diversity helps but not monotonically, weak diversity can be worse than no diversity, maximum diversity can create capitulation cascades, and the highest-scoring pair is not automatically the safest pair.
What The Fourth Model Revealed The strongest pair was stronger than expected. DeepSeek + Mistral dominated the full run: 0.982 average score, 97% verdict rate, 2,352 concessions. Without the fourth model, that pairing never exists, and the thesis looks weaker than it really is. Maximum diversity had a failure mode. The same pair that looked best on paper also had the most dangerous behavior: 44 capitulation cascades, 65% capitulation rate within the pair. The fourth model did not just strengthen the thesis — it exposed the limit of the thesis. Without it, the release would have shipped a much cleaner and much less honest story. The homogeneous control became more meaningful. Once the spectrum widened, GPT + GPT stopped looking like a curiosity and started looking like evidence that weak diversity was a specific failure mode, not just a weak result.
| Pair type | Example | Avg score | |---|---|---:| | Homogeneous | GPT + GPT | 0.688 | | Weak diversity | GPT + Gemini | 0.357 | | Moderate diversity | GPT + Mistral | 0.754 | | Strong diversity | DeepSeek + Mistral | 0.982 | The field test stopped being a pipeline check and started being a product test. Three models were enough to validate that the system ran. Four models were what made it possible to learn something product-level about pairing strategy. That is the difference that matters.
I do not want to romanticize this. Adding a model after the run started created real waste. By the time Mistral was added, GPT, Gemini, and DeepSeek had each already reviewed about 148 PRs. Mistral started late and only reviewed about 70 to 73 PRs. The Mistral-containing pairs could only use the 70 PRs that all four models had in common.