Twenty-five tasks could not tell any of nine AI models apart

On our own bench of nine language models, run on 25 checkable tasks, not one pair could be statistically distinguished from another using a sign test at the 5% level[1]. Even the closest case (gpt-4.1-mini losing 3 tasks to each top scorer, winning none) gave p = 0.25, far from significant[1]. To tell models apart reliably, you need more tasks: at least 150 for a 4-point gap, in the best case[1].

What the test found: no model pairs stood apart on 25 tasks

Running nine models through 25 checkable tasks (reasoning, extraction, dates and units, instructions, knowledge) at temperature 0, we found five models answered all 25 correctly, two missed one, one missed two, and one missed three[1]. With 36 possible model pairs, none differed at the 5% significance level on a sign test over these tasks[1]. The closest pair (gpt-4.1-mini vs the perfect scorers) lost 3 tasks and won none, which translates to a p-value of 0.25; far above the 0.05 threshold for significance[1].

Repeating each run three times did not help: no run disagreed on whether the answer was right, so repeating did not add information[1]. For this task set, you cannot statistically tell these models apart on 25 questions, even when a model misses up to three that others get right[1].

Table: Score distribution and pairwise differences

Based on [1], here is what the scores looked like:

No two models were statistically distinguishable on this set[1].

How many tasks does it take to see a difference?

To reach significance at the 5% level with a sign test, the weaker model must lose at least six tasks that the stronger model gets right (and win none back)[1]. This sets a hard lower bound on the number of tasks needed to detect a given gap.

The best-case formula: Tasks needed = 6 / (gap as share of tasks)

The table below gives the best case for each gap.

Each task the weaker model wins back raises the number needed[1]. This assumes tasks are independent, and the weaker model always loses to the stronger one on the differing tasks.

Table: Tasks needed to detect a gap (best case)

*Assumes weaker model never wins a task back; real task sets may need more[1].

Why repeating runs doesn't help when answers are deterministic

In this setup (API at temperature 0), no model-and-task pair had runs that disagreed on whether the answer was right: 0 of 225[1]. This means repeating the exact same run cannot reveal more about model differences; you only learn more by adding new tasks[1].

What changes if your tasks or models are different

These results are for one bench: 25 checkable tasks, nine models from a single provider, strict programmatic scoring, and temperature 0[1]. If your tasks are less clear-cut, or if scoring requires human or model judgment, more tasks may be needed—or the tests may become noisier, changing the numbers[1].

How the numbers work

Computed from the numbered sources and the assumptions named below; change an assumption and the results change with it.

Sources

1. Our own data, 8 Nov 2026: Nine models on 25 checkable tasks: which pairs can be told apart
2. Power (statistics) - Wikipedia)