Same coding model, same benchmark, different score: what should you compare?

Separate the reported evaluation contexts

Mistral released Large 4 weights on October 29, 2026. Fortune reports a Terminal-Bench 4.0 result of 41%, up from 28% for the preview, attributing the figures to Mistral. VentureBeat separately reports 25% in Artificial Analysis's Intelligence Index evaluation at high reasoning. The reports concern different evaluation contexts. Their available details do not isolate the contribution of each configuration difference.^[1][2][3]

The difference does not establish that either evaluator is wrong, or identify the harness as the cause. Preserve who supplied each figure. Also distinguish the preview from the released checkpoint: the sources describe additional reinforcement learning after the preview. Reusing the family name can conceal a model change before you even reach the agent settings.^[1][2][3]

Keep score families separate. Artificial Analysis's Intelligence Index is a broad capability composite. Its Coding Agent Index combines coding evaluations and pairs models with agent software. A Terminal-Bench percentage is an individual benchmark result. Those three quantities answer different questions, even when a news article discusses all of them together. Record the measure's full name and unit before copying its value.^[2][4]

Fill the comparison worksheet without guessing

For each published score, record checkpoint or revision, reasoning setting, harness and version, benchmark and version, task sample, tool permissions, execution limits, attempt budget, scoring unit, measurement date and evaluator. Keep publication date separate when the experiment date is unavailable. Mark missing fields unknown. This worksheet is an editorial method for inspecting a comparison, not a claim that those fields explain a particular score difference.^[2][4]

A filled row for the Fortune figure starts: Terminal-Bench 4.0; 41%; post-preview Large 4 release; vendor-provided result; reported October 29. Harness version, task sample, attempt budget and experiment date remain unknown from that article. A second row records VentureBeat's reported 25% under the AA Intelligence Index context and high reasoning. Do not import a harness from a different benchmark to fill either row.^[2][3]

Unknown cells limit attribution. A comparison can still help someone choose a complete configured agent, provided its task and operating constraints match their needs. It cannot automatically establish that the underlying model improved. Before accepting a model-specific explanation, identify which components changed and which the source actually held fixed.^[2]

Classify three fictional comparison cards

These cards are invented for a static reasoning exercise and contain no benchmark results. Card A keeps the benchmark version, tasks, harness, tools, attempt budget and reasoning setting fixed; only the model checkpoint changes. Card B keeps the checkpoint fixed while changing the harness and tool budget. Card C compares the same model's old Intelligence Index score with its score on a revised index. Choose the narrowest conclusion for each before continuing.^[2][4]

Card A supports a bounded checkpoint comparison inside that harness and task set. It does not establish a universal model ranking. Card B compares configured systems; it cannot isolate a model effect because other components changed. Card C cannot establish a rise or decline in capability from the score difference alone. Its measurement scale changed.^[2][4]

Card C has a current real-world reason. On October 27, Artificial Analysis introduced Intelligence Index v5 with new and upgraded benchmarks, alongside a separate Progress Index intended for comparisons over time on a continuous scale. An old composite value and a revised value use different benchmark compositions and cannot alone establish a capability change. Check the method of a longitudinal measure before using it.^[4]

Choose the next comparison you actually need

If your decision concerns a model swap, propose a small matched trial: one permitted toy task, fixed harness version, identical tools, runtime, attempts and reasoning policy. Define correctness checks before the runs and retain failures as well as successes. This is a proposed follow-up, not an executed benchmark. One toy task will not establish performance across large repositories. If instead you intend to adopt a complete agent, compare complete systems under the constraints you would actually use.^[2][4]

Save the worksheet with the source links and unresolved cells. When a new score arrives, ask whether it fills a missing condition or changes the comparison unit. That gives you a specific information request before a download, paid evaluation or product choice, without treating a vendor chart or an independent composite as evidence of performance across all tasks.^[1][2][4]