評価 Set Aging / Refresh / Contamination

Benchmark scores are versioned observations, not timeless model properties. Enterprise evaluation records should preserve the exact benchmark release, item-set identity or lawful hash, scorer and harness version, model/revision, tool or web policy, inference settings, run date, and environment.

Public exposure creates contamination risk, but it does not prove that a particular model is contaminated. Benchmark maintainers can also repair defective items, change scoring, alter refresh cadence, or revise harnesses after release. Those changes can break direct longitudinal comparability even when the model is unchanged.

FrontierMath v2 is a concrete example: its maintainer reported that the v2 release addressed errors affecting 42% of problems. LiveBench likewise uses explicit release lineage and refreshes its question set, while its original paper and current operating cadence differ. These are reasons to bind every score to the benchmark version actually used rather than silently replacing historical results.

Treat item correction, scoring-rule change, harness/runtime change, tool or web policy change, model revision, and inference-compute change as comparison boundaries unless a bridge study demonstrates equivalence. Never overwrite an old score when a benchmark refreshes. Create a new evaluation revision and retain the old run with its original provenance.

When licensing permits, a bridge study should run old and new sets under the same model, harness, and settings; report paired unchanged items separately from new or repaired items; record exclusions and reasons; and avoid collapsing non-equivalent releases into one trend line.

A public benchmark implies an opportunity for exposure. A model-specific contamination claim requires additional evidence such as training-data evidence, controlled canaries, memorization or overlap tests, or another specific method. Score movement after a refresh is not diagnostic: it can result from contamination removal, difficulty shift, item repair, scorer changes, or harness changes.

Fresh, private, rephrased, synthetic, and dynamically generated items can reduce some exposure or aging risks, but none is a universal contamination cure. Refresh methods can preserve leakage or change task fidelity, so their own version and generation policy belong in the run record.

Do not compare mixed benchmark releases without stratification. Other invalidating conditions include silent item replacement, scorer or harness changes treated as model gains or losses, benchmark saturation, leaked solutions, broken or ambiguous items, hidden tool/web-access differences, inference-compute changes, post-release benchmark edits, and non-equivalent dynamic variants.

Private test sets, canaries, training-data attestations, historical harness artifacts, and item hashes may be unavailable. Record unavailable fields as UNKNOWN rather than inferring them.

“Benchmark scores are versioned observations, not timeless model properties. Preserve benchmark-release, item/scorer/harness provenance and inference conditions. A refresh or item correction starts a new comparison revision unless equivalence is demonstrated. Public exposure is a contamination risk, not proof that a particular model is contaminated.”

• Epoch AI, FrontierMath: Tiers 1–4, v2 release 2026-06-12.
• LiveBench, release lineage observed 2026-10-03; latest release then shown as 2026-06-25.
• White et al., “LiveBench: A Challenging, Contamination-Free LLM Benchmark,” arXiv:2406.19314.
• Yang et al., “Rethinking Benchmark and Contamination for Language Models with Rephrased Samples,” arXiv:2311.04850.
• OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities,” 2026 audit report.
• Sun et al., “The Emperor’s New Clothes in Benchmarking?,” arXiv:2503.16402.
• Qian et al., “VarBench,” arXiv:2406.17681.

This guidance does not establish the contamination status of any named closed model, does not assume access to training corpora, and does not prescribe one organization-wide refresh cadence or quantitative bridge-equivalence threshold.