# Evaluation Set Aging / Refresh / Contamination

CURRENT_STATUS: CURRENT
OBSERVATION: SPEC_ONLY
FRESHNESS: CURRENT
PAGE_TYPE: reference
CANONICAL_URL: /tools/ea-683-evaluation-set-aging-refresh-contamination
CLAIM_IDS: none
EVIDENCE_IDS: EA-683-CURRENT-APPROVAL, EA-683-CURRENT-BODY-READBACK, EVD-EA-683-URL-84CF1132C66C, EVD-EA-683-URL-130364A0FC6E, EVD-EA-683-URL-9162797DA302, EVD-EA-683-URL-5064F6E55C99, EVD-EA-683-URL-A09FB6FD80E8, EVD-EA-683-URL-91B5C8E2E332, EVD-EA-683-URL-9385E26C5247
UNKNOWN_IDS: none
OPS_IDS: none

## Summary
“Benchmark scores are versioned observations, not timeless model properties. Preserve benchmark-release, item/scorer/harness provenance and inference conditions. A refresh or item correction starts a new comparison revision unless equivalence is demonstrated. Public exposure is a contamination risk, not proof that a particular model is contaminated.”

## Current judgment
“Benchmark scores are versioned observations, not timeless model properties. Preserve benchmark-release, item/scorer/harness provenance and inference conditions. A refresh or item correction starts a new comparison revision unless equivalence is demonstrated. Public exposure is a contamination risk, not proof that a particular model is contaminated.”

## Scope
Evaluation Set Aging / Refresh / Contamination

## Confirmed

## UNKNOWN
- none listed on this page

## Evidence
- EA-683-CURRENT-APPROVAL (RESEARCH): 現行本文の監査・revision対応
- EA-683-CURRENT-BODY-READBACK (IMPL): 現行本文の収録照合
- EVD-EA-683-URL-84CF1132C66C (RESEARCH): E1 | official benchmark maintainer | Epoch AI, "FrontierMath: Tiers 1-4" | — https://epoch.ai/frontiermath/tiers-1-4
- EVD-EA-683-URL-130364A0FC6E (RESEARCH): E2 | benchmark site / author-maintained | LiveBench | — https://livebench.ai/
- EVD-EA-683-URL-9162797DA302 (RESEARCH): E3 | primary paper | White et al., "LiveBench: A Challenging, Contamination-Free LLM Benchmark" | — https://arxiv.org/abs/2406.19314
- EVD-EA-683-URL-5064F6E55C99 (RESEARCH): E4 | primary paper | Yang et al., "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples" | — https://arxiv.org/abs/2311.04850
- EVD-EA-683-URL-A09FB6FD80E8 (RESEARCH): E5 | provider evaluation audit / primary operational report | OpenAI, "Why SWE-bench Verified no longer measures frontier coding capabilities" | — https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
- EVD-EA-683-URL-91B5C8E2E332 (RESEARCH): E6 | research paper / counterevidence | Sun et al., "The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination — https://arxiv.org/abs/2503.16402
- EVD-EA-683-URL-9385E26C5247 (RESEARCH): E7 | primary paper | Qian et al., "VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation" | — https://arxiv.org/abs/2406.17681

## Failure / limits
Do not compare mixed benchmark releases without stratification. Other invalidating conditions include silent item replacement, scorer or harness changes treated as model gains or losses, benchmark saturation, leaked solutions, broken or ambiguous items, hidden tool/web-access differences, inference-compute changes, post-release benchmark edits, and non-equivalent dynamic variants.


Private test sets, canaries, training-data attestations, historical harness artifacts, and item hashes may be unavailable. Record unavailable fields as UNKNOWN rather than inferring them.
## 要点と前提
Benchmark scores are versioned observations, not timeless model properties. Enterprise evaluation records should preserve the exact benchmark release, item-set identity or lawful hash, scorer and harness version, model/revision, tool or web policy, inference settings, run date, and environment.

## Why this matters
Public exposure creates contamination risk, but it does not prove that a particular model is contaminated. Benchmark maintainers can also repair defective items, change scoring, alter refresh cadence, or revise harnesses after release. Those changes can break direct longitudinal comparability even when the model is unchanged.


FrontierMath v2 is a concrete example: its maintainer reported that the v2 release addressed errors affecting 42% of problems. LiveBench likewise uses explicit release lineage and refreshes its question set, while its original paper and current operating cadence differ. These are reasons to bind every score to the benchmark version actually used rather than silently replacing historical results.

## Operating rule
Treat item correction, scoring-rule change, harness/runtime change, tool or web policy change, model revision, and inference-compute change as comparison boundaries unless a bridge study demonstrates equivalence. Never overwrite an old score when a benchmark refreshes. Create a new evaluation revision and retain the old run with its original provenance.


When licensing permits, a bridge study should run old and new sets under the same model, harness, and settings; report paired unchanged items separately from new or repaired items; record exclusions and reasons; and avoid collapsing non-equivalent releases into one trend line.

## Contamination claims
A public benchmark implies an opportunity for exposure. A model-specific contamination claim requires additional evidence such as training-data evidence, controlled canaries, memorization or overlap tests, or another specific method. Score movement after a refresh is not diagnostic: it can result from contamination removal, difficulty shift, item repair, scorer changes, or harness changes.


Fresh, private, rephrased, synthetic, and dynamically generated items can reduce some exposure or aging risks, but none is a universal contamination cure. Refresh methods can preserve leakage or change task fidelity, so their own version and generation policy belong in the run record.

## Failure envelope
Do not compare mixed benchmark releases without stratification. Other invalidating conditions include silent item replacement, scorer or harness changes treated as model gains or losses, benchmark saturation, leaked solutions, broken or ambiguous items, hidden tool/web-access differences, inference-compute changes, post-release benchmark edits, and non-equivalent dynamic variants.


Private test sets, canaries, training-data attestations, historical harness artifacts, and item hashes may be unavailable. Record unavailable fields as UNKNOWN rather than inferring them.

## Safe wording
“Benchmark scores are versioned observations, not timeless model properties. Preserve benchmark-release, item/scorer/harness provenance and inference conditions. A refresh or item correction starts a new comparison revision unless equivalence is demonstrated. Public exposure is a contamination risk, not proof that a particular model is contaminated.”

## Evidence basis
- Epoch AI, FrontierMath: Tiers 1–4, v2 release 2026-06-12.
- LiveBench, release lineage observed 2026-10-03; latest release then shown as 2026-06-25.
- White et al., “LiveBench: A Challenging, Contamination-Free LLM Benchmark,” arXiv:2406.19314.
- Yang et al., “Rethinking Benchmark and Contamination for Language Models with Rephrased Samples,” arXiv:2311.04850.
- OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities,” 2026 audit report.
- Sun et al., “The Emperor’s New Clothes in Benchmarking?,” arXiv:2503.16402.
- Qian et al., “VarBench,” arXiv:2406.17681.

## Limits
This guidance does not establish the contamination status of any named closed model, does not assume access to training corpora, and does not prescribe one organization-wide refresh cadence or quantitative bridge-equivalence threshold.

NEXT: /agent/start.json
STOP: This is a knowledge projection, not an execution order.