← Reference notes

Quantifying the Effect of Test Set Contamination on Generative Evaluations

Summary

Pretraining Qwen3-architecture models (up to 344M params) from scratch with 0–1000s of MATH test-set replicas shows contamination gains are superficial memorisation: they vanish when problems are reworded or numbers change, and are wiped out by temperature sampling, longer solutions, or more fresh-data training (Abstract; §3–5).

When to apply

• Before trusting a surprisingly high benchmark score (your own model, a paper, an agent-improvement run like RSIGym).
• When building evals for agents or self-improvement loops.

Procedure

1. Create a rephrased version (same numbers, new wording) and a perturbed version (same wording, new numbers/answers) of the benchmark.
2. Score original vs rephrased vs perturbed; a large drop on both variants signals memorisation (§3, Table 1).
3. Re-score at temperature ~1 vs greedy; disproportionate collapse suggests contamination (§5, Finding #6).
4. Check score vs solution length (§5, Finding #7).
5. If using EleutherAI LM Evaluation Harness minerva_math scores from before Aug 23, 2025, re-run with fixed code (§2, App. C).

Costs and limits

• Only MATH, dense decoder models ≤344M, one pretraining mix; thresholds may shift at frontier scale (§6 Limitations).
• Rephrasing/perturbing requires care so the new problems stay valid.

Evidence (detail)

• Rephrased/perturbed MATH: performance falls to uncontaminated baseline at all contamination levels and model sizes (§3, Table 1).
• Scaling-law analysis: even one test replica lets models beat the irreducible loss of uncontaminated training (Abstract; §3 Finding #3).
• Temperature: stable up to τ≈0.56; at τ=1, high-contamination (1000 replicas) performance collapses ~40×, to uncontaminated levels; ~2× drop at ≤10 replicas (§5, Fig. 5).
• Overtraining on fresh data dilutes contamination; SFT on the train set helps at low contamination, hurts at high (Abstract; §4).
• Eval-harness bug: gold reference solutions scored only ~70% under the buggy Math Verify (§2; App. C).

Facts

• id: F1 claim: Rephrased/perturbed MATH: performance falls to uncontaminated baseline at all contamination levels and model sizes (§3, Table 1). confidence: medium # single source: author-reported date: 2026-01-07 sources: [S1] license: CC BY 4.0
• id: F2 claim: Scaling-law analysis: even one test replica lets models beat the irreducible loss of uncontaminated training (Abstract; §3 Finding #3). confidence: medium # single source: author-reported date: 2026-01-07 sources: [S1] license: CC BY 4.0
• id: F3 claim: Temperature: stable up to τ≈0.56; at τ=1, high-contamination (1000 replicas) performance collapses ~40×, to uncontaminated levels; ~2× drop at ≤10 replicas (§5, Fig. 5). confidence: medium # single source: author-reported date: 2026-01-07 sources: [S1] license: CC BY 4.0
• id: F4 claim: Overtraining on fresh data dilutes contamination; SFT on the train set helps at low contamination, hurts at high (Abstract; §4). confidence: medium # single source: author-reported date: 2026-01-07 sources: [S1] license: CC BY 4.0
• id: F5 claim: Eval-harness bug: gold reference solutions scored only ~70% under the buggy Math Verify (§2; App. C). confidence: medium # single source: author-reported date: 2026-01-07 sources: [S1] license: CC BY 4.0

Related

• 2610.10310 RSIGym (scores there are explicitly not held-out).
• 2610.04074 IdeaScientist (temporal cutoff evaluation to avoid leakage).

Review notes

• Built from private assessment notes; social-media hype check and source-post links omitted. Paraphrase only; no verbatim quotes beyond short attributed terms.
• Figures not re-checked against the PDF in this pass beyond the original private notes.

For language models and agents: start at /llms.txt. This page in Markdown: /library/reference/arxiv/2601-04301-test-set-contamination-generative-evals.md.