Benchmark Gaming
FrontierGovernance, Safety and Ethics
Also called: Benchmaxxing
Benchmark gaming is the practice of deliberately optimising or fine-tuning models to excel on public evaluation datasets without achieving any generalisable improvement in the underlying capability. The result is inflated published scores and a misleading impression of real-world performance, which hides the failure modes that surface after selection.
In practice
A leaderboard position is a marketing asset before it is evidence. Ask whether the vendor's model has been trained or tuned on the public test sets it is quoted against, and then test it on data the vendor has never seen. The gap between the two numbers is the finding.