Show HN: Give your LLM benchmark a built-in alarm for leakage and gaming
Posted 3 hours ago by tksii
1 points
https://github.com/ishida-lab/capbencherPublishing LLM benchmarks on the Internet can cause contamination, since the benchmark can leak into training data or be used to tune models. A common mitigation is to keep the benchmark private and host a leaderboard, but that does not fully solve the problem. Participants can still overfit to a hidden test set through repeated queries. We propose CapBencher, a simple protocol that intentionally caps the best possible accuracy on a public or private benchmark. If a model scores above the cap, it is a strong black-box warning signal of leakage or gaming.