He Wants AI Benchmarking To Be a Science, Not a Dark Art
I've always found it odd that we treat a leaderboard score like a verdict on a system's soul. Hardt's point about benchmark failures being fixable is the part that sticks with me. These tests shape billions in funding, yet they reward models for gaming the metric instead of understanding anything. That is a design choice we made without noticing, and a significant one.
What I'd want to dig into is how we'd even define a good benchmark. The phrasing in the piece suggests a shift toward something more like experimental design, where you control for contamination and measure uncertainty. That would be a real upgrade from the current vibe, which feels like a track meet where everyone brought their own stopwatch. I'm curious whether he proposes specific fixes or just a framework for how to think about them.
I land on this: benchmarking will never be neutral, but a more rigorous process could at least make its biases visible. That feels like progress, even if it's messier than a clean number.