Did Xai Lie About Grok 3\’s Benchmarks?
Debates over AI Benchmarks – and how they are reported by ai labs – are spilling out into public view.
This week, an openai employee accused Elon Musk\’s Ai Company, Xai, of Publishing Misleading Benchmark Results for its latest ai model, Grok 3. One of the co-for founders of xai, igor babushkin, Insified That the company was in the right.
The Truth Lies somewhere in Between.
In a Post on Xai\’s BlogThe company published a Graph Showing Grok 3\’s Performance on Aime 2025, a collection of challenging Math questions from a recent invitational mathematics exam. Some experts have Questioned aime\’s validity as an ai benchmarkNevertheless, AIME 2025 and older versions of the test are commonly used to probe a model\’s math ability.
Xai\’s Graph Showed Two Variants of Grok 3, Grok 3 Reasoning Beta and Grok 3 Mini Reasoning, Beating Openai\’s Best-Farforming Awailable Model, O3-Mini-Highon Aime 2025. But openai employEEs on x was quick to point that xai\’s Graph did not incline 3-mini-high\’s aime 2025 score at \”Cons@64.\”
What is Cons@64, You Might Ask? Well, it\’s short for \”Consensus@64,\” and it basically Gives a model 64 tries to answer Each Problem in a Benchmark and takes the answers generated most frequent as the fame. As you can imagine, Cons@64 tends to boost models\’ Benchmark scores quite a bit, and omitting it from The case.
Grok 3 Reasoning Beta and Grok 3 Mini Reasoning\’s Scores for Aime 2025 at \”@1\”-meaning the first score the models got on the benchmark-Fall Bell Bell BELOW O3-HIGH\’S Score. Grok 3 Reasoning Beta also trails evr-So-allly behind Openai\’s O1 model Set to \”medium\” computing. YET XAI is Advertising Grok 3 As the \”World\’s smartest ai.\”
Babushkin argued on x That opinai has published Similarly Misleading Benchmark Charts in the Past – Albeit Charts Comparing The Performance of Its Own Models. A more neutral party in the debate put togeether a more \”accurate\” Graph showing Nearly Every Godel\’s performance at consce at consh@64:
Hilaous how some people see my plot as attack on openai and others as attack on Grok while in Reality in Reality it\’s Deepseek Propaganda
(I actually believe Grok Looks Good There, and Openai\’s Ttc Chicanery Behind O3-Min-*High*-Pass@\”-Pass@\” \”\” \”1\” \”\” \”\” \”deserves more scrutiny.) https://t.co/djqljpcjh8 pic.twitter.com/3WH8FOUFIC– Teortaxes ▶ ️ (Deepseek 推特🐋铁粉 2023 – ∞) (@Teortaxestex) February 20, 2025
But as ai researchr nathan lambert pointed out in a postPerhaps the most important metric remains a mystery: the computational (and monetary) cost it took for each model to achieve its best score. That just goes to show how Little Most Ai Benchmarks Communicate About Models\’ Limitations – and their strengths.