\n<\/p>\n
Debates over AI benchmarks \u2014 and how they\u2019re reported by AI labs \u2014 are spilling out into public view. <\/p>\n
This week, an OpenAI employee accused<\/a> Elon Musk\u2019s AI company, xAI, of publishing misleading benchmark results for its latest AI model, Grok 3. One of the co-founders of xAI, Igor Babushkin, insisted<\/a> that the company was in the right. <\/p>\n The truth lies somewhere in between. <\/p>\n In a post on xAI\u2019s blog<\/a>, the company published a graph showing Grok 3\u2019s performance on AIME 2025, a collection of challenging math questions from a recent invitational mathematics exam. Some experts have questioned AIME\u2019s validity as an AI benchmark<\/a>. Nevertheless, AIME 2025 and older versions of the test are commonly used to probe a model\u2019s math ability. <\/p>\n xAI\u2019s graph showed two variants of Grok 3, Grok 3 Reasoning Beta and Grok 3 mini Reasoning, beating OpenAI\u2019s best-performing available model, o3-mini-high<\/a>, on AIME 2025. But OpenAI employees on X were quick to point out that xAI\u2019s graph didn\u2019t include o3-mini-high\u2019s AIME 2025 score at \u201ccons@64.\u201d <\/p>\n What is cons@64, you might ask? Well, it\u2019s short for \u201cconsensus@64,\u201d and it basically gives a model 64 tries to answer each problem in a benchmark and takes the answers generated most frequently as the final answers. As you can imagine, cons@64 tends to boost models\u2019 benchmark scores quite a bit, and omitting it from a graph might make it appear as though one model surpasses another when in reality, that\u2019s isn\u2019t the case.<\/p>\n Grok 3 Reasoning Beta and Grok 3 mini Reasoning\u2019s scores for AIME 2025 at \u201c@1\u201d \u2014 meaning the first score the models got on the benchmark \u2014 fall below o3-mini-high\u2019s score. Grok 3 Reasoning Beta also trails ever-so-slightly behind OpenAI\u2019s o1 model<\/a> set to \u201cmedium\u201d computing. Yet xAI is advertising Grok 3<\/a> as the \u201cworld\u2019s smartest AI.\u201d<\/p>\n Babushkin argued on X<\/a> that OpenAI has published similarly misleading benchmark charts in the past \u2014 albeit charts comparing the performance of its own models. A more neutral party in the debate put together a more \u201caccurate\u201d graph showing nearly every model\u2019s performance at cons@64:<\/p>\n Hilarious how some people see my plot as attack on OpenAI and others as attack on Grok while in reality it\u2019s DeepSeek propaganda \u2014 Teortaxes\u25b6\ufe0f (DeepSeek \u63a8\u7279\ud83d\udc0b\u94c1\u7c89 2023 \u2013 \u221e) (@teortaxesTex) February 20, 2025<\/a><\/p>\n<\/blockquote>\n But as AI researcher Nathan Lambert pointed out in a post<\/a>, perhaps the most important metric remains a mystery: the computational (and monetary) cost it took for each model to achieve its best score. That just goes to show how little most AI benchmarks communicate about models\u2019 limitations \u2014 and their strengths.<\/p>\n<\/div>\n\n
(I actually believe Grok looks good there, and openAI\u2019s TTC chicanery behind o3-mini-*high*-pass@\u201d\u201d\u201d1\u2033\u201d\u201d deserves more scrutiny.) https:\/\/t.co\/dJqlJpcJh8<\/a> pic.twitter.com\/3WH8FOUfic<\/a><\/p>\n