\n<\/p>\n
A discrepancy between first- and third-party benchmark results for OpenAI\u2019s o3 AI model is raising questions about the company\u2019s transparency<\/a> and model testing practices.<\/p>\n When OpenAI unveiled o3 in December<\/a>, the company claimed the model could answer just over a fourth of questions on FrontierMath, a challenging set of math problems. That score blew the competition away \u2014 the next-best model managed to answer only around 2% of FrontierMath problems correctly.<\/p>\n \u201cToday, all offerings out there have less than 2% [on FrontierMath],\u201d Mark Chen, chief research officer at OpenAI, said during a livestream<\/a>. \u201cWe\u2019re seeing [internally], with o3 in aggressive test-time compute settings, we\u2019re able to get over 25%.\u201d<\/p>\n As it turns out, that figure was likely an upper bound, achieved by a version of o3 with more computing behind it than the model OpenAI publicly launched last week.<\/p>\n Epoch AI, the research institute behind FrontierMath, released results of its independent benchmark tests of o3 on Friday. Epoch found that o3 scored around 10%, well below OpenAI\u2019s highest claimed score.<\/p>\n OpenAI has released o3, their highly anticipated reasoning model, along with o4-mini, a smaller and cheaper model that succeeds o3-mini.<\/p>\n We evaluated the new models on our suite of math and science benchmarks. Results in thread! pic.twitter.com\/5gbtzkEy1B<\/a><\/p>\n \u2014 Epoch AI (@EpochAIResearch) April 18, 2025<\/a><\/p>\n<\/blockquote>\n That doesn\u2019t mean OpenAI lied, per se. The benchmark results the company published in December show a lower-bound score that matches the score Epoch observed. Epoch also noted its testing setup likely differs from OpenAI\u2019s, and that it used an updated release of FrontierMath for its evaluations.<\/p>\n \u201cThe difference between our results and OpenAI\u2019s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time [computing], or because those results were run on a different subset of FrontierMath (the 180 problems in frontiermath-2024-11-26 vs the 290 problems in frontiermath-2025-02-28-private),\u201d wrote<\/a> Epoch.<\/p>\n According to a post on X<\/a> from the ARC Prize Foundation, an organization that tested a pre-release version of o3, the public o3 model \u201cis a different model [\u2026] tuned for chat\/product use,\u201d corroborating Epoch\u2019s report.<\/p>\n \u201cAll released o3 compute tiers are smaller than the version we [benchmarked],\u201d wrote ARC Prize. Generally speaking, bigger compute tiers can be expected to achieve better benchmark scores.<\/p>\n Re-testing released o3 on ARC-AGI-1 will take a day or two. Because today\u2019s release is a materially different system, we are re-labeling our past reported results as \u201cpreview\u201d:<\/p>\n o3-preview (low): 75.7%, $200\/task Above uses o1 pro pricing\u2026<\/p>\n \u2014 Mike Knoop (@mikeknoop) April 16, 2025<\/a><\/p>\n<\/blockquote>\n OpenAI\u2019s own Wenda Zhou, a member of the technical staff, said during a livestream last week<\/a> that the o3 in production is \u201cmore optimized for real-world use cases\u201d and speed versus the version of o3 demoed in December. As a result, it may exhibit benchmark \u201cdisparities,\u201d he added.<\/p>\n \u201c[W]e\u2019ve done [optimizations] to make the [model] more cost efficient [and] more useful in general,\u201d Zhou said. \u201cWe still hope that \u2014 we still think that \u2014 this is a much better model [\u2026] You won\u2019t have to wait as long when you\u2019re asking for an answer, which is a real thing with these [types of] models.\u201d<\/p>\n Granted, the fact that the public release of o3 falls short of OpenAI\u2019s testing promises is a bit of a moot point, since the company\u2019s o3-mini-high and o4-mini models outperform o3 on FrontierMath, and OpenAI plans to debut a more powerful o3 variant, o3-pro, in the coming weeks.<\/p>\n It is, however, another reminder that AI benchmarks are best not taken at face value \u2014 particularly when the source is a company with services to sell.<\/p>\n Benchmarking \u201ccontroversies\u201d are becoming a common occurrence in the AI industry as vendors race to capture headlines and mindshare with new models.<\/p>\n In January, Epoch was criticized<\/a> for waiting to disclose funding from OpenAI until after the company announced o3. Many academics who contributed to FrontierMath weren\u2019t informed of OpenAI\u2019s involvement until it was made public.<\/p>\n More recently, Elon Musk\u2019s xAI was accused<\/a> of publishing misleading benchmark charts for its latest AI model, Grok 3. Just this month, Meta admitted to touting benchmark scores for a version of a model that differed from the one the company made available to developers<\/a>.<\/p>\n Updated 4:21 p.m. Pacific: Added comments from Wenda Zhou, a member of the OpenAI technical staff, from a livestream last week.<\/em><\/p>\n<\/div>\n\n
\n
o3-preview (high): 87.5%, $34.4k\/task<\/p>\n