\n<\/p>\n
Not even Pok\u00e9mon is safe from AI benchmarking controversy. <\/p>\n
Last week, a post on X<\/a> went viral, claiming that Google\u2019s latest Gemini model surpassed Anthropic\u2019s flagship Claude model in the original Pok\u00e9mon video game trilogy. Reportedly, Gemini had reached Lavender Town in a developer\u2019s Twitch stream; Claude was stuck at Mount Moon<\/a> as of late February.<\/p>\n Gemini is literally ahead of Claude atm in pokemon after reaching Lavender Town<\/p>\n 119 live views only btw, incredibly underrated stream pic.twitter.com\/8AvSovAI4x<\/a><\/p>\n \u2014 Jush (@Jush21e8) April 10, 2025<\/a><\/p>\n<\/blockquote>\n But what the post failed to mention is that Gemini had an advantage.<\/p>\n As users on Reddit<\/a> pointed out, the developer who maintains the Gemini stream built a custom minimap that helps the model identify \u201ctiles\u201d in the game like cuttable trees. This reduces the need for Gemini to analyze screenshots before it makes gameplay decisions.<\/p>\n Now, Pok\u00e9mon is a semi-serious AI benchmark at best \u2014 few would argue it\u2019s a very informative test of a model\u2019s capabilities. But it is<\/em> an instructive example of how different implementations of a benchmark can influence the results.<\/p>\n For example, Anthropic reported<\/a> two scores for its recent Anthropic 3.7 Sonnet model on the benchmark SWE-bench Verified, which is designed to evaluate a model\u2019s coding abilities. Claude 3.7 Sonnet achieved 62.3% accuracy on SWE-bench Verified, but 70.3% with a \u201ccustom scaffold\u201d that Anthropic developed.<\/p>\n More recently, Meta fine-tuned<\/a> a version of one of its newer models, Llama 4 Maverick, to perform well on a particular benchmark, LM Arena. The vanilla version<\/a> of the model scores significantly worse on the same evaluation.<\/p>\n Given that AI benchmarks \u2014 Pok\u00e9mon included \u2014 are imperfect measures<\/a> to begin with, custom and non-standard implementations threaten to muddy the waters even further. That is to say, it doesn\u2019t seem likely that it\u2019ll get any easier to compare models as they\u2019re released.<\/p>\n<\/div>\n\n