Thought Pok\u00e9mon was a tough benchmark for AI<\/a>? One group of researchers argues that Super Mario Bros. is even tougher. <\/p>\n Hao AI Lab, a research org at the University of California San Diego, on Friday threw AI into live Super Mario Bros. games. Anthropic\u2019s Claude 3.7<\/a> performed the best, followed by Claude 3.5. Google\u2019s Gemini 1.5 Pro<\/a> and OpenAI\u2019s GPT-4o<\/a> struggled.<\/p>\n It wasn\u2019t quite the same version of Super Mario Bros. as the original 1985 release, to be clear. The game ran in an emulator and integrated with a framework, GamingAgent<\/a>, to give the AIs control over Mario.<\/p>\n GamingAgent, which Hao developed in-house, fed the AI basic instructions, like, \u201cIf an obstacle or enemy is near, move\/jump left to dodge\u201d and in-game screenshots. The AI then generated inputs in the form of Python code to control Mario.<\/p>\n Still, Hao says that the game forced each model to \u201clearn\u201d to plan complex maneuvers and develop gameplay strategies. Interestingly, the lab found that reasoning models like OpenAI\u2019s o1<\/a>, which \u201cthink\u201d through problems step by step to arrive at solutions, performed worse than \u201cnon-reasoning\u201d models, despite being generally stronger on most benchmarks.<\/p>\n One of the main reasons reasoning models have trouble playing real-time games like this is that they take a while \u2014 seconds, usually \u2014 to decide on actions, according to the researchers. In Super Mario Bros., timing is everything. A second can mean the difference between a jump safely cleared and a plummet to your death.<\/p>\n Games have been used to benchmark AI for decades. But some experts have questioned the wisdom<\/a> of drawing connections between AI\u2019s gaming skills and technological advancement. Unlike the real world, games tend to be abstract and relatively simple, and they provide a theoretically infinite amount of data to train AI.<\/p>\n The recent flashy gaming benchmarks point to what Andrej Karpathy, a research scientist and founding member at OpenAI, called an \u201cevaluation crisis.\u201d<\/p>\n \u201cI don\u2019t really know what [AI] metrics to look at right now,\u201d he wrote in a post on X<\/a>. \u201cTLDR my reaction is I don\u2019t really know how good these models are right now.\u201d<\/p>\n At least we can watch AI play Mario.<\/p>\n<\/div>\n![\"Super](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/03\/ezgif-12b952f5417751.gif?w=640\")