The Arc Prize Foundation, a nonprofit co-founded by prominent AI researcher Fran\u00e7ois Chollet, announced in a blog post<\/a> on Monday that it has created a new, challenging test to measure the general intelligence of leading AI models.<\/p>\n So far, the new test, called ARC-AGI-2, has stumped most models.<\/p>\n \u201cReasoning\u201d AI models like OpenAI\u2019s o1-pro and DeepSeek\u2019s R1 score between 1% and 1.3% on ARC-AGI-2, according to the Arc Prize leaderboard<\/a>. Powerful non-reasoning models including GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Flash score around 1%.<\/p>\n The ARC-AGI tests consist of puzzle-like problems where an AI has to identify visual patterns from a collection of different-colored squares, and generate the correct \u201canswer\u201d grid. The problems were designed to force an AI to adapt to new problems it hasn\u2019t seen before. <\/p>\n The Arc Prize Foundation had over 400 people take ARC-AGI-2 to establish a human baseline. On average, \u201cpanels\u201d of these people got 60% of the test\u2019s questions right \u2014 much better than any of the models\u2019 scores.<\/p>\n In a post on X<\/a>, Chollet claimed ARC-AGI-2 is a better measure of an AI model\u2019s actual intelligence than the first iteration of the test, ARC-AGI-1. The Arc Prize Foundation\u2019s tests are aimed at evaluating whether an AI system can efficiently acquire new skills outside the data it was trained on.<\/p>\n Chollet said that unlike ARC-AGI-1, the new test prevents AI models from relying on \u201cbrute force\u201d \u2014 extensive computing power \u2014 to find solutions. Chollet previously acknowledged this was a major flaw of ARC-AGI-1.<\/a><\/p>\n To address the first test\u2019s flaws, ARC-AGI-2 introduces a new metric: efficiency. It also requires models to interpret patterns on the fly instead of relying on memorization.<\/p>\n \u201cIntelligence is not solely defined by the ability to solve problems or achieve high scores,\u201d Arc Prize Foundation co-founder Greg Kamradt wrote in a blog post<\/a>. \u201cThe efficiency with which those capabilities are acquired and deployed is a crucial, defining component. The core question being asked is not just, \u2018Can AI acquire [the] skill to solve a task?\u2019 but also, \u2018At what efficiency or cost?\u2019\u201d<\/p>\n ARC-AGI-1 was unbeaten for roughly five years until December 2024, when OpenAI released its advanced reasoning model, o3<\/a>, which outperformed all other AI models and matched human performance on the evaluation. However, as we noted at the time, o3\u2019s performance gains on ARC-AGI-1 came with a hefty price tag<\/a>.<\/p>\n The version of OpenAI\u2019s o3 model \u2014 o3 (low) \u2014 that was first to reach new heights on ARC-AGI-1, scoring 75.7% on the test, got a measly 4% on ARC-AGI-2 using $200 worth of computing power per task.<\/p>\n The arrival of ARC-AGI-2 comes as many in the tech industry are calling for new, unsaturated benchmarks to measure AI progress. Hugging Face\u2019s co-founder, Thomas Wolf, recently told TechCrunch that the AI industry lacks sufficient tests to measure the key traits of so-called artificial general intelligence<\/a>, including creativity.<\/p>\n Alongside the new benchmark, the Arc Prize Foundation announced a new Arc Prize 2025 contest<\/a>, challenging developers to reach 85% accuracy on the ARC-AGI-2 test while only spending $0.42 per task.<\/p>\n<\/div>\n![\"\"](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/03\/Screenshot-2025-03-24-at-3.16.48PM.png?w=680\")
![\"\"](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/03\/Screenshot-2025-03-24-at-3.18.29PM.png?w=680\")