Every Sunday, NPR host Will Shortz, The New York Times\u2019 crossword puzzle guru, gets to quiz thousands of listeners in a long-running segment called the Sunday Puzzle<\/a>. While written to be solvable without too<\/em> much foreknowledge, the brainteasers are usually challenging even for skilled contestants.<\/p>\n That\u2019s why some experts think they\u2019re a promising way to test the limits of AI\u2019s problem-solving abilities.<\/p>\n In a recent study<\/a>, a team of researchers hailing from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastern University, Charles University, and startup Cursor created an AI benchmark using riddles from Sunday Puzzle episodes. The team says their test uncovered surprising insights, like that reasoning models \u2014 OpenAI\u2019s o1, among others \u2014 sometimes \u201cgive up\u201d and provide answers they know aren\u2019t correct.<\/p>\n \u201cWe wanted to develop a benchmark with problems that humans can understand with only general knowledge,\u201d Arjun Guha, a computer science faculty member at Northeastern and one of the co-authors on the study, told TechCrunch.<\/p>\n The AI industry is in a bit of a benchmarking quandary at the moment. Most of the tests commonly used to evaluate AI models probe for skills, like competency on PhD-level math and science questions, that aren\u2019t relevant to the average user. Meanwhile, many benchmarks \u2014 even benchmarks released relatively recently<\/a> \u2014 are quickly approaching the saturation point.<\/p>\n The advantages of a public radio quiz game like the Sunday Puzzle is that it doesn\u2019t test for esoteric knowledge, and the challenges are phrased such that models can\u2019t draw on \u201crote memory\u201d to solve them, explained Guha.<\/p>\n \u201cI think what makes these problems hard is that it\u2019s really difficult to make meaningful progress on a problem until you solve it \u2014 that\u2019s when everything clicks together all at once,\u201d Guha said. \u201cThat requires a combination of insight and a process of elimination.\u201d<\/p>\n No benchmark is perfect, of course. The Sunday Puzzle is U.S. centric and English only. And because the quizzes are publicly available, it\u2019s possible that models trained on them can \u201ccheat\u201d in a sense, although Guha says he hasn\u2019t seen evidence of this.<\/p>\n \u201cNew questions are released every week, and we can expect the latest questions to be truly unseen,\u201d he added. \u201cWe intend to keep the benchmark fresh and track how model performance changes over time.\u201d<\/p>\n On the researchers\u2019 benchmark, which consists of around 600 Sunday Puzzle\u00a0riddles, reasoning models such as o1 and DeepSeek\u2019s R1 far outperform the rest. Reasoning models thoroughly fact-check themselves before giving out results, which\u00a0helps them\u00a0avoid some of the\u00a0pitfalls<\/a>\u00a0that normally trip up AI models. The trade-off is that reasoning models take a little longer to arrive at solutions \u2014 typically seconds to minutes longer.<\/p>\n At least one model, DeepSeek\u2019s R1, gives solutions it knows to be wrong for some of the Sunday Puzzle questions. R1 will state verbatim \u201cI give up,\u201d followed by an incorrect answer chosen seemingly at random \u2014 behavior this human can certainly relate to.<\/p>\n The models make other bizarre choices, like giving a wrong answer only to immediately retract it, attempt to tease out a better one, and fail again. They also get stuck \u201cthinking\u201d forever and give nonsensical explanations for answers, or they arrive at a correct answer right away but then go on to consider alternative answers for no obvious reason.<\/p>\n \u201cOn hard problems, R1 literally says that it\u2019s getting \u2018frustrated,\u2019\u201d Guha said. \u201cIt was funny to see how a model emulates what a human might say. It remains to be seen how \u2018frustration\u2019 in reasoning can affect the quality of model results.\u201d<\/p>\n The current best-performing model on the benchmark is o1 with a score of 59%, followed by the recently released o3-mini<\/a> set to high \u201creasoning effort\u201d (47%). (R1 scored 35%.) As a next step, the researchers plan to broaden their testing to additional reasoning models, which they hope will help to identify areas where these models might be enhanced.<\/p>\n \u201cYou don\u2019t need a PhD to be good at reasoning, so it should be possible to design reasoning benchmarks that don\u2019t require PhD-level knowledge,\u201d Guha said. \u201cA benchmark with broader access allows a wider set of researchers to comprehend and analyze the results, which may in turn lead to better solutions in the future. Furthermore, as state-of-the-art models are increasingly deployed in settings that affect everyone, we believe everyone should be able to intuit what these models are \u2014 and aren\u2019t \u2014 capable of.\u201d<\/p>\n<\/div>\n![\"NPR](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-06-at-12.41.18AM.png?w=680\")
![\"NPR](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-06-at-12.31.38AM.png?w=680\")