\n<\/p>\n
One of the new flagship AI models<\/a> Meta released on Saturday, Maverick, ranks second on LM Arena<\/a>, a test that has human raters compare the outputs of models and choose which they prefer. But it seems the version of Maverick that Meta deployed to LM Arena differs from the version that\u2019s widely available to developers. <\/p>\n As several<\/a> AI<\/a> researchers<\/a> pointed out on X, Meta noted in its announcement that the Maverick on LM Arena is an \u201cexperimental chat version.\u201d A chart on the official Llama website<\/a>, meanwhile, discloses that Meta\u2019s LM Arena testing was conducted using \u201cLlama 4 Maverick optimized for conversationality.\u201d<\/p>\n As we\u2019ve written about before<\/a>, for various reasons, LM Arena has never been the most reliable measure of an AI model\u2019s performance. But AI companies generally haven\u2019t customized or otherwise fine-tuned their models to score better on LM Arena \u2014 or haven\u2019t admitted to doing so, at least. <\/p>\n The problem with tailoring a model to a benchmark, withholding it, and then releasing a \u201cvanilla\u201d variant of that same model is that it makes it challenging for developers to predict exactly how well the model will perform in particular contexts. It\u2019s also misleading. Ideally, benchmarks \u2014 woefully inadequate as they are<\/a> \u2014 provide a snapshot of a single model\u2019s strengths and weaknesses across a range of tasks.<\/p>\n Indeed, researchers on X have observed stark<\/a> differences in the behavior<\/a> of the publicly downloadable Maverick compared with the model hosted on LM Arena. The LM Arena version seems to use a lot of emojis, and give incredibly long-winded answers.<\/p>\n Okay Llama 4 is def a littled cooked lol, what is this yap city pic.twitter.com\/y3GvhbVz65<\/a><\/p>\n \u2014 Nathan Lambert (@natolambert) April 6, 2025<\/a><\/p>\n<\/blockquote>\n for some reason, the Llama 4 model in Arena uses a lot more Emojis<\/p>\n on together . ai, it seems better: pic.twitter.com\/f74ODX4zTt<\/a><\/p>\n \u2014 Tech Dev Notes (@techdevnotes) April 6, 2025<\/a><\/p>\n<\/blockquote>\n We\u2019ve reached out to Meta and Chatbot Arena, the organization that maintains LM Arena, for comment.<\/p>\n<\/div>\n\n
\n