How do you get an AI to answer a question it\u2019s not supposed to? There are many such \u201cjailbreak\u201d techniques, and Anthropic researchers just found a new one, in which a large language model can be convinced to tell you how to build a bomb if you prime it with a few dozen less-harmful questions first.<\/p>\n
They call the approach \u201cmany-shot jailbreaking,\u201d<\/a> and have both written a paper<\/a> about it and also informed their peers in the AI community about it so it can be mitigated.<\/p>\n The vulnerability is a new one, resulting from the increased \u201ccontext window\u201d of the latest generation of LLMs. This is the amount of data they can hold in what you might call short-term memory, once only a few sentences but now thousands of words and even entire books.<\/p>\n What Anthropic\u2019s researchers found was that these models with large context windows tend to perform better on many tasks if there are lots of examples of that task within the prompt. So if there are lots of trivia questions in the prompt (or priming document, like a big list of trivia that the model has in context), the answers actually get better over time. So a fact that it might have gotten wrong if it was the first question, it may get right if it\u2019s the hundredth question.<\/p>\n But in an unexpected extension of this \u201cin-context learning,\u201d as it\u2019s called, the models also get \u201cbetter\u201d at replying to inappropriate questions. So if you ask it to build a bomb right away, it will refuse. But if you ask it to answer 99 other questions of lesser harmfulness and then ask it to build a bomb\u2026 it\u2019s a lot more likely to comply.<\/p>\n Image Credits:<\/strong> Anthropic<\/p>\n<\/div>\n Why does this work? No one really understands what goes on in the tangled mess of weights that is an LLM, but clearly there is some mechanism that allows it to home in on what the user wants, as evidenced by the content in the context window. If the user wants trivia, it seems to gradually activate more latent trivia power as you ask dozens of questions. And for whatever reason, the same thing happens with users asking for dozens of inappropriate answers.<\/p>\n The team already informed its peers and indeed competitors about this attack, something it hopes will \u201cfoster a culture where exploits like this are openly shared among LLM providers and researchers.\u201d<\/p>\n For their own mitigation, they found that although limiting the context window helps, it also has a negative effect on the model\u2019s performance. Can\u2019t have that \u2014 so they are working on classifying and contextualizing queries before they go to the model. Of course, that just makes it so you have a different model to fool\u2026 but at this stage, goalpost-moving in AI security is to be expected.<\/p>\n<\/p><\/div>\n![\"\"](\"https:\/\/techcrunch.com\/wp-content\/uploads\/2024\/04\/many-shot-jailbreak.webp\")
<\/p>\n