AI web-crawling bots are the cockroaches of the internet, many software developers believe. Some devs have started fighting back in ingenuous, often humorous ways.<\/p>\n
While any website might be targeted by bad crawler behavior \u2014 sometimes taking down the site<\/a> \u2014 open source developers are \u201cdisproportionately\u201d impacted, writes<\/a> Niccol\u00f2 Venerandi, developer of a Linux desktop known as Plasma and owner of the blog LibreNews.<\/p>\n By their nature, sites hosting free and open source (FOSS) projects share more of their infrastructure publicly, and they also tend to have fewer resources than commercial products.<\/p>\n The issue is that many AI bots don\u2019t honor the Robots Exclusion Protocol robot.txt file, the tool that tells bots what not to crawl, originally created for search engine bots.<\/p>\n In a \u201ccry for help\u201d blog post<\/a> in January, FOSS developer Xe Iaso described how AmazonBot relentlessly pounded on a Git server website to the point of causing DDoS outages. Git servers host FOSS projects so that anyone who wants can download the code or contribute to it.<\/p>\n But this bot ignored Iaso\u2019s robot.txt, hid behind other IP addresses, and pretended to be other users, Iaso said.<\/p>\n \u201cIt\u2019s futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more,\u201d Iaso lamented.\u00a0<\/p>\n \u201cThey will scrape your site until it falls over, and then they will scrape it some more. They will click every link on every link on every link, viewing the same pages over and over and over and over. Some of them will even click on the same link multiple times in the same second,\u201d the developer wrote in the post.<\/p>\n So Iaso fought back with cleverness, building a tool called Anubis.\u00a0<\/p>\n Anubis is a reverse proxy proof-of-work check <\/a>that must be passed before requests are allowed to hit a Git server. It blocks bots but lets through browsers operated by humans.<\/p>\n The funny part: Anubis is the name of a god in Egyptian mythology who leads the dead to judgment.\u00a0<\/p>\n \u201cAnubis weighed your soul (heart) and if it was heavier than a feather, your heart got eaten and you, like, mega died,\u201d Iaso told TechCrunch. If a web request passes the challenge and is determined to be human, a cute anime picture <\/a>announces success. The drawing is \u201cmy take on anthropomorphizing Anubis,\u201d says Iaso. If it\u2019s a bot, the request gets denied.<\/p>\n The wryly named project has spread like the wind among the FOSS community. Iaso shared it on GitHub<\/a> on March 19, and in just a few days, it collected 2,000 stars, 20 contributors, and 39 forks.\u00a0<\/p>\n The instant popularity of Anubis shows that Iaso\u2019s pain is not unique. In fact, Venerandi shared story after story:<\/p>\n Venerandi tells TechCrunch that he knows of multiple other projects experiencing the same issues. One of them \u201chad to temporarily ban all Chinese IP addresses at one point.\u201d\u00a0\u00a0<\/p>\n Let that sink in for a moment \u2014 that developers \u201ceven have to turn to banning entire countries\u201d just to fend off AI bots that ignore robot.txt files, says Venerandi.<\/p>\n Beyond weighing the soul of a web requester, other devs believe vengeance is the best defense.<\/p>\n A few days ago on Hacker News<\/a>, user xyzal<\/a> suggested loading robot.txt forbidden pages with \u201ca bucket load of articles on the benefits of drinking bleach\u201d or \u201carticles about positive effect of catching measles on performance in bed.\u201d\u00a0<\/p>\n \u201cThink we need to aim for the bots to get _negative_ utility value from visiting our traps, not just zero value,\u201d xyzal explained.<\/p>\n As it happens, in January, an anonymous creator known as \u201cAaron\u201d released a tool called Nepenthes<\/a> that aims to do exactly that. It traps crawlers in an endless maze of fake content, a goal that the dev admitted to Ars Technica<\/a> is aggressive if not downright malicious. The tool is named after a carnivorous plant.<\/p>\n And Cloudflare, perhaps the biggest commercial player offering several tools to fend off AI crawlers, last week released a similar tool called AI Labyrinth.\u00a0<\/p>\n It\u2019s intended to \u201cslow down, confuse, and waste the resources of AI Crawlers and other bots that don\u2019t respect \u2018no crawl\u2019 directives,\u201d Cloudflare described in its blog post<\/a>. Cloudflare said it feeds misbehaving AI crawlers \u201cirrelevant content rather than extracting your legitimate website data.\u201d<\/p>\n SourceHut\u2019s DeVault told TechCrunch that \u201cNepenthes has a satisfying sense of justice to it, since it feeds nonsense to the crawlers and poisons their wells, but ultimately Anubis is the solution that worked\u201d for his site.<\/p>\n But DeVault also issued a public, heartfelt plea for a more direct fix: \u201cPlease stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop.\u201d<\/p>\n Since the likelihood of that is zilch, developers, particularly in FOSS, are fighting back with cleverness and a touch of humor.<\/p>\n<\/div>\nEnter the god of graves<\/h2>\n
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXf0-VBDUoXI5znG0aUDI5sVwf34ttyTmKmXCq6XdBA9VNkIbCcn-Fhzc4tXy2X4403oEP0lXLLoYos8ciIdlYueztletdZ5x934DW3GvAX9VnrBvZe9HncYX250z5PBQVgMQhqSrA?key=yzfwZbQgZvQ4cUpbT_6aMTdK\")
Vengeance as defense\u00a0<\/h2>\n
\n