A traditional web crawler will traverse your website in order to index it. The crawler saves the content together with the link to it. Then, the content would be indexed according to some algorithm, and given a user interface for search.

We’re a small web application software company based in Amsterdam. Recently our alert system noticed one of our projects in trouble. Response times were way up.

Investigating, we quickly found the issue. The application was being clobbered by thousands of requests per minute. By a crawler marked OpenAI.

Being crawled by indexers was always a good thing for your business. But being crawled by AI not so much. Because it’s after your language, not your content.

Why AI Crawlers Are Different

Traditional crawlers, like Googlebot, index your content to help users discover your site. They respect robots.txt, crawl at reasonable rates, and generally aim to drive traffic to your site.

AI crawlers from companies like OpenAI, Anthropic, or Claude are different. Their primary goal isn’t to index your content for search, but to scrape your language to train their models.

Unlike search engines, AI crawlers don’t send users back to your site. The money they make from your content will never come back to you.

AI crawlers are interested not in your content but in your language because they’re inherently uninterested in the meaning you wish to convey. Their interest is in the pattern that creates meaning.

They’re after the statistical patterns. Your carefully crafted documentation, unique phrasing, or even proprietary logic is just raw material for their models. They dissolve your message it into probabilities and correlations.

This is why content creators find AI so unfair.

Imagine you’re an artist, you spend days, weeks, in your studio, working your way to perfection on a painting that tells your story in vivid colour.

Now an AI comes, rips the canvas into shreds, and reassembles it a million times at the request of paying users. Utter disrespect not only for your effort and time, but even more importantly for the personal story you wished to tell.

It’s not just about labour or ownership, but about the erosion of personal expression and intent. When an AI dismantles content into statistical fragments, it erases the humanity behind it. The story, struggle, deliberate choices: all reduced to a probability distribution in a neural network.

Yes, there are ways to mitigate crawlers. Robots.txt, rate limiting for AI user agents, legal action. Some AI companies offer to exclude domains from scraping.

The point is, there’s a huge divide between how AI approaches language and what we as individuals appreciate in it. When Hegel said language is the spirit of humanity, he might have been prescient of AI’s focus on pattern rather than meaning.

In the news recently, an article about fire being used purposefully much earlier than was thought. Not by us, Homo sapiens, but by our near cousins in the tree of life, most probably Neanderthals. The article imagines a group sitting around a fire, perhaps sharing stories.

Four hundred thousand years ago.