Blog

Bots Now Outnumber Humans on the Web

In June 2026, Cloudflare shared a number that got a lot of attention. Cloudflare sits in front of a large share of the internet’s traffic, so it has a good view of the whole web. Its data showed that bots now make up 57.5% of web page requests. Humans are the minority. Cloudflare’s own CEO said it happened about 18 months earlier than the company expected.

The short version

  • Bots pass humans: more than half of web traffic is now automated.
  • AI crawlers are a growing slice of that. About 4 out of 5 AI crawler visits are collecting text to train AI models.
  • These crawlers take a lot and send back almost nothing. For one AI company, Cloudflare measured tens of thousands of crawl visits for every one visitor sent to a site.
  • You can control some of this with two files and a setting. None of it is urgent for a normal small-business site, but it is worth understanding and making a deliberate choice.

What a bot actually is here

A bot is any program that visits a website without a person driving it. Not all bots are bad. Googlebot is a bot, and you want it to visit so your site shows up in search. Bing has one too. Uptime checkers are bots. So are the previews that appear when you paste a link into a text message or a Slack channel.

The new growth is in two kinds of AI bot:

  • Training crawlers. These copy text from your pages to help build AI models like ChatGPT and Claude. Cloudflare found training is about 80% of AI crawler activity. Common names in your logs: GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider.
  • Live fetchers. When you ask an AI assistant a question and it goes and reads a few pages to answer you, that is a live fetch. Names here include ChatGPT-User and Perplexity-User. This slice is smaller today but growing fast, and it is the one that can actually send you a visitor.

Why this matters for a small business

Three practical effects.

Your analytics get noisier. If your traffic numbers jump but your leads do not, a wave of bot visits may be the reason. Tools like Google Analytics filter most known bots automatically, so the numbers there are usually close to real. Your hosting dashboard’s “visits” number often does not filter anything, which is why the two never match. When you judge your site, use real actions like form fills and phone calls, not raw visit counts from your host.

Your hosting works harder. Every bot visit uses a little of your server’s time and your bandwidth. For most small sites this is background noise. For a large site, or a small site on a weak shared hosting plan, heavy crawling can slow real visitors down or push you over a plan limit. If your host emails you about “excessive resource usage,” aggressive crawlers are worth checking as a cause.

Your content trains AI whether you get credit or not. AI crawlers copy your words to build tools that then answer questions without linking back. Cloudflare measured the imbalance directly. One major AI company’s crawlers visited sites tens of thousands of times for every single visitor they referred back. Search engines have always been a trade: they take your content and send you visitors. Training crawlers mostly take.

The controls that exist

There are three tools, and it helps to know what each one really does.

robots.txt. This is a plain text file at the root of your site (yoursite.com/robots.txt). It lists which bots are allowed where. Big, well-behaved crawlers like Googlebot, GPTBot, and ClaudeBot check it and follow it. To ask the main AI training crawlers to stay out, you add a few lines like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

That does not touch Googlebot, so it has no effect on your search ranking. The catch: robots.txt is a request, not a wall. A badly behaved bot can ignore it.

llms.txt. This is a newer file that is supposed to point AI tools to your best content. Adoption is low, around 10% of sites, and studies show the main AI crawlers mostly ignore it and just read your normal pages. It does not hurt to have one, but do not expect it to do much yet.

Blocking at the network level. A service like Cloudflare sits in front of your site and can actually stop AI crawlers before they reach your server. This is the only option that enforces rather than asks. Cloudflare now turns AI-crawler blocking on by default for new sites, and is testing a “pay per crawl” system where AI companies would have to pay to access your content. Standards for letting publishers set content-use rules are still being worked out.

What most small businesses should actually do

For a normal service business, the answer is: not much, on purpose.

  • Keep letting search and social bots in. Blocking them hurts you.
  • Decide how you feel about AI training crawlers. If you would rather not feed them, blocking GPTBot and similar crawlers in robots.txt is a reasonable, low-risk choice. It will not affect your Google ranking. If you would rather be everywhere an AI might quote you, leave them alone.
  • Do not block the live fetchers (ChatGPT-User, Perplexity-User) if you want a chance at referral traffic from AI assistants.
  • If your hosting struggles under load, or you see strange traffic spikes, put your site behind a service that can filter bots.
  • Judge your site by real leads, not visit counts.

The one thing not to do is nothing-while-worried. Either make a clear choice about AI crawlers or decide on purpose that you do not care. Both are fine. Vague worry is not a plan.

How I can help

A free website health check includes a look at your robots.txt, your crawler traffic, and how your hosting handles load. I will tell you whether bots are actually a problem for your site or just a headline, and set your crawler rules to match what you want, without touching the search and social bots you need.

If your hosting is the weak point, that often comes up in the same check, and moving to a stronger setup is usually a quick project.

Sources

← Back to the blog

Get a second opinion on your own site.

Get a free website health check