AI Bots Are Taking Over the Internet and Websites Are Losing Human Visitors
For years, online publishers worried about search algorithms and social feeds deciding who gets seen. Now a different kind of visitor is surging across the web, and it does not read like a person or click like one. New tracking shared by Tollbit suggests AI driven bots are rapidly becoming a major share of total traffic while human visits to many sites are slipping. The shift is not just a technical nuisance, because it threatens the basic bargain that kept much of the open web funded.
Tollbit says that in the last months of 2025 there was roughly one AI bot visit for every 31 human visits, a dramatic jump from about one bot visit for every 200 human visits at the beginning of the same year. At the same time, human traffic fell by around 5 percent between the third and fourth quarters of 2025. The pattern fits how people are increasingly using AI assistants as the first stop for answers instead of opening multiple tabs. When the assistant summarizes what it finds, many users never reach the publisher that produced the reporting in the first place.
Not all bot traffic is the same, and the mix is changing in ways that matter. Tollbit reports that bots used for training models declined by about 15 percent, while retrieval focused bots rose strongly. RAG bots, the kind that fetch and summarize outside sources for an AI response, grew by 33 percent, and AI indexing robots climbed by 59 percent. That means more automated activity aimed at scanning and re packaging information rather than only collecting data for training sets.
One company stands out in the scale of its activity. Tollbit identifies OpenAI as the leading collector in this category, with its RAG bot known as ‘ChatGPT User’ being far more active than rivals. The same data indicates that this bot was about five times more active than the next most active bot associated with Meta and about 16 times more active than the agent tied to Perplexity. The takeaway for publishers is simple, because the most aggressive bot traffic may be coming from products that many readers now treat as a replacement for conventional browsing.
User behavior research points in the same direction. A separate study cited by TechRadar found that 37 percent of active AI tool users now start their searches through AI systems such as ChatGPT or Gemini rather than through traditional search engines. Even if those users still care about trustworthy sources, the path to those sources is being mediated. That mediation often results in fewer page loads, fewer ad impressions, and fewer chances for readers to subscribe or share.
Publishers have tried to manage automated access for decades, usually with a simple file called robots.txt. It is a set of instructions that tells compliant bots which areas of a site they are allowed to crawl and which areas they should avoid. The problem is that robots.txt relies on voluntary cooperation, and the new wave of AI related scraping does not always cooperate. Tollbit’s analysis found that robots.txt instructions are ignored on average about 30 percent of the time, and the figure rises to 42 percent for ‘ChatGPT User’.
That kind of noncompliance turns what used to be a gentleman’s agreement into a weak suggestion. For publishers, the damage is not only bandwidth costs and server strain, though those can add up quickly when bots hit pages at scale. The bigger issue is losing control over how and where content appears, especially when it is summarized without context or attribution strong enough to drive readers back. When an AI answer satisfies the user, the publisher may get none of the value despite bearing the reporting costs.
Click through rates are already showing the impact. Tollbit says websites without direct licensing agreements with AI companies saw their CTR drop by roughly three times between the second and fourth quarters of 2025. Even publishers that do have licensing deals are not immune, because their CTR is also declining. In practice, a licensing check may help with compensation or permissions, but it does not automatically restore the habit of clicking through to the original page.
This is why Tollbit is urging regulators to step in and define acceptable patterns of AI behavior while protecting intellectual property. The question is not whether bots should exist, because search and archiving depend on crawling. The question is how much copying, summarizing, and redistribution should be allowed when it replaces the visit that once paid for the content. Without clearer rules and enforceable standards, the web could drift toward a future where sites are primarily written for machines that ingest them, not for people who read them.
To understand the stakes, it helps to know a few basics about how these systems work. RAG, short for retrieval augmented generation, combines a language model with live or recent information fetched from external sources. That can reduce hallucinations, but it also increases the incentive to harvest vast amounts of publisher content in real time. CTR, or click through rate, measures how often users click a link after seeing it in search results or recommendations, and it is a key driver for ad revenue and subscription funnels. When AI answers replace the link entirely, CTR naturally falls, and the economic pressure shifts from distribution platforms to the publishers who still pay for reporting, editing, and hosting.
Share your thoughts in the comments on whether AI assistants should be required to send more readers back to original websites.
