• By ignoring certain protocols, AI agents and crawlers overload processors and bandwidth, forcing publishers to fund costly infrastructure that generates no revenue.
• Web players are attempting to turn these bots into paying customers through paid feeds, at the risk of creating a two-tier internet between large corporations and small and medium-sized businesses.
Popular AI tools like Claude, ChatGPT, and Gemini access websites to summarize their content, aggregate news, and more. Publishers place great importance on search engine optimization (SEO) for their sites — including through these tools — to improve their visibility. However, these activities place a strain on hosting infrastructure, particularly because AI bots often ignore standard caching protocols to ensure they retrieve the most recent version of a page. In fact, a bot increases both bandwidth usage and CPU load (rendering, databases, APIs) on infrastructure. According to the 2026 AI Bot Impact Report, bots account for 52% of all global web traffic (between 30% and 50% according to Akamai). A resource crisis is developing: the report indicates that AI and LLM indexing bots have quadrupled their share of traffic in just eight months, rising from 2.6% to 10.1%. OpenAI’s GPTBot alone has seen 305% growth.
Publishers face a paradox: if they decide to block some bots, their content will not be indexed by AI tools and the next generation of search engines
Alex Taylor, the founder of Blankspace, notes that a new wave of bots is consuming web content in a way that cannot be detected: “These LLM agents crawl and retrieve content directly from the HTML/DOM without executing JavaScript, thereby bypassing popular analytics platforms (such as Google Analytics) that rely on in-browser scripts.” In other words, traffic generated by AI bots is likely being underestimated.
Additional costs are affecting publishers
The rapid requests from bots, explains Philippe Ensarguet, VP of Software Engineering at Orange, in an op-ed, “put a heavy strain on databases and content delivery networks. Worse still, they often bypass or override caching mechanisms, forcing requests to go all the way back to the origin servers. This route is much more costly.” In other words, publishers must absorb a rapid rise in infrastructure costs caused by AI bots.
“For companies operating on tight margins or fixed IT budgets, these unforeseen costs quickly turn into painful trade-offs: which services should they continue to run? How should resources be allocated between paying customers and indexing bots that will never buy anything? And that is where the real problem lies. These additional costs generate no revenue.” According to Akamai, 63% of recorded AI bot triggers target the publishing sector, which impacts the business model of publishers that rely on advertising.
Toward mandatory monetization
Publishers face a dilemma: if they decide to block some bots, their content will not be indexed by AI tools and the next generation of search engines like Perplexity. This is why we are seeing the emergence of new tools that allow publishers to monetize traffic from AI bots. The idea is to transform AI bots into paying feed or license customers through the implementation of dedicated APIs or “AI feeds,” via contracts, subscriptions, or content licensing agreements for model training, in exchange for access to better-structured data. This requires the ability to identify AI agents and distinguish a legitimate bot from an aggressive scraper. In any case, cloud providers will need to adapt and offer specific solutions, whether through filtering options or partnerships with AI companies.
Existing solutions designed to help publishers regain control require a coherent strategy — technical, economic, and legal — to avoid undermining their business models. Companies that treat AI bots as a new category of customers will transform a cost center into a strategic asset, provided they are able to measure this traffic, have protocols in place, and are capable of negotiating with AI providers. This requires resources that only certain publishers possess. “The risk is that a two-tier internet will emerge. Large companies have the means to adapt, but SMEs are seeing their cloud costs rise without understanding why,” notes Philippe Ensarguet. “Edge computing is often cited as a solution. The idea is appealing: serving bots from nodes close to them to reduce transfer costs. While this reduces the unit cost of the request, it doesn’t address the traffic volume at the source. A bot that makes 10 million requests a day will continue to do so.”
This text has been translated by an artificial intelligence.
