Cloudflare Declares War on AI Crawlers! Enables HTTP 402 to Charge for Content Access

GM,

I’m happy to be back at work. Over the past two weeks, I participated in an interview, recorded two podcast episodes, and cleared a long backlog of to-dos. It might not seem like a break, but for me, getting long-pending tasks off my plate is the most relaxing kind of rest.

Let’s dive in.

If you’ve ever published content online, starting today you should get to know a company called Cloudflare. As the global leader in web firewalls, Cloudflare handles over 20% of all HTTP traffic worldwide. Governments and enterprises alike rely on it to fend off DDoS attacks from hackers.

Recently, Cloudflare announced a new “Pay per Crawl” model — essentially telling AI web crawlers: “No payment, no access.” CEO Matthew Prince even revealed that in the future, Cloudflare may support stablecoin payments — and might even launch its own stablecoin as a payment method.

This move has sparked global debate. Some see it as a long-overdue effort to stand with creators and defend original content. Others criticize Cloudflare for potentially hurting creators, arguing that AI crawlers won’t pay — they’ll simply avoid these sites, making original content even harder to find. But is that really the case?

To understand why Cloudflare launched this service, and how it's connected to cryptocurrency, we need to start with the unraveling of what was once the “social contract of the web.”

The Social Contract of the Web

Thirty years ago, as the internet was just taking shape, two graduate students at Stanford University launched a project called Backrub. Its goal was to build a library-like query system for the web — users could type in keywords and find relevant webpages. The system’s design was based on the citation logic of academic papers: it ranked results by the number of inbound links, and whether those links came from authoritative sources. The more backlinks a page had — especially from trusted sites — the higher it ranked.

Those two students, of course, were Larry Page and Sergey Brin, the founders of Google. Google established a system of reciprocity on the web: content in exchange for traffic. Creators allowed Google to index their work, and in return, the search engine drove users to their websites. Creators could then monetize that traffic through ads, subscriptions, or simply the satisfaction of being read.

But this “social contract” has been rapidly breaking down in the age of AI. AI crawlers scrape content from creators, and present direct “answers” on screen. Users no longer need to click links or visit original websites. As a result, website traffic has plummeted — people aren’t seeing ads, and no one knows who the original author even was.

Cloudflare CEO Matthew highlighted the severity of the problem with a startling statistic: ten years ago, Google brought one visitor for every two times it crawled a webpage — a very high conversion rate. But today, it takes 18 crawls to bring in just one visitor, meaning the conversion difficulty has increased ninefold. Even more extreme, OpenAI needs 1,500 crawls per visitor, and Anthropic requires a staggering 60,000.

The more people trust the answers AI delivers, the less they feel the need to return to the source and verify — and website traffic continues to fall. No wonder Matthew remarked that AI is rapidly dismantling the web’s original “content-for-traffic” social contract. Even if some creators aren’t writing for money, it’s hard to stay motivated when no one reads your work.

Pay-Per-Crawl

To address this issue, Cloudflare announced a Pay-per-Crawl mechanism and is inviting publishers and creators to opt in:

In the past, content creators had only two choices — allow AI crawlers to access their content for free, or block them entirely. What they wanted was a third option: allow access, but get paid for it. Cloudflare’s new Pay-per-Crawl mechanism enables just that. When an AI crawler accesses content, the server returns an HTTP 402 status code, signaling that payment is required. The AI agent can then pay directly or be billed via Cloudflare on behalf of the site. Website owners can decide whether to charge, how much, or to block crawlers altogether — putting control back in the hands of creators.

This system complements the x402 payment protocol previously covered by Blocktrend. x402 is a stablecoin-based payment system specifically designed for AI crawlers, guiding them to pay for the content they access. However, x402 only defines how sites charge and how crawlers pay — it doesn’t erect a fence to stop unpaid access. Think of it like a theme park with a ticket booth and posted entry fees, but no actual gates or fences.

Fortunately, Cloudflare is the world’s most capable company when it comes to building digital fences — keeping unpaid AI crawlers out. As Matthew memorably put it: “For the past decade, we’ve been blocking malicious bots. Now we’re just dealing with AI crawlers instead.” Whether the bot behind the request is from a hacker or an AI firm, Cloudflare has the power to stop them at the door.

Cloudflare’s statements were bold — directly challenging the way AI crawlers currently operate. The prevailing logic among AI crawlers is: if your door isn’t locked, it means I’m free to come and go as I please. Many people intuitively feel there’s something off about this logic, but can’t quite put their finger on it. Matthew explained the fundamental difference between traditional search engines and AI crawlers through the lens of the internet’s social contract. Since AI has broken that contract, he argues, it’s time to rewrite the entire value exchange framework — and the simplest form of exchange is to have users pay.

Some critics argue that the Pay-per-Crawl model will only make information less freely accessible. This reminded me of a distress call issued a few months ago by the Wikimedia Foundation — the organization behind Wikipedia — which warned that its infrastructure was being overwhelmed by unpaid AI crawlers:

As demand for AI training data surges, more and more crawlers are harvesting massive amounts of openly licensed images and media from Wikimedia Commons. These non-human requests not only fail to credit the original sources, but also place a heavy burden on our infrastructure. We've found that 65% of our most expensive traffic now comes from AI crawlers. These bots often bypass caching systems and directly request obscure pages from our data centers, increasing operational strain. While our content is open, our infrastructure is not. We call on developers to adopt responsible usage practices — otherwise, they risk diverting resources away from the contributors and readers our platform was built to serve.

Human behavior is very different from that of crawlers. Most people tend to access similar, high-traffic pages, which allows caching systems around the world to handle the load efficiently. But AI crawlers behave differently — they want everything, regardless of whether it's popular or obscure. When caching can’t fulfill their requests, they hit the data centers directly. As the number of AI crawlers continues to grow without contributing donations or financial support, even Wikipedia — a platform that has never charged users — finds itself at a crossroads.

One-sided extraction is clearly unsustainable. But the real question is: will AI crawlers actually be willing to pay? Ultimately, it all comes down to the kind of content you're producing.

Will crawlers pay?

Matthew used a striking analogy: every AI model is like a block of Swiss cheese riddled with holes. Only new and original content that can fill those holes is something AI crawlers are willing to pay for. His argument aligns with what I previously discussed in “AI Doesn’t See Ads—So Who Pays for Online Content?”:

Mass production, SEO-gaming, and clickbait tactics are the dominant strategies for many content websites. The rise of generative AI has only intensified this "content farm" competition... People often say that in this era, those who use AI will replace those who don’t. If you don’t use AI to churn out content quickly, someone else will grab the traffic before you. But that’s just drinking poison to quench your thirst. The more AI is used, the less originality remains. For AI, duplicated content is worthless. Advertisers used to pay for clicks and attention — now, AI is paying for the content itself.

What AI wants is content that’s structurally sound, opinionated, and backed by sources. It doesn’t fall for clickbait headlines and can instantly spot whether a “10,000-word article” is just empty fluff. In other words, high-quality original content will become the preferred data source for AI. As long as the AI determines “this is worth reading,” it will pay for access. Content creators who produce work AI is willing to buy will emerge as the new winners.

Once we understand why AI crawlers are willing to pay, the next question is how they will do so. At present, Cloudflare does not use cryptocurrency. Instead, it requires both AI crawlers and websites to register accounts on Cloudflare and input their bank payment details. Cloudflare periodically bills the crawlers and then distributes revenue to the content creators. But if either party fails to complete the setup, the transaction can’t proceed.

I suspect this is only a temporary solution. Cloudflare’s core business is bot mitigation — not payment processing. As the system scales, it will inevitably encounter complex issues like cross-border payments and compliance. After all, not every content creator around the world has access to a traditional bank account. When that time comes, stablecoins will likely be the most lightweight and natural solution.

Cloudflare isn’t just blocking AI crawlers — it’s helping to rewrite the internet’s social contract. As a creator whose every article is original, I’m excited about this shift. The Pay-per-Crawl model signals a new era for the internet — one that moves away from traffic is king toward value is king. Creators who can consistently produce high-quality original work will no longer need to obsess over traffic; instead, they’ll be rewarded directly based on the intrinsic value of their content.

And what if AI crawlers decide to avoid paid websites altogether? My answer: let them. Just as not everyone is willing to pay for a Blocktrend subscription, AI crawlers are simply reflecting the values of their users. Blocktrend’s mission is to serve those human readers — and their AI crawlers — who are willing to pay for good content.

p.s. The crypto payment card EtherFi Cash is currently running a 14-day referral competition (from July 11 to July 25), with a total prize pool of 20,000 $ETHFI tokens. The rewards will go to the top 4 referrers — either by number of successful referrals or by total spending generated.

Although I’ve already referred 423 readers to sign up and am currently the top global referrer, this competition only counts new referrals made during the event period. The rules are a bit sneaky, so I’ve decided to participate in a laid-back, no-pressure way. If you happen to be interested in applying for the card, feel free to use my referral link to register. Or if you already have the card, spending more will also help me earn extra points. Thanks for your support!