Cloudflare Turned a Dead Status Code Into an Invoice for AI. Brands That Tune Access Now Stay Visible and Get Paid.

For two years, AI crawlers ate for free. They took the content and left almost nothing trackable behind. The worst part is that there was no way to make them pay. Cloudflare stepped in to change this, and now every brand that publishes a word has options.

Flipping this default rewired the economics of the whole open web. Whether a brand appears in AI answers or disappears now comes down to a single setting. That setting belongs to the brand, not the AI company taking the content.

How the model works

New sites on Cloudflare now block AI crawlers unless the owner allows them, and for each crawler a publisher can grant free access, charge a per-crawl fee, or block it outright. The mechanism runs on HTTP 402, a 'Payment Required' response that sat dormant in web infrastructure for three decades and now carries the price an AI company must pay to read a page.

Cloudflare's own data showed OpenAI's crawler hitting sites roughly 1,700 times for every referral it sent back, and Anthropic's ratio running far higher still. Cloudflare's Will Allen frames the model as a 'yes, if' rather than a 'no,' a programmatic way for a crawler to pay for access in real time.

Real companies are already charging

Stack Overflow is already charging. It co-launched pay-per-crawl with Cloudflare to capture demand its formal licensing deals were missing, treating heavy bot traffic as a signal of commercial interest worth a price. The company already licenses its data to model builders, and pay-per-crawl lets it meet the rest of the demand where it shows up, charging the bots scraping for training while keeping the door open to partners.

The choice every brand now has to make

There is no version of this that costs nothing. Block the crawlers and the content stays protected while the brand disappears from AI search, where being cited in ChatGPT, Perplexity, and Google's answers has become a primary way to get found. Allow everything and the work goes out the door while AI search returns a fraction of the referrals that traditional search once did. The smart play is to sort the crawlers: charge the crawlers that train on the content, and let through the ones that drive citations and referrals.

A decision that once meant knowing the right person at an AI company and cutting a one-off deal is now just a setting. It comes down to pairing access control with a self-healing strategy for staying cited in AI answers, because visibility and compensation are now two sides of the same content decision. The one-way deal is done. Whether that pays off or backfires comes down to how the dial gets set.