The Agent Internet: How Cloudflare's Pay-Per-Crawl Changes Who Pays for Your Data

Nobody visits your site anymore. Charge the agents instead.

Cloudflare made it possible to charge the machines instead.

 Here's the full breakdown — concept by concept.

1. THE OLD BARGAIN IS BREAKING

→ You gave data to search engines, they sent humans who bought things.

→ Now agents read the answer and leave. The audience never arrives.

 2. AGENTS TREAT THE WEB AS INFRASTRUCTURE

→ They request data, comparisons, APIs and tools, then pay tiny fees.

→ Humans refuse micro-payments per page. Machines pay without blinking.

 3. HTTP 402 IS THE NEW CHECKOUT

→ AI crawl control lets you see, allow or block every AI crawler.

→ Pay-per-crawl answers the rest with 402 Payment Required.

→ No account, no cart, no invoice. The request is the transaction.

 4. THE STACK BEHIND THE FUTURE

→ Messy inputs first: PDFs, pricing pages, blogs, changelogs.

→ A refinery that turns them into clean, structured, dated records.

→ Then agent-readable delivery: APIs, MCP tools, payment rules, trust signals.

 5. IDEA ONE — NICHE DATA REFINERY

→ Pick a niche where valuable information is fragmented and changes weekly.

→ Med spas: track prices, reviews, hiring, offers. Sell reports to agencies.

→ Start manual in a spreadsheet. Grow into an API and pay-per-use.

 6. IDEA TWO — AGENT READINESS

→ Audit a company's pricing, docs and FAQs the way an agent reads them.

→ Fix stale or hidden info. Ship structured files, feeds, MCP endpoints.

→ Sell the audit first, then monthly reporting, then software.

 7. IDEA THREE — EXPERT ARCHIVES AS TOOLS

→ Experts sit on archives: podcasts, videos, blogs, client notes.

→ Tag them deeply and ship one focused workflow per tool.

→ A cold email critic. An idea validator. Subscription or licence.

 8. THE FOUR QUESTIONS TO ASK

→ What decision is expensive? What information is messy and changes often?

→ Who already pays for help? What can agents do better than humans?

 THE MASTER FORMULA

Messy data = the raw material.

Structure = what makes it usable.

Payment = what makes it a business.

Trust = what makes them come back.

You don't have to wait for the infrastructure to finish being built. Start manual, start small, own the tiny paid doors agents use all day.