AI crawlers pay to read your site.
Add one middleware file. Humans and search engines walk through as before. GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers get a 402 with a price, pay your Solana wallet in USDC, and are let in.
The money goes to your wallet directly. TOLL takes no cut.
- 1Crawler arrives
- 2Pays the toll
- 3Reads the site
Pay per crawl: what changes when a page costs 0.002 USDC
AI crawlers read the web for free and the sites they read get nothing back. A toll puts a price on each page: one HTTP status, one transfer, one retry.
A crawler announces itself in its User-Agent header: GPTBot, ClaudeBot, PerplexityBot, CCBot and Bytespider all do. A site that knows the names can answer each of them differently. Browsers and search engines such as Googlebot are served exactly as before.
The answer to an AI crawler is HTTP 402 Payment Required, a status code reserved since the first HTTP specification and rarely used. The response carries the price, the asset (USDC), the network (Solana) and the owner's wallet address.
The crawler pays that wallet directly. There is no middleman: no balance is held and no fee is taken. A transfer of 0.002 USDC settles in seconds and costs a fraction of a cent in network fees.
The crawler retries with the transaction signature in a header. The gate confirms the transfer on-chain and serves the page. Paying for fifty pages in one transfer gives a pass good for fifty crawls.
Site owners set the price and a rule per bot: charge, allow free, or block. Crawler operators get a protocol they can read in ten minutes and a cost they can budget for.
One gate, four moves
The same exchange as above, slowed down. Left: what happens at the gate. Right: the real HTTP traffic.
Humans and search engines pass
The middleware reads the declared user agent. Browsers, Googlebot and Bingbot are served as before. Nothing changes for your readers or your search ranking.
> GET /article HTTP/1.1 > User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html) < HTTP/1.1 200 OK < Content-Type: text/html > GET /article HTTP/1.1 > User-Agent: Mozilla/5.0 (a browser) < HTTP/1.1 200 OK
An AI crawler gets a 402
GPTBot asks for the page. The gate answers 402 Payment Required with the price, the asset and your wallet. No page content is sent.
> GET /article HTTP/1.1 > User-Agent: GPTBot/1.2 (+https://openai.com/gptbot) < HTTP/1.1 402 Payment Required < X-Crawl-Price: 0.002 < X-Crawl-Asset: USDC < X-Crawl-Network: solana < X-Crawl-Pay-To: <your wallet> < Content-Type: application/json
The crawler pays your wallet
It sends the USDC on Solana straight to the wallet in the response, with a memo that ties the transfer to your site. TOLL never holds the money. Several pages can be paid in one transfer.
Solana transfer 0.002 USDC -> <your wallet> memo toll:<siteId>:<sha256(secret)> Pay for 50 pages at once: 0.100 USDC -> <your wallet> same memo; the gate issues a 50-crawl pass
Retry, verify, serve
The crawler asks again with the transfer signature in a header. The gate checks the transfer on-chain and serves the page. The response carries a pass for any prepaid crawls that are left.
> GET /article HTTP/1.1 > User-Agent: GPTBot/1.2 (+https://openai.com/gptbot) > X-Crawl-Payment: <signature>.<secret> < HTTP/1.1 200 OK < X-Crawl-Pass: <pass token> < X-Crawl-Pass-Remaining: 49 < Content-Type: text/html
Ask the demo gate for a page
Your browser sends a real request to the demo gate. What comes back is printed unchanged.
The demo gate treats every request as an AI crawler, so you will see the 402 a crawler sees. The second button retries with an invalid payment header, which is what a crawler gets when it tries to skip paying.
One file in front of your pages
The snippet reads the user agent, answers 402 to AI crawlers, checks X-Crawl-Payment and X-Crawl-Pass, and passes everyone else through.
Next.js: drop it in as middleware.ts.
Express: app.use() it before your routes.
Cloudflare Worker: put it in front of any origin, no code change on the site itself.
The key goes in as the TOLL_KEY environment variable. You get one when you register a site.
Loading the snippet.Recent crawls
Every row is one request at a gate: challenged, paid, free or blocked. Paid rows link to the transfer on Solana.
Rows arrive over a server-sent event stream as gates report them.
How to pay
Three steps, no account, no API key. Everything you need is in the 402 response.
Read the 402. It carries the price, the asset, the network and the wallet to pay.
X-Crawl-Price, X-Crawl-Pay-To, X-Crawl-Asset, X-Crawl-Network
Send that much USDC on Solana to the wallet. Pick a secret, put its hash in the memo. Pay N times the price to get N crawls.
memo: toll:<siteId>:<sha256(secret)>
Retry with the signature and the secret. Keep the pass token from the response for the remaining crawls.
X-Crawl-Payment: <signature>.<secret>
Full protocol reference, including the bot list and error cases.