AI Crawler Pricing: 9 Checks Before You Allow, Charge or Block

Make the access decision reviewable before optimizing the price

It is tempting to describe an elaborate price curve as a way to defeat autonomous scraping. That claim skips the operational questions: does the rule apply, what happens before it, which price reaches the requester, and what evidence establishes a paid result?

DN’s thesis is that a publisher first needs a traceable access policy. This article uses Cloudflare’s documented Pay Per Crawl flow as a concrete example and supplies a local record for reviewing its declared branches. It does not rank monetization products or promise that pricing can stop every unwanted crawler.

1. Name the publisher decision and request scope

Start with the outcome you want from a specific kind of request. You might want a search service to discover public pages, a licensed crawler to retrieve paid material, or an unwanted request to be blocked. “Stop AI scraping” combines several decisions before the evidence has separated them.

Define the zone, path or content group, crawler classification and observation time. Record the policy version you are reviewing. The same crawler may encounter a free discovery page and a paid article; the same article may be subject to a security rule that overrides its commercial policy.

The worksheet uses one scope alias rather than collecting real request headers or identifiers. Keep the detailed mapping in your own records. The alias should connect the declared rules and any observed charge to the intended request context. It does not authenticate a crawler or prove that the runtime selected those rules.

Decide what would count as an informative result. A configured charge action, a price offered in a response, a successful content retrieval and a publisher payout are different observations. An experiment becomes easier to assess when its objective names one of them.

DN’s agentic web readiness guide covers the broader website question. This article focuses on access-policy precedence and the evidence needed to interpret a price for one documented flow.

2. Verify that the paid-access product applies

A public product page is not proof that a feature is enabled for your account, zone or crawler. Before modelling a commercial result, establish which implementation you are actually using and what conditions apply to participation.

Cloudflare’s Pay Per Crawl overview still labels the feature a closed beta in the documentation reviewed for this article. It describes a zone-based paid-access flow and identifies Cloudflare as the Merchant of Record. This is a specific product example, not a protocol that every website or autonomous scraper necessarily uses. [1]

The tool therefore asks whether the reviewed request falls within the documented Pay Per Crawl flow. “No” means the case is outside this model; it does not mean the page must be freely accessible. “Unknown” keeps applicability unresolved rather than treating a familiar header name as enrollment evidence.

For a hosted service, retain the relevant account and feature evidence, operating terms and current configuration reference. For a custom implementation, write down its own rules rather than borrowing conclusions from a different product’s documentation.

There is no recommendation here to buy a plan or migrate a site. The useful question is whether the chosen implementation can support the intended access decision under its actual scope. Pricing experimentation should start after that question has a documented answer.

3. Check for a block before interpreting a charge

A commercial action cannot be understood in isolation from the controls around it. An operator may select “charge” and still see no paid delivery because another applicable rule blocks the request first.

Cloudflare states that a WAF or Bot Management block overrides Pay Per Crawl’s charge feature. Its crawler controls separately offer allow, charge and block actions. Those declarations are not interchangeable steps toward the same result. [1] [2]

The first invented case pairs a charge selection with an applicable security block. The price branch stays inactive in the tool. It reports the block declaration instead of multiplying a price by imagined successful requests.

A known block remains visible when other details are missing. Conversely, “not checked” is not equivalent to “no block.” If you have not reviewed the relevant security layer, a free-access or bypass declaration cannot become proof that the request will reach the content.

This is a limited precedence trace, not a complete firewall simulator. The page does not inspect rule order, managed challenges, authentication middleware, origin authorization or the deployed request path. Preserve the actual rule reference and request evidence before deciding that the commercial configuration caused an observed outcome.

4. Preserve free paths and deliberate exceptions

A publisher may need some resources to remain discoverable while applying a price to other material. An exception is part of the policy, not necessarily a failure of the paid-access system.

Cloudflare’s advanced configuration documentation supports disabling Pay Per Crawl for URI patterns and lists always-free discovery or security paths. Its June 2026 change notice also introduced path-based exclusions and dynamic pricing. [3] [4]

Record why an exception exists and which requests it covers. A homepage, navigation resource and licensed dataset can have different commercial purposes. Do not assume that the price shown for one article applies to every request from the same crawler.

The worksheet’s path-bypass declaration concerns the charging flow only. It does not override an applicable block. If a request is both within a declared bypass and subject to a declared block, the block stays visible. If the security layer is unknown, the bypass does not manufacture an access grant.

Keep intended exceptions separate from accidental gaps. An operator reviewing a zero-charge event needs to know whether the request matched an approved free path, an allow policy, an unavailable feature or an unresolved configuration. The same zero revenue observation can accompany several different situations.

5. Identify which price source actually wins

A default price is a fallback rule, not necessarily the price associated with every request. A reliable record preserves the selected source and the evidence that made it applicable.

With Cloudflare dynamic pricing enabled, the advanced configuration guide says an origin response’s crawler-price replaces the zone default when present. It also describes using a Worker to select content-based prices. The setup page separately documents the default per-crawl price and a USD 0.001 minimum. [3] [5]

The tool follows only that bounded selection logic. Zone-default mode uses the entered default. Dynamic mode with a present price header uses the entered response price. An absent header uses the default. Unknown header presence leaves the source unresolved. An empty or malformed present-header value is not silently replaced with a more convenient default.

The invented dynamic case has a USD 0.01 default and a USD 0.03 response price. The selected input is USD 0.03. This does not establish that the live system accepted the value, that the buyer agreed to it or that the publisher collected it.

Read dated documents together. The older FAQ describes one zone price for charged crawlers, while the newer change notice and advanced guide describe dynamic content pricing. That supports the documented content-based branch; it is not evidence for every proposed per-crawler discrimination strategy. [4] [8]

6. Keep the price offer separate from payment evidence

A price says what is being requested under a policy. It does not prove who made the request, what the requester authorized or whether a successful paid retrieval occurred.

Cloudflare’s crawler-side documentation distinguishes crawler-exact-price from crawler-max-price, requires payment headers to be covered by Web Bot Auth signatures, and uses crawler-charged on a successful HTTP 200 response to report the amount billed for the request. [6]

This article does not implement those headers or validate signatures. The tool accepts only a separately recorded charge amount and a local evidence reference. If a selected price and reported charge differ, it exposes the literal difference for investigation. It does not declare that the provider overcharged or that the report is authentic.

Make sure a comparison concerns the same request context. A cached screenshot of a default setting and a charge from another path are not enough to establish a billing error. Neither are two values collected across an unrecorded policy change.

A price response is also not a complete content licence. Keep the applicable rights and contract terms in the review packet. The worksheet does not decide whether a buyer can reuse, train on or redistribute the retrieved content.

7. Reconcile delivered access, accrual and cash separately

Revenue reporting should follow the evidence through its stages. A price multiplied by a request count is an assumption. A recorded charge event is a different observation. Actual publisher cash has its own timing and reconciliation record.

Cloudflare’s payout documentation describes charge recording on successful HTTP 200 delivery, aggregation and reconciliation, and monthly payments to publishers in good standing. It also states that payouts are subject to settlement periods and minimum thresholds. It requires a dedicated Stripe Connect setup for this feature. [7]

Those conditions are why this tool stops before calculating publisher revenue. It has no request-volume input, take-rate assumption, payout forecast or profit estimate. A reported charge is retained as evidence to review, not converted automatically into collected cash.

For an operating report, distinguish the period of content access, the billing period and the period when funds arrive. Explain adjustments and costs using the relevant statement. Do not imply that an unchanged price guarantees unchanged demand or that every request is chargeable.

DN’s seller terms and margin-floor guide addresses the broader commercial terms. This access trace identifies the narrower point at which a pricing claim still needs delivery, billing or cash evidence.

8. Measure the cost of losing useful crawler access

A crawler can be a potential payer, a discovery channel or an unwanted consumer of resources. The appropriate decision depends on the publisher’s objective and the evidence for that crawler’s role.

Cloudflare warns that blocking or charging crawlers classified as search-engine crawlers may negatively affect SEO because content may not be indexed properly. The documentation provides a category distinction; it does not forecast the traffic or revenue impact for a particular site. [2]

Before changing a policy, identify the pages and outcomes you want to observe. Keep a dated baseline and a record of other changes that could affect the result. A fall in crawl activity does not alone establish a commercial improvement, and an increase in apparent AI traffic does not alone establish valuable referrals.

The tool deliberately avoids an “optimal price.” There is no demand curve, universal value per crawl or assumed referral conversion rate. A policy that maximizes one immediate metric may reduce another outcome the publisher cares about.

A useful review can conclude that the evidence is not yet sufficient to choose between allow and charge. That is a concrete finding when it names the missing measurement and the affected scope. It is better than describing a speculative pricing formula as a proven defence against autonomous scraping.

9. Preserve the policy revision and test its claim

Finish with a statement that another reviewer can reproduce. Name the intended request, declared controls, selected price source and any observed charge. Keep configuration evidence separate from runtime evidence and from business results.

The export is designed for that handoff. It preserves the raw values, active price branch, unused prices, scope declarations, charge reference, literal comparison and unresolved fields. Editing any input invalidates the previous downloadable result. Loading an invented example is explicit, and modifying it does not erase its invented provenance.

A useful conclusion might be: “This request is declared blocked despite the charge selection,” or “Dynamic pricing is declared enabled, but header presence is unknown.” These findings point to a specific next investigation without changing a live rule.

Do not give an autonomous operator a broader mandate merely because the record is complete. Permission to review a pricing policy is distinct from permission to deploy a WAF rule, change a price, block a search service or alter a licence. Agree who owns each action and what evidence must be reviewed afterward.

The article’s thesis is deliberately narrower than defeating all scrapers. A policy becomes operationally useful when its applicability, precedence and result can be explained. Pricing is one part of that system; it does not replace authentication, access controls, rights management or business judgment.

Invented cases and bounded conclusions

These are authored records, not observations of a real site or proof of a product’s effectiveness.

Try changing a present-header price to blank. The selected branch remains the dynamic response branch, but its amount becomes unknown; the tool does not silently use the default. Change the header state to absent and the documented default branch becomes available.

A reported charge alongside a block declaration prompts reconciliation. It is not automatically discarded or declared fraudulent. The request identity, policy version and actual records may explain why the two observations do not belong together.

Methodology and limits

The model traces one limited documented Cloudflare flow using reader declarations. A separately declared applicable security block remains a block finding. A selected crawler block requires declared flow applicability; an inapplicable-flow declaration creates a scope conflict instead. Otherwise it records whether the flow applies, whether a charging bypass or allow policy is declared, and whether the charge branch has enough information to select a price. It does not simulate the complete WAF, origin or authentication pipeline.

Price selection is active only when applicability is yes, the applicable security block is no, bypass is no and the crawler action is charge. Zone-default mode selects the entered default. Dynamic mode selects a present response price, uses the default when the header is declared absent, and selects neither when presence is unknown. Missing or invalid active price text does not trigger a fallback.

USD inputs accept up to 12 whole-number digits and 6 decimal places. Exact millionth-dollar integer arithmetic calculates an optional literal reported-charge-minus-selected-price difference. These are worksheet bounds, not a provider tariff format. Blank is unknown and zero is known. A zone-default input below the documented USD 0.001 minimum receives a warning; retaining it does not validate the configuration.

Context and arithmetic are independent. The reader declares a match and supplies scope and snapshot aliases; the tool verifies none of them. A reported charge needs a reference before it is labelled reader-recorded evidence. A missing reference, mismatched scope or conflicting policy record remains visible without rewriting the raw observations.

No request counts, expected demand, profit, payout percentage or revenue forecast are modelled. Access granted and publisher cash remain “not assessed” for every result. Price equality proves only equality of entered amounts. The record does not confer a content licence, establish bot identity or defeat evasion.

Clear resets inputs and provenance. Editing invalidates the displayed record and disables its download; download also checks the raw snapshot again. JSON exports include raw entries, inactive values, independent findings, gaps and errors. Schema validation concerns structure, not evidential truth. Download time does not refresh the source review.

Frequently asked questions

Primary sources

The source list establishes the stated documentation scope. It does not authenticate reader-entered records or certify a deployed system.

1. Cloudflare: What is Pay Per Crawl? — Closed-beta status, Merchant of Record role and the documented security-block override. Updated 28 July 2026.
2. Cloudflare: Select crawlers to charge — Allow/charge/block actions and search-engine-crawler indexing caveat.
3. Cloudflare: Pay Per Crawl advanced configuration — URI exclusions, dynamic origin-response price selection and its place in processing. Updated 28 July 2026.
4. Cloudflare changelog: Pay Per Crawl advanced configuration — Dated 16 June 2026 introduction of path exclusions and dynamic pricing; distinguishes this from older FAQ wording.
5. Cloudflare: Set a pay per crawl price — Default price setup, successful HTTP 200 retrieval basis and USD 0.001 default minimum.
6. Cloudflare: Crawl pages as an AI owner — Exact/max payment intent, signed header requirement and successful-response charge evidence.
7. Cloudflare: Manage payouts — Charge recording, reconciliation, dedicated Stripe Connect setup and payout limitations. No numeric payout threshold is assumed.
8. Cloudflare: Pay Per Crawl FAQ — Older 23 April 2026 wording on a single zone price is read with the newer dynamic-pricing documentation, not treated as a prohibition on content-based pricing.

Related DN research

• The agentic web readiness index
• Terms to set before buyer agents negotiate an invoice
• x402 agent-payment cost and reliability boundaries
• Preserving purchase evidence from approval to refund