Cloudflare's AI Crawler Change and How to Stay Visible in AI Search
On July 1, Cloudflare announced that it would block AI crawlers from accessing content without permission by default, and that every new domain would be asked whether to allow them. Publishers welcomed it. For ecommerce brands that want to be found in AI answers, it's a setting worth checking.
Published Jul 9, 2025. Author: Ecomm Guys. 4 min read.
Key Points
- Cloudflare says it handles traffic for about 20% of the web.
- New domains signing up to Cloudflare are now asked whether to allow AI crawlers, and blocking is the default.
- Cloudflare first offered a one-click AI crawler block in September 2024, and more than one million customers turned it on.
- AI crawlers can now state whether they crawl for training, inference, or search.
- Brands that want to appear in AI answers should check their AI crawler settings in Cloudflare.
What Cloudflare Changed
Cloudflare protects and manages traffic for about 20% of the web, by its own figures. In September 2024 it added a one-click option to block AI crawlers, and it says more than one million customers chose it. On July 1, 2025 it went further. Every new domain that signs up is now asked whether it wants to allow AI crawlers, so new sites start with control by default instead of having to opt out.
Cloudflare also says AI companies can now state the purpose of their crawlers, whether training, inference, or search, so site owners can decide which to allow. And it announced a private beta called Pay Per Crawl, which lets publishers charge AI crawlers for access.
Why Publishers Pushed for It
Cloudflare's CEO Matthew Prince set out the reasoning in a blog post. The old deal was that search engines copied content and sent traffic back. He argues AI tools break that deal, because they answer questions without sending visitors to the original source. He also cites Cloudflare's own estimates that it has become far harder for content creators to earn traffic from AI tools than it was from traditional search.
For a news publisher, blocking AI crawlers protects the content they sell. For a store, the calculation is usually different.
Why Ecommerce Brands Are Different
Most stores don't sell their content. They sell products, and their content exists to help people find and choose those products. If shoppers ask an AI assistant which brand makes a good waterproof jacket, you want the assistant to know your products exist, what they cost, and where to buy them.
That doesn't mean every AI crawler should be allowed. Some brands reasonably want to keep training crawlers out while allowing crawlers that power search and answers. The new purpose labels help with that. The point is to make a deliberate choice.
Where to Check
Where your settings live depends on how your sites are set up.
- In your own Cloudflare account, check the AI crawler and bot settings for each domain, including any blog, landing page, or headless storefront you run.
- If you add a domain to Cloudflare after July 1, review the AI crawler question during sign-up before you click through.
- Check your Shopify robots.txt too. If you or a developer edited robots.txt.liquid, confirm you aren't blocking crawlers you want.
- Some security apps and bot management services add their own blocking, so ask your providers what they block by default.
Test What AI Tools Can See
After checking settings, ask a few AI assistants direct questions about your brand and products: what you sell, your prices, your shipping policy. If the answers are wrong or missing, look at whether crawlers can reach your key pages, and whether those pages state the facts plainly in text. Repeat the check every few months, because both the tools and the defaults keep changing.
Keep a simple record of what you asked and what each tool answered. Over time it shows whether changes to your settings or content made a difference. It also gives you something concrete to bring to your developer or SEO partner if a tool keeps getting your prices or policies wrong.
Make It a Deliberate Choice
List every domain your brand uses and who manages it. For each, confirm the AI crawler setting is a choice someone made on purpose. Then decide as a team which kinds of AI crawlers you want to allow, write it down, and apply it consistently.
Want to know how visible your store is to AI search? Book a call with our team.
Sources
- Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large Cloudflare, July 2025
- Content Independence Day: no AI crawl without compensation! Cloudflare Blog, July 2025