We ran into this while testing our own AI Answer Visibility Check against a batch of local business websites, checking what typical results looked like before we put the tool in front of anyone.

Two of those sites came back blocking their entire sites to GPTBot, ClaudeBot, and Google-Extended, which are the crawlers behind ChatGPT, Claude, and Google’s AI answers.

That’s a legitimate thing to do, and plenty of businesses choose it deliberately. What caught our attention was that these two had nothing to do with each other: different industries, different web developers, different stories. Yet their crawler rules were byte-identical, listing the same eight user agents and disallowing them in the same order.

Two owners don’t independently arrive at the same eight-item blocklist. Something upstream had already made the decision for both of them, and neither of them knew it.

Two sites out of a small sample isn’t a rate, and we won’t pretend otherwise. What makes it worth writing up isn’t how often it happens. It’s that when it does, nobody involved can see it.

What robots.txt is, and what these two sites were saying with it

Every website can publish a file at /robots.txt that tells automated visitors which parts of the site they’re allowed to fetch. Search engines read it before they crawl, and so do the AI companies’ crawlers. It’s the closest thing the web has to a posted set of house rules.

Here’s what both sites were serving. The file opened with a long comment block describing a “content signals policy,” then this:

User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The same eight crawlers, in the same order, on both sites: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, meta-externalagent. Read that list again and you’ll notice it’s alphabetical. People don’t alphabetize a blocklist. Software does.

The tell is the second line. Content-Signal: search=yes, ai-train=no, use=reference is not a phrasing anyone arrives at on their own. It’s a verbatim string, and it appears character for character in Cloudflare’s own managed robots.txt documentation as the default its managed robots.txt writes for you. If you find that line in your file, a machine put it there.

A content delivery network, or CDN, is a company that keeps cached copies of your website on servers around the world so pages load quickly wherever your visitors happen to be. The part that matters here is what that arrangement requires: every request for your site passes through them on the way to you, which means they’re in a position to modify what gets served. That’s usually a good thing. It’s how they compress images and absorb traffic spikes. It also means they can rewrite robots.txt in transit.

That’s exactly what was happening. Both sites were behind Cloudflare, and both had its managed robots.txt feature switched on.

Cloudflare announced this publicly, and the owners affected still don’t know

We want to be clear that we didn’t uncover anything here. Cloudflare announced the Content Signals Policy publicly in September 2025, said at the time that it was rolling the policy out across more than 3.8 million domains already using managed robots.txt, and documented the feature in full. It got a fair amount of coverage, from the SEO trade press through to mainstream tech outlets like Windows Central.

So the feature is public knowledge. What isn’t public knowledge is that it’s now sitting in front of a particular plumber in a particular city who has never heard of any of this, and who would be surprised to learn their site is telling ChatGPT to stay out.

That gap between something being announced and something being known by the person it affects is the whole reason we’re writing this up.

One line states a preference, the other actually blocks

Two things are happening in that file, and only one of them does anything.

The Content-Signal line is a statement of preference. It says search indexing is fine, training on this content is not, and AI answers may reference it. It’s a proposed convention rather than a ratified standard, and it carries no technical force: a crawler that ignores it faces no obstacle at all. Google has been blunt about this, telling Search Engine Roundtable that no crawler or large language model uses these directives and that they have no effect whatsoever. Cloudflare, to its credit, says something similar in gentler language: some companies will ignore the signals.

The Disallow lines are the operative part. Those are ordinary robots.txt rules, they predate all of this, and the major AI crawlers do honor them. That’s what actually removes a site from those systems.

So the practical effect is that these two businesses can’t appear in AI-assisted answers about their industry, and neither of them made that decision in any meaningful sense. Somebody ticked a box in a dashboard, whether their developer, their host, or whoever set up Cloudflare in the first place, and Cloudflare wrote the same file for both of them because it writes the same file for everyone who ticks it. That’s why two unrelated sites match byte for byte.

From September 15, 2026, new domains get this by default

Everything above describes a toggle somebody chose. On September 15, 2026, Cloudflare changes the default: for new domains onboarding after that date, AI training and agent crawlers will be blocked automatically on pages that display ads. Search crawlers stay allowed.

That’s a defensible policy, and Cloudflare is being open about it. But it moves this from something a developer did to you to something that arrives with the box. If you sign up after mid-September and never touch a setting, the decision has still been made, and the number of owners who don’t know it will stop being two.

Why your developer can’t find it

A developer looking for this won’t find it, and the reason changes who you should ask.

The rules aren’t in the site’s code. They’re not in the repository, not in the content management system, not in a plugin’s settings, and not in any file sitting on the server. The robots.txt a visitor downloads is assembled by the CDN on its way out the door. If you have an agency and you ask them why your site is blocking AI crawlers, they can look at everything they control and honestly tell you they can’t see it. From where they’re standing, it isn’t there.

That’s what moved this from an interesting finding to something worth writing about. Invisible defaults with commercial consequences are a bad kind of problem: nobody involved is doing anything wrong, and it can persist indefinitely.

Blocking may well be the right call

We’re not arguing that everyone should let AI crawlers in. There are real reasons not to:

  • Your content gets reproduced in answers with no visit, no attribution, and no way to follow up.
  • You publish something proprietary.
  • Crawler traffic is a real load you’re paying to serve.

Plenty of publishers have weighed exactly that and blocked on purpose, and Cloudflare built this feature because a lot of site owners wanted it. That’s a decision, and it’s theirs to make.

Our argument is narrower and harder to disagree with: whichever way it goes, it should be a decision. Right now, for an unknown number of small businesses, it’s a side effect of which hosting provider they ended up with.

For most local service businesses we’d probably lean the other way. If somebody asks an assistant for a plumber in your area, being eligible to be the answer is worth more than protecting the paragraph about your service area. But that’s our read on a typical case, not advice about yours.

How to check yours, in about two minutes

  1. Type your domain followed by /robots.txt into a browser: yourbusiness.com/robots.txt.
  2. Look for User-agent: lines naming GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, each followed by Disallow: /. If you also see a Content-Signal: line, that one is Cloudflare’s, and it wasn’t typed by a person.
  3. If they’re there, find out where they came from. If they’re not in a file your developer controls, check your CDN or hosting dashboard for an AI crawler or bot management setting. In Cloudflare that’s under bot management, as managed robots.txt.

Our AI Answer Visibility Check does the first two steps for you and reports each crawler separately. It’s free, and it doesn’t ask for your email.

We changed our tool to name the hosting layer

Our tool used to report a block like this and add, in effect, if you meant to do that, there’s nothing to fix.

That sentence is technically true and practically useless to somebody who never made the decision in the first place. It now recognizes the signature of a managed robots.txt and says so: that the block appears to have come from the hosting layer rather than from a choice about this business, and that it’s worth finding out which.

It’s a small copy change. It’s also the difference between a tool that reports a fact and one that tells you something you can act on, which is the only reason to build these at all.

We decided to allow every AI crawler by name

Ours didn’t name a single AI crawler: not to allow one, not to block one. Our own tool docked us points for that, and it was right to. Leaving them unmentioned is allowed-by-default rather than allowed-by-decision, which is exactly the thing we’ve spent this whole post objecting to.

So we made the decision. We let them all in, and we say so by name.

That includes the ones a lot of publishers hold back. Retrieval crawlers, the ones that fetch a page to answer a question right now, are an easy yes: that’s how we get cited. Training crawlers are the debatable ones, and we said yes to those too, including Common Crawl, the open dataset that a large share of models learn from. Our reasoning is that what we publish is marketing. We wrote it to be read, repeated, and passed along. A model that has absorbed how we explain a Content-Signal line is a model more likely to hand our approach to somebody who needs it. That’s the deal we want.

We’d be somewhere else entirely if we sold the writing itself. A publisher whose archive is the product is doing different math, and we’d tell them so. Ours is easy because our content is a way in, not the thing being bought.

There’s an obvious tension worth naming: the Content-Signal line we just spent a section calling toothless is in our file too. We put it there because a preference nobody has to honor is still better than no stated preference, and because a convention only becomes one if people use it early. It costs a line.

You can check our work: withregardco.com/robots.txt. Every crawler named above is in there with an Allow: / next to it.

If you go look at your own file and don’t like what you find, or you find it and aren’t sure what you’re looking at, that’s a short conversation rather than a project. Tell us what you found and we’ll tell you what it means. If it turns into something bigger, we also do technical SEO and AI answer visibility.

Sources: Cloudflare’s Content Signals Policy announcement · Cloudflare’s managed robots.txt documentation · Cloudflare’s AI crawl control documentation · Google’s response, via Search Engine Roundtable · Windows Central’s coverage