Two unrelated business websites returned the same crawler blocklist during our tool testing. The pattern resembled Cloudflare's managed robots.txt configuration. It was a reason to inspect the hosting layer, not proof of who enabled the setting or what the owners knew.
The lesson is to check the file visitors receive. A source file in your website project may not be the whole response.
Your host can contribute rules
Cloudflare's Bot Preference Sync, which replaces its managed robots.txt feature, can prepend crawler directives to an existing file or supply a file when none exists. These instructions express preferences; separate access controls enforce restrictions on requests. Cloudflare's robots.txt documentation
If the served file differs from the file in your website project, review the CDN or hosting configuration. Matching text or a Content-Signal line is a clue; a site owner can also copy or write those lines. The file alone does not establish the origin of a setting.
Cloudflare's September update separates three choices
On September 15, 2026, Cloudflare announced the replacement of its legacy Block AI Bots switch with separate Search, Training, and Agent controls, alongside Bot Preference Sync. Existing settings migrate according to earlier selections; a domain's age alone doesn't establish its current policy. Cloudflare's September announcement
For newly onboarded domains, the recommended settings depend on whether the site earns advertising revenue:
| Control | Without ad revenue | With ad revenue |
|---|---|---|
| Search | Allow | Allow |
| Training | Allow | Disallow AI Training |
| Agents acting for users | Allow | Block on pages with ads |
These are adjustable recommendations, rather than a blanket block for every new domain. Cloudflare's explanation of the new recommendations
The setting names matter. Training's Block and Block on pages with ads now include mixed-use crawlers such as Googlebot, Bingbot, and Applebot, so they can affect search access too. Disallow AI Training is designed to preserve search access for those crawlers while expressing a no-training preference. Cloudflare's setting definitions
Training and search are different uses
Crawler names do not all represent the same activity. OpenAI distinguishes GPTBot, used for model development, from OAI-SearchBot, used for search. Blocking one is not the same decision as blocking the other. OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers, although they can still appear as navigational links. ChatGPT-User serves certain user-initiated requests rather than automatic crawling, and robots.txt rules may not apply to those requests. Official OpenAI crawler documentation
Anthropic documents three agents with distinct purposes: ClaudeBot collects content that may be used for model training, Claude-SearchBot supports search results, and Claude-User retrieves pages in response to users' requests. Anthropic says its bots honor robots.txt directives, so you can set different preferences for each. Restricting Claude-SearchBot or Claude-User can limit search indexing or user-requested retrieval, respectively. Apply your preferences to each subdomain you want them to cover. Official Anthropic crawler documentation
Google-Extended controls certain Gemini training and grounding uses. It's a product token for robots.txt, rather than a separate crawler making requests under that name. Google states that it does not affect inclusion or ranking in Google Search. Google's crawler documentation
Read the current documentation for the particular crawler before drawing a conclusion. A blocked crawler does not establish that a business is absent from every answer: systems can use indexes, other sources, or different retrieval paths. Permission to crawl also does not guarantee a citation.
How to review your own file
- Open your domain followed by
/robots.txtand save the response for comparison. - Identify the relevant user-agent groups and their Allow and Disallow rules. A rule for one path may differ from the rule for the homepage.
- Compare the served response with your source file and hosting settings.
- Decide separately which uses of your content you want to permit. On Cloudflare, review Search, Training, Agent, and Bot Preference Sync settings, alongside any firewall or other server-level restrictions.
- After an approved change, retrieve the live file again and verify the effective rules.
An agent does not need to be named explicitly to have access. Applicable wildcard rules and default behavior matter. Matching groups are combined, and the most specific matching path rule applies. For equivalent Allow and Disallow rules, the robots standard favors Allow. A directive's position at the top of the file doesn't give it priority. Robots Exclusion Protocol, RFC 9309
What our check reports
The Website Crawlability Check inspects the returned homepage and selected public files. It reports homepage-path Allow and Disallow rules for selected agents and explains their purposes. It doesn't interpret every content-use signal, inspect your hosting dashboard, impersonate those crawlers, or test every path. Its findings describe the public response, rather than proving access for a particular bot or inclusion in an answer system.
We revised the check to stop treating explicit permission as better than equivalent default permission. Optional llms.txt information is also excluded from scoring. Google does not use that file for Search, including its generative features. Google's AI-search guidance
We checked our live robots.txt: it matched our published source, with 15 user-agent groups and no Disallow directives. We publish explicit permission for the named agents and permit search, AI input, and training through our content signals. That's our content policy, rather than a requirement for other businesses or evidence of better visibility. This file check also doesn't establish whether a separate network rule would block an individual crawler.
Method and limits
The original observation came from public website responses during tool testing. We did not inspect either business's private hosting dashboard or establish the owner's intention. Crawler policies and vendor defaults can change; verify the current served response before acting.
If you want access and search structure reviewed together, explore search visibility and SEO services.