Postmortem: The Bot Block That Kept a Business Out of AI Answers

A company can publish the right answer and still never get cited. If its CDN, firewall, or robots.txt blocks the bot fetching that answer, another site gets the chance instead.

I keep seeing the wrong diagnosis proposed for this problem: write more comparison pages, build more authority, buy a tracker. Those may be useful when an assistant can read the site. They do little when the request dies before the page loads. Over one 30-day window on groas infrastructure, /robots.txt received 1,222 verified requests from Claude-User, Anthropic’s live-answer fetcher. That is a lot of door-checking to dismiss as background noise.

The case below is a typical composite, not a claim about one identified company. Its sequence is familiar: a missing citation, a content-first response, months of work, then the discovery that an anti-AI setting intended to discourage training crawlers also obstructed live retrieval. The distinction matters because the fix starts at the door, not in the editorial calendar.

Setup: a site that looks healthy from inside

An executive asks ChatGPT for the best platforms for mid-market fleet dispatching. Two competitors appear, with links to pricing and case studies. Their company does not. The screenshot lands in Slack with the obvious question: Why aren’t we listed?

The marketing team checks the tools it knows. Google Search Console shows stable organic traffic. The site loads in a browser. Its comparison pages are current, and the content calendar is full. Nothing in that view says a live-answer fetcher might receive a different response at the edge.

That is the first trap. A healthy Google report is not a test of whether an assistant can fetch a page. A site can be available to a human visitor and still challenge or refuse an automated request. The assistant does not send the marketing team a ticket explaining what happened. The brand simply fails to appear in the answer.

First sign: the answer names everyone else

The missing mention looks like a content problem. I understand the impulse. When a competitor gets cited, the instinct is to inspect its pages, rewrite yours, and publish more of them. But before spending the next quarter on that plan, I want to know whether the assistant could retrieve what the company has already published.

That requires separating three jobs commonly collapsed into the phrase AI bot. As OpenAI’s bot documentation illustrates, bots that gather material for model training are not the same as bots involved in search or user-requested page visits:

• Training crawlers, such as GPTBot and ClaudeBot, collect material for model development. A business may choose to disallow them.
• Search crawlers, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, help make pages available for search-based answers.
• User-triggered fetchers, such as ChatGPT-User, Claude-User, and Perplexity-User, visit pages in connection with a user’s request.

A policy written as “block AI bots” can hit all three. That is how a reasonable decision about training data becomes a bad decision about being reachable when a buyer asks for current information. The bot’s function matters more than its label.

The groas log count makes the point concrete, with an important limit. Those 1,222 verified Claude-User requests show repeated checks of /robots.txt; they do not tell us what each person asked or prove that each check would have produced a citation. They do show why a rule at that endpoint deserves attention. DataDome’s account of Claude-User describes its use of robots.txt during conversational retrieval. If access is refused there, better prose on the target page cannot help that fetch.

Wrong call: buy content before checking the door

In the composite case, the agency treats the absent mention as an authority gap. It proposes twenty comparison articles, more third-party coverage, and an AI share-of-voice dashboard. The team approves the work. Writers write; product managers update feature tables; reports show that publishing is on schedule.

Then the same kind of prompt still leaves the company out.

I do not object to comparison content. I object to paying for it before checking delivery. A tracker can show that a brand was not mentioned in an answer. It cannot, from that answer alone, tell the team whether a CDN returned a 403 Forbidden, an edge rule altered robots.txt, or the page arrived without its useful content. “Not mentioned” is an outcome, not a diagnosis.

Here is how the technical failure develops while the content work continues.

1. The anti-AI setting goes on

The original decision usually sits outside the marketing workflow. Someone reviews bot traffic or origin load and enables a protective setting. Cloudflare’s AI-bot blocking announcement made a one-click control available across plan levels. From an infrastructure perspective, blocking unwanted scraping can look like routine housekeeping. The trouble starts when nobody checks which agents the rule affects.

The company meant to keep its material out of training collection. Its live-answer access was never part of the discussion. Intent is not configuration: the edge applies the rule it has, not the distinction the team had in mind.

2. The public robots.txt says something unexpected

Next, someone adds a broad Disallow rule, or a managed setting changes what an outside visitor receives from /robots.txt. The file in a repository or CMS can look permissive while the public edge response says otherwise. Community discussion of Cloudflare’s managed AI crawler blocking is a reminder to inspect the delivered response rather than assume the source file is the whole story.

That distinction is easy to miss. A developer checks the local file, sees User-agent: * and Allow: /, and tells marketing that bots are welcome. The retrieval agent checks the live domain. It receives the live domain’s rules.

A broad refusal at this point can stop a visit before the pricing page, comparison table, or case study enters the conversation. Check what the edge serves, not just what the team saved.

3. The firewall challenges a permitted fetcher

Fixing robots.txt does not automatically fix the connection. A Web Application Firewall may still block a request or demand a browser challenge that a lightweight retrieval fetcher does not complete. The advisory file says yes; the edge response says no.

DataDome’s analysis of Anthropic bots describes the difficulty of managing Claude-User through a fixed IP allowlist. Perplexity’s crawler documentation addresses access for its crawlers, including IP information administrators can use. These details matter when an operator tries to distinguish a verified fetcher from someone merely putting a familiar bot name in a request header. One SignalWord audit found 223 of 1,715 business sites where robots.txt permitted AI crawlers but the server or CDN refused access.

This is the point where the incident becomes especially misleading. A browser visit works. The rules file appears open. Yet the fetch that matters can still end at the perimeter. Permission in robots.txt is not proof of delivery.

4. The page loads, but its answer does not

Some fetchers get through both gates and still receive little useful material. If pricing, product details, or customer proof appear only after client-side JavaScript runs, the initial HTML response may be an almost empty shell. A browser fills it in. A lightweight fetch may not.

That does not mean every JavaScript site is absent from every AI answer. It means the team cannot assume that what it sees after a browser renders a page is what a particular retrieval request received. For commercially important pages, inspect the HTML delivered before client-side scripts run. If the facts a buyer needs are missing there, another content campaign will not put them into that response.

Damage: a full calendar, an empty path to the answer

By now, the marketing team has improved material the assistant may never have read. The dashboard keeps reporting missing mentions. Search Console remains reassuring. Engineering sees no outage because ordinary visitors can still load the site. Each team has a piece of a normal-looking picture.

Meanwhile, a competitor whose relevant page is reachable can supply the answer. I would not conclude that every missing citation is caused by bot blocking, or that access guarantees a citation. The narrower point is enough: content cannot compete in a live fetch that never receives it. Until the team tests that path, more publishing is a bet placed without looking at the table.

Fix: separate training policy from answer access

The corrective decision is not “allow every crawler.” It is to decide what each kind of bot may do, then make the public rules and the firewall match that decision. RunCloud’s discussion of bot management describes the need to manage these layers rather than rely on a single switch.

For a business that chooses to disallow the training crawlers named here while permitting these search and user-triggered agents, a starting robots.txt distinction looks like this:

# Disallow foundation-model training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Permit search and user-triggered retrieval
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

This is an advisory policy, not a firewall bypass. It does not make a blocked HTTP request succeed, and it does not establish that every permitted bot will use a page in an answer. The operator still has to inspect CDN and WAF rules, verify the agents it intends to permit, and ensure those requests can receive useful HTML. Do not turn a bot-name string into an unconditional security exception; a request header alone is not verification.

I would run the checks in the order the fetch encounters them:

1. Inspect the public edge response. Request /robots.txt and a commercially important page from outside the application. Look at status codes, redirects, challenge indicators, and the text actually returned. Compare that text with the file the team believes it published.
2. Review the CDN and WAF decision. Find the rule behind a refusal or challenge, then check whether verified search and user-triggered fetchers can pass it. A permissive robots.txt cannot overrule a 403.
3. Read the served page. Inspect its initial HTML for the pricing, capabilities, comparison details, or proof the assistant would need. If those facts depend on client-side rendering, address delivery before commissioning replacements for the prose.
4. Use logs to confirm what changed. Look for verified bot requests and their response codes. A request to /robots.txt establishes that the bot checked the door; a successful page response establishes more. Neither, by itself, promises a citation.

These quick curl requests help reveal what the public edge returns to requests bearing the named user-agent strings:

curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Claude-User/1.0" -I https://example.com/robots.txt
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" -I https://example.com/

Treat them as a first check, not proof of verified-bot access. The requests are yours, not Anthropic’s or OpenAI’s, and a WAF may treat them differently. Fetch the body of /robots.txt as well; -I shows headers, not the rules. Then pair the result with CDN decisions and verified-bot logs. An empty log search alone is not a diagnosis: the logging scope, retention window, and point of interception all matter.

The owner of each fix also matters. A CMS or web team can change the published robots.txt and put essential information into the initial HTML. A person with CDN or WAF access must change an edge rule or challenge. A mention tracker cannot make either change, however polished its chart looks.

At groas, we treat organic AI visibility alongside paid search as an operational problem, not just a publishing target. That means examining crawler access and edge responses as well as the content a buyer might ask about. The groas MCP integration for ChatGPT and Claude lets teams query live bot traffic and citation health from their workflow. It does not replace the person who controls a client’s firewall when a rule needs changing.

The rule I now apply is simple: never block an AI crawler by category; filter it by function. If you want training crawlers out, keep them out. Do not accidentally lock the door on the fetcher trying to read an answer for your buyer.