AI Search Visibility

D2C robots.txt Mistakes That Block AI Crawlers

Many D2C sites only handle Google's crawler, not GPTBot, ClaudeBot or PerplexityBot. Block any AI crawler and no optimization gets you AI citations.

Simran Rawat31 Oct 20266 min read
D2C robots.txt Mistakes That Block AI Crawlers

Search crawlers and training crawlers are different. Each AI company runs separate bots: one finds pages to cite in answers, another collects content for training. Blocking a training bot does not remove you from that company’s answers; blocking a search bot can. Each row below is from the company’s own crawler documentation, checked 7 November 2026.

Which AI Engines Crawl Your Site and Why It Matters

Crawler

Company

What it does

If you block it

OAI-SearchBot

OpenAI

Surfaces sites in ChatGPT search results

Your pages can drop out of ChatGPT search answers

PerplexityBot

Perplexity

Surfaces and links sites in Perplexity search results

Perplexity recommends allowing it for your site to appear

Claude-SearchBot

Anthropic

Improves search results for Claude users

May reduce your visibility in Claude search results

GPTBot

OpenAI

Collects content for training OpenAI models

Your content is not used for training; ChatGPT search is a separate setting

ClaudeBot

Anthropic

Collects content that may be used for training

Your content is not used for training

Google-Extended

Google

Controls whether Google may use your content for Gemini training

No effect on Google Search inclusion or ranking

Applebot-Extended

Apple

Controls use of Applebot data for training Apple models

Pages can still appear in Apple search results

Crawler documentation, checked 7 November 2026: OpenAI, Perplexity, Anthropic, Google and Apple.

How to Check and Fix

30-Minute robots.txt Audit

Step 1 — Check your robots.txt. Go to yourdomain.com/robots.txt in your browser. Look for any of these danger patterns:

  • User-agent: OAI-SearchBot followed by Disallow: /
  • User-agent: PerplexityBot or Claude-SearchBot followed by Disallow: /
  • A blanket User-agent: * followed by Disallow: / with allow rules that do not explicitly include AI crawlers

Step 2: Fix it. Allow the search crawlers explicitly. Whether you also allow the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) is a separate choice about training, not about appearing in answers:

  • User-agent: OAI-SearchBot / Allow: /
  • User-agent: PerplexityBot — Allow: /

  • User-agent: Claude-SearchBot / Allow: /
  • User-agent: Googlebot / Allow: / (Google-Extended does not affect Search)

For Shopify sites: Go to Online Store → Preferences → robots.txt editor. Some themes set restrictive crawl rules automatically as a performance or security measure.

The Shopify-Specific Problem

When robots.txt Is Open but AI Still Can't Read Your Content

Some Shopify themes use JavaScript-heavy page sections that AI crawlers do not fully render. Product descriptions, FAQ sections, and review widgets loaded via JavaScript may not be indexed even if robots.txt permits access.

Fix: Ensure your product page's most important content ingredient list, outcome claims, FAQ exists in the static HTML, not only in JavaScript-loaded components. Verify what AI crawlers actually see using the URL Inspection tool in Google Search Console and viewing the rendered HTML output.

Beyond robots.txt

The Next Layer: Crawl Budget and Indexation Quality

Getting AI crawlers through your robots.txt is necessary but not sufficient. Two additional technical factors affect whether your content actually gets indexed and cited.

Crawl budget management: large Shopify stores with thousands of product pages and multiple currency/regional variants can exhaust an AI crawler's budget on low-value pages before it reaches your key product pages. Check your Google Search Console Coverage report for "Discovered currently not indexed" pages the same pages Google deprioritises are typically the ones AI crawlers deprioritise. Fix by adding noindex tags to low-value pages (search result pages, filtered collection pages, checkout pages) so crawl budget concentrates on your product and category pages.

Internal linking depth: AI crawlers, like Google, follow internal links. If your most important product pages are only reachable through 4 or 5 clicks from the homepage, they are unlikely to be crawled frequently. Ensure your top 5 product pages are linked from your homepage or your main navigation. A product that generates 40% of your revenue should not be buried in a subcategory three clicks deep.

Noma records the page behind each AI citation of your brand, so you can see which product pages the engines actually cite and which never appear, and focus your crawl fixes there.

"You cannot get cited by an AI engine that cannot read your website. Check your robots.txt before you write another word of GEO-optimised content."

Sources and References

OpenAI – GPTBot: crawler documentation and robots.txt integration

Anthropic – ClaudeBot: web crawler documentation

Perplexity AI – PerplexityBot: crawl access documentation

Google Search Central – Google-Extended: AI training crawler documentation

See it with your own goal

Leave your work email and pick a time.