AI Search Visibility
D2C robots.txt Mistakes That Block AI Crawlers
Many D2C sites only handle Google's crawler, not GPTBot, ClaudeBot or PerplexityBot. Block any AI crawler and no optimization gets you AI citations.

Search crawlers and training crawlers are different. Each AI company runs separate bots: one finds pages to cite in answers, another collects content for training. Blocking a training bot does not remove you from that company’s answers; blocking a search bot can. Each row below is from the company’s own crawler documentation, checked 7 November 2026.

Which AI Engines Crawl Your Site and Why It Matters
Crawler | Company | What it does | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results | Your pages can drop out of ChatGPT search answers |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results | Perplexity recommends allowing it for your site to appear |
Claude-SearchBot | Anthropic | Improves search results for Claude users | May reduce your visibility in Claude search results |
GPTBot | OpenAI | Collects content for training OpenAI models | Your content is not used for training; ChatGPT search is a separate setting |
ClaudeBot | Anthropic | Collects content that may be used for training | Your content is not used for training |
Google-Extended | Controls whether Google may use your content for Gemini training | No effect on Google Search inclusion or ranking | |
Applebot-Extended | Apple | Controls use of Applebot data for training Apple models | Pages can still appear in Apple search results |
Crawler documentation, checked 7 November 2026: OpenAI, Perplexity, Anthropic, Google and Apple.
How to Check and Fix
30-Minute robots.txt Audit
Step 1 — Check your robots.txt. Go to yourdomain.com/robots.txt in your browser. Look for any of these danger patterns:
- User-agent: OAI-SearchBot followed by Disallow: /
- User-agent: PerplexityBot or Claude-SearchBot followed by Disallow: /
A blanket
User-agent: *followed byDisallow: /with allow rules that do not explicitly include AI crawlers
Step 2: Fix it. Allow the search crawlers explicitly. Whether you also allow the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) is a separate choice about training, not about appearing in answers:
- User-agent: OAI-SearchBot / Allow: /
User-agent: PerplexityBot—Allow: /- User-agent: Claude-SearchBot / Allow: /
- User-agent: Googlebot / Allow: / (Google-Extended does not affect Search)
For Shopify sites: Go to Online Store → Preferences → robots.txt editor. Some themes set restrictive crawl rules automatically as a performance or security measure.
The Shopify-Specific Problem
When robots.txt Is Open but AI Still Can't Read Your Content
Some Shopify themes use JavaScript-heavy page sections that AI crawlers do not fully render. Product descriptions, FAQ sections, and review widgets loaded via JavaScript may not be indexed even if robots.txt permits access.
Fix: Ensure your product page's most important content ingredient list, outcome claims, FAQ exists in the static HTML, not only in JavaScript-loaded components. Verify what AI crawlers actually see using the URL Inspection tool in Google Search Console and viewing the rendered HTML output.
Beyond robots.txt

The Next Layer: Crawl Budget and Indexation Quality
Getting AI crawlers through your robots.txt is necessary but not sufficient. Two additional technical factors affect whether your content actually gets indexed and cited.
Crawl budget management: large Shopify stores with thousands of product pages and multiple currency/regional variants can exhaust an AI crawler's budget on low-value pages before it reaches your key product pages. Check your Google Search Console Coverage report for "Discovered currently not indexed" pages the same pages Google deprioritises are typically the ones AI crawlers deprioritise. Fix by adding noindex tags to low-value pages (search result pages, filtered collection pages, checkout pages) so crawl budget concentrates on your product and category pages.
Internal linking depth: AI crawlers, like Google, follow internal links. If your most important product pages are only reachable through 4 or 5 clicks from the homepage, they are unlikely to be crawled frequently. Ensure your top 5 product pages are linked from your homepage or your main navigation. A product that generates 40% of your revenue should not be buried in a subcategory three clicks deep.
Noma records the page behind each AI citation of your brand, so you can see which product pages the engines actually cite and which never appear, and focus your crawl fixes there.
"You cannot get cited by an AI engine that cannot read your website. Check your robots.txt before you write another word of GEO-optimised content."
Sources and References
OpenAI – GPTBot: crawler documentation and robots.txt integration
Anthropic – ClaudeBot: web crawler documentation
Perplexity AI – PerplexityBot: crawl access documentation
Google Search Central – Google-Extended: AI training crawler documentation