D2C robots.txt Mistakes That Block AI Crawlers

The Five Crawlers You Need to Know

Which AI Engines Crawl Your Site and Why It Matters

Crawler Name	AI Engine	What It Enables	Risk if Blocked
GPTBot	ChatGPT (OpenAI)	ChatGPT retrieval database real-time responses	ChatGPT cannot retrieve your content at all
PerplexityBot	Perplexity AI	Perplexity is entirely retrieval-based live web content per response	Perplexity cannot cite you at all the most damaging block
ClaudeBot	Claude (Anthropic)	Claude AI web retrieval	Excluded from Claude citations
Google-Extended	Gemini / AI Overview	Gemini training and retrieval separate from Googlebot	Removed from Gemini responses without affecting Google Search rank
Applebot-Extended	Apple Intelligence	Apple Intelligence features on iPhone, iPad and Mac	Excluded from Apple Intelligence features as they roll out

How to Check and Fix

30-Minute robots.txt Audit

Step 1 — Check your robots.txt. Go to yourdomain.com/robots.txt in your browser. Look for any of these danger patterns:

User-agent: GPTBot followed by Disallow: /
User-agent: PerplexityBot followed by Disallow: /
A blanket User-agent: * followed by Disallow: / with allow rules that do not explicitly include AI crawlers

Step 2 — Fix it. The correct configuration explicitly permits each AI crawler:

User-agent: GPTBot — Allow: /
User-agent: PerplexityBot — Allow: /
User-agent: ClaudeBot — Allow: /
User-agent: Google-Extended — Allow: /

For Shopify sites: Go to Online Store → Preferences → robots.txt editor. Some themes set restrictive crawl rules automatically as a performance or security measure.

The Shopify-Specific Problem

When robots.txt Is Open but AI Still Can't Read Your Content

Some Shopify themes use JavaScript-heavy page sections that AI crawlers do not fully render. Product descriptions, FAQ sections, and review widgets loaded via JavaScript may not be indexed even if robots.txt permits access.

Fix: Ensure your product page's most important content ingredient list, outcome claims, FAQ exists in the static HTML, not only in JavaScript-loaded components. Verify what AI crawlers actually see using the URL Inspection tool in Google Search Console and viewing the rendered HTML output.

Beyond robots.txt

The Next Layer: Crawl Budget and Indexation Quality

Getting AI crawlers through your robots.txt is necessary but not sufficient. Two additional technical factors affect whether your content actually gets indexed and cited.

Crawl budget management: large Shopify stores with thousands of product pages and multiple currency/regional variants can exhaust an AI crawler's budget on low-value pages before it reaches your key product pages. Check your Google Search Console Coverage report for "Discovered currently not indexed" pages the same pages Google deprioritises are typically the ones AI crawlers deprioritise. Fix by adding noindex tags to low-value pages (search result pages, filtered collection pages, checkout pages) so crawl budget concentrates on your product and category pages.

Internal linking depth: AI crawlers, like Google, follow internal links. If your most important product pages are only reachable through 4 or 5 clicks from the homepage, they are unlikely to be crawled frequently. Ensure your top 5 product pages are linked from your homepage or your main navigation. A product that generates 40% of your revenue should not be buried in a subcategory three clicks deep.

Noma records the page behind each AI citation of your brand, so you can see which product pages the engines actually cite and which never appear, and focus your crawl fixes there.

"You cannot get cited by an AI engine that cannot read your website. Check your robots.txt before you write another word of GEO-optimised content."

Sources and References

OpenAI – GPTBot: crawler documentation and robots.txt integration

Anthropic – ClaudeBot: web crawler documentation

Perplexity AI – PerplexityBot: crawl access documentation

Google Search Central – Google-Extended: AI training crawler documentation