Can AI crawlers read my Shopify store?
Check by visiting yourstore.com/robots.txt and confirming OAI-SearchBot, PerplexityBot and Google-Extended aren't disallowed — and that your firewall or CDN isn't blocking unknown bots. If any AI crawler is turned away, that engine can never cite you for any question. It's the first thing to fix, and often a two-line change.
Last reviewed July 2026
Which AI crawlers should I check for?
The crawlers that decide whether AI answer engines can cite you:
- OAI-SearchBot and ChatGPT-User — used by ChatGPT search to find and fetch pages.
- GPTBot — OpenAI's crawler; allowing it lets your content inform ChatGPT.
- PerplexityBot and Perplexity-User — how Perplexity indexes and fetches sources.
- Google-Extended — controls whether Google can use your content in generative features like AI Overviews.
- Googlebot and Bingbot — power traditional search and feed some AI surfaces.
How do I check if they're blocked?
Two quick checks catch most problems:
- 1Open yourstore.com/robots.txt in a browser. Look for any Disallow rule under a User-agent that names an AI crawler, or a blanket block. Shopify serves a default robots.txt you can customize.
- 2Test a live fetch. Some firewalls, CDNs or bot-protection rules block unknown user agents even when robots.txt allows them. If a page loads for you but not for a crawler test, a security rule is the likely culprit.
Does Shopify block AI crawlers by default?
Shopify serves a default robots.txt that generally allows major crawlers, but the real risk is elsewhere: bot-management rules at your CDN or firewall (or an app you've installed) can silently turn away AI crawlers that don't identify as Googlebot.
That's why the live-fetch check matters. A store can look fine in robots.txt and still be invisible to PerplexityBot because a default "block unknown bots" rule is doing its job too aggressively.
What happens if AI crawlers are blocked?
You become uncitable for that engine, full stop. It doesn't matter how clearly your pages answer buying questions — if PerplexityBot can't reach them, Perplexity has nothing of yours to cite, and the answer goes to a competitor it could crawl.
This is the most common and most fixable cause of an answer gap. Unblocking crawlers won't get you cited overnight, but staying blocked guarantees you never will be.
How do I fix it?
Update robots.txt to explicitly allow the AI crawlers you want, and adjust firewall or bot-management rules to permit them. On Shopify, you can customize the default robots.txt via the robots.txt.liquid template; on other platforms, edit the file or your server rules directly.
A free Answer Gap Scan includes this check — it tells you which AI crawlers can and can't read your site today, alongside five buyer questions and who each engine mentions.
Frequently asked questions
Should I allow GPTBot, or does it just train AI on my content?
GPTBot can inform ChatGPT's knowledge, while OAI-SearchBot and ChatGPT-User handle live search and fetching. If your goal is to be cited by ChatGPT, allowing these is generally necessary. Some brands allow search crawlers but weigh training crawlers separately — it's a business call, but blocking everything means no ChatGPT citations.
Does blocking Google-Extended affect my Google ranking?
Google says Google-Extended governs use of your content in generative features and is separate from Googlebot, which handles crawling and Search ranking. Blocking Google-Extended is intended to keep you out of generative uses without changing core Search — but it also removes you from AI Overview citations.
My robots.txt looks fine — why am I still not cited?
Robots.txt is only half the picture. Firewall, CDN or bot-protection rules can block AI crawlers regardless of robots.txt, and even with access you still need clear, specific answers to the questions buyers ask. Crawlability is necessary but not sufficient.
How do I test as an AI crawler?
You can fetch your pages with the crawler's user-agent string to see whether they load, or use a crawlability checker. A free Answer Gap Scan does this for you and reports which AI crawlers can currently read your store.
See which buying questions AI answers without you
Your free Answer Gap Scan shows 5 real buyer questions in your category, who ChatGPT, Google AI Overviews and Perplexity mention today, and whether AI can even read your site. No credit card. Delivered within 24 hours.
Get your free Answer Gap Scan