Every AI Crawler That Reads Your Store, Listed

AI crawlers are the automated agents that OpenAI, Anthropic, Google, Perplexity, Amazon, and Meta send to read your store. In 2026 they arrive in three distinct kinds - training crawlers, search-index crawlers, and live user-fetch agents - and treating them as one thing is how brands accidentally delete themselves from AI shopping answers.

The stakes are asymmetric. One wrong robots.txt line and every answer engine that would have recommended you simply cannot see you. We have audited stores selling on great products with zero AI presence, and the cause was a copy-pasted "block all AI" snippet from 2023.

The three kinds of AI bot

Training crawlers collect content to train future models. Blocking them is a content-rights decision, not a visibility decision.

Search-index crawlers build the retrieval indexes behind AI search and shopping answers. Blocking these removes you from answers. For a store, they are the closest thing to Googlebot in this ecosystem.

User-fetch agents hit your site live when a real person's AI session needs your page: a shopper asks about your product and the assistant fetches it right then. This traffic converts, and much of it lands in GA4 mislabeled as Direct - the dark traffic problem we cover separately.

The list

Per each vendor's published crawler documentation as of 2026:

Names and behavior change; each vendor's docs page is the source of truth. The structural point is stable: know which of the three jobs a bot does before you decide.

The robots.txt policy for stores

If AI-driven revenue is the goal, this is the default we recommend:

# Search-index and user-fetch bots: allow on everything public
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amazonbot
Allow: /

# Training bots: deliberate business call - this example allows
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /

# Everyone: keep private paths private
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Three notes on the judgment calls:

1. Training bots are the only real debate. Product pages in training data teach next year's models your brand exists; that is upside for a store in a way it is not for a newspaper. We lean allow for commerce, block only if you have a specific content-rights concern.
2. Never block by vendor wholesale. "Block OpenAI" snippets that catch OAI-SearchBot are the most common self-inflicted wound we find in audits.
3. Verify in logs. Spoofing is real; match user agents against vendor-published IP ranges (or let your CDN's verified-bots view do it) before drawing conclusions from traffic.

Access is the first gate, not the whole game

An open robots.txt makes you crawlable. It does not make you citable. What the bots find when they arrive - structured product data, real policy pages, a curated llms.txt file - decides whether the answer engines actually use you. And whether any of it shows up in real answers is a measurement question: our free Citation Rank scan tells you where you stand across six AI surfaces in 24 hours.

Check the access gate right now: the agentic readiness scanner reads your robots.txt, your feeds, and your protocol surface in about a minute and shows exactly which bots you are turning away. Brands that fixed this gate and then worked the content layer have moved real numbers; Amazon India's six-week climb is documented in the case study.