AI Crawler Access: Robots.txt, GPTBot, and Search Visibility

Map AI crawlers by purpose before changing robots.txt

The name GPTBot is often used as shorthand for all ChatGPT discovery, but that is not how OpenAI documents its agents. OpenAI's crawler documentation separates OAI-SearchBot, GPTBot, and ChatGPT-User. OAI-SearchBot supports inclusion in ChatGPT search answers. GPTBot collects content that may be used to improve foundation models. Their robots.txt rules can be set independently. ChatGPT-User performs certain requests initiated by a user and is not the automatic search crawler; OpenAI notes that robots.txt rules may not apply to those user-triggered requests.

Provider behavior changes, so use the linked official documentation as the source of truth rather than copying an old bot list from a blog post. Protect every sensitive, licensed, or private resource with authentication and authorization regardless of user agent.

Write a purpose-based robots.txt policy

Start with a written policy, not a collection of user-agent names. Classify URLs as public and indexable, public but excluded from search, private or authenticated, low-value crawl spaces, and assets needed to render public pages. Then decide which documented search and training crawlers may reach each public URL class. Enforce the private or authenticated class with real access control, because robots.txt is not authorization and may not govern user-triggered agents such as ChatGPT-User. This keeps privacy, infrastructure, and acquisition choices separate.

A company that wants ChatGPT and Claude search discovery but elects not to grant training-crawler access might begin with a pattern like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

This is an illustration, not a universal policy. Add disallow rules for administrative routes, staging paths, internal search results, cart and account URLs, or other crawl traps as needed, but do not use those rules as protection for private content. Review every subdomain because a robots.txt file applies to its own host. Keep the file at the root, return a successful response, use valid syntax, and test precedence before deployment. Do not assume a blanket User-agent: * rule expresses the same product preference for every provider.

Separate crawling, indexing, snippets, and canonicalization

robots.txt controls requesting; it is not a universal removal tool. A disallowed URL may still be known from links, and a crawler cannot read a page-level directive on a page it is forbidden to fetch. Google explicitly advises against using robots.txt for canonicalization. Use the control that matches the outcome you need.

• Use robots.txt to manage crawler access and reduce unwanted crawling of URL spaces.
• Use a noindex robots meta tag or X-Robots-Tag where supported when an accessible resource should not appear in search.
• Use rel='canonical' and consistent internal links to consolidate materially duplicate public URLs.
• Use authentication and authorization for private information. Robots.txt is public and is not an access-control boundary.
• Use snippet controls only after checking their effect. Google documents that nosnippet also prevents content from being used as a direct input for AI Overviews and AI Mode.

Google's Search generative AI control is a separate, limited-site pilot. Participating sites are included by default; where the control is available, it manages whether content is eligible for AI Overviews, AI Mode, and generative features in Discover without changing normal Search crawling or indexing. It is an outcome control, not a replacement for technical SEO or private-content protection.

For an AI visibility program, the normal target is a clean 200-status canonical page that can be crawled, rendered, indexed where applicable, and quoted without ambiguity. For a tool that explicitly documents support, the llms.txt proposal may provide a curated resource map. Google Search ignores llms.txt, and OpenAI's current crawler documentation does not state support. It does not replace robots.txt, sitemaps, internal links, or page-level indexing controls.

Remove CDN, WAF, rendering, and navigation blockers

A permissive robots.txt file does not guarantee access. CDNs, web application firewalls, bot-management products, rate limits, JavaScript challenges, CAPTCHAs, geographic rules, authentication, and origin failures can all return a different result to a crawler than to a browser. Check the full request path from edge to origin. OpenAI and Perplexity publish IP ranges for relevant agents; where the provider recommends it, verify both the user agent and the current official IP list rather than trusting a spoofable header alone.

Make the primary answer available in the initial HTML when practical. A crawler should not need to click a tab, submit a form, or execute a fragile client-side application to learn what a page is about. Use descriptive titles, one clear main topic, meaningful headings, ordinary <a href> links, accessible labels, canonical URLs, and a sitemap. Link category pages to detail pages so discovery does not depend on an internal search box.

Also verify that critical CSS, images, and scripts are not blocked when they are required to interpret the page. Keep response sizes reasonable, return accurate status codes, and use caching headers such as ETag or Last-Modified to reduce repeat transfer. Accessibility improvements often help machine interpretation because semantic structure makes controls and relationships explicit.

Test access at the edge and confirm it in server logs

Test each important URL class, not only the home page. Request the production robots.txt file, a representative article, a product or feature page, a documentation page, and a deliberately blocked route. Check status, redirects, canonical destination, response body, content type, and whether the main answer is present. A browser-like test and a user-agent test should produce equivalent public content; serving keyword-heavy content only to bots is cloaking.

1. Validate robots.txt syntax and confirm the intended group applies to each named agent.
2. Send a controlled request using the documented user-agent token, without sending automated queries to a search service.
3. Inspect CDN, WAF, and origin logs for the request, final status, bytes, latency, and rule that fired.
4. Where an official IP list is available, compare the source safely and keep allowlists current.
5. Repeat after cache purges, WAF releases, hosting migrations, and robots.txt edits.

Logs provide evidence of access, not proof of inclusion or citation. A successful fetch can still lead to no visibility if the page is thin, duplicated, stale, poorly linked, or irrelevant. Pair technical testing with the content and entity checks in the GEO audit checklist.

Measure outcomes and treat crawler rules as governed configuration

Record a baseline before changing access: robots.txt version, WAF policy, index coverage, crawl errors, organic landing pages, AI referral sessions, and a fixed set of prompts where the brand or page is relevant. After deployment, annotate the date and allow for provider recrawling. OpenAI and Perplexity note that robots.txt changes can take about a day to propagate in their systems, while content discovery and selection can take longer.

Measure three layers separately: access (successful verified requests), discovery (indexed or surfaced URLs), and selection (mentions, citations, referral visits, and answer accuracy). The AI citation tracking guide explains how to monitor the last layer without treating a single prompt result as a stable ranking.

Put robots.txt and edge rules under version control. Require an owner, review ticket, rollout plan, and rollback for each change. Alert on unexpected global disallows, robots.txt failures, spikes in 403 or 429 responses, and disappearance of verified bot traffic. Recheck provider documentation quarterly because names, IP ranges, and behavior can change. This operational discipline prevents a security release or migration from quietly removing months of SEO and GEO work.

Frequently asked questions