In short
- Make a separate access decision for search crawlers and training crawlers, then check each publisher's rules for user requests. 1 2 3
- Check the delivered content and indexing status of each important page before treating missing citations as an editorial problem.
- Use IndexNow to report changes, and verify Google Search's generative AI setting separately. 10 11
Start with a page that matters
Choose a pricing page or a product page that answers an actual customer question. Work through that URL before applying changes across your site.
Our recommendation is to treat technical work as a prerequisite for access. Google's eligibility rules don't guarantee indexing or inclusion in an AI answer. 6
Keep a record for the selected URL. Save its HTTP response, then compare its content with the version reported by each search engine. Note the test time and what you changed.
A useful finding identifies a specific failure. For example, a pricing page might return a security challenge instead of its prices. That gives your developer something concrete to fix.
Make access decisions by robot role
An instruction for one robot doesn't necessarily control another. OpenAI explicitly separates search access from training permission. Perplexity and Anthropic also document distinct agents for different tasks. 1 2 3
Use this comparison when reviewing your configuration. Each row describes the publisher's documented purpose.
| Agent or token | Purpose | Access decision |
|---|---|---|
OAI-SearchBot | Finds pages for ChatGPT search. | Allow for search participation, independently of GPTBot. 1 |
ChatGPT-User | Fetches pages for certain user actions. | User requests may fall outside robots.txt rules. 1 |
GPTBot | Collects content potentially used for model training. | Set your training preference separately. 1 |
PerplexityBot | Finds websites for Perplexity search, without collecting training content. | Review search access and published IP ranges. 2 |
Perplexity-User | Retrieves pages for user requests, without collecting training content. | Generally ignores robots.txt for those requests. 2 |
Claude-SearchBot | Supports search indexing and search quality. | Review search access. 3 |
Claude-User | Retrieves content at a user's request. | Anthropic says its bots honor robots.txt. 3 |
ClaudeBot | Collects material potentially used for training. | Set your training preference separately. 3 |
Google-Extended | Controls specified Gemini training and grounding uses. | A robots.txt control token without its own HTTP user agent. 4 |
Bingbot | Crawls pages for Bing. | Check access with Bing's robots.txt tester. 5 |
Googlebot handles Google Search crawling. Google-Extended covers Gemini Apps and specified Vertex AI uses, including grounding answers with retrieved content. It doesn't control inclusion or ranking in Google Search. 4
OpenAI says blocking OAI-SearchBot excludes pages from ChatGPT search answers, although navigational links may remain. Allowing ChatGPT-User doesn't replace search access. 1
Our recommendation is to document the intended permission beside each agent. Ask the person responsible for content rights to decide training permissions explicitly. Don't treat a single switch labelled "AI bots" as an adequate record of those decisions.
Check robots.txt and the security layer
Google's guidance covers both robots.txt access and restrictions imposed by hosting infrastructure or a content delivery network. A permissive robots.txt file alone doesn't establish access. 6
For each selected URL, inspect the matching robots.txt rules. Then examine the response delivered through your normal CDN and web application firewall, or WAF. Review any bot protection rule that applies to the request.
Perplexity documents WAF exceptions that match each agent against its own published IP ranges. OpenAI also recommends allowing its published search crawler ranges. 2 1
A user-agent name alone doesn't prove who sent the request. Google's crawler documentation explicitly warns that these strings can be spoofed. 4
Use a simulated request as a diagnostic check. Confirm real crawler access through logs and the publisher's verification method before declaring the issue resolved.
Save the response body as well as the status. In your test record, distinguish the expected page from an error message or verification screen. Compare the response with what an ordinary visitor receives.
Keep the exception narrow. Your goal is to deliver public information to the intended search agent while retaining protection elsewhere. Recheck the affected URL after any security configuration change.
Put the useful answer in the delivered HTML
Google can execute JavaScript and use the rendered HTML for indexing. Its documentation also recommends server rendering or prerendering because some bots cannot execute JavaScript. 7
Our default is to include essential answers in the initial HTML. Keep the price and its billing conditions together. Include product limitations wherever the related feature is described.
Open the raw response and search for a sentence that matters to the buyer. Then inspect the rendered page. If the sentence appears only after interaction, record what loads it and whether the search engine receives it.
For example, imagine a pricing page that initially delivers only an empty application container. Its visible prices arrive after a browser request. Ask your developer whether those prices can be included in the page's initial response.
This is a compatibility recommendation. It doesn't mean Google rejects JavaScript sites, or that every AI service has identical rendering capabilities. 7
Google recommends making important information available as text. Use an image to illustrate a feature, with its essential specifications written alongside it. 6
Verify Google and Bing separately
Search Console's URL Inspection tool distinguishes Google's indexed copy from a live test. The indexed report identifies Google's selected canonical URL. A live test cannot predict that selection. 8
Inspect the exact URL you want discovered. Check its indexing status and whether Google selected another URL as canonical. Compare the reported crawl with the latest substantive edit.
Inspect accidental noindex directives in page tags or HTTP headers. Search Console reports crawling permission separately from indexing permission. A robots.txt block can prevent Google from seeing a noindex directive. 8
After a fix, run the live test. Review the returned content, then request indexing when appropriate. Save the result separately from the older indexed report. 8
Repeat the inspection in Bing Webmaster Tools. Its URL Inspection tool reports indexing details and crawl issues. Its live check tests access without requesting indexing. 9
Keep separate columns for Google and Bing in your working record. Don't infer one engine's status from the other's result.
Google requires a page to be indexed and eligible for a search snippet before it can appear as a supporting AI link. Eligibility still doesn't ensure selection. 6
Check Search Console's generative AI control
Open the relevant property in Search Console, then go to Settings and Search generative AI. Google documents inclusion, exclusion and inheritance from a parent property. 10
If participation is your objective, verify that the effective setting includes your content. When the property inherits its setting, inspect the parent responsible for that choice. Save the property name and a dated screenshot. 10
The control covers AI Overviews and AI Mode, plus generative AI features in Discover. It doesn't govern model training or ranking elsewhere in Search. Google-Extended addresses the separate training choice. 10
Treat this as an access decision. A screenshot showing inclusion cannot establish that any particular answer will use your page.
Notify participating engines when content changes
IndexNow lets a site report URLs that were added, updated or deleted. Ownership verification uses a key file hosted on the relevant site. 11
Check whether your CMS already supports the protocol before commissioning a custom integration. IndexNow's guidance recommends checking existing platform support first. 12
For a custom implementation, connect submissions to actual publication events. Send the affected URLs after their changes are available. Retain the submission result with your publication record. 11
An HTTP 200 response from IndexNow confirms receipt. It doesn't confirm indexing. Participating search engines decide whether to crawl and index the submitted URL. 11 12
Use a changed product page to test the workflow. Compare what you submitted with what Bing subsequently reports. Don't describe the submission receipt as a ChatGPT citation or an indexing deadline.
Leave a record someone else can verify
Close the technical review with the tested URL and the remaining failure, if any. Attach the delivered HTML and the relevant indexing reports. Record the effective access settings.
Schedule a fresh review when platform controls change. Microsoft announced separate training and grounding preferences on October 30, 2026, with enforcement beginning February 1, 2027. That announcement describes a future transition, so check its effective dates before changing your implementation. 13
If the page is accessible and indexed, examine whether its content answers the question. Our article on citations without brand recommendations helps frame that next review.
Fix the demonstrated obstacle first. Then test the same page again.
Sources
- OpenAI, Overview of OpenAI CrawlersAccessed Back to the text, first mention of source 1
- Perplexity, Perplexity CrawlersAccessed Back to the text, first mention of source 2
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?Accessed Back to the text, first mention of source 3
- Google, List of Google's common crawlersAccessed Back to the text, first mention of source 4
- Microsoft Bing, Start Using Bing Webmaster Tools to Improve Your Site VisibilityAccessed Back to the text, first mention of source 5
- Google Search Central, AI features and your websiteAccessed Back to the text, first mention of source 6
- Google Search Central, Understand the JavaScript SEO basicsAccessed Back to the text, first mention of source 7
- Google Search Console Help, URL Inspection toolAccessed Back to the text, first mention of source 8
- Microsoft Bing, Introducing the Bing Webmaster Tools URL Inspection ToolAccessed Back to the text, first mention of source 9
- Google Search Console Help, Search generative AI controlAccessed Back to the text, first mention of source 10
- IndexNow, DocumentationAccessed Back to the text, first mention of source 11
- IndexNow, Frequently asked questionsAccessed Back to the text, first mention of source 12
- Microsoft Bing, Your content, your choice: clearer preferences for training and groundingAccessed Back to the text, first mention of source 13
