1. Home
  2. Blog
  3. Website checklist for AI answers

Search & AI visibility

Will AI tools find your site? Check these 8 things.

Google, OpenAI, Anthropic, Perplexity and Microsoft have all published what their crawlers do. Here is what they say, what you can safely ignore, and a short checklist for your website.

On this page
  1. What Google now says
  2. The 8 checks
  3. Who is crawling you
  4. What you can skip
  5. How to measure it
  6. Questions people ask
Hands holding a phone beside a coffee cup on a white table
Photo: Kristin Hardwick, Wikimedia Commons (CC0)

The short version

  • Google says there is no special trick for AI answers. Its own guide says optimizing for them is optimizing for Search.
  • Your pages must be crawlable, indexed and allowed to show a snippet. Check that first.
  • Each AI company runs separate bots for training, search and user requests. Blocking one does not block the others.
  • Skip the gimmicks: llms.txt and extra markup will not help you in Google.

What Google now says

In 2026 Google published a dedicated guide for site owners on its AI features, last updated in July. If you have read a lot of advice about "GEO" or "AEO", the guide is a useful reality check. Its main message is that optimizing for generative AI search is optimizing for the search experience.

It then rules out several popular tricks. In Google's words, you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Search. There is no requirement to break your content into tiny pieces, and no need to write in a special way for AI. Structured data is not required, and there is no special schema.org markup to add.

What does Google ask for instead? Its "AI features and your website" page says a page needs to be indexed and eligible to be shown in Search with a snippet. Beyond that, it points to the basics: let crawlers in, link to your pages from other pages on your site, give visitors a good experience, keep important content as text, and use good images and video.

Good news for small teams: the work that gets you into AI answers on Google is the same work that helps ordinary search, and you probably need to do it anyway.

Hands holding a phone beside a coffee cup on a white table
More and more questions are asked on a phone, and answered by an AI summary. Photo: Kristin Hardwick, Wikimedia Commons (CC0)

If you want the bigger picture of why clicks are changing, our earlier guide on getting cited by AI search covers it. This post is the practical website checklist.

The 8 checks

Go through these in order. Most take minutes.

  • Open your robots.txt and read itType yourdomain.com/robots.txt into a browser. Make sure it does not block the search bots listed in the next section. Google also notes that your CDN or hosting layer must not block crawlers either.
  • Make sure the page is indexedSearch Console shows which pages Google has indexed. A page that is not indexed cannot appear in AI Overviews or AI Mode.
  • Do not hide a page from snippets by accidentSettings such as nosnippet or noindex limit what Google shows. Check that none are set on pages you want quoted.
  • Keep key text in the HTMLGoogle can process JavaScript. For other AI crawlers, the vendors say nothing, and one older analysis found they did not render it. Put your headline, answers and prices in the page your server sends.
  • Link your pages togetherGoogle lists easy discovery through internal links as a best practice. Every important page should be reachable from another page.
  • Write the page for peopleGoogle says trust is the most important part of its quality guidance. Show who wrote the page, add a byline where readers would expect one, and back claims with real examples and sources.
  • Publish a sitemap with honest datesBing says it fully supports the sitemap protocol, wants the lastmod date to reflect the real time of the last content change, and ignores changefreq and priority.
  • Tell search engines when you publishIndexNow lets you notify participating engines, including Bing, about new or changed pages instead of waiting days or weeks for a crawl. Keep your Google Business Profile and, for local firms, Bing Places up to date.

Who is crawling you

The big AI companies each run more than one bot, and they do different jobs. This matters because you may be happy for your pages to be found in a chat answer but not want them used to train a model. Here is what each company says.

CompanyBotWhat the company says it does
OpenAIGPTBotCrawls content that may be used to train its generative AI models.
OpenAIOAI-SearchBotSurfaces websites in ChatGPT's search features. Opt out and you will not appear in its search answers.
OpenAIChatGPT-UserVisits a page when a user asks. OpenAI says robots.txt rules may not apply to these user-initiated visits.
GoogleGoogle-ExtendedA robots.txt token that controls use for training future Gemini models. It does not affect inclusion in Google Search.
AnthropicClaudeBotCollects web content that could contribute to model training.
AnthropicClaude-SearchBotImproves the quality of search results. Blocking it means your content is not indexed for search.
AnthropicClaude-UserFetches pages when Claude users ask questions. Blocking it can reduce your visibility in user-directed search.
PerplexityPerplexityBotSurfaces and links websites in Perplexity search. Perplexity says it is not used for training foundation models.
PerplexityPerplexity-UserVisits a page to answer a user's question. Perplexity says it generally ignores robots.txt because a user requested it.

So the simple rule: if you want to be found in AI answers, allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and Googlebot. If you do not want your content used for training, you can disallow the training bots (GPTBot, ClaudeBot, Google-Extended) and still stay findable. It is your choice, and the two goals are separate.

Allow time for changes: OpenAI and Perplexity both say a robots.txt change can take about 24 hours to take effect. Anthropic notes the rules must be set for each subdomain separately.

What you can skip

Plenty of advice online is about things that Google says do nothing. Save your time.

  • llms.txt. This is a proposed Markdown file that summarizes your site for AI tools. Google says Search ignores such files, so they neither help nor harm. We found no statement from OpenAI, Anthropic or Perplexity that they use it for answers either. It is harmless to publish, but do not expect results from it.
  • FAQ markup for rich results. Google deprecated the FAQ rich result, and it stopped appearing in Search on May 7, 2026. Keep writing question-and-answer sections for your readers, but the special display is gone.
  • Chopping pages into tiny pieces. Google says there is no requirement to do this.
  • Buying mentions. Google says seeking inauthentic mentions across the web is not as helpful as it might seem.

Structured data is still worth having for other reasons. Google still supports markup for things like your organization, articles, products, local businesses and breadcrumbs. Just do not expect it to be a switch for AI answers, and make sure it matches the text visitors can see.

How to measure it

You cannot manage what you cannot see, and the tools are getting better.

  • Google Search Console. Google says traffic from AI Overviews and AI Mode is counted in the Performance report under the Web search type, mixed in with normal results.
  • Bing Webmaster Tools. In February 2026 Microsoft launched an AI Performance report in public preview. It shows how often your pages are cited in Microsoft Copilot, AI summaries in Bing and some partner services, with a sample of the queries behind them. Microsoft is clear that citation counts do not show placement, ranking or authority.
  • Your own analytics. Check which referrers send visitors. Watch trends over months, not days.
Related guideGoogle sends fewer clicks now. Here is how to get picked by AI instead.Why clicks are changing and how to write pages that AI tools quote.

Questions people ask

Do I need special markup or an llms.txt file to appear in Google's AI answers?

No. Google's guide to generative AI features says you do not need new machine-readable files, AI text files, markup or Markdown, and that doing so will neither help nor harm your visibility because Google Search ignores them. There is also no special schema.org markup to add.

If I block GPTBot, will I disappear from ChatGPT?

Not from ChatGPT search. OpenAI says GPTBot is for training its models and OAI-SearchBot is what surfaces sites in ChatGPT's search features. If you block OAI-SearchBot you will not appear in ChatGPT search answers, though you can still show up as a plain link.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It is a robots.txt token that controls whether your content may be used for training future Gemini models.

Should I still add FAQ markup?

Write the questions and answers on the page, because readers and AI tools like them. But do not expect a special result in Google: it stopped showing FAQ rich results from May 7, 2026.

Do AI crawlers run JavaScript?

The vendors do not say, except Google, which renders JavaScript as long as it is not blocked. One 2024 analysis by Vercel and MERJ found that GPTBot, ClaudeBot and PerplexityBot did not render it. That is a single, older study, so the safe approach is to keep your important text in the HTML your server sends.

Want a second pair of eyes on your site?

Geneki can review how your website is crawled, fix the technical blockers and set up measurement, so you know what is working.

Talk to us

Sources

This guide is general information. AI products change quickly, so check each company's own page before you change your robots.txt. We name no results or rankings you can expect.