How to let AI crawlers read your website: robots.txt, sitemaps, llms.txt, and structured data

An AI assistant cannot cite a page it cannot reach. Before any clever content strategy, there is a plain technical question: can the bots that power AI search actually read your website?

This checklist covers the fixes that matter, in the order we usually work through them. It is the technical groundwork for GEO.

1. Check your robots.txt

Your robots.txt file, at yourdomain.com/robots.txt, tells crawlers what they may visit. Google explains the format in its robots.txt introduction.

Several AI companies publish the names of their crawlers. Names that appear in their documentation include:

• GPTBot, OAI-SearchBot, and ChatGPT-User from OpenAI. See OpenAI's crawler documentation.
• PerplexityBot from Perplexity. See Perplexity's crawler documentation.
• Google-Extended, a control that lets you decide whether Google may use your content for its AI products. It is listed with the other Google crawlers. Ordinary Googlebot still handles search.
• ClaudeBot from Anthropic.

A robots.txt that welcomes them looks like this:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Some crawlers collect data to train models, while others fetch pages to answer a user's question right now. You may decide to treat them differently. Crawler names and behaviour change, so check each provider's current documentation before you decide.

2. Make sure a firewall is not blocking them

A permissive robots.txt does not help if your hosting, CDN, or security plugin blocks bots before they see it. Many services have a bot-protection or "block AI crawlers" setting. If you want to be cited, make sure it is off for the bots you want. Then check your server logs or your host's dashboard to confirm the bots are getting through.

3. Publish a sitemap

A sitemap lists the pages you want crawlers to find. It helps new pages get discovered faster and is easy to create. Google covers the basics in its sitemaps overview, and the format itself is documented at sitemaps.org. Reference it in robots.txt, as in the example above.

4. Keep your answers in plain HTML

Some crawlers do not run JavaScript. If your key content only appears after a script loads, those crawlers may see an empty page. The safest approach is to make sure the important text, such as headings, answers, and service descriptions, is in the HTML the server sends. Pre-rendered or server-rendered pages do this well.

A quick test: view the page source in your browser, or run the command below, and confirm your main text is visible.

curl -s https://yourdomain.com/ | grep -i "your main heading"

5. Add structured data

Structured data is a standard way to label what a page is about, so machines do not have to guess. Google's introduction to structured data explains how it works, and schema.org defines the vocabulary.

Useful types for most businesses:

• Organization or ProfessionalService to state who you are and what you do.
• Article for blog posts, described in Google's article structured data guide.
• FAQPage for real questions and answers on a page. Google limits when it shows FAQ rich results, so use it for clarity, not as a shortcut.

Always mark up only what is visible on the page. Markup that does not match the content can cause problems.

6. Consider an llms.txt file

llms.txt is a proposal for a plain-text file that summarises your site for AI systems. It is cheap to add and harmless. But it is still a proposal, and we would not rely on it: at the time of writing, its use by the big AI providers has not been widely confirmed. Treat it as a small extra, not the main job.

7. Test what a crawler sees

Once the changes are live:

1. Load your robots.txt and confirm the bots you want are allowed.
2. Fetch your key pages with curl and check the text is there.
3. Validate your structured data with a testing tool.
4. Ask an AI assistant a question your page answers, a few weeks later, and see whether it is cited.

That last step is slow and imperfect, since assistants do not always give the same answer twice. It is still the real test, which is why we track a fixed set of questions over time.

Sources and further reading

• Introduction to robots.txt (Google Search Central).
• Google's common crawlers, including Google-Extended.
• OpenAI crawler documentation.
• Perplexity crawler documentation.
• Build and submit a sitemap (Google Search Central) and the sitemaps.org protocol.
• Introduction to structured data (Google Search Central) and Article structured data.
• The llms.txt proposal.

Want someone to run this checklist on your site? Request a free audit call. We will check what the bots can see, and tell you what to fix first.