llms.txt, robots.txt and AI crawlers: what a UK business needs

If an AI engine cannot fetch your site, nothing else in this field matters. This is the ten-minute check, and the one place where a single misconfigured line does more damage than any amount of content work.

The crawlers are not one thing

The mistake is treating "AI bots" as a single group to allow or block. Each major provider runs several agents for different purposes:

• Training crawlers, gathering data to train future models.
• Search or retrieval crawlers, building the index the assistant queries when answering now.
• User-triggered fetchers, retrieving a page because someone in a chat asked about it.

These have separate user-agent strings, and blocking them has completely different consequences. Blocking a training crawler keeps your content out of a future model — a legitimate choice, with no effect on whether you appear in today's answers. Blocking the retrieval crawler removes you from the answers right now.

Plenty of sites have blocked the second while intending only the first. It is the single most consequential misconfiguration in this area, and nothing tells you it happened.

What robots.txt actually does

It is a request, and the major providers honour it. Some rules worth knowing:

• Disallow controls crawling, not indexing. A blocked page can still be referenced from elsewhere.
• Directives are matched by user-agent, and the most specific matching group wins — a specific User-agent: SomeBot group means that bot ignores your User-agent: * group entirely, including the Allow lines you thought applied to everyone.
• An X-Robots-Tag response header can carry noindex for pages that must never appear in results. That is the right tool for anything containing customer data — a Disallow merely asks a crawler not to look.

Read yours today, out loud, and be able to say which agents it allows. If you cannot, that is the finding.

Is llms.txt worth adding?

llms.txt is a proposed convention: a plain-text file at your root, describing what your site is and pointing to your most important pages, aimed at LLM consumers rather than search crawlers.

The honest position: no major engine has committed to using it, so nobody can promise it will get you cited. It is also trivial to write, costs nothing to serve, and can do no harm. We publish one. Treat it as cheap optionality, not as a ranking factor, and be sceptical of anyone selling it as one.

The check that matters most

Fetch your own homepage the way a crawler does — no JavaScript, no cookies, no session:

curl -sA "Mozilla/5.0 (compatible; GPTBot/1.0)" https://yourbusiness.co.uk | wc -c

Then read the body, not just the byte count. What you are looking for is whether your services, your locations and your trading name are present as text. A large response full of script tags and an empty root element is not readable content.

We found exactly this on our own site: a page that looked complete in a browser and arrived as a couple of kilobytes of JSON to a crawler. It was fixed by rendering on the server. If a company selling AI visibility can ship that defect, it is worth checking yours.

A UK-specific trap

A stale IPv6 record left over from an old host is one of the nastier versions of this: dual-stack clients follow it to the old server while your IPv4-only checks reach the real site and report everything as fine. Several AI crawlers resolve over IPv6. If your DNS carries an AAAA record you did not deliberately set, check where it points before assuming your site is reachable.

The order to do this in

1. Fetch your homepage without JavaScript. Read what comes back.
2. Read robots.txt. Know which agents you allow, and check whether a specific group is quietly overriding your wildcard one.
3. Confirm anything private carries noindex as a header, not just a Disallow.
4. Check your DNS for records you did not intend.
5. Then — and only then — worry about content.

A free check tells you what the engines actually say. This list tells you whether they can see you at all, and it is the cheaper problem to fix.