What Is llms.txt? Does It Work? A Guide to the Evidence

What is llms.txt?

llms.txt is a Markdown directory file served at /llms.txt in a site's root. Jeremy Howard of Answer.AI proposed the idea on 3 September 2024. The aim is for a large language model to learn what a site is and which pages matter from one place, without parsing HTML full of menus, ads and code.

The only required element in the proposed structure is the H1 at the top: the site or project name. A short blockquote summary follows, then an optional unheaded note and link lists separated by H2 headings. Each link is written as the page name in brackets followed by the address in parentheses, with an optional colon and short note.

No standards body has approved the file. It spread as a community proposal, so no search engine or AI system is required to read it.

What does llms.txt do?

The proposal targets the inference moment. When a user asks an AI tool a question, or a coding agent reads documentation, the tool can summarise the site from one file. Model training and Google ranking are not the target.

The area with demonstrated practical use is documentation and API sites. In Ahrefs' server logs, GPTBot and Claude Code stand out among the named bots that request the file. Coding agents fetch documentation files, a behaviour distinct from general-purpose chat assistants searching the web.

In the experiment published by Search Engine Land, only a small share of citations went to the file, and those files carried API usage information. Files written for search optimisation produced no citations. The detailed figures are in the evidence section below.

Is llms.txt the same as robots.txt?

No. robots.txt is a protocol that tells a bot which URLs it may crawl, and bots follow it. llms.txt blocks or redirects no bot; it only offers a suggestion. Search Engine Journal also calls the “AI equivalent of robots.txt” comparison wrong.

robots.txt still decides whether a bot may enter your site. OpenAI's bot documentation describes robots.txt rules for OAI-SearchBot and GPTBot and does not mention llms.txt.

Does Google use llms.txt?

No, Google Search does not use it. In April 2025, John Mueller compared the file to the keywords meta tag: what a site owner says about itself is not an independent signal from the page shown to users. Gary Illyes was reported to say the same at a Search Central event in July 2025.

Google's generative AI search guide, published on 15 May 2026, described AEO and GEO as “still SEO” and said llms.txt was not required. In June 2026 Google clarified the guide: creating the file for other systems is fine, but it neither helps nor harms visibility in Google Search.

Chrome is a separate product. Lighthouse's experimental “agentic browsing” audit checks for llms.txt optionally, and Chrome documentation says agents may crawl more to understand a site when the file is missing. This has nothing to do with ranking or AI Overviews.

Does llms.txt increase AI citations?

Independent studies point in the same direction. None is conclusive on its own, but they agree.

• Ahrefs (June 2026, 137,210 domains) — 28% of domains publish a file. Of roughly 38,000 sites with a valid file, 97% received no request in May. 96% of requests came from bots, and only 19.5% of those were a named AI tool. Sites without a file received no request at all from AI bots, so bots do not search for the file. Ahrefs also notes that its customer base skews technical and that it did not test file validity.

• SE Ranking (about 300,000 domains) — 10.13% of domains have the file. No relationship appeared between the file's presence and AI citation counts; prediction accuracy improved when the variable was removed from the model. The result is limited to the data and model used.

• Otterly, 90-day log (reported by Optimizely) — Of 62,100 AI bot visits, 84 — about 0.1% — reached an llms.txt file.

• Landwehr experiment (Search Engine Land) — Six of roughly 18,000 citations went to the file, or 0.03%. Those six files carried API usage information.

Why is that? The first reason is simple: these systems already read your page. Second, the site owner writes the file and it can contradict the page shown to users, so it is not treated as a trustworthy signal. The “I added it but got no citations” experience does not conflict with this data.

Which sites benefit from llms.txt?

Documentation, API and developer-product sites. The documentation sites of companies such as Anthropic and Cloudflare are reported to carry these files. Content that is Markdown-friendly, extensive and clearly structured is the best fit.

On blogs, corporate service sites and e-commerce sites, the list of “important pages” is short and already in the sitemap. I have not seen a study showing a benefit for these sites. The file also needs maintenance: links and notes must be updated whenever content changes.

How do you create llms.txt?

The file should stay a short reading list, not a copy of the sitemap.

• Choose the 10-20 pages a reader should see first. Do not make the file a copy of the sitemap.

• Write the site name in the H1. Put a one- or two-sentence summary directly beneath it.

• Group the pages under H2 headings. Give each link a short note that says what the page actually covers.

• Do not inflate the notes. Claiming something in the file that is not on the page risks showing systems content different from what users see. Mueller raised the same concern.

• Prepare the file as UTF-8 plain text and upload it to the root directory.

For the format check, use the llms.txt generator tool. Enter the site name, a short summary and the pages to compose the file in your browser; nothing is sent to a server. Open it from the link at the end of this article.

Example file

A short example that shows the format. The addresses and brand are fictional.

# Example Brand

> Products, services and contact details. This content exists only to demonstrate the format.

## Guides

- [What Is GEO?](https://example.com/guides/what-is-geo/): An introduction to generative AI visibility
- [What Is Schema Markup?](https://example.com/guides/schema-markup/): JSON-LD examples and implementation

## Tools

- [Free SEO tools](https://example.com/tools/): SERP snippet, meta and title checks

Open the links live after uploading the file. The tool does not test whether pages resolve; check that yourself before publishing.

Where should the llms.txt file be uploaded?

In your domain's root directory, at https://yourdomain.com/llms.txt. The proposal also allows subdirectories, but keeping it at the root is the common practice. In Next.js, Nuxt or a similar framework, place the file in the public folder. If you use a CDN, make sure the cache keeps the file current.

Open it in a browser and confirm the server returns 200. The command below shows response headers only; you still need to open every link separately.

curl -I https://yourdomain.com/llms.txt

Does llms.txt harm SEO?

According to Google's June 2026 statement, it neither helps nor harms Search visibility. The real risks sit elsewhere: a file with stale links can point to the wrong pages and needs manual maintenance on every update.

The file is not a security surface; it is only text. Even so, claims that do not appear on the page risk presenting different content to users.

Should I add llms.txt?

Ask yourself a few questions.

• If you have a documentation or API site, add it; coding agents use these files.

• If your goal is ranking or more AI citations, the file will not do that; start with the fundamentals below.

• If your CMS generates the file automatically, add it but spend no extra effort. If none of these apply, skip it.

If none of the three applies, do not add the file. That does not make the file bad; it means your site is not a fit for this job.

What should I do for AI visibility instead?

In the GEO checklists in the Digital Marketing and Growth Guide, llms.txt is one item and the one with the weakest evidence. The same time is better spent on the following.

• Bot access — Make sure robots.txt and the CDN do not block Googlebot, OAI-SearchBot, Claude-SearchBot or PerplexityBot. GPTBot and ClaudeBot are training bots and a separate decision. Cloudflare began blocking AI crawlers by default on new domains in 2025; check your own panel.

• Server-rendered HTML — The page text should be visible with JavaScript disabled.

• Indexable pages and Search Console settings — According to Google's guide, a page must be indexed and eligible for a snippet to appear in AI features; the generative AI setting in Search Console must also be on.

• Sources, statistics and citations — In the Princeton GEO study, these methods raised visibility in AI answers by up to 40%. That is a laboratory benchmark and does not guarantee the same effect in live systems.

• Real brand mentions — In Ahrefs' study of 75,000 brands, brand mentions showed the strongest relationship with AI Overviews visibility (0.664). This is a correlation, not proof of causation.

• Fresh content — Ahrefs found that URLs cited by AI assistants were on average 25.7% fresher than organic results.

• Measurement — Build segments in GA4 for chatgpt.com, perplexity.ai, gemini.google.com and claude.ai. The generative AI report in Search Console currently provides impressions only, not clicks.

This list helps you see what your site can actually achieve before adding the file. llms.txt does not replace any of it.

What did I see on my own site?

While preparing this guide, I checked whether mervealgan.com had an llms.txt file: as of 4 November 2026, it does not. The site is not a documentation or API product and does not meet the conditions described in this article. I did not add the file just because it appears on a blog checklist.

That also means I have no /llms.txt request log to share. Rather than invent a number, I am leaving this section open. If I launch a documentation page later, I will add the file and update this section with server logs after a few weeks.

Frequently asked questions