llms.txt: What It Is and Should Your Site Have One?

An llms.txt file is a plain Markdown file at the root of your website (/llms.txt) that lists your most important pages for AI systems to read. It sounds like a smart idea — but Ahrefs' analysis of 137,210 domains found that 97% of published llms.txt files received zero requests in May 2026, meaning the file almost nobody reads has still become a checkbox nearly a third of sites tick anyway.

Photo: Archives New Zealand · BY

What Is an llms.txt File, Exactly?

llms.txt was proposed by Jeremy Howard in September 2024 as a lightweight standard for making websites easier for large language models to navigate. The idea borrows the shape of robots.txt and sitemap.xml, but instead of controlling crawler access or listing every URL, it curates: an H1 with your project name, a one-line blockquote summary, and H2-delimited sections of Markdown links pointing to the pages that matter most — docs, pricing, API references, key blog posts.

The pitch was reasonable. Language models have limited context windows and no patience for parsing navigation menus, cookie banners, and JavaScript-rendered layouts. A clean, hand-curated list of links in plain Markdown is, in theory, far cheaper for a model to ingest than crawling your full HTML. Documentation platforms bought in fast — Mintlify generates an llms.txt automatically for every project it hosts, and WordPress, VitePress, Docusaurus, and Drupal all have plugins that do the same.

Two years on, the spec reached version 2 in August 2026, incorporating lessons from real-world use. The problem is what "real-world use" actually turned out to mean.

Who Actually Reads llms.txt Files in 2026?

This is the part most llms.txt guides skip. Ahrefs pulled server-log data across 137,210 domains that received traffic in May 2026, checked which ones served a valid /llms.txt, and then looked at who was actually requesting those files.

28% of the domains studied — roughly 38,360 sites — published a valid llms.txt file, despite no major AI platform ever officially committing to reading it. That adoption curve happened almost entirely on faith: teams added the file because it was cheap, not because they'd seen it move a metric.

Of the files that existed, 97% got zero requests for the entire month. Of the small remainder that did receive traffic, most of it wasn't from AI at all — Ahrefs found 96% of requests to llms.txt files came from bots generally, but only about a fifth of that bot traffic was AI-related, and just 1.1% of requests came from AI retrieval bots (the category that actually answers user queries, as opposed to training crawlers or infrastructure agents). GPTBot alone accounted for 4.51% of the traffic that did arrive, and Claude's crawler ClaudeBot came in even lower.

The honest summary, from Ahrefs' own SEO lead Ryan Law in a companion explainer: "no evidence that llms.txt improves AI retrieval, boosts traffic, or enhances model accuracy," and "no provider has committed to parsing it." His recommendation, in his words: "In my opinion, no, not yet."

The Data: Who's Actually Requesting These Files

Here's the traffic breakdown Ahrefs published for the small share of llms.txt files that got any requests at all in May 2026:

Two things stand out. First, most of the traffic these files get isn't AI at all — it's the general churn of bots that crawl every text file they find on a domain. Second, even within the AI share, the bots doing live retrieval — the ones that would actually cite your site in an answer — are the smallest slice of an already small number.

There's one place llms.txt earns its keep, though: coding agents. Tools like Claude Code, Cursor, and Continue will fetch a project's llms.txt when a developer points them at a domain or repo, using it to orient themselves in documentation. That's a real, working use case — it's just a different audience than "get cited by ChatGPT," which is the pitch most llms.txt guides actually make.

llms.txt vs. robots.txt vs. sitemap.xml

Founders often lump these three together because they all live in the site root as plain text. They're not the same tool:

If you're choosing where to spend an afternoon on technical setup, robots.txt and sitemap.xml are the ones with actual, confirmed readership across the AI ecosystem. Our technical SEO checklist for a brand-new website covers both in the order that matters.

So Should Your Site Have One?

Given the data, here's a practical framework instead of a flat yes or no:

• You run developer tools, an API, or open-source docs. Ship it. Coding agents genuinely use llms.txt to orient in documentation, and if your product's platform (Mintlify, for instance) generates one automatically, there's close to zero marginal cost.
• You run a marketing site, SaaS landing pages, or a blog with no docs product. Skip it for now, or treat it as a five-minute nice-to-have at most. There is no evidence it improves how often ChatGPT, Perplexity, or Google's AI Overviews cite you — the retrieval bots that generate those answers overwhelmingly aren't the ones requesting these files.
• You're deciding between llms.txt and literally anything else on your SEO to-do list. Do the other thing first. The mechanisms that are confirmed to influence how often AI tools recommend and cite sites are ordinary ones: crawlable HTML, clear entity definitions, structured data, and content that directly answers a query in the first sentence — not a separate curated file most crawlers ignore.

This is the same pattern we've written about with how AI Overviews are reshaping blog traffic and with generative engine optimization more broadly: the tactics that move the needle for AI citation are unglamorous — answer-first writing, verifiable sourcing, clean semantic HTML — not a new file format bolted onto the root of your domain.

How to Create an llms.txt File (If You Decide To)

If your site fits the "ship it" case above, the setup is genuinely simple:

1. Create a plain Markdown file named llms.txt and place it at your domain root (yoursite.com/llms.txt).
2. Start with an H1 containing your project or company name.
3. Add a one-sentence blockquote summary directly under the H1.
4. Use H2 headings to group links — for example, "Docs," "API Reference," "Guides" — with each link as a Markdown bullet, optionally followed by a short description.
5. Keep it current. A stale llms.txt pointing at deprecated docs is worse than none at all.
6. If you use Mintlify, WordPress, VitePress, or Docusaurus, check for a plugin — most generate this automatically from your existing docs structure.

That's the entire spec. There's no submission process, no verification step, and — per the data above — no confirmed retrieval benefit to chase. Full disclosure: we build Rankody, an AI SEO autopilot that researches and drafts sourced articles like this one for approval, and we'd rather tell you honestly that llms.txt is optional than pad a checklist to look thorough. If you want the comparison of how autopilot publishing tools stack up against manual and freelance workflows, see our SEObot alternatives roundup.

FAQ

Does ChatGPT read llms.txt files? There's no public confirmation that OpenAI's live retrieval systems parse llms.txt for answering user queries. GPTBot does occasionally request these files, but Ahrefs' May 2026 data shows AI retrieval bots collectively made up just 1.1% of the traffic that reached llms.txt files at all.

Is llms.txt the same as robots.txt? No. robots.txt tells crawlers what they're allowed to access and is respected by essentially every major search and AI crawler. llms.txt is a curated content list with no enforcement mechanism and no confirmed adoption by AI answer engines.

Will adding llms.txt help my site rank or get cited by AI Overviews? There's no evidence for this as of the Ahrefs May 2026 study. Google's own AI optimization guidance has stated llms.txt isn't needed for AI visibility. Answer-first content, clean HTML, and verifiable claims are the factors with demonstrated impact.

Who actually benefits from having an llms.txt file today? Developer-tool and documentation sites see the clearest use, since coding agents like Claude Code and Cursor fetch llms.txt to orient themselves in a project's docs. General marketing and content sites see minimal confirmed benefit.

Does it hurt to add one anyway? No — it's low-cost and low-risk to publish. The caution is about where you spend limited time: llms.txt shouldn't come before robots.txt, sitemap.xml, or content that actually answers your target query in the first sentence.

The bottom line: llms.txt is a real, well-specified format with one confirmed audience — coding agents — and no confirmed benefit yet for AI search visibility, so treat it as optional infrastructure, not an SEO tactic.

Rankody researches your market, writes sourced long-form articles, and publishes on schedule — every piece waits for your approval first. Three articles free, no card. Analyze my site