Growth Strategy
llms.txt Explained: What It Does, What It Doesn't and How to Create One
llms.txt won't lift Google rankings, but it helps AI agents find your best pages. What the file is, Google's 2026 position and a 30-minute setup.
Brilliant Brains · · 7 min read
If an SEO audit flagged your site for a "missing llms.txt", here is the short answer. llms.txt is a plain Markdown file at the root of your website that gives AI agents a curated map of your most useful pages. It does not improve Google rankings or AI Overviews, and Google now says so in its own documentation. It is still worth 30 minutes of your time, for different reasons than most vendors claim.
What is llms.txt?
llms.txt is a proposed standard, first published in September 2024 by Jeremy Howard of Answer.AI. Web pages are built for people, and an AI agent cannot read a whole website before answering a question. A short Markdown file at a predictable address tells it who you are, what matters and where the detail lives. (llmstxt.org)
Version 2 of the specification, released in August 2026, adds two useful points:
- The file can sit at the root (/llms.txt) or in a sub-folder such as /docs/llms.txt.
- Sites are encouraged to offer clean Markdown versions of key pages, for example the same URL with .md added.
llms.txt vs robots.txt vs sitemap.xml
| File | Job | If it is wrong |
|---|---|---|
| robots.txt | Controls what crawlers may fetch | You can block your site from search or ChatGPT search |
| sitemap.xml | Lists every URL you want indexed | New pages may be found more slowly |
| llms.txt | Gives agents a curated, annotated shortlist | Agents may start from outdated pages |
If your goal is to appear in ChatGPT search, robots.txt matters far more than llms.txt. OpenAI says sites that block its OAI-SearchBot crawler will not be shown in ChatGPT search answers, and its crawler documentation does not mention llms.txt. (OpenAI)
What llms.txt does and does not do
It does not affect Google Search
Google's May 2026 guide to generative AI features in Search lists llms.txt among the things you do not need. In June 2026 Google added that keeping one is fine, but it will neither help nor harm your visibility because Google Search ignores it. (Google Search Central, Search Engine Land)
What does matter for AI Overviews and AI Mode is plain SEO: pages that are indexed, eligible for a snippet and genuinely useful.
There is no public evidence it earns citations
As of October 2026, we found no public statement from OpenAI, Anthropic or Perplexity saying their search crawlers use other websites' llms.txt files to decide what to cite. Be sceptical of any promise that llms.txt will get you "recommended by ChatGPT". (Search Engine Journal)
Where it does help
- Agents reading your site on demand: coding assistants already follow llms.txt on documentation sites, and agents comparing clinics, suppliers or software benefit from a clear, owner-written summary.
- Chrome Lighthouse: its agentic browsing audits include an llms.txt check. A missing file is marked not applicable, but a file that returns a server error is flagged. (Chrome for Developers)
- A better website: choosing your 20 most useful pages often exposes thin service pages, outdated pricing or key facts hidden in PDFs.
Who should prioritise it
| Site type | Priority |
|---|---|
| SaaS, APIs and developer documentation | High |
| E-commerce, D2C brands and healthcare providers | Medium |
| B2B services and agencies | Low to medium |
| Small brochure sites | Low |
How to create an llms.txt file in 30 minutes
- Choose 10 to 30 pages. Use your Search Console queries to pick pages that answer real customer questions: services, pricing, locations, booking and your strongest guides.
- Write the heading and summary. Start with an H1 of your site name, the only required line. Follow it with a blockquote of one or two factual sentences on what you do, for whom and where.
- Group and annotate links. Use H2 sections such as Services, Pricing and Guides. Write each link as a Markdown link, a colon and a one-line note. Put secondary links under a section called Optional.
- Publish at /llms.txt. Serve it as plain text with a 200 status. On single-page app sites, check that a catch-all route is not returning your homepage instead.
- Test and maintain. Give an AI assistant only your llms.txt and ask it your customers' questions, run a Lighthouse report, and update the file whenever services or prices change.
On WordPress, Wix or a documentation platform, tools such as Yoast SEO, AIOSEO, Wix, Mintlify and GitBook can generate the file for you. Review the output: a file that lists every page defeats the point of curation.
Mistakes to avoid
- Pasting your whole sitemap instead of a shortlist.
- Writing taglines instead of facts. "Family dentistry in Hyderabad, open Monday to Saturday" helps an agent; "Smiles that shine" does not.
- Letting it go stale with old prices or closed branches.
- Blocking AI search crawlers in robots.txt while expecting llms.txt to earn visibility.
- Including anything private. The file is public by design.
What actually drives AI search visibility
- Keep important pages crawlable, indexed and eligible for snippets in Google.
- Allow AI search crawlers, such as OAI-SearchBot for ChatGPT search, in robots.txt.
- Publish specific, first-hand content that answers real questions better than competing pages.
- Earn genuine mentions and reviews on the third-party sites AI systems draw on.
- Measure it with the Generative AI performance report in Search Console and AI referrals (chatgpt.com, perplexity.ai, gemini.google.com) in GA4.
Go deeper with The Real Difference Between SEO, AEO and GEO, How Answer Engines Are Rewriting the Top of the Funnel and Hybrid Engine Optimization (HEO).
Frequently Asked Questions
Does llms.txt improve Google rankings or AI Overviews?
No. Google states that Google Search does not use llms.txt, and that keeping one will neither help nor harm your rankings or visibility in AI Overviews and AI Mode.
Will ChatGPT, Claude or Perplexity read my llms.txt?
Their agents may read it when a user asks them to work with your site. As of October 2026 there is no public statement that their search crawlers use it to decide citations. For ChatGPT search, allow OAI-SearchBot in robots.txt.
Is llms.txt the same as robots.txt?
No. robots.txt controls what crawlers may access. llms.txt is a voluntary guide to your best content and cannot grant or block access.
How many links should an llms.txt file contain?
For most small and mid-sized business sites, 10 to 30 well-annotated links. Large documentation sites can use more, with secondary material under the Optional section.
What is llms-full.txt?
A convention some documentation platforms use to bundle the full text of their docs into one file. It is not required by the llms.txt specification, and most business websites do not need it.
Can llms.txt hurt my website?
Not in Google Search. The real risks are an outdated file that sends agents to wrong information, and exposing anything you would not want public.
The bottom line
llms.txt is worth having, and not worth overpaying for. Treat it as 30 minutes of housekeeping, keep it current, and put your real effort into crawlable, useful pages, clear access rules and measurement.
Want to know how your brand shows up across Google and AI answers? Our SEO, AEO and GEO team can audit your search visibility and AI referral data and tell you which fixes will move results. Get your growth plan.
Sources
- The /llms.txt file, v2, llmstxt.org (Jeremy Howard, Answer.AI), accessed 27 October 2026
- Google's Guide to Optimizing for Generative AI Features on Google Search, Google Search Central, accessed 27 October 2026
- Google says LLMS.txt files won't harm or help your search rankings, Search Engine Land, accessed 27 October 2026
- llms.txt audit, Lighthouse, Chrome for Developers, accessed 27 October 2026
- Overview of OpenAI Crawlers, OpenAI Developers, accessed 27 October 2026
- Google's llms.txt guidance depends on which product you ask, Search Engine Journal, accessed 27 October 2026
