You have been told to add a file called llms.txt to your site so AI models can understand it.

The advice contradicts itself. Google’s own documentation says you do not need one. Google’s own Lighthouse tool audits your site for one. And the largest study of the format found that almost nobody reads these files at all.

This guide covers what the current specification actually requires, what the crawler evidence shows, and how to decide whether your site needs one.

What is an llms.txt file?

An llms.txt file is a plain text file written in Markdown and published at the root of a website, usually at yourdomain.com/llms.txt. It gives AI agents a short, curated map of the pages a site owner considers most important, along with links to cleaner versions of that content.

Jeremy Howard of Answer.AI published the llms.txt specification in September 2024. It now sits at version 2, rewritten after two years of real-world adoption.

It is a proposal, not a ratified web standard. No standards body has adopted it, and as you will see below, no major AI vendor has committed to reading it.

It is also worth being clear about what llms.txt is not. It does not control access to anything, and it does not replace robots.txt.

The problem llms.txt was designed to solve

The proposal starts from three complaints about how websites serve machines. They are worth taking one at a time, because they explain what the file does and does not try to fix.

Context windows and token cost

An agent working on your behalf can only hold so much text at once. Most websites do not fit.

The specification puts it plainly: agents are best served by concise, expert-level information gathered in a single accessible location. Every token an agent spends parsing your navigation menu is a token it does not spend on your actual content.

That cost is small per page. It compounds when an agent fetches and discards a dozen pages before finding the one that answers the question.

HTML noise: navigation, ads and JavaScript

Web pages are built for people. An HTML page wraps its information in navigation, adverts and scripts, and converting that back into clean text is difficult and imprecise.

There is a harder version of this problem. Most AI crawlers never run your JavaScript at all.

Vercel and MERJ analysed AI crawler behaviour across more than 500 million GPTBot fetches and found no evidence of JavaScript execution. GPTBot downloaded JavaScript files in roughly 11.5 percent of requests and ClaudeBot in roughly 23.8 percent, but neither ran them. PerplexityBot behaves the same way.

The one meaningful exception is Google’s Gemini, which inherits Googlebot’s rendering infrastructure.

So if your main content only appears after client-side rendering, most AI crawlers see an empty shell. An llms.txt file does not fix that. It points at content that is already readable.

Crawlers guessing at which pages matter

A crawler that reaches your site cannot tell your current pricing page from a blog post you published five years ago and forgot about.

It fetches what it finds. If what it finds first is stale, the answer it builds is stale too.

This is the same underlying problem as crawl budget, where a search engine spends limited attention on URLs you do not care about. The difference is that a search engine eventually recrawls and corrects itself. An agent answering a question right now does not get a second pass.

llms.txt vs robots.txt vs sitemap.xml

Three files sit at the root of most websites and people mix them up constantly. They do different jobs, and llms.txt does not replace either of the other two.

What robots.txt does

robots.txt controls access. It tells automated tools which parts of your site they may request, and the major crawlers honour it.

It is also the file Google actually uses for AI crawl control. Google’s list of common crawlers defines a Google-Extended token that publishers add to robots.txt to decide whether crawled content may be used for training and grounding Gemini models.

One detail is worth holding onto for later. Google-Extended is a control token, not a crawler. Google states it has no separate HTTP user agent string, and that crawling happens with existing Google user agents while the token operates in a control capacity.

So robots.txt is a permission layer. It says where a bot may go. It says nothing about what is worth reading.

What sitemap.xml does

A sitemap lists the indexable, human-readable URLs on your site so search engines can find them.

It is a discovery mechanism, not a curation one. A sitemap listing 40,000 URLs is doing its job correctly. It makes no claim about which of those 40,000 matter.

What llms.txt does differently

llms.txt is curated rather than complete. You choose what goes in, and leaving things out is the point.

It is also consumed at a different moment. A search engine reads robots.txt and your sitemap while crawling, ahead of any query. An llms.txt file is meant to be read on demand, when an agent needs information about your site to answer a question in front of it.

The specification is explicit that the expectation was inference rather than training, and that this is how the format has in fact been used.

And it points somewhere different. The links in an llms.txt file are supposed to lead to LLM-friendly content, such as Markdown versions of your pages, rather than to rendered HTML.

Why sitemap.xml is not a substitute

The specification gives three reasons, and each one stands on its own.

A sitemap generally will not list the LLM-readable versions of your pages, because those versions are not what search engines index.

A sitemap does not include URLs on other sites, even when an external page would help an agent understand yours.

And a sitemap will usually cover far more documents than fit in a context window, most of which an agent does not need.

The four files side by side

File

What it is for

Who reads it

Honoured today

Affects Google ranking

Where it lives

robots.txt

Access control

Search and AI crawlers

Yes, widely

Indirectly, by blocking pages

Site root only

sitemap.xml

URL discovery

Search engines

Yes

Indirectly, by aiding discovery

Root, or anywhere referenced from robots.txt

llms.txt

Curated content map for agents

Some AI agents and documentation tools

Inconsistently, see below

No. Google Search ignores it

Root, or any subpath it describes

llms-full.txt

Full content export in one file

Some AI agents

Inconsistently

No

Alongside llms.txt

 

What goes inside an llms.txt file

The specification is prescriptive about order, which is something the popular explainers tend to skip. Sections appear in the sequence below, and all but one are optional.

An optional byte-order mark

A byte-order mark may appear at the very start of the file. It is optional, most editors will not add one, and you can ignore this unless your tooling inserts one automatically.

An H1 with the project or site name

This is the only required section in the entire specification. Everything below it is optional.

A file containing nothing but a single H1 is a valid llms.txt file. Not a useful one, but a valid one.

Use the name of the site or project. This is not a title tag, so there is nothing to be gained by stuffing keywords into it.

A blockquote summary

A short summary in a Markdown blockquote, carrying the key information an agent needs to make sense of everything that follows.

Treat it as the answer you would give if someone asked what your site is for and you had one sentence to do it in.

Free-form detail sections

After the blockquote you can add any number of Markdown sections giving more detail about the project and how to interpret the files you list.

There is one constraint here that is easy to miss. These sections may contain paragraphs, lists and other Markdown, but not headings. Headings are reserved for the link lists that come next.

H2-delimited file lists

Next come the link lists, each introduced by an H2 that names the group: Docs, Guides, Policies, whatever fits your site.

Under each H2 you write a Markdown list. Every item needs a link in the form [name](url). After the link you may add a colon and a short note about what the page covers.

That note is optional in the specification and worth treating as compulsory in practice. A bare URL tells an agent nothing about when to choose that page over another one.

The “Optional” section convention

One H2 carries a conventional meaning. A section named Optional holds secondary links that an agent may skip when it needs a shorter context.

It gives you somewhere to put supporting material without forcing every agent to spend tokens on it.

A complete worked example

Here is a valid file for a small business site, using every section described above.

				
					# Northbrook Dental

> A three-clinic dental practice in Singapore offering general, cosmetic
> and paediatric dentistry. This file lists the pages that best describe
> our services, pricing and clinic details.

We update pricing quarterly. Where a page and this file disagree, the page
is correct.

## Services
- [General dentistry](https://example.com/services/general.md): Check-ups,
  scaling and polishing, fillings and extractions, with typical appointment
  lengths.
- [Cosmetic dentistry](https://example.com/services/cosmetic.md): Veneers,
  whitening and bonding, including what each treatment does and does not fix.
- [Paediatric dentistry](https://example.com/services/paediatric.md):
  Treatment for children under 12 and what to expect at a first visit.

## Practical information
- [Pricing](https://example.com/pricing.md): Current price ranges for every
  treatment, updated quarterly.
- [Clinic locations](https://example.com/clinics.md): Addresses, opening
  hours and parking for all three clinics.
- [Insurance and payment](https://example.com/payment.md): Accepted insurers,
  instalment options and government scheme coverage.

## Optional
- [About the practice](https://example.com/about.md): Founding story,
  clinical team and professional memberships.
- [Patient stories](https://example.com/stories.md): Written case studies
  with before and after context.

Read that against the rules. The H1 names the site. The blockquote summarises it. The plain paragraph adds a caveat without using a heading. Two H2s group the links, every link carries a description, and the Optional section holds the material an agent can skip.

Note the .md extensions on the links. Those point at Markdown versions of the pages rather than the rendered HTML, which is the second half of the proposal covered in the next section.

At the other extreme, this is also a valid llms.txt file:

				
					# Northbrook Dental

It satisfies the specification and does almost nothing for you. The value sits in the links and their descriptions, not in the file existing.

What changed in llms.txt v2

Most explainers on this topic still describe version 1 of the proposal. Version 2 landed after two years of adoption and changed enough to matter.

Here is what is new.

The file can now live at any subpath

The original proposal put the file at the site root. Version 2 of the specification allows it at the root or at any path within the site, covering the pages underneath that path.

So /docs/llms.txt covers everything under /docs/, and you no longer need write access to the root of the domain to publish one.

That matters if you run documentation on a subpath, or if you publish on a shared host where you control a directory and nothing above it.

When several files apply, the most specific one wins

Because files can now sit at multiple paths, you can end up with more than one covering the same page.

The rule is simple: agents should use the most specific file that applies.

If you publish /llms.txt for the whole site and /docs/llms.txt for your documentation, an agent fetching a page under /docs/ should use the docs file. The site-wide file still covers everything else.

This lets you write a broad file for the business and a detailed one for the section that needs depth, without the two fighting each other.

Companion Markdown versions of your pages

This is the half of the proposal most coverage leaves out entirely.

The specification also asks you to publish clean Markdown versions of pages agents might need, at the same URL as the original. You have two options.

Append .md to the existing URL, so page.html gets a companion at page.html.md.

Or replace the extension, so page.html becomes page.md.

For URLs without a filename, append index.html.md or index.md instead.

The point is that the links in your llms.txt should lead somewhere already clean, rather than to a page an agent has to strip down for itself.

Link relations: alternate and describedby

Publishing Markdown versions only helps if agents can find them. Version 2 recommends two standard link relations for exactly that.

rel=”alternate” with type=”text/markdown” points at the Markdown version of a page. rel=”describedby” points at the llms.txt file that covers it.

As elements in your page head, they look like this:

				
					<link rel="alternate" type="text/markdown" href="/docs/page.html.md">
<link rel="describedby" href="/docs/llms.txt">

The HTTP Link header alternative

The same relations can travel as an HTTP Link: response header instead:

				
					Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

Two things make this more useful than it first looks.

It works for resources that are not HTML, including the Markdown files themselves, which have nowhere to put a <link> element.

And you can add it in your web server or CDN configuration without modifying a single page. If you run a large site and have no appetite for touching every template, this is the cheapest route available.

llms-full.txt and Markdown page exports

You will see llms-full.txt mentioned alongside llms.txt constantly. The two have different origins, and it is worth knowing which is which.

What llms-full.txt is and when it helps

Where llms.txt is an index, llms-full.txt is the whole thing: your content concatenated into a single Markdown file.

The current specification does not define it. It emerged as a convention from documentation platforms, with Mintlify generating llms-full.txt automatically for the sites it hosts.

The appeal is obvious for anyone feeding documentation into a coding assistant. One URL, one fetch, the entire set. It also removes the risk of an agent assembling a partial picture from three pages out of forty.

Why llms-full.txt backfires on large sites

The file grows with your content, and nothing caps it.

A documentation set of any real size produces a file too large for the context window it was meant to fit into. Mintlify’s own agent-readiness checks include a test for whether llms-full.txt sits within a reasonable size for agents to process, which tells you this is a live problem rather than a theoretical one.

So ask one question before publishing one. Could an agent load this file and still have room left to do anything with it? If not, you have built something nobody can use.

Per-page .md exports

The third option sits between the other two.

Rather than one enormous file or a bare index, you publish a Markdown version of each page and link to those from your llms.txt.

Markdown parses more reliably than rendered HTML because the structure survives. Headings stay headings, lists stay lists, code blocks keep their boundaries, and nothing has to be inferred from styling.

Several documentation platforms generate these for you. If yours does not, producing them at build time from your content source is usually less work than it sounds, and it stops them drifting out of date.

Does anything actually read llms.txt?

This is the question the rest of the article turns on, so it is worth walking the evidence before the verdict.

What the AI vendors have actually committed to

Nothing, is the short answer. Here is the position vendor by vendor.

OpenAI documents four user agents in its overview of OpenAI crawlers: OAI-SearchBot for ChatGPT search, GPTBot for training, OAI-AdsBot for ad landing pages, and ChatGPT-User for user-initiated fetches. All four are managed through robots.txt. The page never mentions llms.txt as an input or a control.

Anthropic publishes an llms.txt file for its own developer documentation. It has not stated that its crawlers read yours.

Google manages AI crawl behaviour through the Google-Extended token described earlier, and its documentation says plainly that Search does not use llms.txt. More on that below.

Meta publishes no crawler guidance mentioning the format, and Perplexity has made no commitment to reading it either.

One detail is worth pausing on. OpenAI’s crawler page carries a note at the top pointing readers at its own llms.txt, and telling you that Markdown versions of its documentation are available by appending .md to any page URL.

So OpenAI implements both halves of the version 2 proposal for its own documentation, on the very page where it documents nothing about reading yours. That gap between publishing a file and consuming one runs through this entire topic.

The 137,000-domain adoption study

Ahrefs studied 137,000 domains in its analytics data and found 28 percent publishing an llms.txt file. That is roughly 38,000 sites.

The study carries its own caveat, and it deserves repeating. The sample comes from sites running its analytics software, which skews towards owners who already care about AI visibility. Adoption across the web is likely lower.

Even so, more than one in four, for a format no vendor has committed to reading.

97% of those files were never requested

Same study, and this is the number that matters.

Of the roughly 38,000 domains with a valid llms.txt file, 97 percent received no requests for it at all during May 2026. No bots, no humans. All the measured traffic went to the remaining 3 percent.

The Search Engine Land server-log test

Semrush ran its own experiment. It added an llms.txt file to Search Engine Land in March 2025, then went to the logs.

Between mid-August and late October 2025 it recorded no fetches of the file from GPTBot, PerplexityBot or ClaudeBot. Googlebot and Bingbot requested it a handful of times with no sign of special handling. Semrush found no correlation between publishing the file and any improvement in AI visibility or traffic.

One item on that list needs a correction, and it matters if you plan to run the same check yourself. Semrush also reported no visits from Google-Extended. Google-Extended cannot appear in a server log at all, because it is a robots.txt control token rather than a crawler, as covered above. Its absence there is not evidence in either direction.

What Google has said, and where it contradicts itself

Google’s guide to generative AI features contains a section titled “Mythbusting generative AI search: what you don’t need to do”. llms.txt is the first item on the list.

The wording repays close reading, because most coverage quotes half of it.

Google says you do not need machine-readable files to appear in Search, “as Google Search itself doesn’t use them”.

But the same passage goes on to say it is “completely fine” to create and maintain these files for other services that do use them, and that doing so will neither harm nor help your visibility in Google Search.

That is a narrower claim than “llms.txt is dead”. Google is telling you its own Search product ignores the file, not that no system anywhere reads it.

Then, days after that guidance appeared, the Chrome team shipped an llms.txt check inside Lighthouse.

John Mueller addressed the gap publicly. He said llms.txt is “not done for search”, and described it as a “temporary crutch, perhaps to save some tokens” for AI coding tools working through developer documentation. He had made a similar point earlier, comparing the file to the old keywords meta tag: a self-declared claim about a site, when a crawler could simply read the site instead.

The Lighthouse agentic-browsing audit, and why “Not Applicable” matters

Chrome’s Lighthouse llms.txt audit sits in a category called agentic browsing, and its behaviour answers a question people keep asking.

The audit flags your page only if the server errors when the file is requested.

If you have no llms.txt and the request returns a 404, Chrome marks the audit Not Applicable, and says so explicitly on the grounds that providing the file is optional at present.

So not having one cannot cost you a Lighthouse score. If you were worried that a missing file was quietly hurting you, it is not.

How to check your own server logs for llms.txt requests

You do not have to take anyone’s study on trust. This check takes an afternoon.

Pull 90 days of server or CDN logs and filter for requests to the path /llms.txt. Then segment those requests by user agent.

The agents worth looking for are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Leave Google-Extended off the list, for the reason given above.

One warning before you draw conclusions from what you find. User-agent strings are trivially spoofed, and Google says as much in its own crawler documentation. Verify anything claiming to be a major crawler by running a reverse DNS lookup on the requesting IP, then forward-resolving that hostname back to the same address.

For the full method, including how to separate genuine crawlers from imitators at scale, see our guide to log file analysis.

Should you create one? A decision framework by site type

Most guides on this topic give one answer for everyone, and those answers contradict each other.

The more useful question is not whether llms.txt works. It is whether it works for a site like yours. Mueller drew that line himself when he said that “for non-developer sites, I don’t think this makes much sense”.

Here is where each type of site lands.

Documentation and developer-tools sites

Yes.

This is the use case the format was designed for, and it is the one place where the file demonstrably gets read. Coding assistants pointed at your documentation will use it. Google’s own guidance leaves room for exactly this, saying it is fine to maintain the file for other services that use it.

Most documentation platforms also generate and update it for you, so the ongoing cost is close to zero.

SaaS product sites

Yes, but scope it to your documentation.

Version 2 lets you publish at /docs/llms.txt rather than the site root, and for a SaaS company that is the better choice. Your documentation is the part an agent has a real reason to read. Your homepage and pricing page are not what a coding assistant needs at three in the morning.

A scoped file also stays accurate for longer. Marketing pages change constantly. Reference documentation changes on a release cycle you already track.

If your product content is the thing pulling in AI-assisted buyers, it is worth reading this alongside how we approach SaaS SEO more generally.

E-commerce stores

Not yet.

Product catalogues move too fast for a hand-maintained index. A file listing 200 products is out of date within a week, and a stale file actively misleads anything that reads it.

You also already have a better-supported mechanism. Product feeds and structured data are read by systems that have publicly committed to reading them, which is more than llms.txt can claim.

Mueller made this point in blunter terms, noting that a Markdown version of a shoe’s specifications is not going to sell more shoes.

Lead-generation and service business sites

Optional, and low priority.

A short file listing your services, your pricing page and your contact details might take an hour to write. There is no measured return on that hour, and you should go in knowing it.

If you do it, do it for optionality rather than performance. The honest case is that the cost is trivial and the format may matter later. That is a different argument from expecting it to do something for you now.

Small local business sites

No.

A site of a dozen pages does not present an agent with a navigation problem. There is nothing to curate, because everything is already one click from the homepage.

Whatever time you were going to spend on this would do more for you almost anywhere else.

The one risk to weigh before publishing

There is one downside worth naming, and it applies to every site type above.

An llms.txt file is a curated index of your best content, written by you, published openly. That is useful to an agent. It is also useful to a competitor building a scraper, who now has your own list of what matters most.

The risk is small and it is not a reason to avoid the file. But if your content is genuinely your product, factor it in before you publish.

How to create an llms.txt file, step by step

If you have decided the file is worth having, the whole process takes under an hour.

Step 1: Choose the pages worth surfacing

Start by deciding what an agent should read if it only reads five things.

For most sites that means your core service or product pages, your current documentation or guides, pricing, and the contact or about page that establishes who you are.

Leave out thin pages, near-duplicates, drafts, tag archives and anything you would not send a prospect. Inclusion is a signal, and every low-value link dilutes the ones that matter.

There is a useful shortcut for the descriptions. If your pages already have decent meta descriptions, they are usually close to what an agent needs, and you can adapt rather than write from scratch.

Step 2: Write the file in Markdown

Open a plain text editor and save a file called llms.txt. Markdown syntax is all you need, and there is not much of it.

# opens the H1 with your site name. > marks the blockquote summary. ## starts each link-list section. – begins each list item. [text](url) makes the link, and a colon after it introduces the description.

You can go beyond that if it helps. Adding H3s, tables or code blocks is valid as long as the Markdown itself is valid.

Use the worked example earlier in this guide as your starting point rather than writing from a blank file.

Step 3: Upload it to the right location

Where the file goes depends on what it covers.

A file describing your whole site belongs at the root, so it resolves at yourdomain.com/llms.txt.

A file describing one section belongs at that section’s path, so a documentation file sits at yourdomain.com/docs/llms.txt and covers everything beneath it.

Getting it there depends on your platform. On a self-hosted site you upload it to the public web root over FTP or through your hosting control panel’s file manager, usually a directory named public_html. Platform-specific routes are covered in the next section.

Step 4: Verify it is publicly reachable

Open the URL in a browser first. If you see your Markdown as plain text, you are most of the way there.

Then check the response properly from the command line:

				
					curl -I https://example.com/llms.txt

You want a 200 OK in the response. Most sites serve the file as text/plain, and some serve text/markdown. Either is fine.

The failure worth watching for is subtle. Some content management systems answer a request for a missing file with a 200 status and an HTML error page rather than a 404. In a browser that looks obviously wrong. In an automated check it looks like success. If curl returns HTML rather than your Markdown, the file is not where you think it is.

Step 5: Keep it current

A file nobody maintains is worse than no file, because it points agents at content that has moved or gone.

Add a review to the moments when your site actually changes: publishing a major new section, restructuring your navigation, releasing a new documentation version, or retiring old pages.

If your content changes weekly, do not maintain this by hand. Generate it from your sitemap or your content source at build time, so it cannot drift.

How to add llms.txt on each major platform

The route depends entirely on whether your platform lets you write to the web root. Some now generate the file for you, some give you an upload slot, and some need a workaround.

Platform

Native support

Route

WordPress

No, but plugins handle it

Yoast, All in One SEO or Rank Math

Shopify

Yes, generated automatically

Override with an llms.txt.liquid template

Wix

Yes, generated automatically

Managed from the SEO area of the dashboard

Squarespace

Yes, since July 2026

SEO settings

Webflow

Yes, custom file upload

Hosting settings

Drupal

No, contributed module

llmstxt module

Adobe Commerce

No

Manual upload to the web root

HubSpot CMS

No

File manager plus a URL redirect

Ghost

No

Manual upload or route configuration

Next.js and static builds

No

public/ directory or build-time generation

Documentation platforms

Yes

Automatic or via plugin

WordPress

WordPress gives you full access to the web root, so every route is open to you. The question is whether you want the file generated or hand-written.

Yoast SEO

Yoast generates and maintains an llms.txt file for your site, building it from your existing content rather than asking you to write one.

The generated file leans on your published pages and posts, which means it inherits whatever your site structure already says about importance. Review the output before you leave it running.

All in One SEO

AIOSEO ships its own llms.txt generator, and it refreshes the file from your sitemap on a schedule rather than as a one-off.

Scheduled regeneration is the right default for a site that publishes often. It is the wrong default if your sitemap contains a lot of thin archive pages, because those will flow straight into the file.

Rank Math

Rank Math also generates the file, and it is the option most people search for by name.

Whichever of the three you use, the advice is identical: turn it on, then read what it produced. A plugin can see your sitemap. It cannot see which five pages actually win you business.

Uploading manually via FTP or cPanel

If you would rather curate the file yourself, upload it to the WordPress root, the same directory that contains wp-config.php, with permissions set to 644.

A real file sitting in the root is served before WordPress routing sees the request, which makes this the most reliable option.

The common failure is a 404 after a successful upload. That is almost always a security plugin, a caching plugin or a CDN intercepting the path rather than a problem with the file.

One thing that does not work: pasting the content into a page builder’s custom code block. That injects the text into an HTML page, not a file at a URL.

Shopify

Shopify now generates these files for you. Open yourstore.com/llms.txt and you will almost certainly find one already there.

According to Shopify’s documentation on agentic storefronts, /agents.md, /llms.txt and /llms-full.txt all return the same content by default, covering your store name and URL, a sitemap link, your store policies and discovery endpoints for other store information. No third-party app is needed.

To override it, go to Online Store > Themes, click Edit code on your active theme, click Add a new template, and select llms.txt. Your template replaces Shopify’s default response at that URL.

Think carefully before you override. Shopify’s default file points agents at commerce endpoints that a hand-written replacement will silently remove.

If you want your store’s organic and agent visibility working together rather than separately, that is the ground our Shopify SEO work covers.

Wix

Wix generates an llms.txt for your site automatically and gives you a management screen for it in the SEO area of your dashboard.

Because Wix builds the file from your site content, the useful work is reviewing what it selected rather than writing anything. [CITE: unverified – exact menu path, confirm against current Wix support documentation before publishing]

Squarespace

Squarespace added native support in July 2026 for version 7.1 sites, reachable through the SEO settings in your dashboard. Site administrator access is required.

Before that, the only route was uploading the file to the asset manager and creating a URL mapping to redirect /llms.txt to the CDN path, or proxying the site through a Cloudflare Worker. If you set up either workaround previously, remove it now rather than leaving two mechanisms fighting over the same URL. [CITE: unverified – exact settings path, confirm against current Squarespace documentation]

Webflow

Webflow lets you upload custom files that are served from the root of your site, which is exactly what this needs.

Upload your llms.txt through your hosting settings and confirm it resolves at the root URL before you consider the job done.

Drupal

Drupal has no native support, but a contributed module handles it. Install it, configure which content types feed the file, and Drupal generates and updates it as part of your normal content workflow.

That makes Drupal one of the better platforms for this, because the file stays synchronised with your content rather than drifting away from it.

Adobe Commerce (Magento)

No native support and no first-party module, so this is a manual upload to your web root.

The catalogue problem from the decision framework applies with full force here. If you publish this file, keep it to your category pages, policies and support content rather than individual products, and accept that you are maintaining it by hand.

HubSpot CMS

HubSpot does not serve arbitrary files from the root. The workable route is to upload your file through the file manager and then create a URL redirect mapping /llms.txt to the file’s hosted location.

Verify the result with curl afterwards. A redirect chain that ends in an HTML wrapper rather than plain Markdown will look fine in a browser and fail for an agent.

Ghost

Ghost has no native support either. Depending on how you host it, you either place the file in the appropriate static directory or configure a route to serve it.

Self-hosted Ghost gives you the web root and makes this straightforward. Managed Ghost hosting does not, and needs the same redirect approach as HubSpot.

Next.js and other static or headless builds

Drop the file into your public/ directory and it is served at the root. That takes about a minute.

The better option takes slightly longer and is worth it. Generate the file at build time from your content source, so every deploy rebuilds it from what your site actually contains.

Hand-written files drift. Generated files cannot. If you are already building your site from a content source, this is the only approach that keeps the file honest without anyone remembering to update it.

Documentation platforms

If your documentation runs on a dedicated platform, this is probably already handled.

GitBook and Mintlify generate llms.txt automatically for the sites they host, and Mintlify also produces llms-full.txt.

Docusaurus and VitePress both have community plugins that generate the file as part of your build.

Check your published site before doing anything. There is a reasonable chance the file you were about to write already exists.

llms.txt best practices

The specification tells you what a valid file looks like. It says less about what makes a good one. These eight habits are the difference.

Lead with your canonical pages

An agent working with a limited context may not read your whole file. It will almost certainly read the top of it.

Put your most important pages in the first section, and put the most important link first within that section. Order is a signal, and it costs you nothing to get right.

If you find yourself unsure what belongs at the top, that is worth noticing. It usually means the site itself lacks a clear hierarchy.

Write a real description for every link

The specification makes the description optional. Treat it as compulsory.

A bare URL tells an agent that a page exists. It does not tell it when that page is the right one to open, which is the only decision the agent is actually making.

One clause is enough. Say what question the page answers, not what the page is called.

Link to Markdown versions wherever they exist

This is the whole point of the version 2 companion proposal, and it is easy to skip.

If you publish .md versions of your pages, link to those rather than to the rendered HTML. The agent gets clean structure instead of a page it has to strip down.

If you do not publish them yet, the links still work. You are just handing over more work than you need to.

Make the current version unmistakable

Versioning is where documentation files quietly go wrong.

If you maintain more than one version of your product docs, either mark the current one explicitly in the description or leave the older versions out entirely.

An agent given three versions with no signal will pick one, and it will not tell your reader which.

Leave out thin, duplicate and archived pages

Inclusion is an assertion. Every page you list is a claim that the page is worth an agent’s attention.

Tag archives, paginated listings, near-duplicate location pages and posts you would be embarrassed to have quoted back at you all dilute the links that matter.

A file with twelve strong links beats a file with eighty mixed ones, every time.

Keep the file small enough to fit in context

The file exists because most sites do not fit in a context window. A file that does not fit either has defeated its own purpose.

There is no published limit, so use judgement. If your file is running past a few hundred lines, you are probably listing pages that belong behind a link rather than in the index.

Depth belongs in the pages. The file is a map.

Revisit after any information-architecture change

Nothing breaks this file faster than a site restructure, and nothing is easier to forget.

Add it to the same checklist that already covers your redirects and your internal links. If you moved a page and updated the redirect, you should be updating this too.

Documentation version releases, navigation rebuilds and section retirements are the moments that matter.

Test it by asking an agent, using only the file

The specification recommends a test that almost nobody runs, and it is the most useful thing in the document.

Give an agent your llms.txt and nothing else. Then ask it the questions your customers actually ask.

If it cannot answer, or it answers from the wrong page, your file is not doing its job. This takes five minutes and tells you more than any validator.

Common llms.txt mistakes to avoid

The practices above describe what a good file does. These are the failures that show up most often in files that already exist.

Treating it as access control

llms.txt grants nothing and blocks nothing. Listing a page does not invite a crawler in, and omitting one does not keep anything out.

If you need to control access, the tools are robots.txt for cooperative crawlers, authentication for anything that matters, and firewall or CDN rules for anything that does not cooperate.

Dumping your whole sitemap into it

This is the most common failure by a distance, and it usually comes from automation nobody reviewed.

A file listing every URL on your site recreates exactly the problem llms.txt was invented to solve. You have converted a large, noisy XML file into a large, noisy Markdown file.

If a plugin generated your file from your sitemap, open it. What you find is often the sitemap with different punctuation.

Letting it go stale

A file pointing at pages that have moved or gone is worse than having no file at all.

With no file, an agent reads your site. With a stale file, it confidently reads the wrong thing, and nothing in the process tells it that it has.

Expecting it to replace robots.txt or your sitemap

It complements both and replaces neither. Your sitemap still does discovery for search engines, and robots.txt still does access control for everything.

Removing either because you now have an llms.txt would be a straightforward downgrade.

Publishing an llms-full.txt nothing can hold

Covered in detail earlier, and worth repeating in one line because it keeps happening: a full-content export that exceeds every available context window helps nobody.

Check the size before you publish, not after.

Assuming a ranking or citation benefit

There is no measured evidence that publishing this file improves rankings, traffic or AI citations. Google has said directly that Search ignores it.

The real cost of assuming otherwise is not the hour you spend on the file. It is the attention you take away from the work that does move things.

Forgetting it is public

Anyone can read it, including the competitor building a scraper who now has your own ranked list of what matters most on your site.

This is not a reason to skip the file. It is a reason to think about what you put in it.

Generators, validators and directories

You do not have to write this by hand, and for some sites you should not. Here is where the tooling helps and where it does not.

When a generator is good enough

For a small site with a clear structure and a handful of pages worth surfacing, a generator’s output is a reasonable starting point.

Generators are good at the mechanical parts. They produce valid Markdown, they get the structure right, they pull real URLs, and they will not typo a link.

They are also the right answer for any site that changes weekly, because a file regenerated automatically cannot drift the way a hand-written one does.

What to check before publishing generated output

What a generator cannot do is judge importance. It sees your sitemap. It does not know which pages earn your revenue.

Run through this before you publish anything a tool produced:

  • Does the H1 name your site, rather than repeating a page title?
  • Is the blockquote an actual summary, or boilerplate?
  • Does every link carry a description that says what the page answers?
  • Have archive, tag, pagination and duplicate pages crept in?
  • If you run multiple documentation versions, is the current one obvious?
  • Is the file small enough to be read in full?
  • Does it validate as Markdown?

That last one matters more than it sounds. A single malformed link can end a list early, and nothing will warn you.

Where to see real files in the wild

The fastest way to calibrate is to read a few good ones.

Three public directories index live files: llmstxt.site, directory.llmstxt.cloud, and llmstxt hub. All three let you browse real implementations rather than templates.

The more interesting examples come from the AI labs themselves. OpenAI, Anthropic and Google all publish llms.txt files for their developer documentation, and OpenAI additionally serves Markdown versions of every documentation page.

That is worth sitting with for a moment. The companies whose models everyone is trying to reach have all implemented the format for their own docs, and none of them has committed to reading yours.

What to do instead if AI visibility is the goal

If the reason you came to this article was that you want AI systems to represent your business accurately, llms.txt is a small lever at best. These are the larger ones.

Make the on-page content itself citable

Look at what actually gets pulled into AI answers on this very topic. The pages winning citations are the ones with a dated first-party experiment and a comparison table, not the ones with the best-formatted index file.

That pattern holds generally. Answer the question directly near the top of the page. Put comparable things in tables. Publish data nobody else has, with the date attached. Attribute your sources in the sentence rather than in a footnote.

None of that is new advice. It is simply the version of it that survives being read by a machine, which is the ground our AI SEO work covers.

Get the technical fundamentals right for AI crawlers

Given that most AI crawlers do not run JavaScript, server-rendered content matters more for them than it does for Google.

Beyond that, the list is unglamorous and effective: fast server responses, deliberate robots.txt directives for AI user agents, clean canonicals, and no important content locked behind interactions a crawler cannot perform.

These are the levers that are actually honoured today, which is more than the file this article is about can claim. They are also covered properly in our guide to SEO.

Measure what AI crawlers actually do on your site

Speculation about AI visibility is cheap and mostly wrong. Your logs are neither.

Track which AI user agents reach your site, which URLs they fetch, how often they come back, and whether they are what they claim to be. That gives you a baseline you can act on, and it makes every future decision about files like this one an empirical question rather than an argument.

Frequently Asked Questions

An llms.txt file is a plain text file written in Markdown, published at the root of a website or at any subpath within it. It gives AI agents a curated list of the pages a site owner considers most important, each with a short description, along with links to cleaner Markdown versions of that content.

Yes and no. It is a real, published proposal, actively maintained and now at version two, with thousands of sites using it. But it is not a ratified web standard, no standards body has adopted it, and no major AI vendor has committed to reading it.

It depends on your site. For documentation and developer-tools sites, yes, because coding assistants genuinely use them and most documentation platforms generate the file for free. For e-commerce and small local sites, no. For service businesses it is optional and low priority, worth an hour for optionality rather than measurable return.

No. Nothing requires it and nothing penalises its absence. Chrome's Lighthouse audit marks the check Not Applicable rather than failed when the file returns a 404, explicitly because providing it is optional. Google has also confirmed that a missing file neither harms nor helps your visibility in Search.

Not directly. Google's documentation states plainly that Search does not use these files, and that publishing one will neither harm nor help your rankings. No study has found a measurable indirect effect either, including on AI Overview citations. Treat it as a possible future benefit, not a ranking tactic.

At the root of your site, so it resolves at yourdomain.com/llms.txt. Version two of the specification also allows it at any subpath, covering the pages beneath that path, so /docs/llms.txt describes your documentation. Where more than one file applies, agents should use the most specific.

No. llms.txt is an open proposal published by Jeremy Howard in September 2024 and maintained publicly, now at version two. No standards body has ratified it, and there is no compliance process. Anyone can publish a file, and anyone can ignore one.

Probably not. A site of a dozen pages presents no navigation problem for an agent, because everything is already close to the homepage. There is nothing meaningful to curate. Your time will do more for you almost anywhere else on the site.

llms.txt is an index: a curated list of links with descriptions. llms-full.txt is a full export of your content concatenated into one Markdown file. The specification defines only the first. llms-full.txt emerged as a convention from documentation platforms, and grows large enough on big sites to defeat its own purpose.

Where this leaves you

The honest summary is that llms.txt is a well-designed answer to a real problem that the systems it was designed for have not agreed to use.

That does not make it worthless. If you run documentation or developer tools, publish one, because your platform probably generates it anyway and coding assistants genuinely read it. If you run a shop or a local service site, leave it alone and spend the time on content a crawler will actually reach.

What you should not do is treat this as a ranking tactic. Google has told you its Search product ignores the file. Believing otherwise costs you nothing in effort and quite a lot in attention.

If you want to know what AI systems currently make of your site, that is a measurable question rather than a speculative one, and an SEO audit is where the answer starts.