Search engines crawl your site whether you plan for it or not. Left unmanaged, they waste time on admin pages, filter URLs and duplicates, and a new wave of AI crawlers now pulls your content into training sets and chat answers. A single misplaced line in one small file can hide your best pages from Google or invite every bot to help itself. That file is robots.txt. Get it right and you steer crawlers toward what matters. This guide covers robots.txt optimisation from the ground up: syntax, examples, testing, CMS setup and blocking AI bots.
Key takeaways
- robots.txt is a plain text file at your site root that tells crawlers which URLs they may request. It manages crawl traffic, not indexing.
- It cannot reliably keep a page out of Google. A disallowed URL can still be indexed if other pages link to it. Use noindex for that.
- It is not a security tool. The file is public, and only well-behaved bots obey it.
- Google reads only the first 500 KiB and ignores the crawl-delay Bing honours crawl-delay.
- You can allow or block AI crawlers like GPTBot, ClaudeBot and Google-Extended per user-agent, each on its own line.
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which URLs they may or may not request. It follows the Robots Exclusion Protocol, standardised as RFC 9309. Its job is to manage crawler traffic, not to keep pages out of search results.
So when someone asks what is robots txt, the short answer is a set of instructions for bots. You write rules, save them in a file named robots.txt, and place it where crawlers know to look. The file format is deliberately simple, which is also why small mistakes are easy to make.
How does robots.txt work?
Before a compliant crawler fetches anything on your domain, it first requests your robots.txt file and reads the rules that apply to it. If a rule disallows a path, the crawler skips it. If nothing disallows it, the crawler treats the URL as fair game.
The file only applies to the exact host, protocol and port it sits on. Your https://example.com/robots.txt governs the https version of that host, and a subdomain like shop.example.com needs its own file. This one-file-per-host rule catches many people out during migrations.
If you want a short explainer to share with your team, an embedded video near this section works well. A crawler flow diagram (request robots.txt, read rules, then crawl or skip) also makes the sequence clear for non-technical readers.
Why robots.txt matters for SEO
A well-managed file does not push you up the rankings on its own. What it does is spend your crawl budget wisely and keep low-value URLs out of the way, which helps the pages you care about get found and refreshed. There are three practical reasons it matters.
Managing crawl budget
Every site gets a rough share of crawler attention. If bots burn that share on faceted navigation, internal search results and endless URL parameters, your important pages get crawled less often. Disallowing those low-value paths is core robots.txt optimisation: you point crawlers away from the noise so they spend more time on revenue pages.
Preventing duplicate and non-public pages in search results
Staging areas, cart and checkout steps, thank-you pages and admin paths add nothing to search results. Disallowing them reduces duplicate and thin URLs competing for attention. Just remember the limitation below: disallow controls crawling, not indexing, so pair it with noindex when a URL must never appear.
Pointing crawlers to your sitemap
You can declare your XML sitemap in robots.txt with a Sitemap: line. This gives crawlers a direct map of the URLs you want discovered. It is a small robots sitemap tweak that helps new and updated pages surface faster, and it works alongside submitting the sitemap in Search Console.
The limitations of robots.txt (what it can’t do)
This is where most robots.txt damage happens, because the file is widely misunderstood. Three limits matter more than any clever rule you might write.
It doesn’t keep pages out of Google (use noindex instead)
Disallowing a URL stops Google crawling it, but Google can still index that URL if other pages link to it, showing it in results without a snippet. Per Google’s robots.txt specification, the file is not a mechanism for keeping a page out of search. To remove a page, let it stay crawlable and add a noindex meta tag, or protect it with a password.
It sounds backwards, but blocking a page in robots.txt can keep the ugly URL in results forever, because Google never crawls it to see the noindex you added.
It’s not a security tool
Anyone can read your robots.txt by visiting yourdomain.com/robots.txt. If you disallow /secret-admin/, you have just published the path to your admin area. Never use the file to hide sensitive directories. Use real access controls, authentication and noindex where appropriate.
It relies on voluntary compliance
robots.txt is a request, not a wall. Reputable crawlers such as Googlebot and Bingbot obey it. Malicious scrapers and some aggressive bots ignore it entirely. If a bot must be stopped rather than politely asked, you need server-level or firewall rules, not a robots.txt line.
Is robots.txt legal?
robots.txt is not legally binding on its own. It is a voluntary, de facto standard, and there is no enforcement built into the protocol. The academic review The Liabilities of Robots.txt describes it as a request rather than an enforceable contract.
That said, courts have treated robots.txt as a signal of good or bad faith in scraping disputes. In the hiQ Labs v. LinkedIn case, the Ninth Circuit held in 2022 that scraping public pages does not by itself breach the US Computer Fraud and Abuse Act, though LinkedIn later prevailed on a breach-of-contract argument. The picture also varies by country: in Germany, for instance, robots.txt is treated as legally enforceable, with an exception for scientific research.
The practical takeaway: honouring robots.txt costs little and strengthens your position if a dispute arises. This is general information, not legal advice, so check with a qualified professional for your own situation.
Robots.txt syntax and directives
A robots.txt file is a list of rules grouped by crawler. Each rule is a field, a colon and a value. The directives below are the ones you will actually use. Here is a quick reference before we cover each in turn.
Directive | What it does | Example |
User-agent | Names the crawler a group of rules applies to | User-agent: Googlebot |
Disallow | Blocks a path from being crawled | Disallow: /cart/ |
Allow | Carves an exception inside a disallowed path | Allow: /cart/help |
Sitemap | Declares your sitemap’s absolute URL | Sitemap: https://site.com/sitemap.xml |
Crawl-delay | Asks some bots to slow down (not Google) | Crawl-delay: 10 |
User-agent
The User-agent line names the crawler a group of rules applies to. A robots user agent value of * means every crawler. You can also target one bot by name and group several rules under it.
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /private/public-note.html
Disallow
The Disallow rule tells a crawler not to request a path. So what does disallow tell a robot? It says “do not crawl anything that starts with this path.” A lone Disallow: / blocks the entire site, which is the robots txt disallow all pattern you almost never want in production.
User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Allow
The Allow rule carves an exception inside a disallowed directory. It is how you block a folder but keep one file crawlable. To welcome every bot, the robots txt allow all search engines approach is simply an empty disallow.
User-agent: *
Disallow:
# The blank Disallow above allows everything
Sitemap
The Sitemap line points crawlers to your XML sitemap. It must be a full absolute URL, including https, and you can list more than one. It is not tied to any single user-agent.
Sitemap: https://www.example.com/sitemap.xml
Crawl-delay
The Crawl-delay directive asks a crawler to wait a set number of seconds between requests. Support is uneven: Bing honours crawl-delay with whole-number values from 1 to 30, while Google ignores it entirely. To slow Googlebot, use the crawl-rate signals in Search Console instead of a crawl delay robots line.
User-agent: bingbot
Crawl-delay: 10
Wildcards and pattern matching (* and $)
Google, Bing and other major engines support two wildcards in path values. The * character matches any run of characters, and $ anchors the end of a URL. These let you match patterns rather than listing every URL.
User-agent: *
Disallow: /*.pdf$ # block URLs ending in .pdf
Disallow: /*?sort= # block sorted filter URLs
Nonstandard directives (Host and Clean-param)
A few directives sit outside the core standard. Yandex recognises Host (preferred mirror) and Clean-param (ignore listed URL parameters). Google supports neither. Older guides also mention a Noindex directive inside robots.txt: that was never officially supported and no longer works, so use a real noindex tag instead.
Robots.txt examples (copy-paste templates)
Here are common robots txt file format templates you can adapt. Each is a robots file example you can paste in, then edit the paths to match your site. Test any change before you deploy it.
Allow all crawlers
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Block all crawlers
Use this only on staging or a site that must stay out of search entirely. On a live site it removes you from Google.
User-agent: *
Disallow: /
Block a specific directory
User-agent: *
Disallow: /admin/
Disallow: /tmp/
Block a specific file
User-agent: *
Disallow: /private/report.pdf
A real-world robots.txt file, explained
A typical file for a live WordPress site might look like this. The comments explain each choice.
# Block the admin area, but let Google fetch the
# admin-ajax endpoint that some plugins rely on
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Keep internal search results out of crawl
Disallow: /?s=
# Point crawlers to the sitemap
Sitemap: https://www.example.com/sitemap_index.xml
How to create a robots.txt file
Creating the file takes five steps. If you would rather hand off uploading and maintaining robots.txt to someone else, that is a sensible option too, but the process is straightforward.
Step 1: create a file named robots.txt
Open a plain text editor and create a file called robots.txt, all lowercase. Save it as UTF-8 encoded plain text. Word processors can add hidden formatting, so a code or text editor is safer.
Step 2: write your rules
Add your user-agent groups, disallow and allow rules, and a sitemap line. Keep one directive per line and start simple. You can always add rules later.
Step 3: upload it to your root directory
Upload the file to the top level of your domain so it resolves at yourdomain.com/robots.txt. Crawlers do not check subfolders, so a file in a subdirectory is ignored.
Step 4: test it in Google Search Console
Open the robots.txt report in Google Search Console to see the fetched file, when it was last read, and any warnings or errors. Testing here before you rely on the rules catches a broken Disallow before it costs you traffic. Confirm your key pages are still allowed.
Step 5: submit and monitor
Google re-fetches robots.txt periodically and caches it for up to a day. If you make an urgent change, you can request a refresh in the same report, then keep an eye on crawl stats over the following weeks.
How do I find my robots.txt file?
To find your own file, type your domain followed by /robots.txt into a browser, for example https://www.example.com/robots.txt. If a file exists, it loads as plain text. If you get a 404, you do not have one yet, and crawlers will assume everything is allowed.
The same trick works for how to read robots txt on any public site. Visiting a competitor’s file can show you which sections they keep bots away from, though remember every file is public by design.
How to edit robots.txt on common CMS platforms
Most content management systems generate a robots.txt file for you and expose a way to edit it. The path differs by platform, and one popular platform does not let you edit it at all. Here is where to look on each.
WordPress
WordPress serves a virtual robots.txt by default, which you cannot edit as a file. An SEO plugin gives you a proper editor. The two most common are Rank Math and Yoast.
Editing robots.txt in Rank Math
In Rank Math, go to Rank Math SEO → General Settings → Edit robots.txt. You can write your rules straight into the editor. If the field is locked, a physical robots.txt file already exists on the server and needs removing first, as Rank Math’s own guide explains.
Editing robots.txt in Yoast
In Yoast SEO, go to Yoast SEO → Tools → File editor. The file editor lets you create and edit robots.txt from the dashboard. It only appears if your server allows file writing; if it is missing, your host has disabled direct editing.
Shopify
Since 2021, Shopify lets you edit Shopify robots.txt through a theme template. Go to Online Store → Themes → … → Edit code, then under Templates add a new robots.txt.liquid template. You edit the Liquid that generates the file rather than raw text, as Shopify’s help centre documents. Keep Shopify’s default rules unless you have a clear reason to change them. For a deeper store setup, our Shopify SEO service can help.
Wix
Wix generates an editable file. In your site dashboard, open the SEO settings, find the Robots.txt Editor under tools and settings, and write your directives. Wix’s support article walks through it and warns that edits can affect how your pages appear in search.
Squarespace
Squarespace generates robots.txt automatically and, as of writing, does not let you edit or override it. If you need to keep a single page out of search, use the per-page “hide from search results” toggle, which adds a noindex tag. For per-bot control you would need Squarespace’s developer mode and custom headers.
Webflow
Webflow now edits robots.txt natively. Go to Site settings → SEO → Indexing and enter your rules in the robots.txt field, then publish. Webflow’s help centre covers the current options, including the default sitemap link.
Magento
In Magento 2 there is no file to edit over FTP. Go to Content → Design → Configuration, edit the store view you want, and open the Search Engine Robots section. Save, then flush the Magento cache so the new output serves. Each store view can have its own rules, which suits multi-region stores.
BigCommerce
BigCommerce lets you edit the file in the admin. Go to Store Setup → Store Settings, then scroll to the Search Engine Robots section and edit the text. Save when you are done. As with every platform, test before you trust it, because a wrong rule here can deindex product pages.
How to block AI crawlers in robots.txt
AI companies crawl the web with their own named bots, and robots.txt is the main way to allow or block them. Controlling AI crawlers has become a real decision for every site owner, so it deserves its own section.
Why AI crawler blocking is three decisions, not one
Each AI crawler visits your site for a different reason. Some are gathering training data for the next version of a model. Some are building the index that lets an AI assistant cite your page in an answer. Others fetch a single page in real time because someone asked a chatbot a direct question about it.
Treating all of these the same way is the most common mistake we see. Blocking an AI crawler isn’t one decision, it’s three, and they have opposite effects:
- Block a training crawler and your content stays out of future model training. This has no effect on whether you show up in AI answers today.
- Block a search or retrieval crawler and you remove yourself from that engine’s live AI answers going forward.
- Block a user-triggered fetcher and you break the experience for a real person who asked an assistant about your specific page.
A lot of “block all AI bots” advice floating around collapses these into one rule, which usually means blocking the second or third category by accident when the intention was only ever the first.
The same logic applies within Google’s own crawlers: blocking the AI opt-out token Google-Extended keeps your content out of Gemini training without touching your normal Google Search ranking, because Googlebot is a completely separate user-agent. Never block Googlebot to stop AI training; that removes you from Google search results altogether.
The AI crawlers worth knowing about
Training crawlers scrape your site in bulk to build datasets for model training. Blocking these carries no citation cost: you simply opt out of future training data.
Crawler | Company | robots.txt token | Compliance |
|---|---|---|---|
GPTBot | OpenAI | GPTBot | Respects robots.txt |
ClaudeBot | Anthropic | ClaudeBot | Respects robots.txt |
Google-Extended | Google-Extended | Opt-out token only; doesn’t touch regular Google Search | |
Applebot-Extended | Apple | Applebot-Extended | Opt-out token only; doesn’t touch Siri/Spotlight search |
CCBot | Common Crawl | CCBot | Generally compliant; feeds many other companies’ models downstream |
Amazonbot | Amazon | Amazonbot | Respects robots.txt |
Meta-ExternalAgent | Meta | Meta-ExternalAgent | Mixed compliance reports |
Bytespider | ByteDance | Bytespider | Documented non-compliance; frequently ignores Disallow rules regardless |
Search and retrieval crawlers build the index an AI engine actually cites from. Block one of these and you’re opting out of that engine’s answers entirely.
Crawler | Company | robots.txt token | Notes |
|---|---|---|---|
OAI-SearchBot | OpenAI | OAI-SearchBot | Powers ChatGPT Search citations; a separate crawler from GPTBot |
Claude-SearchBot | Anthropic | Claude-SearchBot | Separate from ClaudeBot; blocking ClaudeBot alone won’t block this |
PerplexityBot | Perplexity | PerplexityBot | Its declared crawler generally respects robots.txt |
DuckAssistBot | DuckDuckGo | DuckAssistBot | Powers DuckAssist answer summaries |
YouBot | You.com | YouBot | AI search and chat indexing |
User-triggered fetchers grab a single page in real time when someone asks an assistant about it directly. Blocking these breaks a genuine user request, not a bulk scrape.
Crawler | Company | robots.txt token | Notes |
|---|---|---|---|
ChatGPT-User | OpenAI | ChatGPT-User | Fires when a ChatGPT user asks about a specific page |
Claude-User | Anthropic | Claude-User | Same idea for Claude |
MistralAI-User | Mistral | MistralAI-User | Same idea for Mistral’s Le Chat |
Perplexity-User | Perplexity | Perplexity-User | Contested: Perplexity maintains this is “an agent, not a bot” and isn’t obligated to honour robots.txt, which has led to public disputes with publishers |
A couple of older tokens still show up in copy-pasted examples: anthropic-ai and Claude-Web were both deprecated by Anthropic in favour of the crawlers listed above. Leaving old rules for them in place does no harm, but they’re no longer active.
What robots.txt can’t actually do for AI crawlers
Robots.txt is a request, not a lock. A few important limits are worth knowing:
- Bytespider and some of Perplexity’s crawler traffic have been documented ignoring Disallow rules outright. If you genuinely need to keep a bot out, that requires a server-level or CDN/firewall block, not just a robots.txt entry.
- Agentic browsers like ChatGPT Atlas or Perplexity Comet send standard browser signatures, because at the protocol level they’re indistinguishable from a person browsing: that’s functionally what’s happening. User-agent rules can’t isolate this traffic without also blocking real visitors.
- A user-agent string is self-declared, not verified. Anyone can send a request claiming to be GPTBot. If you need to confirm a crawler’s real identity rather than just filter by name, that takes a reverse-DNS check against the vendor’s published IP ranges.
A starting robots.txt for AI crawlers
Here’s a defensible baseline for a business that wants AI-search visibility without necessarily opting every model into training on its content:
# Traditional search: always allow
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# AI search and retrieval: allow (needed for citations)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
# AI training opt-out tokens: no effect on search or citations
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# AI training crawlers: decide based on your own content policy
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Treat this as a starting point rather than a rule to copy blindly: whether to allow GPTBot, ClaudeBot and CCBot training access is a content-rights decision, and it’s one every business should make deliberately rather than inherit from a template.
Keeping it current
This list changes more often than the rest of your robots.txt. A few habits keep it from going stale:
- Monthly: check your server logs for GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot activity to confirm the crawlers you’ve allowed are actually showing up.
- Quarterly: re-read your robots.txt end to end against an updated list of AI crawlers; new ones tend to appear a few times a year.
- After any SEO plugin update: several popular WordPress and Shopify SEO plugins have shipped “block AI bots” settings switched on by default. Check that an update hasn’t silently re-blocked a crawler you meant to allow.
- Check your CDN or firewall settings separately: Cloudflare’s one-click “Block AI Bots” option and similar CDN-level settings can override an otherwise-correct robots.txt file without you noticing.
Does robots.txt still work in 2026?
Yes. For compliant crawlers, robots.txt works exactly as it always has, and it now does double duty as the main control for AI crawler access. If anything, the file matters more today than it did five years ago, because it governs both classic search bots and the AI bots shaping how your brand appears in chat answers.
What it still cannot do is stop bad actors. A robots txt no crawl rule is a polite request, and dishonest scrapers ignore it. For those, treat robots.txt as step one and add server-side enforcement for anything that truly must be protected.
Robots.txt best practices
A few habits keep your file safe and effective. None are complicated, and each prevents a common failure.
Keep the file under 500 KB
Google reads only the first 500 KiB of your robots.txt and ignores everything after that. Very few sites approach this, but if yours does, consolidate rules and move blocked material into shared directories rather than listing every URL.
Use one directive per line
Each rule belongs on its own line. Combining directives or adding trailing content confuses parsers and leads to rules being skipped.
Don’t block CSS or JavaScript
Google renders pages to understand them. If you disallow the CSS or JavaScript it needs, pages can render badly in Google’s eyes and lose ranking. Leave rendering assets crawlable.
Mind case sensitivity
Paths in robots.txt are case-sensitive, so /Folder/ and /folder/ are different. The filename itself must be lowercase robots.txt, or crawlers will not find it.
Use an absolute URL for your sitemap
The Sitemap line needs a full URL with https and host, not a relative path. A relative sitemap line is ignored.
Always test before deploying
Validate every change in Google Search Console before it goes live. One test run is far cheaper than discovering a stray Disallow: / after your traffic drops.
Common robots.txt mistakes to avoid
These are the errors we see most often, and each has a simple fix. Running robots.txt checks in an audit catches them before they hurt.
Using robots.txt to hide sensitive pages
The file is public, so disallowing a path advertises it. Protect sensitive pages with authentication and noindex, not a Disallow line.
Blocking pages you want de-indexed (instead of using noindex)
Disallowing a page stops Google seeing the noindex tag on it, so the URL can linger in results. To remove a page, keep it crawlable and let the noindex do its job.
Accidentally blocking the whole site (Disallow: /)
A stray Disallow: / under User-agent: * blocks your entire site. This often survives a staging launch and quietly deindexes the live site. Check for it first whenever traffic drops.
Wrong file location or filename
The file must sit at the domain root and be named robots.txt in lowercase. A file in a subfolder or named Robots.TXT is invisible to crawlers.
Forgetting to update after a site migration
Migrations often carry over a staging robots.txt that blocks everything, or leave old disallowed paths that no longer exist. Review the file as a launch-day checklist item every time you move or rebuild a site.
Robots.txt vs meta robots vs X-Robots-Tag
These three controls are easy to confuse, because they overlap. The difference comes down to crawling versus indexing, and where the rule lives. Use this table to pick the right one.
Control | Where it lives | Crawl or index | Use it when |
|---|---|---|---|
robots.txt | One file at site root | Controls crawling | You want to stop bots requesting URLs or save crawl budget |
Meta robots | HTML <head> of each page | Controls indexing | You want a specific page kept out of results (noindex) |
X-Robots-Tag | HTTP response header | Controls indexing | You need noindex on non-HTML files like PDFs, or at scale |
Free robots.txt generator
If you would rather not write rules by hand, a robots.txt generator lets you pick directives and export a ready file. It is a useful starting point, especially for a first file. Whatever a generator produces, still test it in Search Console before you deploy, since a generator does not know which of your paths matter.
Frequently asked questions
Short answers to the questions people ask most about robots.txt.
A robots.txt file tells search engine crawlers which URLs they may or may not request on your site. It manages crawl traffic and helps you steer bots toward important pages, but it does not control whether a page is indexed.
Yes. Compliant crawlers, including Googlebot, Bingbot and the major AI bots, read and obey it. It does not stop malicious scrapers, which ignore it, so pair it with server-side controls for anything that must be protected.
robots.txt is not legally binding by itself; it is a voluntary standard. Courts may still treat it as evidence of good or bad faith in scraping disputes, and enforceability varies by country. This is general information, not legal advice.
Add /robots.txt to your domain in a browser, for example https://www.example.com/robots.txt. If it loads as plain text, you have one. A 404 means you do not, and crawlers will assume everything is allowed.
Not strictly. Without one, crawlers assume they can access everything. Most sites benefit from having a file to manage crawl budget, point to a sitemap and control AI crawler access, even if the rules are minimal.
Name the crawler on a User-agent line and add a Disallow rule beneath it. For example, User-agent: Bytespider followed by Disallow: / blocks that bot from the whole site, provided the bot honours robots.txt.
For a crawler, ignoring robots.txt means crawling pages the owner asked it to skip, which can look like bad faith and, in some cases, raise legal risk. For a site owner, ignoring the file means leaving crawl behaviour and AI access entirely to chance.
Final thoughts
robots.txt is a small file with an outsized effect. Used well, it spends your crawl budget wisely, keeps clutter out of search, and gives you real control over which AI bots see your content. Used carelessly, a single line can hide your best work. If you take one thing from this guide, make it this: test every change before you trust it.
If you would like a second pair of eyes on your setup, our SEO services and AI SEO services cover both classic crawling and the newer AI crawler decisions. Found this useful? Share it with whoever manages your site, and tell us which robots.txt mistake caught you out.
