Crawl Budget: What It Is and How to Optimise It

Your best new pages can sit unseen for weeks. You publish them, wait, check Search Console, and still find them missing from the index. On a large site, that gap between what you publish and what Google actually crawls is a crawl budget problem, and it quietly costs you traffic. This guide explains what crawl budget is, how Google decides yours, and the exact steps to stop wasting it, including how to handle the AI crawlers that now compete for your server’s attention.

What is crawl budget?

Crawl budget is the set of URLs Google both can and wants to crawl on your site within a given timeframe. It is the product of two forces: how much crawling your server can handle, and how much Google wants to crawl you. Get either one wrong and important pages wait longer to be found.

One detail trips people up. Google treats a site as a single hostname, so www.example.com and shop.example.com each get their own separate budget. Your subdomains do not share one pool.

How Google decides your crawl budget

Two mechanisms set your budget. One caps how fast Google may crawl; the other decides how much it bothers to.

Crawl capacity limit (hostload)

The crawl capacity limit is the ceiling on how hard Google will crawl you without overloading your server. Google’s own documentation on managing crawl budget describes it as a limit on the number of parallel connections Googlebot opens and the delay between fetches.

Every site starts on the same conservative default. If your server stays fast and healthy, Google raises the limit over time and crawls more. If response times climb, or the server returns 5xx errors or 429 “too many requests” signals, the limit drops and Google backs off.

So a fast, stable server is not just good for users. It directly buys you more crawling. Note the older term “crawl rate limit” means the same thing; Google’s current documentation calls it the crawl capacity limit.

Crawl demand

Capacity sets the ceiling. Demand decides how close to it Google actually crawls. Even with plenty of capacity, low demand means Google visits less often. Three factors drive it, and you can influence all three.

Perceived inventory. Left to its own devices, Google tries to crawl every URL it knows about. If many of those are duplicates or low-value, it wastes time it could spend on pages that matter. This is the factor you control most.

Popularity. URLs that attract more links and traffic get crawled more often, so Google keeps them fresh.

Staleness. Google tries to recrawl often enough to catch changes, so pages that update regularly earn more frequent visits.

One more trigger: a site move or large-scale URL change can spike demand while Google reprocesses everything under the new addresses.

Does crawl budget actually matter for your site?

Here is the honest answer most guides bury: for the majority of sites, crawl budget is not worth worrying about. Google is good at crawling small and medium sites promptly. Before you spend time optimising, work out whether you are even in the group that needs to.

When you can ignore it

If your site runs to a few thousand URLs or fewer, and new pages tend to get crawled the same day you publish them, you do not need this guide. Google’s documentation says as much.

For most sites that size, keeping your sitemap current and checking the Page Indexing report in Search Console now and then is enough. Spend your effort on content and links instead.

When it genuinely matters

Crawl budget starts to bite in three situations Google names directly:

• Large siteswith 1 million or more unique pages where content changes moderately often, say weekly.
• Medium or larger siteswith 10,000 or more pages where content changes very rapidly, on a daily basis.
• Any sitewith a large share of its URLs sitting in the “Discovered – currently not indexed” state in Search Console, which is often a sign Google knows about pages but has not prioritised crawling them.

If none of these describe you, the sections below are useful background, not an urgent task.

What this means for Singapore businesses

Most Singapore SME websites, a corporate site, a service business, a modest online store, never come close to these thresholds. If that is you, we would rather tell you plainly than sell you an optimisation you do not need.

The local sites that do hit the limit tend to be large e-commerce stores and marketplaces, classifieds and property or job listings, and multi-language sites serving Singapore alongside the wider region in English, Chinese and Malay. If you run one of those, crawl budget is worth a proper look, and our SEO services can help you audit it.

How to check your crawl budget

You cannot manage what you cannot see. There are two practical ways to check how Google crawls your site, plus a simple sum to gauge whether you have a problem.

Google Search Console Crawl Stats

The Crawl Stats report is the fastest starting point. In Search Console, open Settings, then Crawl stats.

You get the total number of crawl requests over time, your average server response time, and an overall host status that flags whether Google hit availability problems while crawling.

Below that, Google breaks the requests down four ways: by response (200s, 404s, 5xx and so on), by file type (HTML, images, CSS, JavaScript), by purpose (discovering new URLs versus refreshing known ones), and by Googlebot type. A high share of requests spent on errors or the wrong file types is a clear sign of waste.

Server log file analysis

Crawl Stats is a summary. Your server logs are the ground truth. They record every request Googlebot makes, so you can see exactly which URLs it hits, how often, and what status codes it gets back.

Log analysis shows you patterns Crawl Stats hides, like a crawler stuck cycling through thousands of filtered URLs. Tools such as Screaming Frog’s Log File Analyser or Semrush’s Log File Analyzer can parse the raw files for you, though on smaller sites you can review them by hand.

A worked example: estimate your crawl budget from Crawl Stats

You can get a rough figure from the numbers above. Take the average crawled pages per day from Crawl Stats and multiply by 30 for a monthly estimate. If Google crawls about 100 pages a day, that is roughly 3,000 a month.

Then divide your total indexable URLs by that daily average. Say you have 5,000 pages and Google crawls 100 a day: that is a 50-day cycle to get through the site once.

As a rough decision rule, a ratio well above 10 means your pages are outrunning Google’s crawling and optimisation is worth it. A low ratio means Google is keeping up comfortably and you can leave it alone.

What wastes your crawl budget

Optimising crawl budget is mostly about stopping waste. Every request Google spends on a duplicate, a broken link or a dead end is a request it did not spend on a page you care about. Here are the usual culprits.

Duplicate and near-duplicate URLs

When the same content sits under several URLs, Google crawls each version separately. Print variants, tracking parameters and session-based URLs all multiply the work without adding anything Google needs to index.

Faceted navigation and URL parameter bloat

Filter and sort options on large sites are the biggest offender. Each combination of colour, size, price and order can generate its own URL, so a few hundred products can spawn tens of thousands of crawlable addresses. This is a common problem on e-commerce sites with faceted navigation, where crawlers get trapped cycling through near-identical pages.

Soft 404s and broken links (4xx)

A soft 404 is a page that returns a “200 OK” status while showing an empty or error-like page. Google keeps crawling these because the status code says they are fine, so they quietly drain budget. Genuine broken links (4xx) waste it too. Check the Page Indexing report for soft 404s flagged by Google.

Long redirect chains

A redirect is cheap. A chain of them is not. When one URL points to another, then another, Google follows each hop, and long chains slow crawling and can cause it to give up before reaching the final page.

Low-quality and thin content

Pages with little useful content still cost a crawl. If a large part of your site is thin, auto-generated or near-empty, Google spends budget confirming those pages are not worth much instead of finding the ones that are.

Slow pages and server errors (5xx)

Slow responses and server errors feed straight back into your crawl capacity limit. When your server struggles, Google reads fewer pages per visit and, if 5xx errors persist, crawls less overall until things stabilise.

Non-indexable pages Google still crawls

Google often still crawls pages it cannot index, like those blocked by noindex or stuck behind parameters, before it works out they will not be indexed. A large volume of these means real crawl effort spent on pages that will never appear in search.

How to optimise your crawl budget

Once you know what wastes budget, the fixes follow. Work through the ones that apply to your site, starting with whichever is bleeding the most crawl effort.

Consolidate duplicate content

Point duplicate and near-duplicate URLs at a single canonical version so Google concentrates crawling on unique content rather than unique URLs. Fewer versions to crawl means more budget for pages that matter.

Block low-value URLs with robots.txt

Use robots.txt to block crawling of URLs you never want Google to spend time on, like internal search results or infinite filter combinations. Blocking a URL in robots.txt stops Google crawling it, which frees that budget for the rest of your site.

Use 404 or 410 for permanently removed pages

When you remove a page for good, return a 404 or 410 status. Google’s guidance notes that a 410 (“gone”) is a stronger signal not to crawl the URL again than a 404, so it drops out of the crawl queue faster. Pages you merely block in robots.txt stay in the queue much longer.

Support HTTP caching with 304 (Not Modified)

Configure your server to return a 304 (Not Modified) response when a page has not changed since Google last fetched it. Google can then reuse its cached copy instead of downloading the page again, which saves your bandwidth and lets it spend the saved budget elsewhere. This is one of the most overlooked fixes.

Keep your XML sitemap clean and current

List only your indexable, canonical URLs in your XML sitemap, and keep it accurate. Including an honest lastmod date helps Google decide which pages to recrawl, so keep it truthful rather than resetting it on every page daily.

Fix redirect chains

Collapse redirect chains so each old URL points directly to its final destination in one hop. It is worth auditing for these periodically, as they build up quietly as a site changes over the years.

Strengthen internal linking

Googlebot finds and prioritises pages partly through your internal links. Pages with few internal links get less attention, so make sure important pages are linked from several relevant places rather than buried many clicks deep.

Improve loading efficiency and server response

Faster server responses let Google read more from your site in the same window. This goes wider than trimming JavaScript: your server response time, caching and overall page speed all feed into how much Google can crawl before it stops.

Prune or consolidate thin content

Audit low-value pages and decide, honestly, which to improve, merge or remove. Cutting thin content reduces the number of low-worth URLs Google has to wade through to reach your good pages.

Use canonical tags and meta robots deliberately

Canonical tags, noindex and robots.txt each do a different job, and mixing them up wastes budget. As a rough guide: use a canonical to consolidate duplicates you still want crawled, noindex to keep a crawlable page out of the index, and robots.txt to stop crawling low-value URLs altogether.

Managing AI and LLM crawlers (2026)

Googlebot is no longer the only heavy crawler hitting your site. A wave of AI crawlers now fetch pages to train models and to answer questions, and on a large site they add real load alongside Google’s.

One distinction matters before you touch anything. Most AI companies run two kinds of bot: a training crawler that collects content to improve their models, and a search crawler that fetches pages at query time to cite in answers. Blocking the training bot keeps you out of the model. Blocking the search bot keeps you out of the citations. You decide each one separately, in robots.txt.

GPTBot (OpenAI)

GPTBot is OpenAI’s training crawler. Per OpenAI’s crawler documentation, it collects content that may be used to train its models, and you can allow or disallow it independently of OpenAI’s other bots. For citations in ChatGPT’s search, the relevant bots are OAI-SearchBot and ChatGPT-User, which are separate tokens.

ClaudeBot (Anthropic)

ClaudeBot is Anthropic’s training crawler. Anthropic’s crawler documentation lists it alongside Claude-User, which fetches pages when someone asks Claude a question, and Claude-SearchBot, which indexes content to improve Claude’s search answers. Anthropic states all three respect robots.txt.

PerplexityBot (Perplexity)

PerplexityBot indexes content for Perplexity’s answer engine, and Perplexity-User handles live user-triggered fetches. Worth knowing: Cloudflare has documented Perplexity using undeclared crawlers that rotate identities to get around no-crawl rules, so robots.txt alone may not reliably keep it out.

Google-Extended (Gemini / AI training token)

Google-Extended is the odd one out. It is not a crawler at all. Google’s Search Central documentation describes it as a standalone token that controls whether content Google already crawled may be used to train Gemini and for grounding its AI answers.

Because Googlebot does the actual fetching, disallowing Google-Extended does not reduce your crawl load and does not affect your Search ranking. It is a content-use control, not a crawl-budget lever.

CCBot (Common Crawl) and Bytespider (ByteDance)

CCBot belongs to Common Crawl, a non-profit that builds an open web dataset many AI models train on. Bytespider is ByteDance’s crawler, and it has a documented history of ignoring robots.txt, so blocking it politely may not be enough on its own.

Should you block AI crawlers? The AEO/GEO trade-off

There is a genuine tension here, and no single right answer. Blocking AI crawlers saves crawl load and keeps your content out of models you would rather not feed. But it also cuts your visibility in AI answers, which is exactly what answer engine optimisation and generative engine optimisation aim to build.

A common middle path is to allow the search and citation bots, so you can still be quoted, while making a separate decision on the training bots. You can also allow AI crawlers on your public content and block them on sections you would rather keep out. If earning AI citations matters to your strategy, our AI SEO work weighs these calls case by case rather than blocking everything by reflex.

Crawl budget vs related terms

A few terms get used interchangeably when they should not. Here is how they differ.

The key thing to hold on to: being crawled is not the same as being indexed. A page can use its share of your crawl budget and still not make it into the index if Google judges it low value.

Is crawling a ranking factor?

No. Google has been clear that crawling more of your site, or crawling it faster, does not by itself lift your rankings. A bigger crawl budget will not push you up the results.

But crawling is the step before indexing, and indexing is the step before ranking. If Google never crawls a page, it cannot index it, and an unindexed page cannot rank at all. So crawl budget is not a ranking factor, yet it is the gate everything else has to pass through.