Your best pages aren’t getting indexed, and you can’t work out why. Crawl tools give you a guess, and Search Console gives you a sample. Neither tells you what search engines actually did on your site. Your server logs do. They record every request, from every bot, in order, and reading them is how you find the crawl budget waste, orphan pages and bot problems that are quietly costing you rankings.
What is log file analysis?
Log file analysis is the process of reviewing your server’s raw log files to see how bots and people request pages on your site, so you can find and fix crawl, indexing, performance and security problems. Every request leaves a record, and reading those records back shows you exactly what happened.
The same practice serves two audiences. IT and DevOps teams read logs to monitor systems, catch errors and investigate security incidents. SEOs read the same files to see how search and AI bots crawl a site. This guide covers both, but leans into the SEO side, because that’s where server logs answer questions no other tool can.
A log file is just a text file. Your web server writes a new line every time something requests a page, an image or a script. Over a day, a busy site produces thousands of these lines. On their own they look like noise. Analysed properly, they’re the most honest record you have of how search engines treat your site.
What log file analysis means for SEO
For SEO, log files are the only record of what search and AI bots really did, rather than what a tool estimates they might do. Crawl simulators like Screaming Frog copy how a crawler would move through your site. They don’t show the historical, live behaviour of the actual bots.
Google Search Console gets you closer, but its Crawl Stats report aggregates the data, limits it to Google’s own crawlers and only holds a short window. Your logs capture every crawler, every request, with no sampling. If you want proof rather than a proxy, that’s where you look.
Why log file analysis matters
Reading your logs surfaces problems that other tools miss and confirms whether the fixes you’ve made actually changed bot behaviour. Here’s what it gives you.
Validate what search engines actually crawl
Logs show you which pages bots visited, how often and what the server returned each time. A crawl tool tells you what a bot could reach. Your logs tell you what it actually reached, which is the only version that affects your rankings.
The value grows when you track behaviour over time. A sudden spike or drop in crawling often points to a deeper technical problem worth investigating before it shows up in your traffic.
Optimise crawl budget and stop wasted crawling
Crawl budget is the number of pages a search engine will crawl on your site in a given window. Bots won’t crawl everything, so how well you spend that budget decides which pages get seen, indexed and ranked.
Logs show where bots waste time, on old paginated URLs, faceted filters or dead parameters, so you can push that attention back to pages that matter. Crawl budget mainly concerns large sites; if yours is small, it’s rarely a worry. When it does need attention, this is the kind of issue a proper technical SEO audit is built to catch.
Catch crawl errors, status codes and redirect chains
Logs expose server-side problems in real time. You can spot frequent 404s (pages not found), 5xx server errors, and long redirect chains that eat crawl budget and block access.
They also pinpoint where the problem sits, down to the exact URL. A path that keeps flipping between a 404 and a 301 (permanent redirect), for example, usually signals a misconfigured redirect you’d struggle to catch any other way.
Find orphan and non-indexable pages
An orphan page has no internal links pointing to it, yet bots may still crawl it if it sits in a sitemap or has an external link. Logs surface these stray URLs so you can decide whether to link them properly or clean them up.
The same view catches pages Google keeps trying to crawl even though you’ve told it not to, which points to a directive that isn’t doing its job.
Validate site migrations and post-launch changes
After a migration or redesign, your logs are the fastest early warning system you have. They show whether bots are finding your new URLs, hitting errors or still crawling old paths that should be gone.
You can also compare logs from before and after the change to see whether crawling improved. If bots are still stuck on legacy URLs a week later, you’ll know before your rankings tell you.
Detect and control AI crawlers
AI crawlers now appear in server logs alongside search bots. Bots like OpenAI’s GPTBot and Anthropic’s ClaudeBot request your pages to train models or answer user queries, and logs are one of the few reliable ways to see what they’re doing.
This matters if you care how your content feeds tools like ChatGPT, Claude and Perplexity. Understanding that activity is the first step to managing it, and it ties directly into any AI SEO services strategy. We cover the how later in this guide.
Support security, compliance and troubleshooting
Beyond SEO, logs are a core security and operations tool. Teams read access logs to spot unauthorised access attempts, unusual traffic spikes and other signs of malicious activity.
They also support compliance. Some standards require you to keep audit-trail records for a set period, and logs are how you prove what happened and when. For developers, a log is often the quickest route to the root cause of a bug.
What a log file contains: anatomy of a log line
Each line in a log file records one request, packed into a single string of fields. Once you know what each field is, a wall of text turns into a readable timeline of who requested what, and how your server answered.
A worked example: reading a Googlebot log line
Here’s a single line showing a Googlebot request, in the widely used Combined Log Format:
66.249.66.1 - - [20/Jul/2025:14:02:05 +0000] "GET /blog/seo-guide HTTP/1.1" 200 8452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Read left to right, it says: the IP address 66.249.66.1 made a GET request for /blog/seo-guide on 20 July 2025 at 14:02:05, the server answered 200 (success) and sent back 8,452 bytes, and the visitor identified itself as Googlebot. That single line is a complete record of one crawl event.
The standard log fields, explained
Most log formats carry the same core fields. Here’s what each one tells you:
Field | Example | What it tells you |
IP address | 66.249.66.1 | Where the request came from |
Timestamp | [20/Jul/2025:14:02:05 +0000] | When the request happened |
HTTP method | GET | The action requested, usually GET or POST |
Requested URL | /blog/seo-guide | The page or file requested |
Protocol | HTTP/1.1 | The protocol version used |
Status code | 200 | How the server responded (200, 301, 404, 5xx) |
Response size | 8452 | Bytes sent back |
Referrer | “-“ | The page that linked to the request, if any |
User-agent | Googlebot/2.1… | The bot or browser making the request |
The two fields you’ll lean on most for SEO are the status code, which flags errors and redirects, and the user-agent, which tells you which bot you’re looking at.
How to identify individual bots by user-agent
The user-agent string names the crawler. Googlebot, Bingbot, GPTBot and others each present a distinct string, so filtering your logs by user-agent lets you isolate one bot’s behaviour at a time.
But a user-agent can be faked. Anyone can set their user-agent to Googlebot, and plenty of scrapers do, so a string alone isn’t proof. Google’s own guidance is to run a reverse DNS lookup on the IP: the hostname should resolve to googlebot.com, google.com or googleusercontent.com, and a forward lookup on that hostname should return the original IP. Google also publishes its crawler IP ranges as JSON files you can match against. Until a request passes that check, treat the user-agent as a claim, not a fact.
Common log file formats
Servers don’t all write logs the same way. Before you analyse a file, confirm which format it uses, because the field order and structure differ, and a parser set up for one format will misread another. These are the ones you’re most likely to meet.
Format | Typical source | Includes user-agent? | Best for |
Combined | Apache, NGINX | Yes | SEO analysis |
Common (NCSA) | Older servers | No | Basic traffic counts |
W3C Extended | Microsoft IIS | Configurable | IIS environments |
Amazon load balancer | AWS | Yes | Sites behind AWS |
JSON / CDN | Cloudflare, Kinsta | Yes | Modern, tool-friendly setups |
Combined Log Format
This is the default on Apache and often NGINX, and it’s the format in the worked example above. It records the standard request fields plus the referrer and user-agent on each line, which makes it a solid default for SEO work.
Common Log Format
Also called the NCSA Common Log Format, this is the older, shorter cousin of Combined. It carries the same core request fields but drops the referrer and user-agent. Without the user-agent, you can’t tell bots apart, so it’s weaker for SEO analysis.
W3C Extended Log Format
This is the default on Microsoft’s IIS web server. Its fields are configurable, and each file starts with header lines that list which fields are present and in what order, so always read the header before parsing.
Amazon load balancer logs
If your site sits behind an Amazon load balancer, the access logs follow Amazon’s own format rather than the Apache one. The fields differ, so a parser built for Combined logs won’t read them without adjustment.
JSON and CDN logs
Many modern setups and content delivery networks (CDNs) output structured logs, often as JSON, where each request is a set of labelled key-value pairs. Cloudflare and managed hosts like Kinsta produce their own variants. These are easy for tools to parse, but you’ll want to confirm the field names before you start.
How to do log file analysis: a step-by-step walkthrough
Analysing logs breaks down into five stages: get the files, clean them, verify the bots, read the behaviour, then act and re-check. Here’s how to work through each one.
Step 1: Access and download your server logs
Where your logs live depends on your hosting setup. Find the right source below, then download a recent window, 30 days is a good starting point.
Self-hosted Apache or NGINX
On your own server, access logs usually sit at /var/log/apache2/access.log for Apache or /var/log/nginx/access.log for NGINX. You can download them directly over SFTP or pull them from the command line. If a developer or agency handles your server through website maintenance, they can retrieve these for you.
Managed WordPress hosts
Managed hosts like WP Engine and Kinsta usually expose logs through their dashboard or over SFTP. If you can’t find them, their support teams can point you to the raw request logs.
cPanel or Plesk shared hosting
On shared hosting, look for a raw access logs option in cPanel or Plesk. Access can be limited, and some providers rotate logs quickly or strip fields, so download what you need promptly and keep your own backups.
Cloudflare and other CDNs
If a CDN sits in front of your site, it sees the bot traffic first, so its logs matter. Cloudflare uses a feature called Logpush to send request logs to a storage bucket you control, such as AWS, Google Cloud or Azure.
Cloud platforms
On AWS, Google Cloud or Azure, logs are often routed to a logging service rather than a file you download. You’ll usually need to configure an export and set access permissions before you can pull them.
Step 2: Clean and prepare the data
Raw logs are noisy. Before you draw conclusions, narrow the file to what matters: isolate the bots you care about, strip out static assets and duplicate hits, and normalise timestamps so everything lines up.
The goal isn’t a perfect file. It’s a data set that reflects real crawl behaviour rather than clutter, ready to load into whatever tool you’ll analyse it with.
Step 3: Verify the bots are genuine
Before you trust what the logs say about Googlebot, confirm the requests really came from Googlebot. As covered above, run the reverse and forward DNS check, or match the IP against Google’s published ranges.
Skip this and spoofed user-agents will quietly skew your analysis, making you act on crawl patterns that were never really Google’s. It’s a quick step that protects every conclusion after it.
Step 4: Analyse crawl behaviour
Now read the file for patterns. The questions worth asking are consistent across sites:
- Which pages and folders do bots crawl most, and least?
- How is crawl frequency spread across your important URLs?
- What’s the breakdown of status codes, and where are the 404s and 5xx errors?
- Which file types absorb crawl attention?
- Are there unexplained spikes or drops over the period?
Sort your most important pages by crawl frequency to see how Google is spending its budget. That single view often surfaces the biggest opportunities.
Step 5: Turn findings into fixes and re-check
Not every finding needs a fix. Your job is to spot the patterns with real SEO stakes, crawl waste, missed pages, error clusters, then prioritise the ones that offer the biggest gain for the effort.
Once you’ve made changes, pull your logs again a week or two later. If bot behaviour has shifted the way you intended, the fix worked. If not, you’ll know to look again.
What to look for and how to act on it
Log analysis surfaces a lot of information, so the skill is knowing which patterns deserve action. These are the common findings and what to do about each.
Remove crawl traps and infinite loops
Crawl traps, endless calendar pages, bloated URL parameters or redirect loops, send bots chasing thousands of near-identical URLs. If your logs show that, tighten your URL rules: disallow the offending paths in robots.txt, fix the internal links, or resolve the faulty redirects.
Improve internal linking to under-crawled pages
If important pages sit deep in your site and rarely get crawled, they’re probably poorly linked internally. Surface them higher by adding links from your homepage, main navigation or popular posts. More internal links usually means more consistent crawling.
Clean up orphan pages
Cross-reference your logs against a fresh crawl of your site. A URL that shows up in the logs but not in the crawl map is likely orphaned. Reconnect the ones worth keeping, and noindex or remove the rest.
Prune or consolidate low-value pages
Logs can reveal pages that get regular bot visits but earn no traffic or rankings. These often dilute your site’s focus. Consider pruning them or merging them into stronger, more authoritative pages.
Fix robots.txt and canonical mismatches
If bots are hitting URLs you disallowed, or ignoring your canonical tags, your directives and reality have drifted apart. Update the rules, then watch your logs after the change to confirm bots are now following them.
Ecommerce: rein in over-crawled filters
Faceted navigation is a classic crawl trap on ecommerce sites, where every filter combination becomes its own URL. Logs show Googlebot burning time on these permutations. Limit crawl access with robots.txt, canonicals or noindex so your budget goes to core category and product pages instead. On platforms like Shopify, this is a common fix in day-to-day Shopify SEO work.
JavaScript-heavy sites: confirm rendering
Search engines render JavaScript, but not always reliably. If your logs show Google fetching only base URLs and not the resources triggered by JavaScript, your dynamic content may not be getting seen. That’s a sign to look at server-side rendering or how your content loads.
Publishers and news: monitor crawl freshness
For publishers, fast crawling of new articles matters for visibility in news and top-stories features. Logs show how quickly bots reach new content. If they’re slow, stronger internal linking and up-to-date XML sitemaps usually help.
After a migration: catch legacy-URL crawling
Following a migration, logs tell you whether bots are still crawling old URLs, hitting 404s or ignoring new content. They also show whether your top pages are getting more or less attention than before, so you can tell early whether the move helped or hurt.
Log file analysis tools compared
The right log analysis tool depends on your site size, your budget and whether your focus is SEO or general system monitoring. The main options, from dedicated SEO analysers to general-purpose platforms, break down like this:
Tool / tier | Type | Cost | Best for |
Screaming Frog Log File Analyser | SEO desktop | Freemium | Most sites |
Semrush Log File Analyzer | SEO (cloud) | Paid | Semrush users |
JetOctopus, Oncrawl, Botify | Enterprise SEO | Paid | Large sites |
Splunk | IT / security | Paid | Operations teams |
ELK Stack, Grafana Loki | Observability | Open-source | Custom setups |
GoAccess | Log analyzer (terminal) | Open-source | Command-line users |
Excel, Python / pandas | DIY | Free | Small or custom jobs |
SEO-specific log analysers
These are built around crawl behaviour and bot verification, which makes them the natural fit for SEO work.
Screaming Frog SEO Log File Analyser
A desktop app made for SEOs. You upload your log files and it verifies search bots, identifies crawled URLs and breaks down bot behaviour. It’s a strong starting point for most sites.
Semrush Log File Analyzer
Part of the Semrush toolkit, this one focuses on Googlebot activity and pairs with the rest of their technical SEO tools, which is handy if you already work in that ecosystem.
Enterprise SEO platforms
Tools like JetOctopus, Oncrawl and Botify handle log analysis at large scale and combine it with wider crawl and site data. They suit big sites with millions of URLs rather than small ones.
General IT and observability platforms
If you need log analysis for system monitoring and security as well as SEO, these platforms cover far more than crawling.
Splunk
An enterprise platform for indexing, searching and visualising large volumes of machine data, widely used for IT operations and security monitoring rather than SEO specifically.
ELK Stack and similar
The ELK Stack (Elasticsearch, Logstash and Kibana) is an open-source suite for storing, processing and visualising logs. It’s flexible and scalable, though it takes setup effort. Grafana Loki is a comparable open-source option.
Free and open-source options
You don’t always need paid software, especially for smaller sites or one-off analysis.
GoAccess
A fast, terminal-based log analyser that shows web server statistics in real time. It’s open-source and a good fit if you’re comfortable on the command line.
Excel or Google Sheets
For a small log set, a spreadsheet can be enough. You can filter by user-agent, sort by status code and spot patterns without any dedicated software.
Python or pandas
For larger or repeated analysis, a short Python script using the pandas library lets you parse and filter millions of lines exactly how you want. It takes some coding, but it scales.
How to choose the right tool for your site size
For a small site, a spreadsheet or GoAccess may be all you need. Mid-sized sites are well served by Screaming Frog or Semrush. Large sites with millions of URLs, or teams that need security and operations monitoring too, are where the enterprise SEO platforms and general observability tools earn their cost.
Controlling AI and LLM crawlers with your logs
Once you can see AI crawlers in your logs, you can decide what to do about them. This is where log analysis meets AI search readiness: spot the bots, then choose whether to welcome, limit or block each one.
Which AI bots to watch for
Several AI crawlers now request pages regularly, and it helps to know who’s who. The main ones and their user-agents:
- GPTBot: OpenAI’s training crawler, which collects public content that may be used to train its models.
- OAI-SearchBotand ChatGPT-User: OpenAI’s other two agents, for surfacing sites in ChatGPT search and for fetching a page when a user asks.
- ClaudeBot: Anthropic’s crawler for Claude.
- PerplexityBot: Perplexity’s crawler.
- Google-Extended: Google’s opt-out control for whether your content trains its generative models. It doesn’t affect your Google Search ranking.
- CCBot: Common Crawl’s bot, whose open data set is used by many AI projects.
- Bytespider: ByteDance’s crawler, which has a documented history of ignoring crawl directives.
A paste-ready robots.txt block for AI crawlers
Once you’ve seen which AI bots hit your site, you can control the compliant ones through robots.txt. Here’s a starting block you can adapt. This example allows the search and user agents (so you stay citable in AI answers) while blocking the training crawlers:
# Allow AI search and user agents
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
Adjust the allow and block decisions to fit your own strategy. Blocking a training bot keeps your content out of a model; blocking a search bot removes you from that tool’s answers, so decide per bot rather than blanket-blocking.
Rate limits, firewall rules and Cloudflare Pay Per Crawl
Robots.txt is a polite request, and not every bot honours it. For crawlers that ignore it, control moves to the server: rate limits to throttle aggressive bots, and firewall rules to block by pattern.
Cloudflare has taken this further. Its Pay Per Crawl feature lets you allow, charge or block each AI crawler, using the HTTP 402 “Payment Required” status code to signal that a bot must pay to access your content. New domains on Cloudflare are set to block AI crawlers by default, so it’s worth checking your own setting.
Common log file analysis mistakes to avoid
A few recurring mistakes can turn a useful analysis into a misleading one. Watch for these.
Trusting spoofed user-agents
The most common error is treating the user-agent as proof. Scrapers fake the Googlebot string constantly, so if you skip verification you’ll draw conclusions from traffic that was never Google’s. Always run the DNS or IP check first.
Analysing too small a window
A single day of logs is a snapshot, and snapshots miss patterns. Crawl behaviour shows its shape over weeks, so a longer window, 30 days or more, gives you far more reliable signals than a day or two.
Ignoring log retention limits
Many hosts keep logs for only a short time before overwriting them, especially on shared hosting. If you wait until you need them, they may already be gone. Set up regular backups so you have history to work with.
Treating Search Console as a substitute
Search Console’s crawl data is useful, but it’s sampled, limited to Google’s bots and short-lived. It’s a complement to log analysis, not a replacement. Only your logs capture every crawler and every request.
Ignoring privacy and PDPA obligations
Log files contain IP addresses, and in Singapore an IP address can count as personal data under the PDPA once it can identify an individual, on its own or combined with other data. The PDPA’s retention limitation obligation means you shouldn’t keep that data longer than you need it. Anonymising IPs and setting a sensible retention period keeps your analysis on the right side of the rules.
Frequently Asked Questions
Download your server logs, clean out noise like static assets and duplicate hits, verify the bots are genuine with a DNS or IP check, then read the file for crawl patterns: which pages get crawled, how often, and what status codes come back. A log analyser tool makes this far faster than reading raw lines.
In SEO, log file analysis is the practice of reading server logs to see how search and AI bots actually crawl your site. It reveals crawl budget waste, orphan pages, crawl errors and bot behaviour that crawl simulators and Search Console can only estimate.
Each line records one request as a string of fields: IP address, timestamp, HTTP method, requested URL, status code, response size and user-agent. Read left to right, one line tells you who requested what, when, and how your server responded. The status code and user-agent are the two fields most useful for SEO.
A common example is diagnosing crawl budget waste. You pull 30 days of logs, filter to verified Googlebot, and find it spending most of its crawls on old filtered URLs instead of your product pages. You block those URLs, then re-check the logs to confirm crawling shifted to the pages that matter.
It's an ongoing task, not a one-off. Your site changes and bots adapt, so a regular review, monthly for many sites, catches issues early. Always run one after a migration or major site change, when crawl behaviour is most likely to shift.
No, though crawl budget concerns mostly affect large sites. Smaller sites still benefit from spotting crawl errors, orphan pages, spoofed bots and AI crawler activity. The scale of the task differs, but the insights are useful at any size.
Search Console's Crawl Stats report is sampled, limited to Google's own crawlers and holds only a short window. Log analysis captures every crawler and every request with no sampling, giving you the full, unfiltered picture rather than a Google-only summary.
Final thoughts
Once you start reading your logs, you stop guessing about crawl behaviour and start acting on what’s actually happening. That’s the real gain here: fixes aimed at problems you’ve confirmed, not ones you assume. Whether you run the analysis yourself or fold it into a wider technical SEO programme, your server logs are one of the most honest signals you have.
Have you found anything surprising in your own logs? If this guide helped, share it with someone wrestling with a crawl problem of their own.
