How to Fix Common Crawl Errors

Why Crawl Errors Matter

Google allocates a crawl budget to every website — the number of pages it will crawl within a given time period. Crawl errors waste that budget by sending Googlebot to URLs that return errors instead of content.

For small sites with a few hundred pages, crawl budget isn’t usually a constraint. But for larger sites — or sites with significant crawl errors — budget waste can mean important pages go uncrawled and unindexed for weeks or months.

Beyond crawl budget, errors create poor user experience. A visitor landing on a 404 page or getting stuck in a redirect loop will leave and may not return.

Finding Crawl Errors

Google Search Console

The Pages report (formerly Coverage report) in Google Search Console shows:

• Errors: Pages Google tried to crawl but couldn’t
• Valid with warnings: Pages that are indexed but have issues
• Valid: Pages successfully indexed
• Excluded: Pages not indexed (sometimes intentionally, sometimes not)

Review the Errors section first — these are the most impactful problems.

Crawl Tools

Run a full-site crawl using dedicated tools for a more comprehensive view:

• Screaming Frog SEO Spider — Crawls your site like a search engine and reports all errors
• Sitebulb — Visual crawl analysis with prioritized recommendations
• Ahrefs Site Audit — Cloud-based crawling with integration into your broader SEO data

These tools catch errors that Google Search Console may not report, including internal issues like redirect chains and orphan pages that Google hasn’t attempted to crawl.

Server Log Analysis

Server logs show every request Googlebot makes to your site, including:

• Which pages Googlebot crawls most frequently
• Which pages return errors
• How Googlebot’s crawl pattern has changed over time
• Whether Googlebot is wasting crawl budget on low-value URLs

Tools like Screaming Frog Log File Analyser or Loggly can parse raw log files into actionable reports.

The Most Common Crawl Errors (and How to Fix Them)

404 Not Found

What it is: The page doesn’t exist. Either it was deleted, the URL was changed, or a link points to a URL that never existed.

Impact: Moderate to high. A few 404s are normal and don’t hurt rankings. But excessive 404s waste crawl budget and lose the link equity that external sites may be passing to those URLs.

How to fix:

1. Check if the page was moved. If the content exists at a new URL, implement a 301 redirect from the old URL to the new one.
2. Check for inbound links. If external sites link to the 404 URL, a 301 redirect preserves that link equity. Use Ahrefs or Google Search Console’s Links report to identify inbound links.
3. Fix internal links. Update any internal links pointing to the 404 URL. Broken internal links waste crawl budget and create poor user experience.
4. If the content is truly gone, return a proper 404 status code (not a soft 404) and ensure your 404 page helps users find what they’re looking for.
5. Clean up your sitemap. Remove 404 URLs from your XML sitemap.

301/302 Redirect Issues

Redirect chains: When URL A redirects to URL B, which redirects to URL C (or longer). Each hop adds latency and can cause Google to stop following the chain.

Fix: Update all redirects to point directly to the final destination. If A → B → C, change A to redirect directly to C.

Redirect loops: When URL A redirects to URL B, which redirects back to URL A.

Fix: Identify the loop in your redirect configuration and break it by pointing one URL to the correct destination.

302 instead of 301: 302 redirects indicate a temporary move. Google may continue indexing the old URL instead of transferring authority to the new one.

Fix: If the redirect is permanent, change it to a 301. Reserve 302s for genuinely temporary situations.

HTTP to HTTPS redirect issues: Some sites have incomplete HTTPS migration, leaving certain URLs on HTTP without redirects.

Fix: Ensure every HTTP URL redirects to its HTTPS equivalent via 301. Check both www and non-www versions.

5xx Server Errors

What they are: The server failed to respond. Common codes:

• 500 Internal Server Error — Generic server failure
• 502 Bad Gateway — Server received an invalid response from an upstream server
• 503 Service Unavailable — Server is temporarily overloaded or under maintenance
• 504 Gateway Timeout — Server didn’t receive a timely response from an upstream server

Impact: High. If Googlebot encounters 5xx errors frequently, it will reduce crawl rate and eventually deindex affected pages.

How to fix:

1. Check server logs for the specific error cause (database timeout, memory exhaustion, PHP fatal error)
2. Monitor server resources (CPU, memory, disk) — 5xx errors often indicate capacity issues
3. Review recent deployments — code changes frequently introduce 500 errors
4. Check third-party dependencies — API failures, database connectivity issues
5. Implement proper error handling in your application code to prevent unhandled exceptions from causing 500 errors
6. If maintenance is needed, return a 503 with a Retry-After header to tell Googlebot when to come back

Soft 404s

What they are: Pages that return a 200 (OK) status code but display error-like content — “No results found,” empty pages, or placeholder content.

Impact: Moderate. Google identifies soft 404s and treats them similarly to real 404s. They waste crawl budget and can indicate thin content issues.

How to fix:

1. Return proper 404 status codes for pages that don’t have meaningful content
2. Add real content to pages that should exist but are currently thin
3. Check search/filter pages — empty search results are a common soft 404 source. Block parameter-driven pages with no results via robots.txt or noindex
4. Review paginated archives — old archive pages with no content should be consolidated or removed

Blocked by Robots.txt

What it is: Your robots.txt file prevents Googlebot from crawling certain URLs.

Impact: Variable. Intentional blocking (admin pages, internal search results) is fine. Accidental blocking of important pages is critical.

How to fix:

1. Review your robots.txt file and verify each disallow rule is intentional
2. Test specific URLs using Google Search Console’s URL Inspection tool — it shows whether robots.txt blocks the page
3. Remove accidental blocks immediately and request re-crawling via Search Console
4. Don’t block CSS or JavaScript files that Google needs to render your pages properly

Crawled but Not Indexed

What it is: Google crawled the page but decided not to include it in the index. This isn’t technically a crawl error, but it appears in Search Console and represents a significant issue.

Common causes:

• Thin or low-quality content
• Duplicate content without canonical tags
• Pages that add no unique value
• Content that Google deems unhelpful

How to fix:

1. Improve content quality. Add depth, unique insights, and genuine value
2. Implement canonical tags to consolidate duplicate pages
3. Add internal links to the page from authoritative pages on your site
4. Check if the content is simply too similar to content on other sites — differentiate your perspective
5. Consolidate thin pages into more comprehensive resources

Crawl Budget Optimization

Beyond fixing errors, optimize how Google spends its crawl budget on your site:

Remove low-value URLs from Google’s crawl queue:

• Block faceted navigation parameters via robots.txt
• Noindex tag pages, author archives, and date archives (unless they drive traffic)
• Clean up URL parameters in Google Search Console

Prioritize important pages:

• Include them in your XML sitemap
• Link to them from high-authority internal pages
• Keep them within 3 clicks of the homepage

Improve crawl efficiency:

• Fast server response times encourage more frequent crawling
• Clean internal linking (no broken links or redirect chains)
• Consistent URL structure without unnecessary parameters

Ongoing Monitoring

Crawl errors don’t just appear once — they accumulate as content changes, URLs are modified, and plugins or code updates introduce issues.

Weekly: Quick review of Google Search Console for new errors Monthly: Run a full-site crawl and compare error counts to previous months After deployments: Spot-check critical pages for new crawl issues Quarterly: Full server log analysis to identify crawl patterns and waste

For a broader look at technical optimization, see our technical SEO complete guide. For speed-specific issues affecting crawlability, our site speed optimization guide covers server response optimization. And for professional diagnostic support, our technical SEO services include comprehensive crawl analysis.