How Crawl Budget Works, and When You Need to Care
On this page
Crawl budget is the number of URLs on your site that Google can and wants to crawl in a given period. It depends on how much crawling your server can handle and how much Google wants to crawl, based on your site's size, freshness and value. Most sites never need to think about it. It matters for very large or fast-changing sites, where wasted crawling can delay important pages. This explainer is part of our technical SEO guide.
Key takeaways
- Crawl budget is the set of URLs Google can and wants to crawl on one hostname.
- It comes from crawl capacity (what your server handles) and crawl demand (what Google wants).
- Google says most sites don't need to manage it; its guide is for very large or fast-changing sites.
- Duplicate URLs, soft 404s and long redirect chains waste crawling.
- Noindex doesn't save crawling; Google still has to fetch the page to see it.
What crawl budget is
Google can't crawl every URL on the web, so it decides how much time and resources to spend on each site. That allocation is what SEOs call crawl budget. Google's own definition is the set of URLs it can and wants to crawl on a site.
Two details are easy to miss. First, Google counts a site as one hostname, so www.example.com and shop.example.com each have their own budget. Second, crawling isn't indexing. After a page is crawled, Google still evaluates it and decides whether it belongs in the index. A page can be crawled often and still not be indexed.
Crawl capacity and crawl demand
Google sets crawl budget from two things. The table below shows what drives each one.
| Part | What it is | What raises or lowers it |
|---|---|---|
| Crawl capacity limit | How hard Google can crawl without overloading you | Fast, stable responses raise it; 5xx and 429 lower it |
| Crawl demand | How much Google wants to crawl your site | Popularity, freshness, quality and URL inventory |
Crawl capacity protects your server. Every site starts with the same conservative limit. If your server responds quickly and consistently, Google raises it over time. If responses slow down or return server errors, it backs off. The limit is shared across Google's crawlers, so heavy demand from one leaves less room for the others.
Crawl demand is about interest. Google tries to keep popular pages fresh and to recrawl pages often enough to catch changes. The factor you control most is what Google calls perceived inventory: every URL it knows about on your site. If many of those URLs are duplicates or pages you don't care about, Google spends its time on them.
Which sites need to think about crawl budget
Google is direct about this. If your site doesn't have a large number of pages that change rapidly, or your new pages tend to be crawled the day you publish them, you don't need its crawl budget guide. Keeping your sitemap current and checking the Page indexing report is enough.
Google's guide is written for three kinds of site. It gives rough sizes, not exact thresholds: large sites of around a million or more unique pages that change weekly, sites of 10,000 or more pages that change daily, and sites where a large share of URLs sit in the "Discovered - currently not indexed" status.
For most business websites, the useful lesson isn't budget management. It's that a tidy URL inventory helps any site get its important pages crawled sooner.
What wastes crawling
Google's guide lists the main causes of wasted crawling. They are worth checking in any technical SEO audit, whatever the size of the site.
- Duplicate URLs. Sort orders, filters, tracking parameters and session IDs can turn one page into hundreds of URLs. Consolidate them with canonical tags or redirects.
- Soft 404s. Pages that say "not found" but return a success code keep being crawled. Return a real 404 or 410 for removed pages.
- Long redirect chains. Each hop costs a request. Link straight to the final URL.
- Endless URL spaces. Calendars and infinite scroll pages that duplicate linked content can generate URLs without end. Block them in robots.txt if they can't be consolidated.
- Slow responses. If your server is slow, Google crawls less to protect it.
How to check crawling in Search Console
The Crawl stats report, found under Settings in Search Console, shows how Google crawls your site. Google calls it a report for advanced users, and says sites with fewer than about a thousand pages shouldn't need it.
It shows total crawl requests over time, average response time and host status, which flags problems with robots.txt fetching, DNS or server connectivity. You can also break requests down by response code, file type and purpose. Purpose splits crawls into discovery of new URLs and refresh of known ones. If response times rise and crawl requests fall at the same time, your server may be the limit.
The Page indexing report helps too. A growing number of URLs in "Discovered - currently not indexed" can mean Google found the URLs but postponed crawling them. Our guide on how to fix indexing issues covers that status.
Crawl budget myths
Robots.txt can move budget to other pages. Google says blocking URLs won't shift crawling to other pages unless it is already hitting your capacity limit. Use robots.txt only for URLs you never want crawled.
Crawl-delay slows Googlebot down. Google doesn't support the crawl-delay rule in robots.txt. It adjusts its crawl rate from your server's responses.
More pages earn more budget. Google says it weighs popularity, overall user value, content uniqueness and serving capacity. Adding thin pages adds inventory to crawl without adding value. The robots.txt and sitemap guide covers how to guide crawlers to the pages that count.
If you'd rather have crawling and indexing problems found and fixed for you, our in-house SEO team handles technical SEO, from site structure to performance.
Common questions
Does crawl budget affect rankings?
Not in the sense of being a ranking signal. Crawl budget decides which of your URLs Google fetches and how often. Google says that not every page it crawls will be indexed, so more crawling doesn't mean better rankings. It matters when important pages aren't being crawled, or are crawled too rarely to stay fresh, which mostly affects very large or fast-changing sites.
Does adding noindex save crawl budget?
No. Google has to request a page to see its noindex rule, so the crawl still happens. Google's crawl budget guide says not to use noindex for this. For URLs that shouldn't be crawled at all, such as endless sort orders, use robots.txt. For duplicates, consolidate them with redirects or canonical tags, and return 404 or 410 for pages you have removed for good.
Can I ask Google to crawl my site more often?
Not as a setting, and Google says it doesn't accept payment to crawl a site more often. You can request indexing for individual URLs in Search Console, within a quota, and keep your sitemap current. Over time, Google says crawl demand rises with a site's popularity and value to users, and capacity rises when your server responds quickly and reliably.