usdigitalmarketingindex.com SEO
When does crawl budget become a realistic concern for a small website?
Updated November 15, 2026
Not because of size alone. It becomes a concern when the site, though small in real pages, exposes a very large number of URLs, or when the server is too slow or too error-prone for Google to fetch pages at a normal pace. Google’s own guidance says who it is written for: large sites with a million or more pages that change weekly, sites of ten thousand or more pages that change daily, and sites with a large share of URLs reported as discovered and not indexed. For everyone else it says that if your pages “seem to be crawled the same day that they are published, you don’t need to read this guide” (Google’s crawl budget guidance). A small site with pages missing from search should look for a different problem first.
Signs of real crawl constraints
On a small site, a true crawl constraint shows as one of these.
- New or updated pages go unrequested by Googlebot for weeks, visible in server logs or in the last crawl date in URL Inspection.
- The number of URLs Google knows about is many times the number of real pages, because of parameters, filters, calendars, session values, or search result pages.
- The server returns errors or responds very slowly to crawler requests, and crawl stats show Google reducing its requests.
- A large portion of the site’s URLs sit under discovered and not indexed.
The second sign deserves the first look when a small site does have a crawl problem. Three hundred real pages can turn into three hundred thousand addresses through a faceted filter. The site is then small in content and large in URLs, and Google’s guidance on what it calls perceived inventory applies: without direction, it tries to crawl all or most of the URLs it knows about.
Google describes its own figures as approximate. The size thresholds are, in its words, “a rough estimate to help you classify your site. These are not exact thresholds”.
Index coverage versus crawl rate
Missing pages can be an indexing matter instead of a crawling one, and the page indexing report says which.
| What the report shows | What it means | Crawl budget issue? |
|---|---|---|
| Crawled, currently not indexed | Google fetched the page and chose not to index it | No; the page was crawled |
| Excluded by noindex | A rule on the page | No |
| Blocked by robots.txt | A rule in the file | No |
| Duplicate, Google chose a different canonical | Canonicalization | No |
| Page with redirect, not found, soft 404 | The URL does not resolve to content | No |
| Discovered, currently not indexed | Google knows the URL and has not fetched it yet | Possibly, if the count is large and persistent |
Only the last row points toward crawling, and on a small site even that can have other explanations, which the triage below works through.
Simple-site priorities
For a site of a few hundred or a few thousand pages, the work that helps is ordinary.
- Make every important page reachable through links from other pages.
- Submit a sitemap of canonical, indexable URLs.
- Remove accidental blocks: stray noindex rules, robots.txt disallows, canonical tags pointing elsewhere.
- Fix server errors and slow responses.
- Make the pages worth indexing.
None of that is crawl budget optimization. Time spent trimming crawl paths on a 200-page site with a healthy server is time taken from the fifth item.
Situations that change the answer
Run the triage. Each line gives the input, the action, and the pass or fail condition, and the list ends at a conclusion.
| Input to verify | Action | Result |
|---|---|---|
| Count of real pages against URLs known to Google | Compare the sitemap count with total URLs in the page indexing report | If known URLs are many times the real count, go to URL sprawl below. Otherwise continue |
| Crawl timing | Check last crawl date for several new pages in URL Inspection | If crawled within days of publishing, continue. If not after weeks, investigate links and server health |
| Server health | Review crawl stats for errors and response time | If errors or slow responses appear, fix the server; this is a hosting problem |
| Indexing reasons | Read the reasons for excluded pages | If the reasons are noindex, canonical, blocked, or crawled and not indexed, fix those; this is not crawl budget |
| Discovered, currently not indexed | Check its share of total URLs and whether it persists | If small or shrinking, wait. If large and persistent with a clean site, review page quality and internal links |
| All checks pass | None | Not a crawl budget problem |
URL sprawl branch, for a small site with too many addresses:
- Identify the pattern generating URLs: filters, sort orders, calendars, session parameters, internal search.
- Remove the links that expose the pattern, or constrain it.
- Point variants to a canonical URL, or keep them from being crawled once the links are cleaned.
- Confirm in the logs that requests concentrate on real pages.
Situations in which a small site should take crawling seriously:
- An ecommerce catalog with faceted navigation, however few products.
- A site with a calendar, a search function, or user-generated pages open to crawlers.
- A site on hosting that fails under modest load.
- A site that publishes time-critical content and needs it fetched within hours.
- A site that has just migrated and needs many redirects processed.
Two of the indexing states mentioned above have their own explanations and remedies: why Search Console labels a URL discovered and currently not indexed and why it labels a URL crawled and currently not indexed.
The conclusion for a typical small site with a healthy server and a few hundred clean URLs: this is not a crawl budget problem. Look at indexing reasons and page quality instead.
Add your comment to this answer