Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

When does crawl budget become a realistic concern for a small website?

Not because of size alone. It becomes a concern when the site, though small in real pages, exposes a very large number of URLs, or when the server is too slow or too error-prone for Google to fetch pages at a normal pace. Google’s own guidance says who it is written for: large sites with a million or more pages that change weekly, sites of ten thousand or more pages that change daily, and sites with a large share of URLs reported as discovered and not indexed. For everyone else it says that if your pages “seem to be crawled the same day that they are published, you don’t need to read this guide” (Google’s crawl budget guidance). A small site with pages missing from search should look for a different problem first.

Signs of real crawl constraints

On a small site, a true crawl constraint shows as one of these.

  • New or updated pages go unrequested by Googlebot for weeks, visible in server logs or in the last crawl date in URL Inspection.
  • The number of URLs Google knows about is many times the number of real pages, because of parameters, filters, calendars, session values, or search result pages.
  • The server returns errors or responds very slowly to crawler requests, and crawl stats show Google reducing its requests.
  • A large portion of the site’s URLs sit under discovered and not indexed.

The second sign deserves the first look when a small site does have a crawl problem. Three hundred real pages can turn into three hundred thousand addresses through a faceted filter. The site is then small in content and large in URLs, and Google’s guidance on what it calls perceived inventory applies: without direction, it tries to crawl all or most of the URLs it knows about.

Google describes its own figures as approximate. The size thresholds are, in its words, “a rough estimate to help you classify your site. These are not exact thresholds”.

Index coverage versus crawl rate

Missing pages can be an indexing matter instead of a crawling one, and the page indexing report says which.

What the report shows What it means Crawl budget issue?
Crawled, currently not indexed Google fetched the page and chose not to index it No; the page was crawled
Excluded by noindex A rule on the page No
Blocked by robots.txt A rule in the file No
Duplicate, Google chose a different canonical Canonicalization No
Page with redirect, not found, soft 404 The URL does not resolve to content No
Discovered, currently not indexed Google knows the URL and has not fetched it yet Possibly, if the count is large and persistent

Only the last row points toward crawling, and on a small site even that can have other explanations, which the triage below works through.

Simple-site priorities

For a site of a few hundred or a few thousand pages, the work that helps is ordinary.

  1. Make every important page reachable through links from other pages.
  2. Submit a sitemap of canonical, indexable URLs.
  3. Remove accidental blocks: stray noindex rules, robots.txt disallows, canonical tags pointing elsewhere.
  4. Fix server errors and slow responses.
  5. Make the pages worth indexing.

None of that is crawl budget optimization. Time spent trimming crawl paths on a 200-page site with a healthy server is time taken from the fifth item.

Situations that change the answer

Run the triage. Each line gives the input, the action, and the pass or fail condition, and the list ends at a conclusion.

Input to verify Action Result
Count of real pages against URLs known to Google Compare the sitemap count with total URLs in the page indexing report If known URLs are many times the real count, go to URL sprawl below. Otherwise continue
Crawl timing Check last crawl date for several new pages in URL Inspection If crawled within days of publishing, continue. If not after weeks, investigate links and server health
Server health Review crawl stats for errors and response time If errors or slow responses appear, fix the server; this is a hosting problem
Indexing reasons Read the reasons for excluded pages If the reasons are noindex, canonical, blocked, or crawled and not indexed, fix those; this is not crawl budget
Discovered, currently not indexed Check its share of total URLs and whether it persists If small or shrinking, wait. If large and persistent with a clean site, review page quality and internal links
All checks pass None Not a crawl budget problem

URL sprawl branch, for a small site with too many addresses:

  • Identify the pattern generating URLs: filters, sort orders, calendars, session parameters, internal search.
  • Remove the links that expose the pattern, or constrain it.
  • Point variants to a canonical URL, or keep them from being crawled once the links are cleaned.
  • Confirm in the logs that requests concentrate on real pages.

Situations in which a small site should take crawling seriously:

  • An ecommerce catalog with faceted navigation, however few products.
  • A site with a calendar, a search function, or user-generated pages open to crawlers.
  • A site on hosting that fails under modest load.
  • A site that publishes time-critical content and needs it fetched within hours.
  • A site that has just migrated and needs many redirects processed.

Two of the indexing states mentioned above have their own explanations and remedies: why Search Console labels a URL discovered and currently not indexed and why it labels a URL crawled and currently not indexed.

The conclusion for a typical small site with a healthy server and a few hundred clean URLs: this is not a crawl budget problem. Look at indexing reasons and page quality instead.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *