Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How does an XML sitemap support discovery of URLs on a very large website?

It gives search engines a direct list of the URLs you want found, so that discovery does not depend on a crawler reaching every page by following links. On a site with millions of pages, some are many clicks deep, some are new and not yet linked widely, and some change faster than they are recrawled. A sitemap puts all of them in front of the crawler at once, with a date showing when each last changed. What it does not do is make pages get crawled or indexed. Google’s documentation is exact about the limit: “A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee” that every item will be crawled and indexed (Google’s overview of sitemaps).

Discovery versus indexing

Three stages are separate, and a sitemap acts on the first.

Stage What happens What a sitemap contributes
Discovery Google learns that a URL exists Lists the URL directly
Crawling Google requests the page Nothing guaranteed; the URL becomes a candidate
Indexing Google decides whether to keep the page Listing it is a weak signal about which URL is preferred

On the third row, Google’s canonicalization guidance describes sitemap inclusion as “A weak signal” toward a URL becoming canonical, weaker than a redirect or a canonical annotation (Google’s guidance on consolidating duplicate URLs).

The same overview page notes that if a site’s pages are properly linked, Google can usually discover most of it. A sitemap complements links. On a large site it matters most for pages that links reach slowly.

The separation runs both ways. A sitemap does not guarantee indexing, and a page left out of every sitemap can still be indexed, for reasons given in the guide to why Google might index a page absent from every submitted XML sitemap.

Lastmod reliability

Each entry can carry a last-modified date, which tells a crawler which pages changed since it last looked. Google’s sitemap documentation says it uses the value if it is consistently and verifiably accurate, and that it ignores the priority and change-frequency values (Google’s guidance on building sitemaps).

That makes accuracy a condition of usefulness.

  • Set the date when the page’s main content changes, not when a template, an advert, or a sidebar changes.
  • Do not stamp every URL with today’s date on each build. A file in which everything always changed today tells the crawler nothing.
  • Generate the date from the content record, not from the time the file was written.

Sitemap partitioning

One file has limits. Google’s documentation states that “All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs”. Larger inventories are split into several files, listed in a sitemap index. The documentation on large sitemaps adds that the referenced sitemaps “must be hosted on the same site as your sitemap index file”, and that a sitemap index may list up to 50,000 sitemaps (Google’s guidance on sitemap index files).

Split by something meaningful, so that each file can be monitored. A layout for a large inventory, with the decisions that shape it:

/sitemap-index.xml
├── /sitemaps/products-1.xml        up to 50,000 URLs
├── /sitemaps/products-2.xml
├── /sitemaps/products-new.xml      items added in the last 30 days
├── /sitemaps/categories.xml
├── /sitemaps/articles-2026.xml
├── /sitemaps/articles-2025.xml
└── /sitemaps/locations.xml

Decision points:
  Does a section exceed 50,000 URLs or 50MB?
    yes -> split it into numbered files
  Do you need to see how one page type is indexed?
    yes -> give that type its own file
  Is there content that must be found fast?
    yes -> a small file of recent URLs, regenerated often
  Do older sections rarely change?
    yes -> group them by year so their files stay stable

Splitting by page type pays off in reporting. Search Console shows indexing figures per submitted sitemap, so a file containing only product pages tells you how product pages are faring, which a single mixed file cannot.

These files are written for crawlers. A sitemap page written for visitors is a different thing with a different purpose, discussed in the guide to when an HTML sitemap is useful in addition to a website’s primary navigation.

Submission and monitoring

  1. Submit the index file in Search Console, and reference it in robots.txt.
  2. Check that each child file is read without errors.
  3. For each file, compare the number of URLs submitted with the number indexed. A file with a low share points to a problem in that section.
  4. Regenerate files when content changes, on a schedule that matches how fast the section changes.
  5. Monitor the files themselves: a build that fails silently can leave a sitemap empty or months out of date.

Keep the listed URLs clean. Google’s documentation asks for fully-qualified, absolute URLs and says it will attempt to crawl them exactly as listed. On a very large site, a sitemap padded with redirects, duplicates, and error pages spends crawler attention on addresses that lead nowhere. Which URLs belong in the file and which do not is set out in the guide to the URL types that should be excluded from an XML sitemap.

Three expectations to set with stakeholders.

  • Submitting a sitemap does not schedule a crawl.
  • A URL in the sitemap can remain unindexed. The reason will be in the page indexing report, not in the sitemap.
  • Removing a URL from the sitemap does not remove it from the index.

Build in this order: decide the partitions, generate from the content database with accurate dates, submit the index, then watch submitted against indexed per file.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *