Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

Which URL types should be excluded from an XML sitemap?

Exclude every URL you do not want shown in search results as its own page: redirects, error pages, pages marked noindex, pages blocked by robots.txt, duplicates whose canonical points elsewhere, parameter variations of a page, and anything behind a login. A sitemap should be the list of your preferred, indexable URLs and nothing else. Google’s documentation describes sitemap inclusion as a signal toward a URL becoming canonical, so listing a URL you do not want chosen sends a signal against your own intent. Leaving a URL out, on the other hand, does not remove it from the index. A sitemap states preferences. It does not issue removals.

Canonical and indexability rules

A URL belongs in the sitemap when it passes every one of these tests.

  • It returns a 200 status.
  • It is the canonical URL for its content: its canonical tag points to itself.
  • It carries no noindex rule in a meta tag or HTTP header.
  • It is not blocked by robots.txt.
  • It is reachable without logging in.
  • It is a page you want people to land on from search.

A URL that fails any test is excluded. The logic is consistency: Google’s documentation notes that sitemap inclusion is “A weak signal that helps the URLs that are included in a sitemap become canonical” (Google’s guidance on consolidating duplicate URLs). A URL listed in the sitemap and marked noindex, or listed while its canonical tag names another address, gives two instructions that disagree.

Redirect and error URLs

URL type Include? Reason
Returns 301 or 302 No List the destination instead
Returns 404 or 410 No The page does not exist
Returns a server error No, and fix the error A listed URL should work
Returns 200 with an empty or not-found message No A soft error; fix the page or return a real 404
Old URLs after a migration No, with one exception Old URLs can be listed for a short period so the redirects are found sooner, then removed

Sitemaps drift out of date as content is retired. A file generated once and left alone accumulates redirects and errors.

Parameterized duplicates

One page can be reached at many addresses. List one.

Variation Include?
Tracking parameters, such as campaign tags No
Session identifiers No
Sort orders and view options No
Filter combinations No, unless a specific filtered page is a deliberate landing page with its own canonical
Pagination pages Depends on your pagination approach; include them if they are self-canonical pages you want crawled
HTTP when the site is HTTPS; www when the site is non-www No; list the preferred host and protocol
Trailing slash and case variants No; list the canonical form
Print, AMP, or mobile alternates No, unless they are the canonical version

Google’s documentation asks for “fully-qualified, absolute URLs” and says it will try to crawl them “exactly as listed” (Google’s guidance on building sitemaps). Exactly as listed means a wrong protocol or a stray parameter is crawled as written.

Also leave out pages that are technically indexable and are not meant as search landing pages: internal search results, cart and checkout steps, account pages, thank-you pages, and thin tag or archive pages you have chosen to keep out of search.

Automated sitemap QA

Check the file against the live site on a schedule. A pass or fail table for a sample of entries, with example rows:

URL in sitemap (example) Status Canonical points to Robots Verdict
/products/cordless-drill/ 200 Itself Indexable Pass
/products/cordless-drill/?color=red 200 /products/cordless-drill/ Indexable Fail: not canonical
/old-category/ 301 Not applicable Not applicable Fail: redirect; list the target
/sale-2023/ 404 Not applicable Not applicable Fail: remove
/account/orders/ 200 Itself Noindex Fail: noindex
/search?q=drill 200 Itself Blocked in robots.txt Fail: blocked
/guides/choosing-a-drill/ 200 Itself Indexable Pass

The checks to automate:

  1. Fetch every URL in the sitemap and record the status. Anything other than 200 fails.
  2. Read the canonical tag. Anything not self-referencing fails.
  3. Read the robots meta tag and header. Noindex fails.
  4. Test the URL against robots.txt. Blocked fails.
  5. Compare in the other direction: canonical, indexable pages on the site that are missing from the sitemap.
  6. Confirm the file is within the size and URL limits, and that its dates are plausible.

The fifth check matters as much as the others. A sitemap can be perfectly clean and still omit the newest section of the site.

Fix failures at the source. If the sitemap is generated from the content database, the generator should apply the rules: published, canonical, indexable. Editing the file by hand does not last.

Two limits on what exclusion achieves.

  • A URL removed from the sitemap stays in Google’s index until something else changes, such as a noindex rule, a redirect, or the page’s removal.
  • A URL excluded from the sitemap can still be crawled if links point to it.

To take a page out of search, use the mechanism made for that. The sitemap only stops recommending it.

How a sitemap helps a very large site get its URLs discovered in the first place, and how to partition one, is covered in the guide to how an XML sitemap supports URL discovery on a very large website.

The rule: if you would not want the URL to appear in search results exactly as written, it does not belong in the sitemap.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *