Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How can JavaScript faceted filters expose too many crawlable URL states?

By writing every filter combination into the URL and rendering a link to every further combination. A listing with a handful of filters has thousands of possible states, and if each state has its own address and each page links to its neighbors, a crawler following links is led through all of them. Google’s documentation describes the result: faceted navigation based on URL parameters “can generate infinite URL spaces”, and because the URLs look new, crawlers “can’t determine whether the URLs are going to be useful without crawling first” (Google’s guidance on faceted navigation). The answer is to decide which few filter states deserve to be pages, expose those as real links, and keep the rest from becoming crawlable addresses.

Filter state and URL proliferation

A JavaScript filter component has to keep its state somewhere. Each choice has consequences.

Where state is kept URLs created Crawl exposure
In memory only None; one URL for the listing None, and filtered views cannot be shared or bookmarked
In the fragment, after a hash None that crawlers treat as separate None; Google's guidance says it generally does not support fragments in crawling and indexing
In query parameters, updated with the History API One per combination Depends on whether links to them are rendered
In query parameters, with every option rendered as a link One per combination Full; every state is discoverable

The fourth row is where the problem lives. Count a modest example, a shoe listing with these filters.

Filter Options States including "none selected"
Brand 12 13
Size 15 16
Color 10 11
Price band 5 6
Sort order 4 4

Single-selection combinations: 13 x 16 x 11 x 6 x 4 = 54,912 URLs for one category. Allow several options per filter, let parameters appear in any order, and add pagination, and the figure has no practical ceiling.

Recognizing useful landing states

A small number of filter states match how people search. Those are worth having as pages.

A filter state is a candidate landing page when:

  • People search for it by name: waterproof trail shoes, a brand plus a category.
  • It contains a meaningful number of products that will remain available.
  • It can carry its own title, heading, and introductory content.
  • It is a single filter, or at most a stable pair.

A state is not a candidate when it is a size, a price band, a sort order, a combination of three or more filters, or a state with few or no results.

State Candidate? Reason
Category plus brand Yes Searched by name; stable range
Category plus a defining attribute, such as waterproof Yes A recognized subtype
Category plus size No Narrows a list; not searched as a page
Category plus color plus size plus price No A permutation
Any state, sorted by price No Same items, reordered
A state with zero results No Empty

Restricting irrelevant combinations

Treat the two groups differently in the markup.

Candidate states get real, clean URLs and real links.

<a href="/shoes/trail-running/waterproof/">Waterproof</a>
<a href="/shoes/trail-running/brand/northgate/">Northgate</a>

These pages are self-canonical, listed in the sitemap, and given their own titles and headings.

Other states are applied without creating crawl paths.

<button type="button" data-filter="size" data-value="10">Size 10</button>

A button with a handler filters the list for the visitor and offers no href for a crawler to follow. The state can still be written to the URL with the History API, so that the view can be shared.

Rules and trade-offs for the non-candidate states:

Measure Effect Trade-off
Render filter controls as buttons, not links No crawl paths are created The views are not discoverable, which is the intent
Keep state in the fragment Crawlers treat every state as the base URL Cannot become a landing page later without changing the scheme
Disallow the filter parameters in robots.txt Stops requests for those URLs Blocked URLs cannot pass a canonical or noindex signal; they may still be listed if linked from elsewhere
Canonical tag from filtered URLs to the unfiltered listing Consolidates signals where the URLs do get crawled The URLs are still crawled; a hint only
Return 404 for combinations with no results Removes empty states Needs server-side handling
Fix parameter order and reject unknown parameters Prevents duplicates of the same state Engineering effort

Google’s guidance offers the first of the blocking options plainly: “Use robots.txt to disallow crawling of faceted navigation URLs”, observing that there is frequently no good reason to allow crawling of filtered items. The cleanest approach, though, is not to render the links at all, so that there is nothing to block.

Whatever the mechanism, be consistent. Internal links, canonical tags, and the sitemap should all point at the same small set of URLs.

QA of selected facets

Verify both halves: the chosen states are reachable, and the rest are not multiplying.

Check How Pass condition
Candidate links Rendered HTML of the category page Each candidate state is an anchor with a clean path
Non-candidate controls Rendered HTML Buttons or controls without href
Crawl size Crawl the category with rendering enabled URL count equals the category, its pagination, and the candidate states
Canonicals Inspect a filtered URL Points to itself for candidates; to the base listing for others
Empty states Request a combination with no results Not-found status, or noindex
Parameter order Request the same state with parameters reordered Redirects to one form, or canonicalizes to it
Server logs Verified Googlebot requests to the category Requests concentrate on base, pagination, and candidate URLs
Search Console Page indexing report filtered to the category path No large count of filter URLs in duplicate or not-indexed states
Visitors Use the filters in a browser Filtering, sharing a filtered view, and the back button all work

The crawl size check is the decisive one. If a crawl of one category returns tens of thousands of URLs, the filters are exposed.

Sort orders are a particular kind of non-candidate state, since they change nothing but sequence. Their handling is covered in the guide to how sorting parameters produce duplicate ecommerce category URLs.

The rule: a few filter states are pages and get links. The rest are controls and get none.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *