Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How can client-side routing prevent crawlers from discovering distinct page URLs?

In two ways. The application can change what is on screen without giving each view a real address, or it can give views addresses that crawlers have no link to follow. A single-page application swaps content with scripts, so a visitor moves from the product list to a product without the browser requesting a new document. If those views live behind URL fragments, or are reached only through buttons with click handlers, a crawler sees one URL and nothing pointing anywhere else. Google’s documentation gives the remedy in a sentence: “For single-page applications with client-side routing, use the History API to implement routing between different views of your web app” (Google’s JavaScript SEO basics).

URL state and navigation

A page that can be found in search needs three properties.

  1. Its own URL, with the view identified in the path or query string.
  2. A server response for that URL when it is requested directly.
  3. A link to it from some other page, as an anchor element with an href.

Client-side routing can break any of them.

Pattern What a crawler finds
Views switch with no change to the URL One page; the other views do not exist as addresses
Views identified by a fragment, such as /#/products One URL; the part after the hash is not treated as a separate page
Real paths, with navigation by click handlers only The paths exist and nothing links to them
Real paths and real links, and the server returns an error for a direct request Links are found; the pages cannot be fetched
Real paths, real links, server responds to direct requests Each view is a page

History API and anchors

The documentation is specific about fragments: “don’t use fragments to load different page content”. It labels fragment-based navigation a bad practice because Googlebot cannot reliably resolve those URLs.

Bad pattern, with fragments:

<nav>
  <a href="#/products">Products</a>
  <a href="#/services">Services</a>
</nav>

Good pattern, with real paths and the History API handling the transition:

<nav>
  <a href="/products">Products</a>
  <a href="/services">Services</a>
</nav>

<script>
  document.querySelectorAll('nav a').forEach(function (link) {
    link.addEventListener('click', function (event) {
      event.preventDefault();
      history.pushState({}, '', link.getAttribute('href'));
      renderView(link.getAttribute('href'));
    });
  });
</script>

In the good pattern each link is a normal anchor with a real path. A visitor’s click is intercepted and handled in the page, so the experience stays fast. A crawler, which does not click, reads the href and requests the path. The function renderView stands for the application’s own view loader.

Internal path discovery

Google’s documentation describes how links are collected: Googlebot “parses the response for other URLs in the href attribute of HTML links and adds the URLs to the crawl queue”, and after rendering it “parses the rendered HTML for links again”. So links added by script are acceptable, provided they end up in the rendered HTML as anchors with href values.

Check that every route is linked.

  • Navigation, listings, and pagination use anchors with real paths.
  • Views reachable only by typing in a search box, choosing from a dropdown, or pressing a button have no crawl path. Give them links, or list them in the sitemap.
  • The sitemap contains every route you want found, as a second route to discovery.

The server side matters equally. When a crawler requests /products/trail-shoe directly, the server must answer with a 200 and a document that produces that view, not an error and not the home page’s content.

Crawl simulation

Test both halves: discovery through links, and direct requests.

Test case How Pass condition
Direct request to a deep route Request the URL in a fresh browser tab, or with a command-line fetch 200 status; the correct view renders
Direct request to a nonexistent route Request a made-up path A not-found status, not the app shell with a 200
Links in rendered HTML URL Inspection live test on a listing page; search the rendered HTML for anchor elements Each item appears as an anchor with a real path
No fragment routes Search the rendered HTML for href values beginning with a hash None used for navigation between views
Crawl without scripts Crawl the site with JavaScript disabled Shows what is linked in served HTML
Crawl with scripts Crawl with JavaScript rendering enabled Reaches every route
Distinct content per route Compare the rendered title and main heading of several routes Each differs
Back and forward Navigate in a browser, then use the back button The URL and the view stay in step

Compare the two crawls. Routes found only with rendering depend on Google’s rendering step for discovery. Routes found in neither are invisible.

The second row guards against a related fault. An application that serves the same shell for every path returns a 200 for addresses that do not exist, which creates soft errors.

Also check titles and canonical tags. An application that keeps the first page’s title and canonical tag as the visitor moves between views is telling crawlers that every route is the first page.

What an anchor is, and why a click handler on another element does not substitute for one, is examined in the guide to why crawlable HTML anchor links are preferable to click-handler-only navigation.

The rule: every view worth finding has a real path, answers a direct request, and is the target of a real link.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *