usdigitalmarketingindex.com SEO
How can server logs distinguish verified Googlebot requests from crawlers impersonating Googlebot?
Updated November 15, 2026
By checking the IP address of each request, not its user-agent string. Any client can send a user agent that says Googlebot. Only Google’s crawlers come from Google’s addresses. Google documents two ways to confirm this: a reverse DNS lookup on the IP followed by a forward lookup on the returned hostname, or matching the IP against the address ranges Google publishes (Google’s instructions for verifying its crawlers). A request that claims to be Googlebot and fails both checks is an impersonator, and it should be left out of any analysis of how Google crawls the site.
Why user-agent strings can be spoofed
The user agent is a header the client writes. A scraper, a monitoring tool, or a competitor’s crawler can set it to Googlebot’s published string in one line of configuration, in the hope of receiving the access a site grants to search engines. A log filtered on the user agent alone therefore counts every pretender as Google.
Reverse and forward DNS verification
Google’s documentation gives the manual method in four steps.
- “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.”
- “Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”
- “Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name.”
- “Verify that it’s the same as the original accessing IP address from your logs.”
The documentation’s own example shows a passing check:
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
The forward lookup is the step that cannot be skipped. An operator who controls an IP address can set its reverse DNS record to any name, including one ending in googlebot.com. Only the forward lookup, answered by Google’s own name servers, confirms that the name maps back to that address.
For large volumes, the same page describes the automatic method: match the crawler’s IP address against the list of published Google IP addresses, which Google provides as JSON files for its different crawler groups. This avoids a DNS query per log line. The published lists change, so refresh them on a schedule.
Logging request source
Verification depends on the log holding the right address.
- Log the client IP for every request, along with timestamp, method, URL, status, response size, and user agent.
- If the site sits behind a CDN, load balancer, or reverse proxy, the address in the default field is the intermediary’s. Configure the server to record the original client address from the header your intermediary sets, and accept that header only from the intermediary’s own addresses, since a forwarded header can also be forged.
- Keep logs long enough to investigate. A question about last month’s crawling cannot be answered from a week of logs.
Audit interpretation and caveats
Run the audit with this checklist. Each row gives the input, the action, and the pass or fail condition.
| Input to verify | Action | Pass or fail |
|---|---|---|
| Log sample filtered to user agents containing Googlebot | Extract the distinct client IPs | Pass if the IP field holds real client addresses, not the proxy's |
| Each distinct IP | Reverse DNS lookup | Pass if the hostname ends in googlebot.com, google.com, or googleusercontent.com |
| Each returned hostname | Forward DNS lookup | Pass if it resolves to the original IP |
| Alternative for volume | Compare IPs with Google's published ranges | Pass if the IP falls inside a published range |
| Published ranges file | Check its retrieval date | Fail if stale |
| Requests that failed | Label as unverified | Pass when they are excluded from crawl analysis |
Read the results with four caveats.
- Google runs several crawlers and fetchers. The documentation groups them, and user-triggered fetchers resolve to different hostnames from the main crawler. A request that is from Google is not necessarily from the crawler that feeds Search indexing.
- A failed verification shows the request is not from Google. It does not show who sent it.
- Results depend on the day. DNS records and published ranges change, so record when the check was run.
- Do not block on the user agent in either direction. Blocking everything that says Googlebot and fails a quick check risks blocking the real crawler if the check was wrong. Verify, then act on the IP.
What to do with the two groups: analyze crawl behavior using verified requests only, and treat unverified ones as ordinary bot traffic, subject to whatever rate limits apply to unknown clients.
Verification is the first step of any log investigation. Once the real Googlebot requests are isolated, a change in their volume can be examined, as described in the guide to what an unexpected rise in Googlebot requests can reveal in server logs.
The rule: the user agent tells you what a request claims to be, and the IP, checked both ways, tells you what it is.
Add your comment to this answer