Fake Googlebot in Your Access Log: How to Tell It From the Real Crawler

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/130.0.0.0 Safari/537.36

What it is

Googlebot is the user agent of Google’s web crawler, and one of the most copied: any client can send it, because the User-Agent header is whatever the sender writes. The strings above are two of the forms Google documents for Googlebot (the rarer short form, and the desktop crawler with a Chrome version in place of Google’s W.X.Y.Z); a fake one is byte-for-byte the same. Google’s crawler documentation says it plainly: the user agent string can be spoofed.

On our own server the difference is stark. In the last 30 days, thousands of requests carried a Googlebot user agent from addresses outside Google’s published crawler ranges, while only a handful came from Google itself (the counts are above). The impostors did not crawl pages. The paths they asked for most were /.env and copies of it such as /.env.production and /www/.env, /.aws/credentials, /.ssh/id_rsa, /.npmrc and /config/database.yml — files that hold secrets when a deployment leaves them inside the web root. Even the most requested of these was asked for only a couple of dozen times, so the thousands of requests were spread over hundreds of different paths: a wordlist, not a crawl.

Who sends it

Two different parties. The real Googlebot is Google’s crawler, and Google documents how to confirm it. Its common crawlers, Googlebot among them, resolve to names in googlebot.com; for a manual check, a reverse DNS lookup of the address must give a name in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name must return the same address. For automated checks Google publishes its ranges as JSON files, Googlebot’s in common-crawlers.json. Our counter uses that file.

Everyone else using the string is pretending. On our server the spoofed requests came from about ten addresses, hundreds of requests per address on average, which is the pattern of scanning scripts rather than of many independent clients.

Is it a threat?

The request itself is reconnaissance (MITRE ATT&CK T1595.003, wordlist scanning). The risk is in what it finds: a readable .env or credentials file hands over keys at once (T1552.001, credentials in files), and the sender does not need to be clever to use them. A fake Googlebot that only received 404 responses learned nothing; one that received a 200 for a secret file is an incident.

How to handle it

1. Find requests that claim to be Googlebot and check their addresses against Google’s ranges, or with reverse and forward DNS.
2. For every address that fails, look at the paths and the status codes. Any 200 for a file such as /.env or /.aws/credentials means the file was served: move it out of the web root and rotate every secret in it.
3. Block the impostors by address check, never by user agent alone, so Google’s crawler keeps working.

List the addresses that sent a Googlebot user agent, busiest first, then check one of them:

grep -i 'googlebot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head
host 203.0.113.9   # real Googlebot: a name in googlebot.com, google.com or googleusercontent.com

Build an nginx include from Google’s published ranges and refuse a Googlebot user agent from anywhere else:

curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
  | jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' > /etc/nginx/google-crawler-ranges.conf

# http block: 1 when the client address is in Google's published crawler ranges
geo $google_crawler {
    default 0;
    include /etc/nginx/google-crawler-ranges.conf;
}
map "$http_user_agent|$google_crawler" $fake_googlebot {
    default 0;
    "~*googlebot.*\|0$" 1;
}

# server block
if ($fake_googlebot) {
    return 403;
}

Refresh the include on a schedule, since the file carries its own creationTime and Google’s ranges change. The free log analyzer groups an access log’s scanners by source and flags any secret-file probe that got a 200.