The command-line check on Google's own page accepts names that any rented Google Cloud machine receives. We repeated the test on November 9, 2026. What to change in your firewall rule, and the opposite mistake: blocking the real Googlebot.
If your firewall lets Googlebot in using the reverse DNS steps from Google's page, anyone with a rented Google Cloud machine can walk through the same door. A company called SenseGap flagged the problem on November 5, 2026, and we repeated the test on November 9: it holds up. The fix takes a morning. Swap the rule for the IP list Google itself publishes, or require a name ending in .googlebot.com.
In the second of the four steps. The page has you look up the IP's reverse name, accept an ending of googlebot.com, google.com or googleusercontent.com, look that name up going forward, and see whether the same IP comes back. The third ending is the trouble: every Google Cloud virtual machine gets a name under bc.googleusercontent.com, so whoever rents one meets all four steps without doing anything clever.
Further down the same page, a table says something else. There, Googlebot uses names under googlebot.com, and googleusercontent only shows up as gae.googleusercontent.com, set aside for user-triggered fetchers such as the site verifier. The steps carry a long list because they cover all of Google's crawlers at once. Anyone who copied them just to admit Googlebot also took a suffix they never needed. We found no comment from Google on this as of November 9, 2026.
Three out of five Google Cloud IPs passed all four steps. The sample is small on purpose, since we only wanted to know whether the door exists. We queried DNS and the public lists and sent no request to any site.
The field report comes from SenseGap, which sells a bot dashboard, and it is worth reading with that in mind. Over eight days, from October 28 to November 5, 2026, its site received 89 requests from Google Cloud machines that claimed to be Googlebot and passed the check. They were after files like .env and .git/config, not pages. That is four sites and sampled data: enough to know it happens, not how often it happens on yours.
Because a rule that admits the crawler usually puts it beyond whatever holds other visitors back: rate limits, challenges, country blocks. In October 2026, Gary Illyes of Google said on a podcast that scrapers pose as Googlebot precisely because sites clear the way for it, and that changing a user agent takes seconds.
There is a quieter cost too. Anyone reading logs to understand crawling starts counting fake visits as Google's, and a "heavily crawled" folder may just be someone sweeping the server for credentials.
Blocking the right crawler tends to cost more than admitting the wrong one. According to Google's documentation, 403, 404 and the other 4xx codes, except 429, are treated as content that does not exist, and a URL already indexed drops out of the index. The same page asks you not to use 401 or 403 to slow crawling; 429 and 503 do that job.
The VLC site gave an example, reported on November 5, 2026. It returned 418, a 4xx code, and the result stayed in Google with a catch that Gary Illyes described in a LinkedIn comment: Search held on to the URL for a while because it is so important and, not knowing what was on the page anymore, was left with "no snippet, no favicon, no nothing". The link and the last good title remained. The reason for the 418 has not been made public, and nothing suggests a firewall was involved.
Two things follow. Google does not document which URLs get this protection or for how long, so do not plan around it. And a result with no description and no icon is a good reason to open URL Inspection: the office browser may get a 200 while the crawler gets something else.
It does, and they have their own lists. OpenAI publishes one per crawler (gptbot.json, searchbot.json, chatgpt-user.json), Perplexity does the same with two files, and Anthropic keeps a single one without saying which of its crawlers it covers. On SenseGap's four sites, 32% of visits claiming to be GPTBot and 87% of those claiming to be PerplexityBot failed the check published by the company itself. For OAI-SearchBot it was 1%. These are one vendor's numbers on a few sites, useful as a warning and not as a market average.
One caution before acting: Anthropic warns that blocking its IPs may not guarantee an opt-out, because its crawler can no longer read your robots.txt. An IP list tells you who is who. Saying what each one may read is still the job of robots.txt.
3,000 pages a month on the Basic plan, no card.