Is GPTBot Actually Crawling You? How to Verify AI Crawler Traffic in 2026

Is the AI Crawler Traffic in Your Logs Actually Real?

The reason is simple enough to be uncomfortable. A user-agent string is a claim, not a credential. DataDome tested roughly 698,000 reachable sites in February 2026 with a spoofed ChatGPT-style user-agent, and 79.7% let it straight through without a block or a challenge. If your analytics counts a header, it is counting whatever the requester chose to type.

This guide assumes you already have bot traffic and want to know how much of it is honest. If you are earlier than that, our primer on what GEO is and how it works is the better starting point, and the brand tracking guide covers the visibility side that logs cannot see.

Why Does a User-Agent String Prove Nothing?

Because the requester writes it. User-Agent is a free-text header, set by whoever makes the request, and nothing in HTTP checks it against anything. HUMAN Security's Satori team measured 5.7% of all traffic presenting an AI crawler user-agent as spoofed, peaking near 7.7% on individual days — and for ChatGPT-User specifically, 16.7% of requests were impersonators.

The good campaigns don't look sloppy either. Satori documented operators routing through the correct autonomous systems, using addresses adjacent to official ranges, and spreading low-volume requests across serverless functions so no single source stands out. That is not a curl script with a copied header. It is someone who read your detection rules first.

There's a quieter version of the same problem, and it's in the live stream above from our own site. Every row is a bot pulling blog URLs in sequence, several per second, and every row carries a plain Mozilla/5.0 (Macintosh; Intel …) browser string. Platform and bot type are both blank, because nothing in the request declares anything. Those requests are not fake exactly — they simply refuse to say who they are, which leaves them in a triage bucket rather than in a metric.

What Did We Find Verifying 90 Days of Our Own Logs?

We ran every AI crawl event from 90 days of production against published vendor IP ranges and reverse DNS. Of 7,202 events across 1,708 bot-and-IP pairs, 71.5% verified, 21.8% were spoofed, 4.3% came from edge-run agents we cannot verify by address, and 2.4% carried identities with no published feed to check against.

The split by identity is where it gets useful. Bytespider, GrokBot, DeepSeekBot, CohereBot and Google-Extended have never once sent us a verifiable request — every appearance was an impostor. PerplexityBot failed 52% of the time and GPTBot 42%. The genuine signal is solid on the other side of the line: Amazonbot verified 92%, ChatGPT-User 85%, ClaudeBot 83% and OAI-SearchBot 78%.

How Do You Actually Verify an AI Crawler?

Two mechanisms cover almost everything. Most AI vendors now publish their crawler IP ranges as JSON at a fixed URL, so verification is a set-membership test against a list you refresh daily. Google and Amazon instead expect forward-confirmed reverse DNS: resolve the address to a hostname, confirm the hostname belongs to the vendor's domain, then resolve that hostname forward and check it returns the same address.

Every feed in the table below returned HTTP 200 when we fetched it on 23 August 2026. None of this requires a vendor relationship, an API key, or a security product — it is six HTTP requests and a nightly cron.

The Trap That Breaks a Naive IP Allowlist

Here is the mistake we nearly shipped. While testing, one of us fetched yozigo.ai through Claude Code and watched the row land in the database a second later, correctly classified. The source address belonged to Bharti Airtel in Bengaluru — a consumer broadband line, not Anthropic's infrastructure. Agent tools that run on the user's machine fetch from the user's connection, so a strict allowlist would have marked a completely genuine AI-assisted visit as an impostor.

That single observation is why verification has to be tiered rather than binary. The four tiers below each need different evidence, and treating them as one class produces false confidence in one direction or false accusations in the other.

How Should Tier C Agents Be Handled?

Label them, don't block them. Tier C is the fastest-growing slice of AI-assisted traffic and the only honest verification path is cryptographic: Web Bot Auth, an IETF draft built on RFC 9421 HTTP Message Signatures, where the agent signs each request with a key whose public half is published at a known endpoint and identified by a Signature-Agent header.

Adoption is running ahead of the paperwork. The working group had no adopted documents as of August 2026, yet Cloudflare, Akamai, AWS WAF, HUMAN and Vercel already verify these signatures in production. If you sit behind a major CDN, you can switch on signature verification today and get a real identity for the agents that sign — while everything unsigned stays where it belongs, in triage.

Two Other Ways a Bot Dashboard Misleads You

Spoofing is the loud problem. Two quieter ones cost us more accuracy. The first was a classification error of our own making: we had GoogleOther and Google-CloudVertexBot mapped to "Gemini citations", which sounds right and isn't. Their reverse DNS resolves to ordinary **crawl-*.googlebot.com** infrastructure, and they are general-purpose fetchers, not answer grounding. That mislabel accounted for 691 events — 9.1% of our AI crawls — all of it counted as citation evidence.

The second was self-crawl pollution. Our own crawlers, running under four different names, generated around 5,724 events, or 27.5% of all bot traffic hitting our site. Nothing excluded them. Every SEO auditing tool, uptime monitor and internal scraper you run does the same thing to your numbers, and none of it is AI interest in your brand.

So how much of a typical AI bot report survives contact with verification? In our case: remove the spoofed requests, remove the mislabelled Google fetchers, remove our own crawlers, and roughly a third of the original figure was left standing. The full catalogue of self-inflicted measurement problems is worth a read in our guide to how not to write content for AI citations.

Does a Crawl Mean You Were Cited?

Usually not, and the arithmetic that suggests otherwise is a trap we fell into first. Matching 30 days of grounded citations against crawl events within a 72-hour window gave a 64.6% match rate, which reads like proof. Then we shuffled the timestamps and re-ran it: random data produced 58.0%. The real lift was 1.11x — statistically indistinguishable from coincidence, because at that window width almost everything matches almost everything.

Tightening to a one-hour window separates the platforms sharply. OpenAI's grounding fetches showed a 3.78x lift over the permutation baseline (z = 13.1) and Perplexity's 7.57x (z = 5.9). Google AI Overviews came in at 0.99x and Gemini at 0.77x — no signal at all, and not a bug to be fixed. Those surfaces answer from the search index without fetching your page, so no crawl event exists to find. For Google surfaces the honest instrument is Search Console's generative AI reporting, not your access log.

What Is All That Crawling Actually Worth?

Less than the volume implies. Cloudflare's network-wide measurement for 1–7 August 2025 put Anthropic's crawl-to-refer ratio near 50,000:1 across all industries, OpenAI at 887:1 and Perplexity at 118:1. News and publications fared better — 2,500:1, 152:1 and 32.7:1 — but the direction holds everywhere: crawling vastly outpaces referral.

The purpose mix explains it. Cloudflare attributed nearly 80% of AI bot crawling to model training rather than answering a live question, with user-action and undeclared purposes together under 5%. Ratios have compressed through 2026 as the assistants started sending more traffic — Cloudflare Radar data reported by SEOmator for the 28 days to 21 July 2026 has Anthropic around 2,237:1 and OpenAI 217:1 — though that reading is second-hand and worth treating as a trend line rather than a measurement.

What Changes Once You Verify?

Every number moves, and not all of them downward. Applying verification, self-crawl exclusion and proper sessionisation to our own brand cut the AI crawl count by more than half while raising measured AI-driven visits thirty-fold — because the old pipeline had been counting fake crawls as engagement and misclassifying real AI-referred humans as bots.

How Do You Start Checking Your Own Logs?

Five steps, in order, and the first three cost nothing but engineering time. One: pull the six vendor JSON feeds nightly and store them with a change guard, so a feed that suddenly shrinks by a fifth doesn't silently convict every real crawler overnight. Two: for Google and Amazon, implement forward-confirmed reverse DNS rather than trusting the hostname alone.

Yozigo does this natively — tiered verification against live vendor feeds, self-crawl exclusion, crawl-to-citation attribution restricted to the platforms where the signal is real, and a live stream of every bot request hitting your site. If you want the visibility side alongside it, our comparison of the dedicated GEO platforms covers the wider category. See how it works.

Frequently Asked Questions About AI Crawler Verification