How to Detect Bots and AI Agent Traffic on Your Website
In the first post of our agentic browsing series, we explained what an agentic browser is and why it matters. The second explored the hidden risks agentic browsing poses to your analytics stack. Now we’re going to get practical and answer the question: how do you actually detect bots and agentic browsers?
What follows is drawn from our recent session on AI agent and bot behavioral intelligence featuring Snowplow Co-Founder & CEO Alex Dean and Senior Product Manager Nick Stanchenko. Between them, they laid out a detection framework built on what Snowplow sees across more than 2 million websites and over a trillion events per month.
By the end of this post, you’ll understand why detection beats blocking, three ranked methods for identifying agent traffic, and how to build a composite bot score that separates your data into actionable segments. We’ll also show where each method lives in Snowplow today, and how to measure the AI agents that never show up in your analytics. If you’re trying to figure out how to detect bots in your analytics, separate real user behavior from bot activity, and actually do something useful with that knowledge, this is the framework to follow.
Why Detection Beats Blocking
To date, the default industry response to bots has been defensive with companies deploying web application firewalls (WAFs), edge protection, and bot blocking. For malicious bots, these tactics make absolute sense. But when it comes to AI agents, blocking them creates a blind spot. You can’t understand or optimize for traffic you’ve shut out entirely.
Why is this an issue? Well, McKinsey projects that 20–50% of traditional search traffic is at risk of displacement by AI-powered search. But the key thing here is that this traffic is expected to bring $750 billion in consumer spend by 2028. As we’ve touched on before, HUMAN Security has documented a four-digit percentage increase in agentic traffic over the past year alone. So it’s clear, blocking all of this growing traffic means blocking commercial intent.
To address this, Alex Dean discussed the importance of playing offense whereby you proactively detect the agents, analyze what they’re doing constructively, in real time, while they’re still on site. So rather than treating every non-human visitor as a security risk, you start treating AI agents as first-class visitors whose behavior is worth modeling and responding to. A traditional bot detection solution focuses on blocking threats and minimizing the negative impacts on user experience. That's necessary but insufficient. This shift from defense to offense is the foundation everything else in this post builds on.
Three Bot Detection Methods, Ranked by Effectiveness
During the webinar, Nick Stanchenko framed detection as a spectrum. The methods below apply to bots and agents that show up in your browser-based analytics, those that execute JavaScript and generate client-side events. AI agents that fetch pages without running any scripts are a different problem, which we’ll cover later in this post.
The three bot detection techniques below are arranged by how hard they are for a bot to fake and also how difficult they are for you to implement. You can think of it as a trade-off between simplicity and reliability. Each method catches a different tier of bot.
Method 1: Identity Signals: Who They Appear to Be
This is the most simple check. If a user agent string says GPTBot, it’s probably GPTBot. If automation flags are turned on in the browser, or the screen size is anomalous, or the browser fingerprint has irregularities, those are signs you’re looking at a bot rather than a human in a traditional browser.
New standards are emerging here too. RFC 9421 HTTP Message Signatures let an agent sign its requests to prove who it is. Web Bot Auth builds on this, and in September 2026 the IETF’s Web Bot Auth working group adopted its first draft, HTTP Message Signatures for automated traffic. It’s still a draft, and few agents sign their requests yet. As more do, you’ll be able to check that a bot claiming to be ChatGPT really is ChatGPT, and not a scraper borrowing the name.
However, the limitation with this method is obvious: this only catches bots that self-identify or are sloppy. If a bot or agent says it’s Chrome, it doesn’t necessarily mean it’s Chrome. Identity signals are the easiest layer to fake.
Method 2: Network-Origin Signals: Where They Come From
The next layer examines IP addresses and the autonomous systems they belong to. If traffic originates from an internet service provider, it’s probably a human. If it’s coming from a cloud provider like AWS or DigitalOcean, it’s probably a bot. This touches on what Nick calls the sweet spot for reliability versus complexity, meaning it’s harder to spoof than a user agent string, but simpler to implement than full behavioral analysis.
Meta’s Muse is a live example. DataDome found 99.3% of its sampled requests came from cloud and CDN infrastructure, even though it doesn’t identify itself as an AI agent.
The caveat here is that VPN users create false positives. A real user browsing through a cloud-hosted VPN will look like a bot at the network-origin level. This is why no single method is sufficient on its own.
Method 3: Behavioral Signals: How They Act
This brings us to our final method, which is the hardest layer for bots and agents to fake. Whether it’s a scraper, a search agent, or an AI assistant acting on behalf of a user, the underlying constraint is the same: efficiency. Bots simply cannot afford to behave like humans and wander aimlessly around pages. They navigate directly, process content quickly, and move on.
The behavioral signals to watch for include the presence or absence of mouse movements or screen taps, scrolling patterns, and click or tap intervals. We know humans exhibit irregular timing and indirect paths. But when you look at bot and agent behavior, they tend to browse pages in rapid succession with regular intervals.
HUMAN Security's threat intelligence team found that ChatGPT's agent, for example, always moves its mouse in increments of 0.25 pixels, with smooth, linear traces that are visually distinct from the messy, unpredictable paths humans produce. Combined with device fingerprinting, these behavioral signals create a layered picture that’s extremely difficult to spoof.
The more sophisticated the agent navigates and the more it tries to mimic human behavior to handle multi-step workflows across web pages, the more computational overhead it incurs. Efficiency and stealth are at odds, making behavioral signals the most reliable detection method.
Combining Signals Into a Composite Bot Score
Nick stresses the point that you cannot expect to look at your analytics data and separate human users and automated bots with 100% accuracy. It’s a best-effort, diminishing-returns exercise. He suggests establishing a composite bot score, which is a probability that combines all three signal types into a single metric. This is the core of effective bot detection. There isn’t a binary yes/no, but a confidence score.
By this, he means weighting signals by how reliable they are.
Crucially, the weights need to be tunable per business. For example, marketplaces typically see short-session price crawlers that hit a single listing and disappear. Media companies, on the other hand, see longer-running bots that traverse content libraries. As agentic browsing scales, a one-size-fits-all threshold doesn’t work.
Once you have a bot score, the downstream value is significant. You can segment analytics by threshold to isolate human KPIs like bounce rate, click-through rate, and conversion rate. You can filter agent traffic out of the sensitive data feeding your ML models. These models were designed to learn from human behavior, not from agents browsing every page in rapid succession.
As we explored in the second post in this series, tangled data doesn’t just skew dashboards, it degrades every decision built on top of it.
The Invisible AI Problem: Why Your Analytics Can’t See Most AI Agent Traffic
Everything we’ve discussed so far assumes the AI agent is running in a browser or uses a browser-like automation framework that executes JavaScript and therefore shows up in your analytics. Most don’t. The majority of AI agents - training crawlers, AI search agents, retrieval agents - fetch web pages without ever executing JavaScript or triggering your client-side tracking. As a result, they’re completely invisible to traditional web analytics tools like GA4.
During the session, Nick illustrated this concept as an iceberg. Consider the full journey when a user gets an answer from ChatGPT that references your content. First, a training crawler (GPTBot) visited your page and added it to the model’s training data. Later, a user fetch agent (ChatGPT-User) pulled the current version of the page to answer someone’s question. Then the user got their answer from the chatbot. Finally, the user clicked through to your site. That last step, the referral with utm_source=chatgpt, is the only part your analytics sees. The rest is below the waterline, visible only in server logs.
Why does this matter? Because optimizing for AI requires data on the full picture. You need to know which platforms are most active on your site, which content they’re indexing, and which pages they’re ignoring. Without server-log data, you’re basically making decisions on the tip of the iceberg. But if you deploy a standalone server-log tool, you end up creating a data silo. So the moment you need to answer a question that spans both datasets, like “which AI platforms are most active on my highest-engagement pages?”, you’re stuck. The solution, therefore, is a unified platform that ingests both client-side events and server-side logs into the same warehouse, with schemas that support different data types natively.
The Hardest Problem: Detecting Mid-Session Handoffs
Historically, a session on your site was either human or bot. Agentic browsing breaks that assumption entirely.
Let’s say a user visits your site. They browse normally, then hand off to Perplexity Comet to book a flight or fill out forms for a subscription renewal. At this point, the agent takes over, and your analytics sees one continuous visit.
In this scenario, behavioral data can tell you the story of the handoff if you know what to look for. During the human phase, you’ll see wandering across the page, irregular scrolling, indirect paths, engagement with promotions and ads. But when the agent takes control, you’ll see the pattern changes sharply. The agent will navigate directly to target elements, with consistent timing between actions, and won’t engage with anything peripheral. As Nick put it:
“the agent just processes the content, doesn’t click on anything considered secondary to its primary objective, and ignores all your ads and promotions.”
Detecting this handoff takes event-level behavioral data, scored as the session goes on. When the score flips from “human” to “agent,” you can flag the transition and report on the two phases separately. Your conversion rate then reflects what people did, and the agent’s part of the visit stays visible for analysis rather than skewing your human metrics.
Where Each Method Lives in Snowplow Today
Here’s how the three layers map to what you can run in Snowplow now.
Identity signals. The YAUAA enrichment parses the user agent string and names the agent. The IAB enrichment checks the IP address and user agent against the IAB/ABC International Spiders and Bots List. For AI agents, the agent classification enrichment adds who operates the agent, such as OpenAI, and what it’s for: AI training (GPTBot), AI search indexing (OAI-SearchBot), fetching a page for a user’s question (ChatGPT-User), link previews or traditional search indexing.
Network-origin signals. The ASN lookup enrichment checks each event’s network against a list of cloud hosting, data center and abusive networks. A match is flagged as a likely bot, not a definite one, because VPN users share those networks. Treat it as one input to your score.
Behavioral signals. The bot detection plugin for the JavaScript tracker uses FingerprintJS BotD to flag automated browsers such as Selenium and headless Chrome. The finer signals in this post, like click timing and mouse paths, are something you model yourself. Snowplow gives you every raw event to do it, in your own warehouse.
Combining them. The bot detection enrichment rolls YAUAA, IAB, ASN and the plugin into one bot field. If any source says bot, the event is a bot. That makes it a good filter for your human analytics. For the weighted, tunable score Nick describes, build it in your warehouse on top of these fields.
Agents that don’t run JavaScript. None of the above can see an agent that never loads your tracker. That’s the gap Agent Intelligence fills. CDN trackers for Cloudflare, CloudFront and Fastly capture every request at the CDN, including agents that skip JavaScript. They send events through the same pipeline as your JavaScript tracker, and the Agent Analytics dbt package combines both into BI-ready tables in your warehouse. Agent Intelligence is an add-on and needs a Snowflake warehouse. Read more in the launch post.
Bot filtering and Agent Intelligence do opposite jobs. Bot detection flags automated traffic so you can take it out of your human metrics. Agent Intelligence keeps known AI agents in so you can study them.
How to Measure AI Crawler and Referral Traffic by Page
Once agents and people sit in the same warehouse, you can answer the question most teams can’t: which AI platforms read your content, and do they send anyone back? Here’s the process.
- Capture requests at the CDN. This picks up training crawlers, search agents and user fetch agents that never run your JavaScript.
- Classify each agent. Tag every request with its operator and purpose, so you can tell OpenAI’s training crawler apart from the agent fetching a page for a live ChatGPT question.
- Count by page. Look for pages agents read but people don’t, such as old archive, tag and feed pages, and pages people read but agents ignore. The first group may be wasting agent attention. The second is a list of candidates to optimize for AI search.
- Compare crawling with referrals. For each operator, set pages requested against visits its products send back. The Agent Analytics dbt package builds this as a scorecard over rolling 7, 30 and 90 day windows.
Treat CDN counts as a lower bound. They’re best for direction and comparing platforms, not exact totals.
That crawl-to-referral ratio is what you need before deciding how to treat each crawler. Block it, allow it, or charge it: Cloudflare’s pay per crawl, in closed beta, answers AI crawlers with an HTTP 402 Payment Required response and a price. Whichever you pick, you’ll want the data first.
See How Snowplow Measures AI Agent Traffic
Agent Intelligence is now available as an add-on. Agent Intelligence has three parts: CDN trackers that capture requests from agents that don’t run JavaScript, an agent classification enrichment that names each known agent’s operator and purpose, and an Agent Analytics dbt package that turns that data into BI-ready tables in your warehouse.
Frequently Asked Questions
How do you detect AI bot traffic on your website?
Combine three kinds of signal. Identity signals, like the user agent string, catch bots that announce themselves. Network-origin signals flag traffic from cloud and data center networks. Behavioral signals, like click timing and mouse movement, catch automation that pretends to be human. Weight them into a single bot score, and capture requests at your CDN too, because most AI agents never run the JavaScript your analytics relies on.
Can GA4 see AI crawlers?
Mostly not. Training crawlers, AI search agents and user fetch agents usually request pages without running JavaScript, so client-side tools like GA4 never record them. You need server-side or CDN data to see them.
How can you detect agentic browsing sessions?
Agentic browsers run JavaScript, so they do show up in your analytics, but they can look like normal visitors. Behavioral data is the best clue. When an agent takes over a session, navigation becomes direct, timing between actions becomes regular, and anything secondary, like ads or promotions, gets ignored.
How do you get traffic from AI bots?
AI bots rarely send visitors themselves. The products behind them do, when an assistant cites your page and someone clicks through. To see which platforms send people back, compare how many pages each operator’s agents request with how many visits its products refer.
What is Web Bot Auth?
Web Bot Auth is a proposed standard that lets bots and agents prove their identity by signing their HTTP requests, based on RFC 9421 HTTP Message Signatures. The IETF’s Web Bot Auth working group adopted its first draft in September 2026.