Real-Time, Highly Scalable, and Built for Behavioral Data Quality

Should we build our own data pipeline or buy one?

It depends on whether your team wants to own a pipeline's upkeep for as long as it runs, including traffic spikes, schema changes, failed events, monitoring, and upgrades. Snowplow Customer Data Infrastructure is a managed alternative with high availability and auto-scaling, data quality monitoring, and SLAs for collector uptime and warehouse loading latency. It can run in Snowplow's cloud or in your own cloud account. See how Snowplow compares with building your own pipeline.

Is a data pipeline the same as ETL?

No. ETL (extract, transform, load) is one type of data pipeline: it copies data from existing systems, transforms it, and loads it into a warehouse, usually on a schedule. A data pipeline is any process that moves data from one system to another, in batches or in real time. An event pipeline like Snowplow collects behavioral events as they happen, validates and enriches each one, and loads them continuously.

What are the main components of a data pipeline?

Most data pipelines have three parts: sources that produce data, processing that checks and transforms it, and destinations that store or use it. In Snowplow, trackers and webhooks send events to a Collector. The Enrich stage validates each event against its schema and adds context such as campaign attribution or location. Loaders then write events to your warehouse or lake, and enriched events are also available on real-time streams.

Is there an out-of-the-box enrichment for stripping out PII?

Yes. The PII Pseudonymization enrichment replaces the fields you choose, such as user IDs, IP addresses, or properties in your custom events, with hashed values before data is loaded. The IP Anonymization enrichment masks part of each IP address. For personal data inside free text, such as a phone number in a message, you can add your own logic with the JavaScript enrichment, which can modify event fields directly.

How quickly do events reach our warehouse?

With a streaming loader, events can reach your warehouse within seconds of collection. Loading into Snowflake, for example, takes seconds rather than the minutes batch loading takes. Each event is validated and enriched as it arrives and written to a real-time stream, where your applications can read it before it's loaded. Snowplow Customer Data Infrastructure includes a warehouse loading latency SLA.

How does Snowplow detect bot traffic?

Snowplow flags likely bots in the pipeline, before events reach your warehouse. Enrichments check each event's IP address and user agent against the IAB/ABC International Spiders and Bots List, parse the user agent, and match the network against known data center and bot ASNs. A web tracker plugin can also detect automated browsers. The Bot Detection enrichment combines these signals into one bot field, so you can filter bots in SQL or drop them in the pipeline.

Can we fix and reprocess events that fail validation?

Yes, when the error can be corrected, such as a misspelled property name. Events that fail validation are kept as failed events rather than loaded into your main events table or discarded, and each one records what went wrong. You can load failed events into a separate table in your warehouse or lake, fix them with SQL, and recover them into your main events table.

Can the Snowplow pipeline run in our own cloud account?

Yes. With the Private Managed Cloud option of Snowplow Customer Data Infrastructure, the pipeline runs in your own cloud account and loads into your own warehouse or lake, while Snowplow manages upgrades and security patches. If you'd rather not run any infrastructure, Snowplow can host the pipeline for you instead, and data still loads into your warehouse or lake.