Snowflake Streaming Loader: The fastest, most cost-effective way to load Snowplow data into the Data Cloud
Today, Snowplow is excited to expand the availability of our Snowflake Streaming Loader into Public Preview, allowing companies to load their behavioral data into the Data Cloud with extremely low latency and at a fraction of previous costs. Snowflake Streaming Loader is a fast, fully-streaming architecture that gets event data into Snowflake in mere seconds.
A few months ago, Snowplow first announced this release with our strategic investment into Snowflake Native Apps, evolving our loading capabilities even further by supporting Snowflake’s latest Snowpipe Streaming API. While it was then in Private Preview, today all Snowflake users can now leverage the real-time benefits.
Unlocking Real-time Use Cases
The Snowflake Streaming Loader revolutionizes how Snowplow customers work with their first-party behavioral data in Snowflake. It allows data to land in the Data Cloud in seconds rather than minutes, opening up new real-time use cases while also reducing data loading costs up to 80% compared to our previous RDB Loader.
Features and benefits of the new Snowflake Streaming Loader:
Reduce costs
🎉 Reduce your data loading costs by up to 80%.
]The new Snowflake Streaming Loader features a fully streaming architecture (surprise!), which significantly helps teams keep costs significantly lower. Our RDB Loader made use of higher cost map-reduce technologies and required Snowflake to load events in large batches, which was not as efficient.
Ultra-low latency
🎉 100× faster! Sync Snowplow behavioral data into the Snowflake Data Cloud in a matter of seconds.
In contrast, the RDB Loader would batch data and send it to Snowflake every few minutes. This is a major step change in how quickly you can sync Snowplow data into your single source of truth, and it unlocks new opportunities for analytics, AI/ML, and marketing workloads. Real-time accessibility and insights into customer behavior will allow your teams to make quicker decisions and better personalize the experience for your customers. (And it’s very handy for testing and debugging!)
Schema evolution
🎉 Interpret Snowplow data and automatically create new columns in your Snowflake Tables when you evolve your data schemas in a major way.
Data contracts are a very important concept for Snowplow. Our product is designed so that you can evolve the schema of your data and have that automatically reflected in the data warehouse or lake. (Large, sophisticated data teams that move quickly will especially appreciate this.) The new Loader is no exception and is fully compatible in its schema management approach with the previous loader and all our data models.
Start Streaming!
The Snowflake Streaming Loader is now available to Snowplow Behavioral Data Platform (BDP) customers on Snowflake. Reach out to the Snowplow support team to start your migration or simply fill out the set-up form in Console under “Destinations catalog”.
With Snowflake Streaming Loader, our customers can unlock the full power of their real-time event streams and behavioral data in the Snowflake Data Cloud. This is a game-changer for data teams building data applications and products. Similar to all of our core pipeline components, the sources of this application are available for exploration under the Snowplow Limited Use License Agreement.
Using a different cloud data warehouse?
Snowflake Streaming Loader is part of Snowplow’s wider initiative to radically redesign all of its data warehouse and lake loaders, making them more efficient, scalable and cost-effective. Stay tuned for future announcements this year regarding loaders for BigQuery, Databricks, Redshift, and data lakes.