Introducing the New Snowplow BigQuery Loader: Lower Costs, Better Schema Evolution, and Multi-Cloud Support

We’re excited to announce the release of our new BigQuery Loader - a major upgrade designed to boost your performance, reduce costs, and provide you with greater flexibility as a Snowplow user. With this latest release, BigQuery is now available as a destination in Snowplow BDP Cloud, and we’ve made significant improvements to schema evolution, infrastructure costs, and cloud compatibility.

What’s New?

Multi-Cloud Support

One of the biggest updates is expanded cloud support. While the previous loader was limited to GCP, the new BigQuery Loader can now run on both AWS and GCP. This means that organizations using Snowplow BDP Cloud can now send their data directly to BigQuery, no matter which cloud provider you use - giving you more choice and control.

Lower Infrastructure Costs

We’ve upgraded how data loads into BigQuery, moving from the Streaming Inserts API to the more cost-efficient BigQuery Storage Write API. This translates to lower operational costs for you -previously, the API charged $0.01 per 200 MB of data(roughly $1 per 4 million events). You can check your BigQuery bill under “Streaming Inserts” to see the savings yourself. 

Improved Schema Evolution

Schema evolution is a core feature of Snowplow loaders, keeping your event schemas adaptable as your data needs evolve. en. The new BigQuery Loader introduces a smarter approach to schema evolution:

• Instead of creating a new column for every schema version, the new loader only adds columns for major schema versions.
• This results in a more compact atomic events table, making queries and data models easier to manage.
• The new approach also helps you to avoid hitting BigQuery’s 10,000-column limit.

If you’re transitioning from the old loader, we’ve included a compatibility mode that mimics the previous schema behavior to ensure a smooth migration. Learn more about schema mapping in our documentation here.

Seamless Migration for Snowplow BDP Users

For Current BigQuery Users

All existing Snowplow BDP customers running BigQuery Loader v1 will be automatically migrated to v2 in the coming weeks. A separate maintenance notification will provide additional details—be sure to follow that page for updates.

By default, the migration will deploy the new loader in full compatibility mode, meaning no immediate changes to your data structure or workflows.

For New Deployments

For users not currently using the old BigQuery loader but looking to get started, BigQuery Loader v2 is now available as a self-serve deployment in Snowplow BDP. Simply navigate to Destinations and select “BigQuery (Loader v2)” in the Available tab. You’ll need to provide authentication credentials and a BigQuery dataset in your own GCP project.

Handling Failed Events in BigQuery

Another major benefit of the new loader is the ability to send failed events directly to BigQuery. You can configure it to store failed events in a separate BigQuery dataset, providing greater visibility and debugging capabilities. This feature is independent of the migration, meaning you don’t have to wait for your primary loader to be upgraded before taking advantage of it.

For more details on setting up failed event destinations, see our previous announcement here.

Get Started Today

The new BigQuery Loader is now live for Snowplow BDP customers! Whether you’re transitioning from v1 or setting up a new BigQuery destination for the first time, we’re excited for you to experience the benefits of this cost-effective, scalable, and cloud-flexible solution.

To learn more, check out our migration guide here or visit our documentation here. If you have any questions, reach out to our support team—we’re happy to help!