Snowplow 75 Long-Legged Buzzard released with support for Urban Airship Connect and SendGrid

We are pleased to announce the immediate availability of Snowplow 75 Long-Legged Buzzard. This release lets you warehouse the event streams generated by Urban Airship and SendGrid, and also updates our web-recalculate data model.

The new webhook integrations are as follows:

1. Urban Airship – for tracking mobile app-related events from Urban Airship using the new Urban Airship Connect product
2. SendGrid – for tracking email-related events delivered by SendGrid via SendGrid webhooks

Here are the sections after the fold:

1. Urban Airship Connect support
2. SendGrid webhook support
3. Updated data model
4. Upgrading
5. Roadmap and contributing
6. Documentation and help

1. Urban Airship Connect support

The Urban Airship Connect adapter in lets you track mobile app events delivered by Urban Airship Connect. Using this functionality you can warehouse all of your Urban Airship mobile app and push notification events alongside your existing Snowplow events.

For help setting up Urban Airship Connect support, see the Urban Airship Connect setup wiki page.

For technical details on this adapter, see the Urban Airship Connect adapter wiki page.

2. SendGrid webhook support

The SendGrid webhook adapter lets you track all email-related events emitted by SendGrid through their SendGrid webhooks. It’s great for keeping track of your email campaigns, allowing you to see which emails generate the most interest.

For help setting up SendGrid support, see the SendGrid webhook setup wiki page.

For technical details on this adapter, see the SendGrid webhook adapter wiki page.

3. Updated data model

Community member Bernardo Srulzon has contribued an update to one of our SQL data models: web-recalculate.

The updated data model is more efficient (consuming less disk space), and now creates a helpful cookie-ID-to-user-ID map. Thanks Bernardo!

4. Upgrading

4.1 Upgrading EmrEtlRunner and StorageLoader

The latest version of the EmrEtlRunner and StorageLoader are available from our Bintray here.

In your EmrEtlRunner’s config.yml file, update your hadoop_enrich job’s version to 1.5.0, like so:

 versions: hadoop_enrich: 1.5.0 # WAS 1.4.0

For a complete example, see our sample config.yml template.

4.2 Redshift

You’ll need to deploy the Redshift tables for any webhooks you plan on ingesting into Snowplow. You can find the Redshift table deployment instructions on the corresponding webhook setup wiki pages:

• SendGrid webhook Redshift setup
• UrbanAirship Connect webhook Redshift setup

5. Roadmap and contributing

We welcome any contributions of webhook adapters for other services – for details on getting started, see our wiki page How to integrate a webhook into Snowplow. Please note that contributing will require some experience in Scala at this time.

Similarly, if you would like to sponsor the Snowplow team to build a webhook adapter, do get in touch!

Upcoming Snowplow releases include:

• Release 76 Bird TBC, which will refresh our EmrEtlRunner app, including updating Snowplow to using the EMR 4.x AMI series
• Release 77 Bird TBC, which will bring the Kinesis pipeline up-to-date with the most recent Scala Common Enrich releases. This will also include click redirect support in the Scala Stream Collector
• Release 78 Bird TBC, which will introduce basic event deduplication in Hadoop prior to loading Redshift

Note that these releases are always subject to change between now and the actual release date.

6. Documentation and help

Documentation relating to the new webhook support is available on the wiki:

• Setting up a webhook
• Technical documentation on webhooks

As always, if you do run into any issues or don’t understand any of the new features, please raise an issue or get in touch with us via the usual channels.

For more details on this release, please check out the R75 Release Notes on GitHub.