Snowplow Mini 0.4.0 released

We are pleased to announce the 0.4.0 release of Snowplow Mini, our accessible “Snowplow in a box” distribution.

Snowplow Mini is the complete Snowplow real-time pipeline running on a single instance, available for easy deployment as a pre-built AMI. Use it to:

1. Set up an inexpensive and easily discardable Snowplow stack for testing your tracker and schema changes
2. Learn about Snowplow without having to set up a horizontally-scalable, highly-available production-grade pipeline

This release focuses on addressing the long-standing NSQ migration. It also adds new features to the Control Plane and makes it much more functional. As a reminder, the Control Plane, which was introduced in release 0.3.0, lets you control a Snowplow Mini instance through an API interface.

Read on for:

1. New Control Plane features
2. NSQ migration
3. Creating index mappings
4. Other updates
5. Documentation and getting help

1. New Control Plane Features

In order to make Snowplow Mini easier to use and more accessible, we have added plenty of new features to the Control Plane in this release. In this post, only new Control Plane features are explained. If you want to learn more about the Control Plane API, you can visit the documentation page for Control Plane API.

1.1. Adding an external Iglu schema registry

One of our goals for Snowplow Mini is making it stateless, meaning that all the required services such as Iglu schema registry, Elasticsearch, Postgres etc live outside the actual box running Snowplow Mini.

As part of this, you can now configure an external Iglu schema registry via the Control Plane. Simply submit a vendor prefix, URI, name, priority and optional api key of the external Iglu schema registry, like so:

After that, you can start to use this schema registry in your Snowplow Mini instance.

1.2. Uploading custom enrichments

Configurable enrichments are an important part of Snowplow. We added custom enrichments to Snowplow Mini in the previous release.

With this release, we’re adding the ability to upload custom enrichments directly through the Control Plane, like so:

1.3. Adding an API key for the embedded Iglu schema registry

Another shortcoming we’re addressing with this release is the ability to add or change the internal Iglu schema registry’s API key, again through the Control Plane.

This replaces the cumbersome process of having to specify the initial API key through the user data script or changing it by SSHing into the box. Use the new functionality like so:

1.4. Changing the username and password for basic HTTP authentication

In 0.3.0, we made HTTP basic authentication for Snowplow Mini mandatory. We’re making changing those credentials available in the Control Plane with this release:

1.5. Adding a domain name for automatic TLS

Lastly, we’re improving on the TLS support introduced in 0.3.0, by letting you specify the the Snowplow Mini’s domain name via the Control Plane, thus:

2. NSQ migration

Up until this release, we were using Unix named pipes for inter process communication. Even though they are easy and straightforward to use, they were causing unexpected behaviors, and were difficult to debug and reason about.

Therefore, we decided to use NSQ instead of named pipes, making use of the new NSQ functionality released in R96 Zeugma. This migration will make the pipeline more robust; this change has not caused any user-visible changes.

3. Creating index mappings

Before this release, Snowplow Mini users had to create index mappings for their Snowplow data manually in Kibana. Creating those index mapping is now handled automatically during the Snowplow Mini setup; users will now start to see events appearing directly in Kibana.

4. Other updates

Version 0.4.0 also includes some internal changes and minor enhancements under the hood, including:

• Bumping the included JavaScript Tracker version to 2.8.2 (#71)
• Ensuring links in the UI adhere to the currently used protocol (HTTP or HTTPS) (#127)
• Using Caddy instead of Nginx for serving the Snowplow Mini dashboard (#130)
• Updating the build process to build Caddy from source (#132)
• Adding libffi-dev, libssl-dev, python-dev and markupsafe as dependencies (#133)

5. Documentation and getting help

To learn more about getting started with Snowplow Mini, check out the Quickstart guide.

If you run into any problems, please raise a bug or get in touch with us through the usual channels.