Snowplow Micro 1.0.0 released

We are pleased to announce the v1.0.0 release of Snowplow Micro.

Please note: If you are still using version Snowplow Micro 0.1.0 then you need to update your scripts to use the snowplow/snowplow-micro:0.1.0 docker image, not the latest image. Existing users might notice their tests will fail if they are still using the latest tag. More information on the breaking changes in version 1.0.0 can be found below.

Snowplow Micro is built to enable companies running Snowplow to build automated test suites to ensure that new releases of their websites, mobile apps and server-side applications do not break tracking / Snowplow data collection.

Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried to understand:

• How many events have been received?
• How many of them were successfully processed vs ended up as “bad” (e.g. because the events failed validation against the corresponding schemas in the Iglu Schema Registry)
• For any events that have been successfully processed, what type of events they are, what fields have been recorded etc.
• For any events that have not been successfully processed, what errors were generated on processing the events. (So these can be surfaced back via the test suite.)
• This means companies can build automated test suites to ensure that specific events in an application generate specific events that are successfully processed by Snowplow.

For examples of how to use Snowplow Micro, take a look at the ‘How to’ guides for Cypress and Nightwatch as well as the examples repository.

Read on below for:

1. Updates to the validation logic
2. Breaking changes since version 0.1.0
3. Other updates and bug fixes
4. Documentation and help

1. Updates to the validation logic

In version 1.0.0, Snowplow Micro now uses the exact same validation the production Snowplow pipeline uses. Specifically, it uses EnrichmentManager.enrichEvent to validate events (GitHub issue #23) and outputs the post-enrichment canonical event (but with all enrichments deactivated).

This effectively makes the validation in Micro even stricter. In version 0.1.0, an event validated by Micro could fail during enrichment due to an invalid input field (such as a timestamp). In version 1.0.0, using EnrichmentManager.enrichEvent to validate the event, we can ensure an event that is validated by Micro will not unexpectedly fail in the enrichment process.

2. Breaking changes since version 0.1.0

The good event format has changed

The /micro/good endpoint returns a JSON array of good events. The structure of the JSON objects in this array has changed (GitHub issue #28) due to the improvements in the validation logic.

In version 0.1.0, each GoodEvent used to contain the following four fields:

• event: the RawEvent, in the format of a validated event before enrichment
• eventType: the type of the event
• schema: the schema of the event in the case of an unstructured event
• contexts: the contexts attached to the event

In version 1.0.0, each GoodEvent now contains the following four fields:

• rawEvent: the RawEvent, in the format of a validated event before enrichment (previously event)
• event: the Canonical Snowplow Event, in the format of a validated event after enrichment (with all the enrichments deactivated)
• eventType: the type of the event
• schema: the schema of the event in the case of an unstructured event
• contexts: the contexts attached to the event

The format of the configuration file has changed

Micro’s configuration file has been brought up to date to match version 1.0.1 of the Snowplow stream collector. Several new configuration options are present in the newest file format, as described in a previous blog post, some of which a required fields.

When switching from Snowplow Micro version 0.1.0 to 1.0.0, the easiest way to upgrade your configuration is to download the latest example from the GitHub repository. Alternatively, if you are already using a recent version of the stream collector, you might choose to copy over that configuration to match your settings in production.

3. Other updates and bug fixes

• Fix filtering on event_type and schema (GitHub issue #24)
• Add Snowplow Bintray and Snowplow Maven to resolvers (GitHub issue #26)
• Bump sbt to 1.3.12 (GitHub issue #25)
• Bump snowplow-stream-collector-core to 1.0.1 (GitHub issue #22)
• Bump circe to 0.13.0 (GitHub issue #21)
• Bump specs2-core to 4.9.4 (GitHub issue #20)
• Bump Scala version to 2.12.11 (GitHub issue #10)

4. Documentation and help

For more information on Snowplow Micro, please take a look at the technical documentation.

For help with getting started, check out
the Snowplow Micro examples repo.

If you have any questions or run into any problems, please visit our Discourse forum.

Please raise any bugs in Snowplow Micro’s issues on GitHub.