Iglu Server 0.6.0 released

We’re tremendously excited to announce the new 0.6.0 release of the Iglu Server, a next-generation of Iglu Registry, providing consistency and migration guarantees to Iglu schemas.

In the rest of this post we will cover:

1. Performance and stability improvements
2. API changes
3. Upgrading
4. Roadmap and Upcoming Features
5. Getting Help

1. Performance and stability improvements

During development cycle of 0.6.0 we rewrote most of Iglu Server components from scratch, replacing old monolithic Akka HTTP dependency with modern tools provided by Scala community, such as http4s, Rho and fs2. As a result, we have gained much more efficient performance and resource utilization. Our benchmarks show that t2.small EC2 instance running a new version of Iglu Server can easily handle 1.000 concurrent requests per seconds without dropping them. On the same instance, version 0.5.0 of Iglu Server starts to choke at 500 requests per second, either by dropping requests or responding within an inappropriately long time. As a consequence, the Snowplow pipeline slows down and sends some events into a bad bucket.

Another big change on the performance front is configurable thread and connection pools. While previous versions hardcoded default settings everywhere, the new Server gives a lot of configuration options to tune the performance.

• Use single DB connection or HikariCP (recommended)
• connectionTimeout, maxLifetime, maximumPoolSize settings from HikariCP
• Configurable thread pool for DB transactions
• Configurable thread pool for DB connections
• Configurable thread pool for HTTP server

All thread pools can have one of the following formats: global for global Scala ForkJoinPool, cached for cached JVM thread pool or fixed N for connection pool with fixed N amount of threads.

Iglu Server provides sensible defaults for all these options, but it’s now possible to tune the performance for specific usage scenarios.

2. API changes

We had two main aims for API changes in this release:

1. Preserving the core API for fetching schemas unchanged
2. Reducing the chance of accidental mistakes by making API stricter
3. Adding more flexibility

There’s not much to add for the first item – all our tools, including old versions of the Snowplow pipeline, Snowplow Mini, igluctl (with very minor inconsistencies) and Iglu Clients are working with the new version of Iglu Server just fine. If you’re interested only in stability and performance improvements – you can proceed to upgrade section.

Previous versions of Iglu Server were extremelly permissive about inconsistent state, providing just basic CRUD operations without semantic checks. This could lead to various hard to debug incidents with mutated schemas and inconsistent warehouse state. In this release we tried to narrow down the most common user mistakes and provide defensive checks for them in Iglu Server:

• Users cannot create a schema if the previous one does not exist, e.g. if there’s only 1-0-0 schema available and user tries to submit 1-0-2 it will result in 409 CONFLICT HTTP response
• Users cannot create a schema using HTTP PUT method if its URI and metadata do not correspond
• Users cannot create a private schema if previous ones were public and vice versa (otherwise there will be gaps between versions for public requests)
• Users cannot override a schema unless special patchesAllowed flag is set to true, marking the registry as dev-only

The next major step is to reject JSON Schemas where changes are incompatible with previous versions, e.g. if a user tries to add 1-0-1 which is clearly a 2-0-0 compared to its previous version.

We didn’t just make API stricter, we also added new features!

• Health-check endpoints: /api/meta/health for HTTP health and /api/meta/health/db (recommended) for HTTP+DB health
• Meta-information endpoint at /api/meta/server, providing server version and some safe to expose configuration options
• Debugging mode, enabled by setting debug flag to true and switching database.type to dummy (transient in-memory implementation). Extremely useful for debugging and testing purposes
• Webhooks. You can add a list of HTTP endpoints to a configuration to force a Server to push notifications about created schemas
• Ordering endpoints at /api/schemas/vendor/name/format/model enabling our Redshift automigrations, planned for the next release of RDB Loader
• DELETE endpoints for dev-only registries (enabled by patchesAllowed = true)

3. Upgrading

Unfortunately, due to a big internal refactoring and an attempt to make codebase more type-safe we had to completely overhaul the table structure Iglu Server uses, so it cannot use your old tables anymore. Good news is that we provide tooling for automatic migration and consistency checks.

In order to perform the migration you’ll need to have a new config file. The new config file is mostly compatible with old one, but requires database.type and debug

database { type = "postgres" # or "dummy" for debugging } debug = false 

All other options are optional, a full list can be found in our setup guide.

In order to migrate tables you need to run a special setup subcommand:

$ ./iglu-server_0.6.0.jar setup --config $IGLU_CONFIG_PATH --migrate 0.5.0

This should migrate all your schemas and API keys or let you know about consistency issues that have to be fixed. Withou --migrate option it will just create empty tables with new schema.

The Iglu Server 0.6.0 is also available on our Docker registry:

$ docker pull snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.6.0

5. Roadmap and Upcomming Features

The 0.6.0 release is planned to be a last one in the 0.x series. Unless critical bugs will be reported, the next version should be 1.0.0.

In 1.0.0 we plan to add:

• DynamoDB backend to make Iglu Deployment cheaper (#23)
• Schema version recognition (#30)

6. Getting Help

If you have any questions or run into any problems, please raise an issue or get in touch with us through the usual channels.