It was great to have the opportunity to speak at Span Conference this week in London on the topic of “Why your company needs a Unified Log”. Span is a single-track developer conference about scaling, organized by Couchbase developer evangelist Matthew Revell; Tuesday’s was the inaugural Span and it was great to be a part of it.
Below the fold I will (briefly) cover:
Why your company needs a Unified Log
As community regulars will know, at Snowplow we are hugely excited about the potential for Apache Kafka and Amazon Kinesis to be more than just message queues — they can serve as a unified log which you can put at the heart of your business, effectively creating a “digital nervous system” which your company’s applications and processes can be re-structured around:
In my talk, I provided an introduction to unified log technology, highlighted some killer use cases and also showed how Kinesis is being used “in anger” at Snowplow. Of course, my talk was heavily influenced by Jay Kreps’ unified log monograph, and by my recent work penning Unified Log Processing, from Manning Publications. Hopefully I went some way to showing how event streams inside a unified log are an incredibly powerful primitive for building rich event-centric applications, unbundling local transactional silos and creating a single version of truth for a company. I finished with a live demo (always fraught!) of Snowplow’s beta Kinesis support, showing events from the Snowplow website flowing through Amazon Kinesis and being enriched in real-time.
It was a very technical and experienced audience, which made for some great questions immediately afterwards and in the various one-on-one chats afterwards. I came away from the event with a renewed sense of the importance of robust event modeling, and the importance of our work on JSON Schema, Iglu and SchemaVer.
My highlights from Span
Matthew and the Span Conference team succeeded in packing a huge amount of content and discussion into Span’s single day format. There were too many great talks and interactions to mention all of them – but here were some of my highlights:
- Richard Astbury of Two10 Degrees introducing Microsoft’s Project Orleans actor framework. Richard provided a great introduction to the power and utility of actor frameworks, and followed it up with some nice hands-on coding on the Orleans platform. Orleans looks really interesting – the virtual actor model in particular seems novel. My only real concern about Orleans was that none of the persistence options seemed particularly pragmatic/compelling
- Martin Kleppmann, author of Designing Data-Intensive Applications and Apache Samza committer, spoke incredibly articulately about the difficulties of scaling technology architectures as more and more databases, caches and queues are added to our applications. He made a strong case for the unifed log pattern, and illustrated the role that a stream processing framework like Samza can play in this architecture. I came away with a much clearer understanding of the importance of ordering in our unified log’s queues, and a better handle on the increasingly important area of Change Data Capture
- Robert Virding, co-inventor of Erlang, gave a great and evocative talk on the origins of the Erlang platform. The most surprising anecdote was that the Erlang triumvirate had been unaware of the actor model when creating Erlang – they just happened to coincide with the very same design
- Elliot Murphy, CTO of CommonGround, gave a very frank and engaging talk about common security, encryption, privacy, and regulatory concerns relating to running distributed systems in the cloud. Elliot was willing to discuss important security topics like key management and employees-gone-bad which many in the devops community seem to want to gloss over – very refreshing!
Of course these were just my highlights – the day was jam-packed with great talks, one-on-one and group discussions, continuing into the evening. Many thanks to Matthew Revell, Philipp Fehre and all the organizers for a truly excellent Span!
