There’s no getting around it: driving value in an organization with behavioral data is hard. It’s so challenging, in fact, that we often hear that even with the best tools and the right people in place, teams can still struggle to get behavioral data projects off the ground when beset by obstacles around communication, culture and organizational challenges.
Let’s look briefly at some of the key reasons why driving value with behavioral data is difficult.
Why is driving value from behavioral data challenging?
There is a lot of data
Working with behavioral data requires the collection of high volumes of data from multiple platforms and products. Each platform from which you capture data, be it your web and mobile experiences, your support desk or your server-side applications, will produce huge heterogeneous sets of data. They will be disparate in nature, shaped and structured differently. And if the data sets are not captured, stored, or managed properly, the end result can be a tangle of messy data in your data lake or data warehouse, or both, that requires hours of cleaning and reorganizing before it can be used to drive value.
Data, and data collection, is constantly evolving
From Apple’s Intelligent Tracking Prevention (ITP), to the deprecation of third-party cookies and more recent privacy measures arriving on iOS, the landscape for data collection is in constant flux. Organizations who don’t stay on top of these shifts and overcome them may find their data is incomplete, affecting their ability to make sense of their data and put it to use. The most disruptive measures can result in user data going missing altogether, which not only limits business intelligence, but can impact a company’s ability to make the best decisions.
Aggregating behavioral data is hard
There is no straight-forward way to aggregate individual “events” to create higher level segments of data that can be used for analysis, such as for ‘funnels’ and ‘sessions’. There is no one-size-fits all, and yet packaged analytics tools attempt to prescribe a logic to model data that is often incongruous with your business logic . Organizations can (and should) invest in tooling and expertise to model their own data, but that isn’t easy either. Data modeling and transformation requires a deep familiarity with the business, the userbase and the nuances of the behavioral data itself.
There are ‘people and process’ challenges in behavioral data
Working with people within the business to deliver successful outcomes with behavioural data is a huge challenge. Internal teams such as product, marketing, and others work with behavioural data very differently, often bringing different levels of understanding and unique use cases that they require the data to serve. Often the responsibility of delivering behavioral data to these teams falls on a centralized data engineering team, who are overwhelmed with internal requests and become a bottleneck in the data lifecycle. There is also the enormous challenge of governing data, which necessitates communicating and enforcing a universal language and structure for the behavioral data in the business. All of this must be underpinned by a strong data culture and a company-wide effort to buy into the potential of behavioral data.
Data is a sensitive asset that needs to be protected
Behavioral data is often personal in nature. Personal identifiable information (PII) is tightly regulated and users have the right to request their information, or opt out of being tracked altogether. New regulations such as Europe’s General Data Protection Regulation (GDPR) and California’s Consumer Privacy Act (CCPA) are two examples of recent regulatory measures that enforce strict rules about how companies capture and manage behavioral data, but new legislation is being published all the time.
These challenges are significant, and while they are laid out separately, often they overlap and compound each other. For example, multiple teams within the organization will produce different data sets, from different platforms, with varying requirements and expectations for how that data should be used.
To take one example, the marketing team might want to understand the role different channels and campaigns contribute to revenue, while the product team may want to conduct funnel analysis to understand what drives user retention within their application. In each case, the tools, the type of data, the tracking methodology, the regulatory requirements, and the people involved might all look totally different. And that’s to say nothing of the data infrastructure required to deliver behavioral data.
What are three most common challenges organizations face?
Managing a complex data platform
Technology plays an integral role in managing and driving value with behavioral data. But building, purchasing and maintaining a data platform made up of multiple, siloed tools is a challenge in itself. When behavioral data first became an important part of business intelligence, packaged solutions like Google and Adobe Analytics paved the way for data teams. Since then, the need for robust, flexible tooling has grown – organizations want to own their data, and take control of how it’s captured, processed and modeled. They want to build assurance in the quality of their data and the capabilities to ensure its accuracy and completeness. And they need the tools to make data accessible to those who need it, when they need it.
Organizations venturing out from the packaged analytics solutions they may have started with are faced with countless choices and questions.
- Should they build or buy their data pipeline?
- What are the best ways to deliver data at low-latency, in a performant manner, that data consumers can easily work with?
- How can they monitor and observe their data platform for data quality issues, breakages, and red flags?
- How can organizations protect their data pipeline against volume spikes, cloud outages and build infrastructure that prevents data loss?
- And many more!
Only by addressing these questions can the organization prepare a foundational data platform that can support the whole business – ideally in an environment where individual teams can fish for themselves.
Enabling a self-serve data culture
Reaching a self-serve data culture is a widely held ambition for many organizations, for good reason. The ones who achieve it can facilitate a seamless flow of data productivity – where business teams can help themselves to data to empower decision making without going through a data practitioner. The snag is that building such a culture presents a number of challenges. First among these challenges is the task of building company-wide trust in the integrity of the data, its accuracy and completeness.
As mentioned above, data quality can be affected by so many outside factors, from changes to the way data must be collected (browser privacy updates, ad-blockers and changes to mobile tracking to name a few) to the way data structures are governed and adhered to. For instance, if a front-end developer, setting up tracking creates an entity describing a clothing product “White T-shirt”, but another developer names another product “Blue Tee-shirt”, inconsistent naming conventions can cause confusion when analysts try to query the data. Companies must therefore form a universal language that outlines how events and entities should be named. This can lead to sprawling data dictionaries, which make it extremely difficult for multiple stakeholders to understand, and adhere to a unified approach to tracking. Without a consistent, trusted approach to maintaining high data quality, it can be difficult to build assurance in the data that feeds reports and decision making, let alone powering advanced data use cases.
Deploying advanced data use cases
Advanced use cases, such as deep personalization, recommendation engines or artificial intelligence are part of what makes working with behavioral data so exciting. The most successful companies in the world have been able to use behavioral data to build market-leading products, while the organizations below them can only pin up these use cases as an aspiration.
While the tech industry is rife with vendors who offer big promises with little effort, in reality deploying advanced data use cases is not a quick win. Rather it is the result of a systematic effort to build a robust data platform that can deliver high quality behavioral data to the right place, in the right format, at the latency each particular use case requires. Without this foundation in place, data teams struggle to deliver the data needed to support a use case like personalization reliably in a production setting. Even the best algorithms and recommendation systems cannot perform without a strong data set to drive them – the behavioral data in this respect is essential, like fuel for an engine.
How we tackle modern data challenges at Snowplow
It’s our belief that organizations need to build the strongest possible behavioral data asset to tackle these challenges. They need
- Robust data infrastructure that can support the needs of a growing, evolving business with a single source of truth, optimized for delivering low-latency, behavioral data at scale.
- The best behavioral data, assured in its accuracy, completeness and structure, so analysts and data consumers can immediately work with the data with confidence.
- Tools for data governance to ensure that data is managed in a way that protects customers and empowers the organization to leverage behavioral data safely.
At Snowplow, the challenges for the modern data team are at the forefront of our minds in everything we do. We built our products to streamline the process of tracking and working with high-quality behavioral data so organizations can focus their time on driving value for their customer, their teams and their business.
