The ultimate guide to effective product analytics

Not long ago, the best way to witness firsthand how the general population interacted with your product was to assemble a random sample of potential users and stick them in a room while you observed from behind a two-way mirror.

What is Product Analytics?

As a process, product development has grown apart from digital analytics into a discipline in its own right. It is the process of analyzing product usage in order to make better decisions regarding features, releases, and the overall product roadmap.

Different approaches differ greatly in sophistication, with some companies taking basic data from analytics products to approximate product usage with generic metrics, and others using data warehouses to create real-time predictive models to better understand customers.

At the latter end of the spectrum are market leaders such as Airbnb, Netflix and Spotify, which have used product analytics to create unparalleled personalization and growth.

The need for product analytics has never been so great, with Product Managers and Owners increasingly seen as “mini-CEOs”, each taking responsibility for one of a company’s products. With this comes the responsibility to make rational decisions regarding product choices, such as which features to release or prioritize. Additionally, cycles of iteration are now incredibly fast, and data-led decision-making is distinguishing certain products in a hyper-competitive market.

Data and digital products

But when free analytics tools hit the market in the early 2000s, companies of all sizes began to look at product data as a way to aid the decision-making process. This has now evolved into a rich ecosystem of product analytics tools capable of answering different levels of questions about digital products.

In this article, we’ll take a broad overview of what’s possible at the different ends of this ecosystem and provide some best-practice tips for effective product analytics.

Key areas in product analytics, what is now possible?

What is possible in the world of product analytics evolves as companies mature. This often starts with free tools like Google Analytics and progresses to highly-customized pipelines revolving around a data warehouse, lake, and/or real-time stream.

The more basic end of the product analytics spectrum

Free tools like Google Analytics provide users with an approximate picture of the usage of a digital product. There is some basic customization, but the big advantage is that they are really easy to use and self-serve from.

You can observe certain user behaviors, generally aggregated to ‘sessions’ and ‘page view’ level – i.e. not specific to a given user or users. You can use metrics such as ‘time on page’ or ‘number of conversions’ to gain insights into user demographics and interests.

Limitations

Many companies start out using these solutions and don’t mind sacrificing accuracy and granularity of data for speed and convenience. At a certain point, however, most companies find they need an analytics solution that can answer more sophisticated questions.

The advanced end of the product analytics spectrum

At the other end of the spectrum are digital product pioneers such as Netflix, who build their own data pipelines in-house.

Netflix evolved their tracking from relatively low sophistication to one of the most advanced behavioral data platforms in the world.

Their initial recommendation engine was built on a basic API, which took data from IMDB to gauge a movie’s popularity and recommend it to users. This was a simple but wildly successful strategy at the time as it meant they were more than just a content library – they were giving users more of what they wanted.

Netflix’s current behavioral data platform monitors every user interaction to the smallest detail. They then stitch each interaction together to build detailed user profiles. These, in turn, provide the fuel for automation and personalization – or ‘data apps’.

“At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations. A large number of batch workflows run daily to serve various business needs. These include ETL pipelines, ML model training workflows, batch jobs, etc. As Big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have increasingly become more important for our data scientists and the company.”

Netflix tech blog |

‍

Limitations

All this is achieved, however, with enormous engineering resources, as a large and knowledgeable team is needed to build and manage these pipelines. Smaller companies trying to imitate this home-built success are often overwhelmed by the difficulty of building a pipeline and scaling data capture simultaneously. (Snowplow was built to bridge this gap and allow more data teams to access advanced analytics without the huge technical overheads.)

Problems companies face with product analytics

1. Product-market fit

Airbnb founder Joe Gebbia described what he referred to as the trough of sorrow, “where you have a product and a market and they don’t fit, like two gears that don’t touch and you can’t figure out how to close the gap… it’s completely flat, there’s zero growth.”

This is a crucial period in the launching of a new product or piece of technology, when you determine your product-market fit. However, knowing what to measure, what metrics are significant, and collecting that data are challenging and make determining product-market fit tough. In Joe’s words, “This is when people tend to quit.”

2. Asking the right questions

Knowing what to measure requires several factors to align which are not always within an analyst’s control:

• Creativity (outside-of-the-box thinking around what features are to be experimented on and why)
• Deep product understanding
• An openness to challenge all of your assumptions
• Valuing experimentation (sometimes facts are hard to hear, or they can be difficult to wait for when they are slow to discover).
• Allowing for complexity. (A/B testing is much more extensive than simple experiments around the color or placement of buttons. When taken to advanced levels, A/B testing is dramatic but this complexity takes time and resources.)

Learn more about advanced A/B testing

3. Communication

Another problem that frequently arises is communication. As teams scale and data collection grows, data can become siloed or poorly understood, leading to errors in documentation which go on to cause bad decisions to be made. Maintaining effective communication between teams is something that requires a mixture of good tooling, effective leadership and a strong operational culture.

4. The increasing complexity of products

Twenty years ago, digital products were simple websites that were little more than linked documents. Now, the web is far richer and more interactive, with devices like smartphones, tablets and the Internet of Things bringing a lot of additional complexity.

Most analytics platforms struggle to describe the complex user journeys that arise as users move seamlessly between devices. This creates an inaccurate picture of product usage, often artificially inflating the number of sessions and splitting one user into many (at Snowplow we store all data in one giant ‘atomic’ table for this very reason).

‍

Which is the best product analytics tool?

Boring answer – there is no best tool. A better question is: ‘how can I choose the best product analytics tool for my situation?’

If you mean ‘the best tool for a data mature company with a data team of 5+’, that’d be a different answer to ‘the best tool for my new startup’. That being said, here’s a rundown of some of the top tools:

1. Snowplow

Snowplow is an open-core tool that captures data from all over the business, including product analytics. They pioneered a new way of viewing data pipelines – Data Creation – in which companies purposely create the data they need, rather than extracting it from 3rd-party sources.

It is designed for growing data teams looking to create data applications and stands out for its data quality, richness, and full ownership. Open-source users and customers can track user behavior in real-time and collect rich data about their users.

Snowplow’s data modeling feature allows businesses to define their own events and attributes, providing complete flexibility over their data. Snowplow also integrates with a range of other analytics tools, such as Looker and Tableau, to provide deeper insights into user behavior.

2. Mixpanel/Heap/Amplitude

These tools are sometimes referred to as ‘packaged analytics solutions’, as they try to package many features and types of functionality into a convenient user interface (UI). Snowplow data can also be pushed into these tools after the warehouse to benefit from the carefully created and granular data at the same time as the great UIs these tools offer.

They all have extensive functionality, as the UIs are intuitive, packed with features, and are relatively easy to get started with. If you are starting to explore product analytics or looking for intermediate use cases, they can be very effective.

However, these tools do limit organizations that want to tackle advanced use cases, so we recommend caution when adopting these tools, as they become pricey and technically limited as you grow and can be expensive to migrate from due to vendor lock-in. It all depends on your long-term data strategy.

3. Google Analytics

Google Analytics (GA) is a free tool that provides businesses with basic data about their websites and mobile app. It does not specialize in product analytics, but can be used to track events within a product.

Combining data from across mobile and web can be challenging with GA, however, and creates inconsistencies within your data. Google has also come under scrutiny recently for GDPR compliance issues in Europe, which have led to hefty fines. Many companies have decided to migrate away from GA as a result of this. Another factor has been difficulties migrating from the soon-to-be-outdated Universal Analytics to GA4 with the uncertain future of GA.

That being said, GA has long been to go-to product analytics tool of choice for those just getting started, as it’s super easy to access rough data in no time.

4. Hotjar

Businesses can create heatmaps, scroll maps, and session recordings to gain insights into user behavior.

Hotjar’s polls and surveys also allow businesses to gather feedback from their users regularly and with a centralized tool.

This differs from the previous two solutions, in that it does not collect ‘events’ in a database format, but offers different ways of visualizing user behavior. It can be seen as a complimentary, not a stand-alone, tool for product analytics.

Top product analytics use cases

1. Cohort Analysis

Cohort analysis is a longitudinal study. It compares two or more user groups over a period of time.

Cohorts can be used to analyse the product onboarding process. For example, in the first month, you don’t give any tips to new users; in the second you design tips to gradually expose features to new users. At the 30 day mark for each group, what is the retention rate between the two groups? Did new users with the tips stay longer?

It is instructive to compare cohort analysis with time-series analysis. As an example, a time series analysis of a metric such as retention might show an increase over time, but a cohort analysis can show that new users acquired are more likely to be retained than old users.

Therefore, cohort analysis gives a more precise answer to specific questions because you’re controlling for the changing composition of your user base.

Cohort analysis is often used in early-stage startups or early in a product’s life to identify product-market fit, however, it frequently continues to be valuable as products mature. Any A/B tests run on the product are a form of cohort analysis, with the control group and the test group being two cohorts.

2. A/B Testing

This is the most important analysis technique available for product teams and one heavily relied upon by the most successful companies.

The potential benefits of A/B testing are nearly limitless. From small tweaks in the user interface to massive changes to core features, A/B tests, sometimes called split tests, allow product analysts to scrutinize the effectiveness of every little detail of a product with the ultimate goal of creating a user experience optimized for growth.

A/B testing provides a formal mechanism for measuring if any given update or change to a product succeeds in generating a desired effect. Each new release and update must then be rigorously A/B tested to ensure its quality and impact.

Early on in a product’s life A/B testing will help you rapidly iterate toward product-market fit. You can gather quantitative evidence of the effectiveness of different features and how your users respond by measuring things like time engaged and retention.

As a product matures, A/B testing is instrumental in keeping it competitive and relevant. Products are never static, as new features and competition change the landscape, along with changing user sentiment.

3. Retention analysis

Retention analysis looks closely at how long your customers are staying engaged with your product and can help you understand what percentage of your users come back after signing up or downloading your product.

This kind of data is essential for building an accurate customer lifecycle model and determining which factors are keeping people engaged.

This is important in particular for any type of subscription business, ad revenue, or virtual goods business where Customer Lifetime Value is a function of the length of time a user engages with a product.

Retention is fundamentally a metric: retention rates are often compared across different cohorts of users. Typically, retention curves are modeled so based on any given 1-, 2-, or 7-day retention rate, a longer-term retention rate can be predicted with some level of accuracy.

4. Heat mapping

Examining a heat map is one of the most powerful ways for your product team to see exactly what the most popular and commonly used features or buttons are within your product.

A digital heat map is a visual representation of where certain events take place within a defined area: for example, where the most clicks occur on your website’s homepage.

Heat mapping is one way of visualizing how groups engage with individual pages or features in an app or website, showing you where your users spend most of their time.

This kind of data is especially valuable in the first stage of the product development process when you’re generating ideas.

Looking at heat maps is a great way to develop hypotheses: it’s a type of analysis that can turn up surprising insights, such as finding that a region of a page you don’t think is interesting actually captures a disproportionate amount of user attention or that users are struggling with a particular part of a workflow because their attention is being directed to the wrong part of the screen.

There are some limitations, however. In single-page apps or websites where there are a low number of options, heat maps are not as revealing.

5. Funnel-based analytics

This is the bread and butter of product analytics; you can see the percentage of a specific group that moved between the various stages in a product journey.

The concept behind it is straightforward: your platform is designed to support users working through one or more different workflows. Each workflow is a series of steps necessary to deliver a specific outcome. Examples of workflows include:

• Buying a product
• Paying a bill
• Researching the answer to a specific question
• Signing up for a service
• Checking your bank balance

Funnel analysis is very commonly employed in online shops, for example, to understand what fraction of visitors to the website ‘convert’ (i.e. make a purchase), and how far other users make it through the purchase funnel.

This analysis can inspire a robust set of actionable insights, from identifying conversion areas that aren’t functioning properly to finding new ways to acquire new users and fill the top of your funnel. Sometimes, just the act of conducting a funnel analysis can help reveal any discrepancies between what you believe your funnel to be, and the steps and stages your users actually follow in their journey using your product.

Being aware of the health of your funnel is also useful, especially in conjunction with something like cohort analysis or retention modeling, to project revenue and inform growth strategies.

Learn more about how to conduct funnel-based product analytics

6. Session replay

A session replay is a reconstruction of one user’s entire engagement session, from loading an app or website to closing it or navigating away. Session replays are particularly useful for discovering how users engage with your product.

Session replays can help you see your app directly from a user perspective, confirming or refuting what you believe about how they interact with different features.

This type of macro-level, open-ended question can be difficult to answer using only graphs and charts. Further, testing often reveals that users engage with your product in ways you hadn’t counted on, revealing potentially new avenues for development.

When analyzing session replays, it’s important to note that it’s possible to uncover unique edge cases that don’t apply to your wider audience and sometimes it can be difficult to discern between surprising insight and bizarre behavior

7. User survey

Though clickstream data can tell you exactly what a user did, surveys give you the chance to find out what they were trying to do, why they took a particular action, and how they feel about what happened.

Often, if you want to know more about your users, simply asking them can be very powerful.

Customer surveys, through email or modal boxes on a website, can be invaluable sources of customer information that you can’t extract from traditional digital metrics. Clickstream data, for example, can’t tell you why a user is on your site or if they’re happy with their experience. Intent and motivation are powerful forces, and understanding your users’ feelings and mindset can help you improve their experiences.

When using customer surveys, it’s important to take sample size into consideration as one user’s experience is not indicative of everyone’s.

8. Form analysis

Form analysis can mean drilling down into individual form fields and working out where users are getting stuck and which fields users are revisiting. This helps you identify and improve bottlenecks and where your instructions lack clarity.

Many digital products include forms such as sign-up forms, contact sheets, support forms, or even customer feedback surveys.

On a top level, form analysis can help you identify if the number of form submissions you receive is unusually high, low, and how to nudge that number in a more favorable direction.

While form analysis is not always necessary, many different types of forms are essential for moving customers through your funnel, and a frictionless form submission process can help improve your customer feedback and increase user acquisition.

6. Predictive analytics

This is a specialty of Snowplow BDP. Predictive analytics examples include:

• predicting expected customer lifetime value
• the likelihood to churn
• best product or service to recommend
• best content to show a user
• best ad to serve a user

This spans all types of analytics, not just product, however predictive data applications are fantastic at informing the evolution of your product; in essence, they increase activation and retention by better predicting user behavior.

Predictive product analytics can also identify important characteristics of a user journey which make high-value events such as sales more likely to occur. Snowplow has created pre-made use case recipes to accelerate your adoption of predictive use cases – check out this predictive ML recipe.

How can predictive analytics boost product development?

As you can see in the diagram above, predictive analytics is a step up in data maturity. You begin your product analytics by simply describing what happened. Once you get efficient enough at this, you can begin to use predictive models – providing you have data of succinct quality.

In terms of developing your product, predictive analytics can be used to learn more about specific user groups. If certain behaviors are very predictive of churn, for example, you can target groups exhibiting these behaviors with the information/nudge they need to fall back in love with your product. Similarly, if a behavior is predictive of a high-value customer, you can build that into your product experience.

How to price a product with predictive analytics

Pricing a product requires balancing revenue goals with customer demand and market competition. Predictive analytics can help businesses make informed pricing decisions by analyzing customer behavior, purchase history, and market trends.

To price a product with predictive analytics, ML algorithms can be used to create pricing models that take into account various factors, such as customer demographics, purchasing power, and willingness to pay.

These models can be continuously updated, allowing businesses to adjust their pricing strategy in real-time based on changing market conditions.

By using predictive analytics to price their products, you can optimize revenue and improve customer satisfaction by offering the right price at the right time.

Using product analytics to improve in increments

Agile workflows and incremental improvements have become extremely popular, and this is no different for product analytics. As more and more companies set up shop and compete for the same audiences, data-driven product teams attempt to gain more granular insights than their competitors by winning thousands of small victories. Any tool that can give product teams an edge becomes a force multiplier.

Andrew ‘Boz’ Bosworth, Chief Technology Officer at Meta, describes how the product teams are in a constant stream of testing and are continually updating the many products that comprise Meta platforms.

“When you see such dramatic results from the smallest tweaks, you realize how much opportunity there is to improve things—and we feel a constant sense of urgency to do so. When a test goes out we look at the data immediately and adapt the products quickly. We do this on a daily basis. This cycle of iteration is the engine of progress and the people who use Facebook are not just the beneficiaries but are also intimately a part of the process. We don’t just develop this product for them, we develop it with them.”

Andrew ‘Boz’ Bosworth | Chief Technology Officer at Meta

What is product-launch analytics?

In terms of measuring the success of a product launch, the main thing to consider is the initial aim of the creation of the product – what problem is it trying to solve? At release, the amount of attention a product gets, as well as the resulting customer activation are often taken as north-star metrics – or KPIs. As the product evolves, product teams often move on to optimize different stages in the customer lifecycle (see below).

Analyzing stages in a product user’s lifecycle with data

1. Acquisition – how well are your website and/or marketing channels performing in terms of generating interactions with your website or product?

Example metrics:

• ROAS or return on advertising spend
• Mindshare or brand awareness metrics such as ‘mentions’
• Marketing-funnel metrics such as click-through rates

2. Activation – turning prospects or users into customers – normally sign-ups or purchases, but includes any desired first step

Example metrics:

• No. of sign-ups
• No. of interactions with digital product

3. Engagement – how successfully is your product engaging your users once they’ve signed up?

Example metrics:

• No. of logins
• No. of actions taken
• No. of features used
• Time in app/on site

4. Retention – how long are users using your product for?

Example metrics:

• Customer lifetime value
• Average subscription length

5. Referral – are customers recommending your product to others

Example metrics:

• Referrals per customer

6. Revenue – at this stage, your cost of acquisition is lower than cost of retention. It means companies have correctly identified which products and features will create loyal customers and therefore create profits.

Example metrics

• Net Revenue Retention (NRR)
• Gross Revenue Retention (GRR)
• Annual Recurring Revenue (ARR)

What is churn analytics?

Churn analytics refers to the rate at which customers stop using a product or service. Churn analytics can help businesses understand why people are leaving and then to take steps to prevent this.

Churn analytics typically involves analyzing customer behavior and demographic data to identify patterns and trends. This data can then be used to create targeted retention strategies, such as personalized offers or improved customer service. Churn analytics is particularly important for businesses with a subscription-based model or those that rely on repeat customers.

Who is responsible for product analytics?

Sometimes the question arises whether product marketing or product management should be responsible for conducting analysis on the product.

This very much depends on the skills and responsibilities of the individuals concerned, that being said, often product marketing’s main focus will be on acquisition, activation and referral, while Product Managers will be more interested in engagement and retention. This differs greatly between companies at different stages of development, however.

The most important thing is that product analysis is conducted and the results are shared with the entire team or organization in order to create a data culture that puts trust in data-led decision-making.

Data Product Management

Data product management is a new way of seeing data. A company’s data is viewed in the same way a product manager views a product – that is with a sense of structure, evolution, testing and ownership.

This role has emerged from a need to manage the growing technology stacks that are required to compete in modern markets. It is a multidisciplinary practice that involves data science, engineering, and business strategy. As part of this, Data Product Managers identify data-driven opportunities, develop data products, and manage the data lifecycle from conception to retirement.

The product-development loop: how a product manager can use product analytics to update a product

Before launch, the intended impact of the update should be made clear. There should then be a formal process for A/B testing the new features with a subset of users prior to any rollout. Only if the A/B test shows that the update had the desired effect on the user base should it be launched – otherwise, it should be revised.

Having the discipline to stick to this formal process is the only way product teams can ensure that over time successive releases drive significant incremental improvements to the products.

Product analytics and company culture

A strict process for managing product development with data is only possible with the right company culture. It is essential that:

1. Arguments evidenced by data hold weight, especially with senior management who often have their own ideas about how a product develops. (“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.” Jim Barksdale.)
2. Experimentation is encouraged.
3. Failures are treated as learning experiences.

Some organisations value data and experimentation more than others. Too many organisations only pay lip service to the value of data – data is quoted in arguments, but is often not decisive. Too many organisations are also not committed to rigorous experimentation.

Product teams in organisations that do not have that deep cultural commitment to data and experimentation will constantly be under pressure to cut corners. This means, even if the team is committed to data and experimentation, success is not guaranteed.

“Committing the resources necessary to run all the different experiments a process like this requires necessitates a very high level of organizational commitment. Only those organisations that really value the solid insight that derives from that experimentation will make that commitment. For those organisations that do commit the resources, that commitment, to both data and experimentation, will be an important part of the organization’s culture.”

Yali Sassoon | Snowplow cofounder

Case Study 1: using data to inform product development

This service, which enabled customers to share their photos and videos, needs to find the parts of the product that really made a difference to their users. The core rationale was to grow their user base more rapidly. The team knew they were spending a lot on acquiring new customers, but their user base wasn’t growing as fast as they were expecting.

Naturally, the first question to ask was: what’s going wrong?

Read more about this product analytics journey

Case Study 2 (technical read): how to develop better games with level analytics

Product managers and game designers generally aim to design game levels in such a way that they challenge gamers enough to make completing a level satisfying, but not so challenging that they drop out and stop playing the game. This complex task requires a constant state of testing and iteration once a game is released in order to find what works and what creates churn.

Learn how CodeCombat used product analytics to make their game more ‘sticky’.

A word about product analytics with Snowplow

Snowplow is an advanced Behavioral Data Platform (BDP). The aim of Snowplow is to give product teams access to the type of custom, granular and accurate data used by data industry leaders, without the technical overheads.

Snowplow allows teams to take a step up from basic product analytics and iterate on far more advanced use cases – ramping up the complexity as they go. This can culminate in cutting-edge data applications, such as churn prediction models which can automatically serve the right customers with the right words at the right time.

Take a look at our ultra-granular data for yourself

‍