Part two of our series on product analytics. Read: Part 1, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8
“Every employee should be empowered to make data informed decisions,” wrote Jeff Feng, PM Lead for Data at Airbnb, in a post on the roomshare service’s Engineering & Data Science blog. Calling it one of Airbnb’s fundamental beliefs, Feng identifies this desire for empowerment as the driving force behind the Data Science team’s mission to scale their skillset, giving every team member access to the skills necessary to make data informed decisions. Their solution, Data University, is a means by which employees can receive training in acquiring, manipulating, and analyzing data.
If you build it (for them), they will come
Any technology-based startup will face the same challenge at some point right after launch: you offer a good product or service, one you’re confident there’s a desire for, but there’s little to no user adoption. Establishing product-market fit can be incredibly challenging and requires much strategic forethought around your intended users: the better you know them, the more clearly you understand their needs and motivations, the better you can optimize your product for them. It’s no surprise, then, that the companies who can understand their users better than their competitors can develop a product that’s a better fit for their market.
Think of the companies with the biggest digital products in the world today, companies like Facebook, LinkedIn, and Airbnb: all known for the innovative ways they use data, these companies are savvy and creative about how they improve their products. Facebook, for example, doesn’t introduce a new feature to its two billion plus users at random, no matter how good of an idea it might seem. Through a rigorous and systematic process, Facebook product teams (which consist of around eight engineers, one or two designers, one data engineer, one data scientist, and a product manager) run hundreds of tests daily to identify the features and adjustments most likely to have a positive, and measurable, impact. They use sophisticated and powerful tools to coordinate testing and collect the results, allowing Facebook to roll out only the best features. The data collected, even from failed tests, helps Facebook deepen their understanding of how people use their platform, allowing them to make it “easier to use, faster, and more engaging.”
Culture counts
In what Boz, VP of AR/VR, describes as a ship early, ship often process, the product teams at Facebook are in a constant stream of testing and updating the many products that comprise the Facebook platform.
When you see such dramatic results from the smallest tweaks, you realize how much opportunity there is to improve things—and we feel a constant sense of urgency to do so. When a test goes out we look at the data immediately and adapt the products quickly. We do this on a daily basis. This cycle of iteration is the engine of progress and the people who use Facebook are not just the beneficiaries but are also intimately a part of the process. We don’t just develop this product for them, we develop it with them. -Boz
Consider again Airbnb and their relatively new Data University. Their approach to democratizing data internally is to spread the skills, letting each employee learn as much as is desired or necessary to interact with data while Facebook embeds equipped professionals within their product teams. With almost three hundred open positions within the data sector at Facebook, they are rapidly and aggressively growing their data team, allowing for product teams, as well as other areas, to have dedicated data professionals because the company believes this is a crucial skill for each team to have.
Contrasting with most companies where you have a centralized analytics or data science function and people in that function then farm out to serve multiple teams and products, Facebook has placed a strong emphasis on data related skills where other companies have not. Though these two methodologies may be different, they converge at the same point: data has a critical role to play in the successful decision making process. Data University is just about one year old now but the cultural focus on the importance of data is hardly anything new for Airbnb.
Product development home run
Talking about the early days of the startup on NPR’s How I Built This podcast, Airbnb founder Joe Gebbia described what he referred to as the trough of sorrow, “where you have a product and a market and they don’t fit, like two gears that don’t touch and you can’t figure out how to close the gap… it’s completely flat, there’s zero growth.” This is a crucial period in the launching of a new product or piece of technology, when you determine your product-market fit. However, knowing what to measure, what metrics are significant, and collecting that data are challenging and make determining product-market fit tough. In Joe’s words, “This is when people tend to quit.”
Joe recognized that there was a problem with adoption, one that could be identified. “We looked at the stats and the data around the early adopters and we noticed a pattern: people don’t know how to take good photos of their home.” The data alone didn’t tell them this. Arriving at the conclusions that the most successful homes on Airbnb had better images, both in quality and ability to display the home’s environment, but the majority of listings lacked quality photos and was blocking adoption was not purely an analytical exercise. Taking the appropriate action required a combination of high quality data along with the intuitive insight gained from thoroughly understanding their product vision and who they want the product to serve. Seeing how people used the Airbnb website showed them that what they thought was perfectly designed was confusing and wrong. It turned out the gap in the market was a poor UI. Armed with that understanding, after making minor tweaks informed by their data they were able to double revenue in the space of one week.
Analytics drives iteration and innovation
Today, data at Airbnb remains so important that the team has their own dedicated Twitter account and blog space. As the platform continues to grow and spread around the world, the data scientists at Airbnb are constantly identifying and improving pain points to keep the end to end user experience frictionless and positive. It’s easy to focus on the happy user, the “correct” user path. Product analytics becomes a very simple, reductive process when everyone does everything right and everything goes well. You have your product funnel and you track conversions from point to point.
But even a product that seems as simple as Airbnb (from a user perspective) has deep complexity to it. “The range of challenges that people can face is pretty large, it’s not a short list,” Elena Grewal, Head of Data Science, explains to the host of Developer Tea. “Airbnb is not a simple product to use.” Whether the data scientists are tracking transactional payment data to monitor for cross-border, cross-currency integrity or ensuring users who need specialized help are able to connect with the appropriate customer support representatives, these are multi-dimensional problems that can only be solved effectively with good data along with deep understanding of the users and their varied experiences.
Play to win
The user journey is complex, as the example above shows. Using data in the product development process is difficult because your users and their journeys are far from straightforward. This is the crux of why product analytics is crucial: strong, data-driven product teams will have the insight to build products that better serve your users, outclassing your competitors. In this highly competitive space, inches matter, and any tool that can give product teams an edge becomes a force multiplier. Data, as Facebook, Airbnb, and others have shown, has become one of those things that can give product managers and their companies an edge. We’ve seen, across the companies that we work with, that those companies that are better at using their Snowplow data to drive their product development process, frankly, grow faster and do better than those that don’t.
