It’s an exciting time to be working in data. The opportunity for data teams to make an impact is huge. They have more tools at their disposal, with exciting technology hitting the market each day, than ever before.
Hiring in data, despite a slowdown in 2020, continues to be hot, with new job titles emerging like the Analytics Engineer and growing investment in data capabilities, and the market is projected to be worth $103 billion by 2023.
| Watch the webinar: People and process in data with Alex Dean and Scott Breitenother |
But it’s not all roses. Building and managing a modern data capability is a tremendous challenge. Aside from the technology involved, it requires working closely with technologists with a broad range of skill sets. It means working with (i.e. winning over) internal teams and stakeholders and educating them around the value of data. Sometimes it’s a challenge just to get a seat at the table and make the case for data in the business.
And even with all the right things in place: the tools, the people and buy-in from the right stakeholders, there is only so much one team can achieve. Small data teams can find themselves stretched, under-resourced while overwhelmed by company demands.
To get a better understanding of the challenges of working in data, we spoke to some of our customers, asking about the key pain points in their everyday working lives. Throughout the article, you’ll find featured snippets and quotes from conversations with our customers.
The answers below were not gathered in a formal or scientific way, nor is the list of challenges exhaustive. Nonetheless, we hope they shed some light on the experience of a data professional in 2021.
N.B. Our quotes below have been anonymized and paraphrased to protect the privacy of our interviewees.
Internal customers and responsibilities
During our conversations, the difficulties of working with internal customers made up around 21% of the challenges mentioned.
When we think of ‘customers’ we might imagine people buying products or services. But pretty much all the data professionals we spoke to listed their primary ‘customers’ as internal colleagues. That could mean working with finance, marketing, sales teams, operational teams, or sometimes even the C-suite.
A crucial part of working with those teams was winning their trust. The data professionals we talked to spoke about the need to win over these colleagues, build strong relationships with them and serve them efficiently.
‘I want my customers [internally] to trust the data. I want them to be able to get the answers they’re looking for from the data, and get them faster, eventually through self-service. Marketing, finance, business development all depend on us on a daily basis.’
In a sense, data teams have to act a bit like internal detectives – investigating what their internal customers need and building a plan to deliver it.
‘I try to understand what my counterparts in finance, accounting, operations and marketing are trying to accomplish – and then look ahead to see if we have the technology to do what they want to do.’
‘I often say “if we want to deliver what product wants in 9 months, we have to start today’
Time is of the essence. There is pressure for data teams to deliver data efficiently to their data consumers. Marketing and product teams don’t want to wait around for data they need to execute use cases. Equally important is getting the data at the right latency.
‘We are responsible for working with other engineering teams to enhance and/or create data sets. We ingest the data, process it and deliver it to where it needs to be’
‘I want to make sure we can deliver data at the speed our teams need it’
‘Our job is to help the business to work with data in the most efficient way possible’
But above all, data teams mentioned that the data needs to be clear and coherent. It needs to be a single source of truth, so all teams can be on the same page.
‘Our main customers are our business units. We provide support to partners, marketing, sales, account managers – we make sure they’re all speaking the same language and looking at the same data’.
Tool evaluation
Evaluating and purchasing tools made up over 10% of the challenges mentioned by our customers. With technology constantly evolving in the data industry, perhaps this comes as no surprise.
A quick look at Indicative’s recent post on the ecosystem of modern data infrastructure shows you that the challenge of choosing the right tools for the data stack is no joke. Even with Indicative’s selection and helpful breakdown of each tech category, it’s clear from their diagram that modern data teams face a number of choices for how they build out their data capability.
Our customers explained that tool evaluation (and selection) was an ongoing challenge for them. They told us that picking the right tools demands constant research, investigation, trialing and careful planning to ensure their teams are well equipped.
‘[When I’m evaluating tools] I’m looking at what is our business need, where are we going, how are we growing? – what projects do we have on the table and are they staffed properly?’
‘A really big part of my job is “how do we not spend tons of money on tools we’re not going to use?”’
When it comes to buying tools, our customers were clear that they preferred to put in the research themselves before contacting a sales team. Not only does this save them time in the long run, it also gives them the opportunity to investigate not just pricing and features, but the ‘softer’ side of the tools – e.g. is there a community? Are there other teams using this solution who I can reach out to?
‘I do a lot of research. I like to know a lot about the product before I call the sales person.’
People and communication challenges
Challenges around people and communication were brought up the most, making up around 29% of the difficulties our data professionals mentioned.
Communication was actually a common thread in all of our conversations with our customers. It boils down to the need to be able to educate and inspire confidence in internal clients – demonstrating the value in data and data processes.
And data professionals, despite the stereotypes, are good communicators. One customer summed it up perfectly:
‘Working agile forces you to be better at working with people. The stereotype of the engineer working alone with headphones on is totally wrong’.
According to the customers we spoke to, technical ability isn’t enough. Data professionals need to back it up with strong communication skills and the ability to to guide people towards a common goal.
‘You can have the greatest technical skills but if you are an ineffective communicator or ineffective at having influence to move a group of smart people forward toward a common goal – it’s really hard to get the job done’
‘Communication is 50% of the work. Understanding one’s audience – whether executive level or at the front-line level, I have to be able to calibrate my message for the audience.’
That’s not to say that technical skills are not important. But for data leaders, it’s also about amplifying your team’s abilities with appropriate team structures and processes in place. Put another way, the best tools cannot be leveraged without the right people.
‘Tech is still really important. Tools position you well, but you can have all the chess pieces in place but if the people aren’t ready, you’re going to have real challenges’
One of the biggest challenges for data teams is to prove their value, or the value of their work. Our customers mentioned the constant battle to win buy-in from their colleagues and stakeholders.
Sometimes it’s about showcasing a new, exciting way of working with data. But often (and more challenging) it’s the case that data leaders need to convince colleagues that an investment in a certain data project is worth the time and resources. In both cases, communication and the ability to ‘sell’ the value in data are key.
‘We run showcases for the business to show new builds, new features, how they work and how they impact peoples’ jobs’.
‘It’s hard to get stakeholders excited about a large investment just for accurate insights’
‘When you talk to a developer their initial thoughts [around tracking] are “this is going to take a lot more time.” It’s our job to convince them of the value of that investment.’
At other times, education around data is crucial. Working with data means handling a vital business asset that should be treated as such, especially when sensitive data and customer information (PII) is involved.
And while ‘self-service’ data is often the dream, data professionals are tasked with coaching the rest of the organization on how they can find, understand and work with their data. While some in the business (e.g. developers and engineers) may already be data-proficient, that’s not always the case for other stakeholders.
‘Self-serve is a huge challenge. We want lots of departments to be able to access data and work with it. But it’s really hard to teach everyone to be competent with data and use data safely.’
‘Building consensus can be challenging when there are so many stakeholders involved with different levels of knowledge.’
Without education around data practices, resources can go to waste and teams can grow frustrated with their data product. As one customer found, it’s a long term challenge when internal teams are ‘sold the dream’ by a packaged solution, only for it to go wrong in the longer term. This could have been avoided if the data team was involved earlier in the evaluation process.
‘I would see marketing go out and buy some tool, get sold on the pretty pictures and then they’d have us make a bunch of changes to implement it, only then not see use it in the long run. This happened a few times.’
Working in data is a constant challenge
Between the continual demands of the organization, ongoing tool evaluations, hiring, decision making and dealing with internal stakeholders – data teams have it tough.
Thankfully the data industry is maturing. Each year there are more tools available, more budding communities of helpful contributors and examples of content (hopefully including this one!) to guide data teams towards success.
Sadly there’s no silver bullet to the challenges of working in data. But as one customer summed it up, one key element might lie in hiring, educating and equipping the right people, arguably the most important part of your data capability.
‘Getting the ‘people bit’ right is actually the hardest part for companies when it comes to data.’
Despite the challenges, data teams are still driving huge value in their organizations, from empowering their colleagues with insights to turning game-changing use cases into reality.
At Snowplow, we want to make it as easy as possible for data professionals to manage and work with their behavioral data.
This is a 5-part series
Click below to navigate to the next chapter:
-
Chapter 1 The challenges of working in data in 2021
-
Chapter 2 A guide to data team structures with examples
-
Chapter 3 Breaking communication barriers with a universal language
-
Chapter 4 Reducing data downtime with data observability
-
Chapter 5 How data storytelling can make your insights more effective
