Everything has changed. It’s now hard to sum up how fundamentally different our world has become since the spread of COVID-19 ground everyone’s pre-crisis plans to a halt. It is a difficult and chaotic time for many of us.
Recently I gave a talk at the Open Data Science Conference (ODSC) 2020, addressing the global challenges we’re facing, outlining how data teams can continue to deliver value in a post-COVID world. The problems we’re facing are serious, but I believe that despite those challenges, and without any intention to trivialize the situation, there are grounds for hope for modern data teams. I’d like to share some of my thoughts from my recent talk, some tangible steps data professionals can take, and some suggestions for how to navigate the next 12 months.
To watch my full talk, click below.
Understand the trend towards digital
COVID-19 is accelerating a drive towards digital in many businesses. Although every major company has been trying to implement digital transformation over the last 20 years, many of these projects have failed. What we can see today is that digital transformation is no longer a nice to have – it’s now essential for survival. Many of our face-to-face interactions – doctors appointments, cinema, high-street stores, our office environments – none of these will come back to the same extent after the crisis. Those behavioral changes have accelerated underlying challenges and have brought us to a new normal. That’s an important trend for us to be aware of in the data industry.
Embrace creativity
On a positive note, life under quarantine has unleashed so much learning and creativity, as well as reskilling. People are dusting off old hobbies, learning new skills, new capabilities and it’s a great time to invest in personal and people development. Many of us are going deeper on the topics that we always wished we had time to look at, but had otherwise been too busy to focus on. My feeling is that we will see a lot of developers reskilling and pivoting into data engineering, analytics engineering and data ops, with many people migrating to the data space, attending conferences, downloading books from Manning and O’Reilly and taking online courses.
Articulate your value
There are two lenses to consider in terms of value data teams deliver to their business. The first is business impact; the second is weight and variance.
Digital expert Avinash Kaushik has a great metaphor of the rosebush to explain this. He argues that data teams present a beautiful rosebush while the house behind it is going up in flames. Data teams need to focus on presenting the ‘rosebush’ right now – if you have 76 slides to present, only pick the stories that show the business impact in what you are doing.
Secondly, I recently listened to an excellent talk by Professor Sonia Marciano, who discussed weight and variance. She explained that project attributes can be perceived in terms of not just their ‘weight’ – how much value an attribute has in overall estimation – but also their ‘variance’ – how differently the value of an attribute a beholder is willing or able to see. As an example, Professor Marciano spoke about her daughter’s school assignment to make a puppet. The assignment was worth 15% of her overall grade (high weight), but Marciano noticed the best puppets scored an A, while the worst puppets (socks with eyes) scored a B+ (low variance).
The takeaway here is that it’s best to choose data projects with high weight _and _high variance, to convey maximum value to your organization.
What to do over the next 12 months
Here are some tangible actions I recommend for data teams over the course of the next 12 months:
- Speed up decision making in your company with data. In the current global climate, companies are having to make more decisions at a faster rate. Companies that were making 12-month budgets are now working on 3-month budgets. These decisions will be made whether or not data is available – make sure that data is available!
- Focus on forward-deploying data, not just looking back. Most data teams use historical data to look back, but I recommend also instituting a forward-looking data culture, sometimes known as ‘operational analytics’. It’s really important that we look for ways to operationalize our data to inform and improve fundamental aspects of our business. For example, if you’re a subscription business, data can play a key role in predicting and reducing churn – check out Carl Gold’s (Chief Data Scientist at Azure) book Fighting Churn with Data for more on this.
- Build that awesome data product that everyone said was impossible. It won’t be the same for all of us, but some data teams will find much-needed breathing space this year to build that killer product or undertake that killer data project. These unusual times present an opportunity to build and create difficult things that your ‘business-as-usual’ has always got in the way of.
- Figure out how to stress test your business. How resilient is the business you’re working for right now? Stress testing is a worthwhile exercise to figure out the main stressors and drastic scenarios for your business and flag them to the right people in time for appropriate intervention. We data professionals are especially well-placed to help our organizations to look for spikes or anomalies to catch certain situations before they become disruptive.
- Do what you can to help in the global recovery. There are too many things to call out and too many ways to get involved, but the Alan Turing Institute is a good one – they are calling for data professionals to support their initiatives and live projects in the fight against COVID-19.
What not to do over the next 12 months
- Avoid creating a data breadline within your organization. A data breadline forms when departments and managers have to queue up to get ‘their’ data, reports and insights from the data team. It’s vitally important that data teams figure out how they can make high-quality data as self-serve as possible: think about training, data catalogues and tutorials – anything that can help to lift the burden from the data team and reduce contention.
- Don’t over-engineer your data remit for future black swans. It’s common right now for companies to think about the next ‘black swan’, in other words the next unexpected high impact event. Whatever lies next over the horizon, it likely won’t look like the last three black swans, so I’d recommend not focusing too much time on speculation or hypotheses.
Post-crisis predictions
A retreat from shiny objects. I think we are going to see a retreat from ‘shiny objects’. These projects (such as AI, neural networks, blockchain, distributed ledger, deep learning and fog computing) can have great merit for some businesses, but are overkill for many others. I think we will see many companies taking a step back from many of these bleeding-edge projects.
We’ll move back to basics. Instead, I think we will see a move back to basics. Previously I’ve discussed the data science ‘hierarchy of needs’, and how
we, as data professionals want to move up the ladder from basic data being available, to industry-leading work and advanced use-cases. Funnily enough, I think we will see companies temporarily moving down the ladder, focusing more on getting the plumbing right, ensuring high-quality data and pulling back from aspirational things to the more ‘nuts and bolts’ of data.
A contracting data landscape. Snowplow is a software vendor in the hugely crowded data landscape, and a number of startups are having a difficult time in this period. Sadly, I feel we are going to see some consolidations, pivots and some vendors closing their doors. If you’re exploring vendors and solutions in this space, I would always suggest looking first at open source projects and open-core vendors.
The road just got more bumpy. Our data journeys as data professionals, data teams and the organizations we work for, have got a lot more bumpy. We will all be following the situation closely to see how it develops. I wish everybody the best in these uncertain times.
