The changing role of data scientists: from astronaut to race-car driver

To unlock commercial value from their data, businesses are scrambling to hire data scientists who are in short supply, write Amit Joshi, Magali Deloof and Henriëtte Hoving

This article is republished with permission from I by IMD, the knowledge platform of IMD Business School. You may access the original article here.

Demand for data scientists has skyrocketed as many organisations struggle to make sense of – and reap value from – the unprecedented volume of data they have access to. Data analytics empowers businesses with actionable insights, enabling them to make better decisions, improve operational efficiency, understand customers, stay competitive, manage risks, forecast trends, and enhance marketing and sales effectiveness. It has become an invaluable tool for driving business growth and success in today’s data-driven world.

However, many of the skills and activities performed by data scientists are being commoditised – easily replicated or substituted. Most companies can leverage analytics with basic data knowledge combined with the appropriate software tools. Consequently, they no longer need to compete for this elusive talent.

Instead, they can shift their recruitment efforts to a much smaller pool of technical experts, combined with hiring or upskilling existing employees and using software tools to crunch their data. This can help them significantly alleviate their shortage of analytics expertise.

The sexiest job in the world?

About a decade ago, Harvard Business Review declared the role of data scientist to be the sexiest job of the 21st century. These professionals create automated actions or insights from typically large volumes of data. And they are still in high demand today.

 According to the US Bureau of Labor Statistics, employment of data scientists is projected to grow 36 per cent between 2021 to 2031, much faster than the average for all occupations. Companies are prepared to pay handsomely for this talent: the US median wage for a data scientist is $103,500, compared with $97,430 for overall IT occupations, according to the Bureau.

However, organisations need not fork out on human capital anymore: easy-to-use software is democratising even those complex tasks that previously required data scientists. Artificial intelligence tools like DataRobot and AutoML help build models without the need to code. There are solutions to convert speech to text or detect objects, made available free of charge by opensource pioneers such as HuggingFace. This is great news for organisations that want to put data at the core of their business model.

Read more: Three ways data analytics will change the future of accounting

The commoditisation of data science

For most data scientists, a considerable chunk of the work involves “scrubbing” the data: collecting, cleaning, structuring, and processing it to avoid creating fatally flawed algorithms. But in the next few years, we expect that menial data preparation will be carried out using software tools, such as IBM InfoSphere Information Server or innovative solutions from newcomers like Sweephy, a low-code data cleaning startup. Gartner predicts that 70 per cent of new applications developed by organisations will use low- or no-code solutions by 2025, up from under 25 per cent in 2020.

Even the tasks where data scientists once stood out due to their creativity, analytical strength, and business acumen are slowly being automated. That includes the creation of algorithms, which transform clean and reliable data into predictions and recommendations. The rapid growth and adaptation of generative AI tools, such as ChatGPT, has given even more impetus to this automation. Specifically, three forces are slowly eroding the need for these individuals.

First, we are starting to see large-scale commoditisation of frequently used data services such as recommender engines, chatbots, and matching systems. The argument for building a customised data service is quickly losing ground in the face of emerging libraries of affordable and off-the-shelf algorithms and low-code solutions.

Second, the sheer volume of data, rather than the algorithm itself, is what is most valuable to organisations, especially those using advanced machine-learning techniques. The importance of volume over sophistication is visible in many winning applications, including TikTok, Spotify, and Netflix, which all leverage simple recommendation engines to generate value for users.

Third, many data scientists are technical specialists who lack a business background. This can lead to a lack of appreciation of the actual problems their tool is meant to address, as well as a misunderstanding of how it will be used by customers.

The data scientist: 2.0

Taken together, these three forces are leading to a different role for the data scientist: from that of an astronaut (using state-of-the-art technology to venture into unchartered territory) to that of a champion race-car driver (using standardised technologies for real-world navigation).

A growing responsibility of the data scientist 2.0 is to bridge the gap between business and technology by translating broad business questions into clear data opportunities, as well as selecting the right set of services and tools to grasp the opportunities.

Read more: What technology skills are needed to future-proof business?

This implies different skill sets are needed, with a greater focus on adaptability, creativity, and business acumen. Organisations no longer need data scientists with PhDs in math or statistics to get clean data and smart algorithms, at least not to the extent they once did.

That means it has never been easier for companies to compete in today’s data-driven world. But there are some important considerations, such as starting early and seriously factoring investment in data into one’s business model.

We have three further recommendations for companies that want to harness the power of data analytics.

Train employees to act as ‘data translators’

When it comes to building data capabilities, start by upping your average employee’s data skills rather than narrowly focusing on recruiting hard-to-find data scientists and then training them to understand your business. This will automatically reduce the number of data scientists you need.

Given the commoditisation of data services and automation of tasks, it is becoming easier and more effective to teach functional experts the basics of adopting data science to address specific problems. In other words, data science is becoming easier to understand and deploy for the data-savvy businessperson. For instance, a Japanese consumer electronics giant has now made it compulsory for all new hires in the organisation to learn how to use a basic data analytics tool, irrespective of function and level.

So, train suitable employees to act as “data translators,” who can bolster the essential bridge between technology and business that most organisations lack. It will also help you make the right trade-offs between off-the-shelf solutions and customised development.

Keep hiring data scientists, but for specific tasks

However, this doesn’t mean you don’t need data scientists anymore — you do, but they should focus more on bringing technically sophisticated applications, scalability, and innovation to bear.

As a large Dutch bank discovered, highly-skilled -technical roles are best organised as part of a separate team that can be tapped by different departments as a shared resource. The bank created a system in which the data team connects with the business through data translators, who act as data product owners. This ensures the data scientists stay focused, and work on projects that create real business value.

Subscribe to BusinessThink for the latest research, analysis and insights from UNSW Business School

Invest in analytics infrastructure

Imagine you are a consumer goods company that wants to increase sales by cross-selling to customers. You will need granular data to understand what they are buying and what their interests are, across your various products. This needs to be combined with simple models that provide individual-level recommendations. But far too often, we see companies embarking on an elaborate “data crusade” to collect consumer data, without having defined up front what they will use it for and how it will help deliver business results.

So, invest in systems that allow for the modularisation of the most commonly used analytic tools and the data needed to drive them.

Ultimately, every company can benefit from the application of analytics to solve pressing problems. However, not all of them need a cadre of cutting-edge scientists to do so. As data analytics is democratised, organisations should rethink the mix of skills and capabilities they need to successfully navigate their business through the digital era.

Amit Joshi is Professor of AI, Analytics, and Marketing Strategy at IMD and Program Director of the Digital Analytics program, AI for Business Sprint, and the Business Analytics for Leaders course. This research builds on his 2020 paper Hollywood studio filmmaking in the age of Netflix: a tale of two institutional logics, which was co-authored by Allègre L. Hadida, Joseph Lampel, W. David Walls) and published in Journal of Cultural Economics. Magali Deloof is a Partner at Simon-Kucher, a strategy consultancy company. Henriëtte Hoving is an associate partner at SparkOptimus, a pan-EU digital strategy consultancy, and leads the firm’s data transformation practice.