The role of data scientist – one who creates information and makes discoveries from data – has been declared the sexiest job of the 21st century by the Harvard Business Review.
For what ? Well, two years ago the study authors revised their prognosis: they believe that data scientists have become absolutely vital to their companies, especially in the era of artificial intelligence and machine learning ( ML).
The job has also evolved, partly for the better, but also for the worse.
Still cleaning data, despite AI
“It has become more institutionalized, the scope of work has been redefined, the technology behind it has made huge advances, and the importance of non-technical expertise, such as ethics and change management, has increased”, observe the authors of the study.
At the same time, data scientists report “spending much of their time cleaning and manipulating data, and this is still the case despite some advances in using AI to improve data management “.
More importantly, “many organizations do not have a data-driven culture and do not take advantage of the insights provided by data scientists,” the study finds.
“Many of them are frustrated, which leads to high turnover”
And draw a rather harsh observation. “Being hired and paid well doesn’t mean data scientists will be able to make a difference for their employers. As a result, many become frustrated, leading to high turnover.”
Professionals respect data scientists within the company. But they tend not to act on their recommendations or ideas, as another recent survey from Rexer Analytics confirms.
Only 22% of data scientists say their initiatives – often models developed to enable a new process or capability – typically result in deployment, the survey observed. More than four in ten (43%) respondents say 80% or more of their new models are not deployed.
The interaction between professions and data scientists at the heart of the problem
In many cases, even changes to existing models do not make it into production. “On all types of ML projects – including refreshing models for existing deployments – only 32% say their models are usually deployed,” the study said.
So what’s the problem ?
Interaction between the business and data science teams – or lack thereof – appears to be at the heart of many problems.
A problem of definition
Only 34% of data scientists say the goals of data science projects “are usually well defined before they start,” according to the survey.
In addition, less than half (49%) say that business leaders who must approve the deployment of models “are sufficiently informed to make such decisions fully.”
Overall, the top reasons cited for not deploying recommended machine learning models include:
- Policymakers are not willing to approve modifying existing operations with new models.
- Lack of sufficient planning to carry out projects.
- Lack of understanding of the correct way to execute model deployment.
- Problems with availability of data necessary for model evaluation.
- No one person is designated to manage the deployment.
- Staff are unwilling or unable to work effectively with the model results.
- Technical barriers to implementing/integrating the model into existing systems.
Struggle for deployment
The struggle for deployment stems from two main factors says the study:
- “Rampant under-planning and a lack of concrete visibility from stakeholders. Many data professionals and business leaders have not yet understood that ML operationalization must be planned down to the smallest detail and pursued with determination from the start of each ML project” says the study.
- Business leaders or professionals need greater visibility “into precisely how ML will improve their operations and the value this improvement is expected to deliver,” the study adds. “They need it to give the green light to deploy a model and, before that, to influence the execution of the project throughout the pre-deployment phases.”
It is also important to note that the performance of the ML project is often not measured.
Too often, performance measures are based on obscure technical metrics, rather than business metrics, such as ROI.
An increasingly high level of professional satisfaction
However, the job of data scientist is a great job, which is constantly improving, as the Rexer survey suggests.
In the previous survey, in 2020, 23% of corporate data scientists reported having a high level of job satisfaction. A percentage that almost doubled to 41% in this most recent survey. Only 5% of them say they are dissatisfied, compared to 12% in 2020.
The appetite for data science skills also continues to grow.
Data scientists continue to be hard to find – 40% say they are concerned about talent shortages within their companies. Half say their organization has increased internal training to improve data science skills, while 39% are collaborating with universities to promote interest in data science.
Source: “ZDNet.com”