Why Many Data Scientists Quit Good Jobs at Good Companies?

Examine why the sexiest jobs of the 21st century have lost their appeal

dream job?

We often hear about how popular data science is as a career. You’ll often find articles about data science being the ” sexiest job of the 21st century, ” and about how you can expect higher salaries with years of experience.

Data science has many attractions. It’s a rewarding job, you learn a lot, and you never get bored. Compared to many other professions, data scientists are given a lot more autonomy to explore and solve interesting problems. It also often gives you the opportunity to work alongside talented veterans in a variety of fields.

Despite this, Kaggle’s research shows that many data scientists spend several hours a week looking for a new job. In fact, machine learning people consistently rank high on the list of developers looking for a new job, with 20.5% according to Stack Overflow’s 2020 developer survey . , second only to academic researchers.

If data science is a dream job, then these findings raise questions…

Why are so many data scientists looking for another job?

As a data scientist, I’ve also had the experience of looking for another job, so I’m writing this article in the hope that sharing my experience will improve the situation even a little.

After working as a data scientist for several years, I was promoted to director of a startup company. He is now in a more managerial and leadership role. Both my experience as a data scientist and managing a team of data scientists (as well as a wide range of developer and data teams) give me a unique perspective.

When I was working as a data scientist, I felt pain that made me want to quit my job. He mostly worked at start-ups, but changed ships several times early in his career. The reason was a combination of several factors, but many of them I’d heard from other companies as well.

In this article, I’d like to share some of the more common reasons why data scientists want to leave, and some advice on how to improve the situation. This advice is useful for people who:

• A data scientist who is currently dissatisfied and unsure whether to leave
• Managers and organizations unable to secure excellent data science talent
• Aspiring professionals looking to step into unfamiliar roles (if any of these are red flags for you, the situation is a big investment)

First let me say that I still love my job as data science. Data science can be a very rewarding career if you know how to make the most of it.

・・・

difference between reality and expectations

Using cutting-edge technology to solve difficult problems in interesting ways, and using new algorithms to develop machine learning solutions that have a big impact on organizations – data science is amazing because it allows us to do just that.

But this is often too much better than the truth.

I have hands-on experience with data science and have spoken to many in the industry, but too often reality and expectations do not align. This discrepancy is arguably the number one reason data scientists get frustrated and quit.

Well, there are many reasons for this situation. It should also be remembered that data science has two sides.

unrealistic expectations

Many early-career data scientists have never worked in a real organization. In the same way that social media presents an unrealistic picture of other people’s lives, it’s easy to become overwhelmed with exciting stories about data science and assume it’s normal.

These misconceptions are common among recent college graduates and those entering the data science industry from academic research positions. They tend to fall into the mindset of unlimited timescales and infinite budgets. I have often heard data scientists protest that there is no timeline for when a job will be finished, but that the job will take as long as it needs to. These complaints are not true and do not fit into the culture of many organizations.

Either you fix the scope of what you’re trying to accomplish and change the timescale accordingly, or you fix the timescale and change the scope of the solution.

Another big factor is the realization that most jobs aren’t that exciting. Most organizations have to juggle time between technical work and less interesting work. If you’re not good at writing reports, giving presentations, repeating the basics of models and approaches, managing projects and clerical work, and getting buy-in from other departments in your organization, these things can be a nuisance. becomes (*Translation Note 2).

harsh reality

Also, most of the infrastructure and data processing you would expect to have is often absent.

I used to work at a start-up as a second-class data scientist. My colleagues (who are older than me) who have been with us for a year and a half have spent their time building basic data pipelines. Luckily for me, they were able to persuade stakeholders to approve the budget, solve the security and IT pain points associated with the introduction of new cloud technologies, and understand what security and IT mean. Thousands of times, and took all the pain for me.

A data scientist may sometimes be used as a clever engineer who can successfully respond to rough requests and move things forward. At times like this, the reality is that the ability to do data science may become secondary.

These problems are exacerbated when you don’t have an experienced data scientist on your team or your organization’s management doesn’t have experience managing data scientists. If you’re the only data scientist in the company, it can be hard to get your point across in a way that resonates with you.

This kind of isolated situation often creates an unhappy work environment and the data scientist’s expectations are disappointed.

When you join a company as a data scientist, you might think your goal is to build smart models and get as much value out of your data as possible. But the first few months can be daunting as you have to build the necessary infrastructure and pipelines to get the data.

For senior stakeholders in the company, it feels like a lot of time goes by without results. In practice, however, they are content with simply creating simple charts for their regular board meetings. Then you start to realize that expensive resources don’t add value right away.

A disconnect between data scientists and management results in frustration on both sides.

If you have the opportunity, ask questions during the interview about the following points that may cause the above disconnection.

• Who is working on data science at the highest levels of the organization?
• Is management experienced in data science, or are they hiring data scientists because of the hype?
• How many other people are on the data team?
• Do you have a data engineer/analyst/DevOps engineer or are you expected to do it all yourself?

At any rate, the gap between expectations and reality may seem pessimistic, but it is not. In many organizations, this gap is small, so it’s important not to expect too much, but to land in positions where you’re well-supported and set up for success.

・・・

wonderful world of politics

Politics in the office. This is troublesome.

I’ve heard time and time again how well-managed, capable teams get completely emaciated, withered, and die for political reasons. I’ve heard first-hand stories of the only senior leader in an organization with a passion for data science being kicked out and the team he led quickly repurposed for menial work that doesn’t make the most of his skills.

Unfortunately, politics is an integral part of many careers. That said, you don’t have to “play games,” so to speak. Data scientists have a rare and in-demand skill set, so they can always go elsewhere.

If you’re unfamiliar with the ” leave, speak, loyalty, ignore ” decision-making model, I highly recommend giving it a read. It grew out of the work of Albert Hirschman and describes an abstract model of how individuals respond to unacceptable situations. Published in 1970, the book has been long debated and expanded .

When things go wrong, the response can be summarized in four options.

• Quit – Quit the job and look for another. Leaving companies continue to have problems, which is made worse by the loss of skills and experience of those who leave.
• Patience – hang in there and see if it gets better. If things don’t turn for the better before the cord of patience runs out, it often leads to one of the other options.
• Negligence – Reluctance to take responsibility for what is happening, resulting in being put on the sidelines for a period of time or fired.
• Speak up – Stand up and try to make a difference.

Of the four options, only “speak” actively tries to improve things. In this case, it means solving political problems in the workplace.

In many cases, the political situation within an organization may seem far beyond one’s job rank. This situation can be very uncomfortable, feeling like you can’t influence big budget cuts or big changes. Times like this are a good time to weigh your options, but sending a well-written message to someone who is a leader may actually be the catalyst for real change.

If you’re in a small organization with immediate access to decision makers, I highly recommend building a relationship with them. Contrary to what many people think, humans generally want to do the right thing for an organization and its people. Rarely do companies hire people who are truly bad guys (trying to make the organization better) and trying to make you suffer.

In many cases, senior stakeholders may not have the opportunity to understand the needs of the data science team. If you take the time to show them how you can add value and build a strong relationship with them, you’ll get the most value out of your skills. Also, having a good relationship with management can help you better understand what the business’s real concerns are as they see it in the highest positions.

In a talk to early-career data scientists, I jokingly advise, “Automate parts of your CFO or finance director’s workflow as early as possible in your career.” That way, you can demonstrate your value directly to budgeters and build allies. I’m half-joking, but honestly, they’re the busiest people in the business and often end up in Excel hell.

You have to get people with business influence to give you good reviews. Most of them don’t care how much you know about algorithms and statistics. If you do mundane tasks for them, basic data retrieval, automation, reporting, etc., they will be liked. If you can do these things with a smile and get a good reputation, things will turn around in the long run.

・・・

Data Science == Data Everything

If you can navigate office politics well, you are likely to get a good reputation. However, high evaluation is also a double-edged sword.

Many people don’t understand (or care about) what data scientist means. As mentioned earlier, you are considered a clever engineer who can get things done. Armed with access to all your data and a wide range of technical tools, you can quickly become the go-to person to solve problems.

If you can solve various problems flawlessly, that’s great. But when people start to rely on you and put pressure on you, the situation becomes a burden and uncomfortable. You might find yourself spending 80% of your time doing what a novice database administrator should be doing.

I often tell corporate executives that data scientists can do anything, but they usually work slower and more expensive than others. And any job can be stressful.

It’s fun to fill broad skill sets and loosely defined roles, but don’t fall into the trap of taking on work that other roles are better suited to, just because your organization isn’t aware of it. When you find yourself in that trap, reach out to your senior stakeholders and ask them to help you hire a database administrator or BI person who’s getting away from what you really want to do and who’s willing to do what you’re doing.

Enabling data scientists to do their job also helps solve the isolation problem. If you’re on a team that’s isolated from other departments with a small group of data scientists, you can become isolated because of your expertise in all things data. Data becomes your domain and people stop being data controllers and users. When you become isolated because of data, improving your organizational structure and expanding the role of data will help you integrate into the wider team with good results.

・・・

At the end

Unfortunately, knowing all the latest tools and algorithms is not enough to get the most out of most data science jobs. If you enter the company with some expectations and understand that the organization needs to be educated a little more, you are more likely to succeed.

I hope this article is useful for data scientists , companies hiring data scientists, or anyone looking to get into data science.

And if you need advice, feel free to contact us.