Is AI Replacing Data Scientists?

AI has created a world where people with no knowledge have suddenly become a subject matter experts.

For me, as a data scientist, watching this bot go through an entire dataset, conduct a certain level of EDA, and create a predictive algorithm, was amazing. Almost intoxicating. What took me hours, even days, was now done in minutes.

It was too good to be true. The best $20 I’d spent.

One hot summer day, one of those days where the AC couldn’t keep up, I was hot, and annoyed. I was working on a project that was getting under my skin.

What made the project so complex was multiple data sources. The ETL process was tedious and, any other time, it would require a lot of brainpower. Not anymore. With a few prompts, the ETL code was done, and data was flowing in.

Before I took up the challenge, I was aware 95% of it was done with AI and little to no human oversight. A machine can do beter than men, right?

At first, everything appeared to be working well, and I was pleased. But on that fateful day, my eyes caught the glimpse of something that didn’t look right. It seems like that everytime I am either annoyed or bored is when my brain catches things.

The machine learning output, though calibrated, was pushing out some questionable results.

The Algorithm Issue

When I suspected something was wrong, I went into problem finding mode.

As a data scientist, the first thing I did was check the usual suspects — F-score, accuracy, precision, recall, ROC curve, etc.

The algorithm, while distinctively not performing the way it should, was passing every metric. Nothing stood out.

The problem was, when I manually checked the results I know it should be included, it weren’t. Certainly, it was me. Feeling crazy, I doubled checked my own work.

I began pulling on threads that I thought would lead me somewhere, but didn’t.

So, I did what any prudent data scientist would do in this situation. I turned to AI. Impressively, its analysis pinpointed multiple issues with the very algorithm it had created in the first place.

Even after the improvements the problem persisted.

I decided to go back to 2019, when I banged my head against the wall and figured things out on my own. Or using Stackoverflow.

The Problem

The problem, as in most cases, happened to be the data. Somewhere along the ETL, some of the critical fields became null. To an AI, the missing values was more of an EDA problem and not critical data. Thus, the ETL code circunvented the null.

The solution to the problem was not removing the null values, but enriching them with the correct data from another source. AI would never know that because it doesn’t have critical thinking.

Trusting Your Business to AI

This was not the first time that I encountered businesses trusting their data for decision making to AI.

Blindly trusting AI output without even checking is something that I still can’t really process.

The crazy part is that the people who we would consider the experts in this subject are delusional. For the better part of a decade, we heard about how AI will change everything.

You have Microsoft AI CEO, Mustafa Suleyman, say that most white colar jobs will be gone in 18 months. Just 18 months. Imagine that. This was not the first time a tech leader has predicted something like this.

But then, when I look around and see things like: Deilotte lost six figures because of AI, it make me think that Suleyman is too optmistic.

To be fair in my assessment, AI has been leaping faster than any other technology. So, anything from last year, it is already old news. However, recently, the advances are flattening.

One problem that AI companies are having is something called Data Walls. Labs are running out of high quality data to train larger system. But even with all the training in the world, AI will never be like a human mind. Intuitive and critical.

Data Quality

During a discovery meeting, a business owner decided that my services could be easily done with AI. And to some extent, I agree. I’m using AI for the bulk of my work.

This potential client, put his excel spreadsheet into one of the LLMs and told it to create a dashboard based on the numbers. A few prompts later, he had this beautiful visualization.

When the numbers were not working, I got a call.

AI had invented a bunch of statistical trends and mathematical relationships that weren’t there. Not to mention that it took his request at face value. It didn’t ask the probing questions to figure out what he really needed.

Conclusion

Can you trust AI to do the data analysis for your business?

The short answer is, no. When using the data to make decisions, you have to make sure that it is good and appropriate to use. Garbage In, Garbage Out is critical and could cost you a lot of money.

While hiring a expert might not be in your budget, you cannot afford not to hire one.

AI is not going discover that you have data silos in your company, or that there is a possibility that your data is mislabeled and missing critical values.

This is where an expert comes in and assist you with your data needs.