Skip to content
Exploring Machine Learning, AI, and Data Science

Self-Healing Pipelines and the Evolving Role of Data Engineers in the AI Era

Welcome back to Data Driven, the podcast exploring the dynamic intersection of artificial intelligence, data science, and the foundational world of data engineering. In this episode, we dive into the realm of data pipelines, enterprise analytics, and the ever-evolving tools that power our most advanced AI models. Our guest is a seasoned data engineering and analytics professional from the College of Southern Nevada, sharing over five years of expertise spanning pharmaceutical supply chain analytics, business intelligence, and leading-edge platforms like Azure Data Factory, Databricks, and Power BI.

The conversation focused on the often under-appreciated work of data engineers, the challenges of maintaining data quality, and the art of making data truly AI-ready. One concept discussed was reporting drift and its impact on business decision-making. A key theme that emerged was the transformative influence of AI integration within analytics platforms such as Power BI, as well as the arrival of AI agents and self-healing data pipelines. The discussion explored the universal lessons from industries like pharma and higher education, the advancing role of dashboards, and just how quickly science fiction is becoming reality in the data space. Get ready for a thought-provoking look at how organizations can future-proof their data infrastructure in the age of AI.

Links

Time Stamps

00:00 Intro and guest introduction

05:47 Copyright challenges in AI training

07:38 Early doubts about LLM effectiveness

13:08 Evolving impact of new technology

16:48 AI simplifying data analysis

18:41 Frustrations with Stack Overflow

22:45 AI agents in data engineering

26:48 AI conversation evolving rapidly

31:26 Talking about kids and technology

34:40 Building RAG for file classification

36:26 Exploring AI in Data Visualization

Transcript
Speaker:

In these systems, I do not have come across any

Speaker:

scenarios wherein like you know we need real-time data. It

Speaker:

is usually the data systems over here are

Speaker:

that no matter if you report it late,

Speaker:

but the data should be accurate. That is the primary

Speaker:

concern. Whereas if you look at the financial sector,

Speaker:

like you know banks or like you know trading firms

Speaker:

over there, those systems like you know those

Speaker:

businesses would require real. So to answer

Speaker:

your question, so far we have not had any

Speaker:

scenarios or any business use case wherein, like, you know, you

Speaker:

have to report the data as soon as it becomes

Speaker:

available. Data engineers aren't the front-end guys. They're the people

Speaker:

behind the scenes who carry out all the action. Pratham Yadav of the

Speaker:

College of Southern Nevada on dirty data, reporting drift, and

Speaker:

self-healing pipelines. This is Data Driven.

Speaker:

Hello and welcome back to Data Driven, the podcast where we explore the emerging

Speaker:

field of artificial intelligence, data science, and of course, without data

Speaker:

engineering, none of this is really possible.

Speaker:

And unfortunately, my favoritest data engineer in the world can't make it

Speaker:

today. So today I'm going solo and

Speaker:

I'm very excited to have a conversation

Speaker:

with a data engineer. We have with us

Speaker:

today Pratham Yadav, who is a data

Speaker:

engineering and analytics professional at the College of Southern

Speaker:

Nevada with 5+ years of experience across data

Speaker:

engineering, pharmaceutical supply chain analytics, and business

Speaker:

intelligence. He's a published author and he's an

Speaker:

expert in all things Azure Data Factory, Databricks,

Speaker:

SQL, Python, and Power BI. Welcome to the show.

Speaker:

Thank you for the lovely introduction, Frank.

Speaker:

No problem, no problem. Uh, hopefully you can't hear my dogs

Speaker:

barking. Um, not at all. Good, because they are

Speaker:

barking quite loudly,

Speaker:

and I'm gonna close the door so we don't hear them.

Speaker:

What— tell me what

Speaker:

We're talking about today, I think, is something that's very, very

Speaker:

important is the— there's a huge gap between the

Speaker:

data that arrives successfully to train a

Speaker:

model and data you can actually trust to feed into

Speaker:

a model that would make decisions. Right. What

Speaker:

can you do about it? Because like, you know, we're 4 years, we're coming up

Speaker:

4 years into the ChatGPT era. And people

Speaker:

still don't appreciate the importance of quality data engineering.

Speaker:

Yeah, I would say so. What, uh, when, when you talk

Speaker:

about enterprise data, right, a lot of, a lot of times what we come

Speaker:

across and what our job as data engineers

Speaker:

is that the data that comes into our systems

Speaker:

are not like enterprise ready. They have a

Speaker:

lot of dirty data. You need a lot

Speaker:

of cleaning, data cleaning, that

Speaker:

goes into, into the picture

Speaker:

before you can actually use that data for your

Speaker:

analytics or, you know, eventually to train your AI models.

Speaker:

What was the question once again? No, it was like, I've just

Speaker:

marveled at the fact that we're 4 years into— we're what,

Speaker:

15 years into the big data You know,

Speaker:

boom. And we're what, 10 years into the

Speaker:

data science boom? We're 4 years into

Speaker:

the ChatGPT kind of LLM era and

Speaker:

data engineering still doesn't get the respect that it deserves,

Speaker:

like, or the quality that it deserves, right? Like, what's it gonna take at

Speaker:

this point? I would rather disagree with that. Like, it does get a lot of

Speaker:

respect. We just are not recognized.

Speaker:

We are not the front-end guys, I would say. Like, you know, We are

Speaker:

behind-the-scenes people who carry out all the action.

Speaker:

I guess they're just behind-the-scenes type of

Speaker:

crowd. Yeah. Interesting. Maybe things are better than I

Speaker:

thought they would be. But I just find myself in a lot of conversations

Speaker:

where the data

Speaker:

is not among the first 3, 4 things to discuss, which I

Speaker:

find interesting. I think we're in that phase of You know, we talk about the

Speaker:

model, and then we talk about the infrastructure to run the model, then we talk

Speaker:

about governance, right? The data

Speaker:

behind it and the source data and where it's stored only comes up in the

Speaker:

context of governance, at least in my perspective,

Speaker:

my view of the world. Yeah, because a

Speaker:

lot of us, especially when we talk about the non-technical

Speaker:

population, they do not

Speaker:

understand how an LLM actually works.

Speaker:

So, but when you talk about, you know, the

Speaker:

accuracy of models or how good a model can produce an

Speaker:

output, it eventually boils down to the quality

Speaker:

of data that is fed into the models. For

Speaker:

example, like, you know, what we saw

Speaker:

with the ChatGPT and the AI race with US and

Speaker:

China is that Eventually

Speaker:

ChatGPT started on like, you know, the front foot, like, you know, they

Speaker:

had great models, especially with the, uh, with the,

Speaker:

uh, you know, the, the trend of Will Smith eating those

Speaker:

noodles videos. Spaghetti. Yeah. Yeah. Yeah. The spaghetti

Speaker:

noodles, those things, the, the video has got

Speaker:

better over time just because the data

Speaker:

was like the training like, you know, the, the data that was

Speaker:

fed to the model, it trained on it and it got eventually better.

Speaker:

But after a certain point, when it comes to

Speaker:

US-based companies, the AI model companies,

Speaker:

these companies hit a roadblock wherein they

Speaker:

cannot use any more data that was

Speaker:

copyrighted. Right. So,

Speaker:

so, so that, that's when we hit a stagnation of like, you know,

Speaker:

the AI output getting, you know, not getting any better.

Speaker:

Whereas when we see like, you know, other countries wherein

Speaker:

the copyright laws and other regulations are

Speaker:

not as strong, the companies, you know, again, we don't

Speaker:

want to name any companies, but the companies were able to use that

Speaker:

copyrighted data to train their models, which eventually

Speaker:

led to their models giving better

Speaker:

output than the US-led models.

Speaker:

Like, you know, uh, where there was, where there was copyright

Speaker:

laws that are not as strictly enforced. Yeah, they definitely have. So you

Speaker:

kind of, you kind of use those, like, you know, copyrighted materials

Speaker:

like Hollywood movies and all of those, you know, clips to train your

Speaker:

model. And eventually your model knows a lot better

Speaker:

than the, the models coming out of US

Speaker:

because again, like over here, there, the, the laws are quite

Speaker:

strict. You cannot use any data without the

Speaker:

consent of the data owner. Interesting.

Speaker:

Yeah, I wonder how— I'm not a lawyer, but I can imagine that that's going

Speaker:

to get real interesting real fast in terms of how that's

Speaker:

handled. And the interesting thing is that when

Speaker:

the LLMs blew up, if you look at the history of

Speaker:

LLM, nobody knew that this is going to work.

Speaker:

Even, right, you know, All they thought

Speaker:

was increasing the number of parameters would

Speaker:

eventually increase the output of the model.

Speaker:

And that's how it was an experiment. I was watching

Speaker:

a video of a professor from one of the

Speaker:

universities in California who was

Speaker:

also involved in the early stages of these LLM development. He

Speaker:

was like, you know, we, we were not sure of like

Speaker:

whether this model will actually work, but it eventually it started giving

Speaker:

out good output. And then we tried to like, you know, increase the

Speaker:

number of parameters and it gives out better

Speaker:

outputs as they increase. They started to increase

Speaker:

the parameters. Like, you know, we even, we started from like, you know,

Speaker:

1 billion parameters, then 2 billion. And I'm not sure,

Speaker:

like, uh, we are approximately at 20 billion

Speaker:

parameters right now. I think the latest model by Meta. And

Speaker:

Things have eventually gotten better. But right now,

Speaker:

when we talk about LLMs, their accuracy of the

Speaker:

output and their relationship with data,

Speaker:

say that it's quite closely linked in a way that we are at a

Speaker:

stage wherein the LLMs do not have enough data to train on.

Speaker:

We're actually running out of data, which is a crazy thing to think of because,

Speaker:eah, you know, we started the:Speaker:

much data we're creating every minute and you know, some ridiculous numbers, right?

Speaker:

You know, enough, if every byte of data was a grain of rice, it would

Speaker:

fill the Pacific Ocean or some ridiculous thing like that. Like just

Speaker:

so far beyond like what you would think. But getting down,

Speaker:

back down to brass tacks, right? In your Beyond ETL

Speaker:

article, you discussed the concept of reporting drift.

Speaker:

What is reporting drift? So basically

Speaker:

it is, you know, you're

Speaker:

the— okay, let me put it in a better way.

Speaker:

I am trying to articulate. So it is basically— Sorry, I know we went from

Speaker:

like LLMs to like good old-fashioned ETL. Sorry about that context shift.

Speaker:

But like, yeah, this seems to be where your center of gravity is in kind

Speaker:

of that. And again, like, you know, it's been a while since I've

Speaker:

like gone through that article, so I don't remember from the top of my

Speaker:

head. Right. But eventually what I was talking about in that article is

Speaker:

that, you know, your requirement changes over time

Speaker:

and as like your reports change because the

Speaker:

backend data changes for some reason, like, you know, somebody made a small

Speaker:

change or like, you know, the business started interpreting a particular

Speaker:

label differently, which started affecting your numbers. Right.

Speaker:

And eventually now the KPI numbers

Speaker:

represent— does not represent the accurate picture of

Speaker:

what's going on in the background of your business.

Speaker:

And do you think LLMs will help with that? Because one of the

Speaker:

things that I've noticed is, you know, there's a, you know,

Speaker:

there's a plugin for Claude for Excel, right? And it will go through and

Speaker:

find— I would say even with Power BI, they have

Speaker:

started integrating, for example,

Speaker:

with the dashboard development because like we as data analytics professionals,

Speaker:

we create a lot of dashboards for stakeholders.

Speaker:

to get an accurate picture of the entire business. If you have a big

Speaker:

business, that's the way to go about it. Excels are

Speaker:

basically the analytical stuff that we do behind the scenes.

Speaker:

Even there, we use a lot of Power BI and Tableau over here at

Speaker:

CSM. Over there, you see a lot of AI

Speaker:

integration and AI plugins. It is not as

Speaker:

good as developing an actual report,

Speaker:

but I What I feel, in my personal opinion, is that we will

Speaker:

eventually get there. Right now, with these AI

Speaker:

integration, you can create a very basic

Speaker:

level of a dashboard. Given that you have connected your

Speaker:

data with the dashboard and everything is in place,

Speaker:

you can write in a natural language to

Speaker:

the AI that you want the dashboard to look a

Speaker:

particular way. These are the KPI metrics, these are their

Speaker:

formulas, and they can

Speaker:

build an initial draft of how your dashboard will look like,

Speaker:

which will eventually give you a starting point of, you know, on how

Speaker:

to develop your dashboard and, you know, better it with the number

Speaker:

of iterations that you go through. Interesting.

Speaker:

I also find it interesting that we went from you won't need dashboards

Speaker:

This is, you know, the hype, the hype news cycle, right? To

Speaker:

now you can design your own. I know Claude has an option now

Speaker:

of a dashboard. It's like one of their products.

Speaker:

It seems like dashboards are here to stay. They

Speaker:

are. They're not going anywhere. They're not going— I, I never

Speaker:

really thought they were going to go away. I thought that was far-fetched that they

Speaker:

would go away. But what's your take on that? That, that flip, the

Speaker:

180 that the, you know, trade press and all the, the

Speaker:

talking heads have done on, on that.

Speaker:

I think that's true with any new technology, right? Like, you know,

Speaker:

there— I, I remember in the '90s there were, there were

Speaker:

like trends that, you know, the, the old businesses

Speaker:

before, before the, uh, like, you know, the, the dot-com bubble,

Speaker:

that the that everything that is offline,

Speaker:

you know, would fade away. But here we are, 20 years

Speaker:

down the line. Like, you know, we have evolved over time wherein,

Speaker:

like, you know, the offline businesses have also have an online presence.

Speaker:

It's the similar situation, like with AI initially when what we

Speaker:

thought was that we would not need data engineers.

Speaker:

And to be honest, I was quite concerned by myself, like, you know,

Speaker:

because with, with AI they can write code. they can

Speaker:

analyze data. All of those things are like, you know, something

Speaker:

that you can do pretty damn well with AI,

Speaker:

even better than a human, a normal human. But

Speaker:

eventually it comes down to a situation wherein

Speaker:

you use that new technology to better the services.

Speaker:

I think the similar thing will happen with dashboards, that the

Speaker:

dashboards where, let's say, in In an

Speaker:

era before the AI, I needed 2 or 3 analysts

Speaker:

to work on a particular dashboard. Now, with the help

Speaker:

of AI, that efficiency will improve, wherein, like, you know,

Speaker:

I can just allocate one resource to

Speaker:

handle multiple dashboards. That will happen,

Speaker:

but dashboards as a techno— like, you know,

Speaker:

as a tool will not go anywhere. That makes a

Speaker:

lot of sense. You mentioned Power BI

Speaker:

has built-in AI tools.

Speaker:

I'm not as deeply embedded in the Microsoft ecosystem now.

Speaker:

What sorts of tools does Power BI have now?

Speaker:

I think there is a GitHub repo

Speaker:

which you can use. I forget the name of it,

Speaker:

but you can What you can basically do is you can

Speaker:

integrate that with your Power BI

Speaker:

ecosystem. And then it's a .md

Speaker:

file wherein the Microsoft developers

Speaker:

have written rules on how to

Speaker:

design a dashboard, what are the things to consider.

Speaker:

And it is not just related to Power BI, you

Speaker:

can use with other Microsoft products. And then you

Speaker:

can basically query the LLM and it will do

Speaker:

the task for you. Interesting.

Speaker:

That's amazing. It's just amazing how

Speaker:

quickly this has gone from kind of a research project, you know,

Speaker:

LLMs, to now a core principle

Speaker:

or foundational tool for enterprise data

Speaker:

systems, which is both scary and exciting at the

Speaker:

same time. Yes.

Speaker:

And yeah, it's called Skills. Yeah, go ahead. Go ahead. No, no, no, go ahead.

Speaker:

So it's called Microsoft Skills, which is

Speaker:

hosted. It's an AI agent skill wherein,

Speaker:

like, you know, you can use those, you can

Speaker:

use that particular project to integrate with your Microsoft ecosystem.

Speaker:

Interesting. Exploratory analysis,

Speaker:

is that still the right tool? Like, is that still a good thing for people

Speaker:

to go through, or are we just going to offload that to the AI?

Speaker:

I think that is already happening, wherein, like, you

Speaker:

know, a lot of cognitive load that humans

Speaker:

had when it came to, you know, analyzing

Speaker:

some data from scratch has gone to AI. In

Speaker:

fact, I myself, you know, when working with ERP

Speaker:

systems or like, you know, a database system would

Speaker:

not know what a particular

Speaker:

keyword means, you know, in the database system. So for

Speaker:

example, we at the institute, like as a

Speaker:

higher-end institute, we use student management systems, and

Speaker:

these student management systems have millions of

Speaker:

tables inside, like, you know, the system,

Speaker:

and these systems would

Speaker:

contain n number of columns which I

Speaker:

might have not seen before. So initially

Speaker:

when I ran into some such columns wherein I had

Speaker:

no idea what this column means, I used to go online,

Speaker:

spend hours tracking like, you know, what this actually

Speaker:

column means, and I'll find someone who has answered

Speaker:

that question on You know, Stack Overflow. You remember those times?

Speaker:

And then, and then you

Speaker:

eventually figure out, okay, this is what is happening into, you

Speaker:

know, the business, or like, you know, this is what is happening with the data.

Speaker:

And you get in that entire picture. Then you analyze that data,

Speaker:

present a report, all of those things. Now with AI,

Speaker:

that has like, you know, reduced to a simple query.

Speaker:

Like, you know, AI will go out scrapping the

Speaker:

internet and finding the right answer for you. So like, you know, those

Speaker:

hours of analysis has reduced to a single query

Speaker:

that can be answered in 5 seconds.

Speaker:

Yeah, I can't tell you how

Speaker:

much I don't miss Stack Overflow or having to

Speaker:

sift through the comments for the actual answer and all the

Speaker:

smug, like, you should read the documentation.

Speaker:

Stack Overflow, I think, was helpful for sure, but there was

Speaker:

definitely a lot of cost, cognitive cost to

Speaker:

using Stack Overflow. One of the things that's really intrigued me,

Speaker:

and I don't use the term very often because

Speaker:

once I said it in a room full of PhD researchers in Silicon

Speaker:

Valley, real-time data, right? Okay.

Speaker:

Because that is, real-time is in the eye of the

Speaker:

beholder, I suppose. I got a whole lecture on the speed

Speaker:

of light and causality. So I'm very cautious about using that, but

Speaker:

the history of data engineering is largely around pipelines and

Speaker:

batch. Um, are

Speaker:

business users demanding something that's either

Speaker:

real-time or very close to real-time? So

Speaker:

from what I have seen, I've worked with, you know, the

Speaker:

pharmaceutical industry as well as the hiring industry. In

Speaker:

these systems, I do not have come across

Speaker:

any scenarios wherein, like, you know, we need real-time data. Right.

Speaker:

It is usually the data systems over here are

Speaker:

that no matter if you report it late,

Speaker:

but the data should be accurate. That is the primary

Speaker:

concern. Whereas if you look at the financial sector,

Speaker:

like, you know, banks or like, you know, trading firms

Speaker:

over there, those systems, like, you know, those

Speaker:

businesses would require real-time data. To answer

Speaker:

your question, so far we have not had any

Speaker:

scenarios or any business use case wherein you

Speaker:

have to report the data as soon as it becomes

Speaker:

available. It is usually the very next day after the refresh.

Speaker:

The data refresh has taken place early in the morning or late night,

:

00 AM. Now,

:

I was just wondering, I guess it is industry-specific. Correct. What

:

lessons do you think transfer or

:

are universal, lessons that you've learned from pharmaceutical and

:

higher ed? What

:

lessons have you learned there do you think are universal that could apply to other

:

domains? I would say

:

that this field is ever-evolving, right? You

:

know, you have to constantly, like, be in touch with the latest

:

trends in the technology and constantly upskill

:

yourself. That is one very important

:

lesson that I've learned throughout my career. The second thing is that

:

the basics of any, any

:

skill remains the same. For example, when I was in

:

pharmaceutical sector in my previous role, I— we used a

:

lot of Power BI as our primary

:

visualization tool. When I moved to my next role in

:

higher ed, We over here use a lot of Tableau,

:

but the data visualization concepts

:

remain the same, although the tool changes, maybe the terminologies

:

change, but the concept overall remains the same. So

:

those are the 2 things I feel like are universal.

:

Like, you know, the, you know, keep your basics right. Number 2,

:

keep learning. I mean,

:

that's very sound advice because this is a very fast-moving field. I mean,

:

every week there's a new paper coming out. It

:

feels like keeping up with what's going on requires a

:

full-time job itself, right?

:

It's interesting. What do you think the

:

future of data engineering is going to look like?

:

Even though I know that the future is very hard to predict, but yeah. Yeah,

:

but I think we will see a lot of AI

:

agents coming into the picture wherein, like, you know,

:

as of now, if you need 5 data engineers to

:

run an entire enterprise data engineering system,

:

maybe that could be done by a single engineer who

:

looks these AI agents. I think we are

:

moving very fast towards that. Eventually we

:

will see a lot of new self-healing

:

healing systems. Some of the papers that I've read recently,

:

they talk a lot about self-healing data

:

pipelines wherein like if a pipeline breaks

:

for some reason, and sometimes it could

:

be something as small as a data schema change. So like,

:

you know, the data that is coming in into your data

:

pipeline, the schema of that data has changed, which is a

:

very small issue whenever we as data engineers go

:

into debugging these pipelines. failures, we find those

:

errors and it essentially requires maybe to change

:

the name of the column or remove a particular section

:

of the code, which is a quick fix. These

:

quick fixes can be done by these AI

:

agents as they get smarter over time

:

and they can rerun those pipelines in that

:

very same moment instead of like, you know, we

:

going into the system trying to debug what's wrong and

:

then running it manually. I just find it

:

interesting that what's possible, right? And

:

even when it comes to data visualization, right? Like, right, you know,

:

there's so much opportunity in what Claude,

:

Code can do, Codex can do. It doesn't really matter, right? They're all

:

roughly the same capacity. It's

:

fascinating to me because one of the things I've recently discovered is

:

you can get Claude Code to generate videos. Yes.

:

And you could probably give it a

:

dataset and tell it to animate a compelling story

:

around your dataset, right? And it's not that much more of

:

a stretch of the imagination to throw in

:

a game engine and a VR headset so

:

you can have visualizations that you would see only in movies, right? Like,

:

it's not— Yeah. And that's the type of work that

:

would just— it wouldn't have been practical in terms of timing and

:

engineering-wise. But if you can offload a lot of that,

:

right, to an AI, then it's not— it's just tokens at that point,

:

right? Like, right, you know, obviously tokens are,

:

are costly, but still, like, it goes from being

:

impractical from the get-go to now it becomes

:

something that's possible. Like, it just blows my mind that we're at

:

that point. And, you know, I've experimented with some

:

Claude-generated videos, and I'm

:

suitably impressed about what's possible now. It's

:

almost ridiculous, you know, in terms of what is

:

possible.

:

That's cool. Eventually— Or, um,

:

no, go ahead. Yeah, go ahead. No, so what I was saying is like,

:

you know, uh, the things that we used to see in the '90s

:

and '60s as science fiction, we are living it right now at this

:

point in time. We used to see in the, in those

:

cartoonish movies that, you know, you can talk to your computer and

:

the computer will reply to you in a natural language with your life problems.

:

We are doing that. People are asking AI with

:

you know, their life problems. And it

:

is pretty good at suggesting you with some of the, like, you know,

:

advice. It does give you a different perspective.

:

Yes. Even if it's not always the right perspective, it does give you

:

a different perspective. One of the things I've noticed is

:

if you watch the Star Trek TV series, the older ones where they have the

:

computer and they have a complete interactive

:

conversation with it. And I remember

:

seeing, thinking that and like, wow, that's, that's definitely way off in the

:

future. And I even saw a demo video

:

for Cortana, which was Microsoft's voice assistant at one point.

:

And they showed kind of like this whole thing. And I was like, yeah, it's

:

going to be a while before that happens. Turns out it was, you know, less

:

than 3 years, you know, after that type of

:

conversation is possible now. It's just, and when you have these

:

ChatGPT dots and things like that, they have the ability to retain

:

context. And like, I've had conversations, both voice

:

and chat, where I've got like multiple threads on in my head, and

:

it's able to keep up with that. I'm suitably impressed.

:

And Mistral earlier, I think earlier this week, announced that they

:

have a Voxtral as their conversational agent

:

that can handle things like interruptions and stops in flow and can

:

pick up on that. It's very we've

:

gotten to the point where it's even better than the way it was presented

:

on Star Trek, right? Like, and that's— this is just like the '90s.

:like, you know, in the early::Yeah, I mean, even till::

uh, we had these— we were working with this machine learning

:

system where we were trying to predict the next word in the sentence,

:

and that was the frontier of the technology. And now

:

we are trying— like, we are able to generate paragraphs of text

:

based on what you ask.

:

Yeah. There was an early version of

:

a transformer. I think the website was called Talk to a Transformer.

:

It was— this would have been

:

2018, early

:,::

remember you can go and you can talk to it, And it would kind of

:

lose the context after a couple of sentences. Correct. But

:

still, it was impressive. You can say, I need a packing

:

list for a trip, and then give me 10 items. And the

:

first 3 to 5 were usually on point. And then after that, it

:

kind of lost the plot and got weird. And

:

it was interesting to see that. Obviously, it's gotten really

:

evolved since then. I am constantly impressed with

:

what LLMs can do. I think calling back to something we

:

early said in, you know, we talked about earlier, we've gone

:

further with LLMs than I thought we would. The reasoning

:

capabilities, the information like that, I

:

didn't think LLMs alone would get us there, but they have.

:

Yes. I used to joke that, you know, we're kind of on borrowed time with

:

LLMs, but I've been Clearly I've been wrong with that because they keep

:

surprising us with some of these things that they come up with.

:

Yeah. Interestingly, if you, you know, I was just

:

reading about like, you know, the history of LLMs, machine learning

:

models. And what we found out is that the initial

:

idea of these deep learning neural networks, the

:

neurons and like, you know, the

:

neuron architecture was developed back in

:

1947. That's right. But we

:

didn't— back in the time, we didn't have the computing power

:

and the data to develop it. And now eventually we are here.

:

Yeah, no, I mean, I remember when, you know, MNIST was like a

:

huge big deal. Like, wow, you can actually tell numbers. You can run that

:

on my computer. It just seems so

:

science fictiony that you would have

:

a computer could read, do OCR, just doing optical

:

character recognition. OCR was science fiction. OCR is not far back.

:It's just::

The whole idea of you can take a picture of a document

:

and my phone does it

:

now. Take a picture of a document and I can scan

:

it where it does the— or it'll just read the text if I need to

:

copy the text. I'm— maybe I'm old-fashioned, but

:

I'm always like amazed that, you know, wow, that actually

:

works. True. It's, it's, uh, it's one of those things

:

where, you know, I'm— I guess I'm at that age where I'll

:

say, oh, these kids don't understand, you know. Like

:

my, my oldest, you know, my

:

oldest, he's a teenager and he remembers when

:

We had to put limits on like how much time you can stream

:

on, watch videos on YouTube in the car. Mm-hmm. Because we would get, we'd get

:

data overcharges. And my second

:

child has no memory of that. And obviously my, my third child

:

who's 4 has no, he's, my 2 younger kids

:

are, we've always had an Alexa device in the home. Mm-hmm. Right? Like

:

we've always, Right. Voice activated technology has always

:

been a thing. And then for my youngest, for him, like AI is always

:

gonna have been a thing. So it's kids today don't

:

understand, you know, like, there's going to be a

:

point when they'll say, what'd you do in your first job? And I was like,

:

well, I programmed it. Like, you talk to an AI? No, you actually

:

had to program things by hand. Even those punch card

:

days, wherein you had to write a program by punching those

:

cards? I'm not that old, but yes.

:

Yeah. I mean, but those, those kind of things

:

fascinate me, like how those systems function. Right? Like, you

:

know, even when I— it's hard to, like, you know, I was, I was trying

:

to develop a collection of, like, you know, my

:

computer. You know, there's a thing called this computer

:

collection that I have where I was trying to get a floppy disk. It's

:

hard to get a floppy disk nowadays. It really is. It's

:

kind of crazy when you think about it. Like, and I had my first

:

digital camera, but my second digital camera was a Sony Mavica.

:

And those were— those had floppy disks as a storage mechanism

:

because at the time when I had it, you know, a floppy disk was

:

like about a dollar per. Oh. And you can get them anywhere,

:

right? So like, you know, you just— but the whole thought of that now is

:

this like— and I get annoyed when I have to move photos off my phone.

:

I have like 128 gig on this and it's just amazing how

:

quickly this has evolved in that space.

:

And I just,

:

I'm just fascinated. But again, like when it comes to

:

the data engineering side of things, right? Mm-hmm. I wanna get your opinion on this,

:

right? Like how do you make data AI ready, right? So

:

clearly a clean table normalized is no longer

:

enough, right? You have to do, if you're doing RAG, Though it's a first

:

start for sure. Yes. But it's not necessarily great training

:

data if we're talking about predictive AI, which people seem to have

:

forgotten exists. But when it comes to generative AI,

:

and, you know, particularly I'm thinking RAG in particular, right? Or RAG-Raft, anything

:

that involves retrieval. There's a whole

:

art and science to the chunking methodology and like, how do you

:

do that, right? And I just don't, I don't

:

know, like what— that is also a kind of data engineering,

:

certainly not classical data engineering. What's your take on that?

:

So you are almost there. Like, you know,

:

we have to classify different types of

:

files. You know, the PDFs go into, you know, one section,

:

the images, you have to classify them differently, the text data goes

:

differently. You like, you know, you have to develop a processing

:

architecture for each one of those in order to develop

:

RAG. So I think,

:

you know, there we already have started

:

employing those, those mechanisms into the

:

industry as of now. For example, you

:

know, we had a chatbot developed in my

:

previous role for training the

:

factory workers who used to work in those pharmaceutical firms

:

so that they can have direct access to their

:

queries instead of going up to their managers. So,

:

you know, we developed a RAG model which was

:

eventually given to a chat, you know, a

:

voice-commanded chat system so that you can just

:

fire your query by asking it a question verbally and

:

then it will reply you back. So yeah, I mean, there are a lot of

:

use cases of RAG. You know, the way data engineering

:

is going. That's my take. Yeah, it's

:

interesting. I think data engineering, I think, is important to

:

keep in mind that it's still here. It's going to be important.

:

It may look a little different, but I think the enterprise data pipelines and all

:

that sort of things have been going on for decades, are probably going to go

:

on for a few decades more. Whether or not you'll

:

need a team of data engineers of 50,

:

The efficiency will increase. You can probably get away with 10,

:

I would say, and maybe even less at some point. But

:

what fascinates me is what AI can do for dashboards

:

and data visualization, right? Like, you know, whether you want to add a 3D

:

aspect to it or, you know, that's something

:

that I'll be burning some tokens this weekend on

:

trying that out. In the winter, I think I

:

keep my home office warm just by running

:

running jobs and things like that. And it's

:

fascinating. Where can folks find out about— you seem to be a

:

prolific writer. I've seen at least 2

:

of your articles. Where can folks find out more about

:

you and what you're up to? And I think you've written a book. Is that

:

true? No, I haven't written a book, but yeah,

:

I've published a few articles.

:

related to data engineering, database management systems, and all of those.

:

And like, people can find me on LinkedIn. Okay.

:

That's my go-to, like, you know, as a professional. Awesome, I will do

:

that. We'd love to have you back in the show, and

:

you and Andy can geek out on some ADF and

:

some SQL stuff. Because Andy is a

:

legend in the SQL Server space. He will blush when he hears this, but like,

:

when I was internal at Microsoft, there was a— we had a group group Teams

:

chat of basically everybody in Northeast. So basically from

:

DC all the way up to Boston. And, um,

:

and maybe about 50 of us on the call. And somebody was like,

:

oh, you know, you got to read this guy's blog. And it was

:

Andy Leonard. And I was like, oh, that's cool. And, uh, and people

:

like, oh yeah, Andy's great. Andy's great. Andy's great. And then I

:

posted. I was like, oh, they even have a podcast. He has a podcast too.

:

And then I said with a wink in the emoji, but his co-host

:

is a bit of a jerk. That would be me.

:

And you could tell who got the joke and who didn't

:

right away. But yeah, but I've had people at Microsoft

:

say like, whoa, you know Andy Leonard? So yeah.

:

So, well, I really appreciate your time

:

and really great interview. Any parting thoughts?

:

No, I think thank you for having me. Anytime. For giving me the

:

opportunity to, like, you know, share my thoughts.

:

Awesome. Well, great having you, and we'll let the outro play.

:

That was a great interview, man. I really appreciate it. Thank you, Frank.

:

Awesome. We're slowly getting our infrastructure together.

About the author, Frank

Frank La Vigne is a software engineer and UX geek who saw the light about Data Science at an internal Microsoft Data Science Summit in 2016. Now, he wants to share his passion for the Data Arts with the world.

He blogs regularly at FranksWorld.com and has a YouTube channel called Frank's World TV. (www.FranksWorld.TV). Frank has extensive experience in web and application development. He is also an expert in mobile and tablet engineering. You can find him on Twitter at @tableteer.