Self-Healing Pipelines and the Evolving Role of Data Engineers in the AI Era
Welcome back to Data Driven, the podcast exploring the dynamic intersection of artificial intelligence, data science, and the foundational world of data engineering. In this episode, we dive into the realm of data pipelines, enterprise analytics, and the ever-evolving tools that power our most advanced AI models. Our guest is a seasoned data engineering and analytics professional from the College of Southern Nevada, sharing over five years of expertise spanning pharmaceutical supply chain analytics, business intelligence, and leading-edge platforms like Azure Data Factory, Databricks, and Power BI.
The conversation focused on the often under-appreciated work of data engineers, the challenges of maintaining data quality, and the art of making data truly AI-ready. One concept discussed was reporting drift and its impact on business decision-making. A key theme that emerged was the transformative influence of AI integration within analytics platforms such as Power BI, as well as the arrival of AI agents and self-healing data pipelines. The discussion explored the universal lessons from industries like pharma and higher education, the advancing role of dashboards, and just how quickly science fiction is becoming reality in the data space. Get ready for a thought-provoking look at how organizations can future-proof their data infrastructure in the age of AI.
Links
- Pratham on LinkedIn – https://www.linkedin.com/in/ypratham4/
- Watch on YouTube – https://youtu.be/G8JdYyV_Cas
Time Stamps
00:00 Intro and guest introduction
05:47 Copyright challenges in AI training
07:38 Early doubts about LLM effectiveness
13:08 Evolving impact of new technology
16:48 AI simplifying data analysis
18:41 Frustrations with Stack Overflow
22:45 AI agents in data engineering
26:48 AI conversation evolving rapidly
31:26 Talking about kids and technology
34:40 Building RAG for file classification
36:26 Exploring AI in Data Visualization
Transcript
In these systems, I do not have come across any
Speaker:scenarios wherein like you know we need real-time data. It
Speaker:is usually the data systems over here are
Speaker:that no matter if you report it late,
Speaker:but the data should be accurate. That is the primary
Speaker:concern. Whereas if you look at the financial sector,
Speaker:like you know banks or like you know trading firms
Speaker:over there, those systems like you know those
Speaker:businesses would require real. So to answer
Speaker:your question, so far we have not had any
Speaker:scenarios or any business use case wherein, like, you know, you
Speaker:have to report the data as soon as it becomes
Speaker:available. Data engineers aren't the front-end guys. They're the people
Speaker:behind the scenes who carry out all the action. Pratham Yadav of the
Speaker:College of Southern Nevada on dirty data, reporting drift, and
Speaker:self-healing pipelines. This is Data Driven.
Speaker:Hello and welcome back to Data Driven, the podcast where we explore the emerging
Speaker:field of artificial intelligence, data science, and of course, without data
Speaker:engineering, none of this is really possible.
Speaker:And unfortunately, my favoritest data engineer in the world can't make it
Speaker:today. So today I'm going solo and
Speaker:I'm very excited to have a conversation
Speaker:with a data engineer. We have with us
Speaker:today Pratham Yadav, who is a data
Speaker:engineering and analytics professional at the College of Southern
Speaker:Nevada with 5+ years of experience across data
Speaker:engineering, pharmaceutical supply chain analytics, and business
Speaker:intelligence. He's a published author and he's an
Speaker:expert in all things Azure Data Factory, Databricks,
Speaker:SQL, Python, and Power BI. Welcome to the show.
Speaker:Thank you for the lovely introduction, Frank.
Speaker:No problem, no problem. Uh, hopefully you can't hear my dogs
Speaker:barking. Um, not at all. Good, because they are
Speaker:barking quite loudly,
Speaker:and I'm gonna close the door so we don't hear them.
Speaker:What— tell me what
Speaker:We're talking about today, I think, is something that's very, very
Speaker:important is the— there's a huge gap between the
Speaker:data that arrives successfully to train a
Speaker:model and data you can actually trust to feed into
Speaker:a model that would make decisions. Right. What
Speaker:can you do about it? Because like, you know, we're 4 years, we're coming up
Speaker:4 years into the ChatGPT era. And people
Speaker:still don't appreciate the importance of quality data engineering.
Speaker:Yeah, I would say so. What, uh, when, when you talk
Speaker:about enterprise data, right, a lot of, a lot of times what we come
Speaker:across and what our job as data engineers
Speaker:is that the data that comes into our systems
Speaker:are not like enterprise ready. They have a
Speaker:lot of dirty data. You need a lot
Speaker:of cleaning, data cleaning, that
Speaker:goes into, into the picture
Speaker:before you can actually use that data for your
Speaker:analytics or, you know, eventually to train your AI models.
Speaker:What was the question once again? No, it was like, I've just
Speaker:marveled at the fact that we're 4 years into— we're what,
Speaker:15 years into the big data You know,
Speaker:boom. And we're what, 10 years into the
Speaker:data science boom? We're 4 years into
Speaker:the ChatGPT kind of LLM era and
Speaker:data engineering still doesn't get the respect that it deserves,
Speaker:like, or the quality that it deserves, right? Like, what's it gonna take at
Speaker:this point? I would rather disagree with that. Like, it does get a lot of
Speaker:respect. We just are not recognized.
Speaker:We are not the front-end guys, I would say. Like, you know, We are
Speaker:behind-the-scenes people who carry out all the action.
Speaker:I guess they're just behind-the-scenes type of
Speaker:crowd. Yeah. Interesting. Maybe things are better than I
Speaker:thought they would be. But I just find myself in a lot of conversations
Speaker:where the data
Speaker:is not among the first 3, 4 things to discuss, which I
Speaker:find interesting. I think we're in that phase of You know, we talk about the
Speaker:model, and then we talk about the infrastructure to run the model, then we talk
Speaker:about governance, right? The data
Speaker:behind it and the source data and where it's stored only comes up in the
Speaker:context of governance, at least in my perspective,
Speaker:my view of the world. Yeah, because a
Speaker:lot of us, especially when we talk about the non-technical
Speaker:population, they do not
Speaker:understand how an LLM actually works.
Speaker:So, but when you talk about, you know, the
Speaker:accuracy of models or how good a model can produce an
Speaker:output, it eventually boils down to the quality
Speaker:of data that is fed into the models. For
Speaker:example, like, you know, what we saw
Speaker:with the ChatGPT and the AI race with US and
Speaker:China is that Eventually
Speaker:ChatGPT started on like, you know, the front foot, like, you know, they
Speaker:had great models, especially with the, uh, with the,
Speaker:uh, you know, the, the trend of Will Smith eating those
Speaker:noodles videos. Spaghetti. Yeah. Yeah. Yeah. The spaghetti
Speaker:noodles, those things, the, the video has got
Speaker:better over time just because the data
Speaker:was like the training like, you know, the, the data that was
Speaker:fed to the model, it trained on it and it got eventually better.
Speaker:But after a certain point, when it comes to
Speaker:US-based companies, the AI model companies,
Speaker:these companies hit a roadblock wherein they
Speaker:cannot use any more data that was
Speaker:copyrighted. Right. So,
Speaker:so, so that, that's when we hit a stagnation of like, you know,
Speaker:the AI output getting, you know, not getting any better.
Speaker:Whereas when we see like, you know, other countries wherein
Speaker:the copyright laws and other regulations are
Speaker:not as strong, the companies, you know, again, we don't
Speaker:want to name any companies, but the companies were able to use that
Speaker:copyrighted data to train their models, which eventually
Speaker:led to their models giving better
Speaker:output than the US-led models.
Speaker:Like, you know, uh, where there was, where there was copyright
Speaker:laws that are not as strictly enforced. Yeah, they definitely have. So you
Speaker:kind of, you kind of use those, like, you know, copyrighted materials
Speaker:like Hollywood movies and all of those, you know, clips to train your
Speaker:model. And eventually your model knows a lot better
Speaker:than the, the models coming out of US
Speaker:because again, like over here, there, the, the laws are quite
Speaker:strict. You cannot use any data without the
Speaker:consent of the data owner. Interesting.
Speaker:Yeah, I wonder how— I'm not a lawyer, but I can imagine that that's going
Speaker:to get real interesting real fast in terms of how that's
Speaker:handled. And the interesting thing is that when
Speaker:the LLMs blew up, if you look at the history of
Speaker:LLM, nobody knew that this is going to work.
Speaker:Even, right, you know, All they thought
Speaker:was increasing the number of parameters would
Speaker:eventually increase the output of the model.
Speaker:And that's how it was an experiment. I was watching
Speaker:a video of a professor from one of the
Speaker:universities in California who was
Speaker:also involved in the early stages of these LLM development. He
Speaker:was like, you know, we, we were not sure of like
Speaker:whether this model will actually work, but it eventually it started giving
Speaker:out good output. And then we tried to like, you know, increase the
Speaker:number of parameters and it gives out better
Speaker:outputs as they increase. They started to increase
Speaker:the parameters. Like, you know, we even, we started from like, you know,
Speaker:1 billion parameters, then 2 billion. And I'm not sure,
Speaker:like, uh, we are approximately at 20 billion
Speaker:parameters right now. I think the latest model by Meta. And
Speaker:Things have eventually gotten better. But right now,
Speaker:when we talk about LLMs, their accuracy of the
Speaker:output and their relationship with data,
Speaker:say that it's quite closely linked in a way that we are at a
Speaker:stage wherein the LLMs do not have enough data to train on.
Speaker:We're actually running out of data, which is a crazy thing to think of because,
Speaker:eah, you know, we started the:Speaker:much data we're creating every minute and you know, some ridiculous numbers, right?
Speaker:You know, enough, if every byte of data was a grain of rice, it would
Speaker:fill the Pacific Ocean or some ridiculous thing like that. Like just
Speaker:so far beyond like what you would think. But getting down,
Speaker:back down to brass tacks, right? In your Beyond ETL
Speaker:article, you discussed the concept of reporting drift.
Speaker:What is reporting drift? So basically
Speaker:it is, you know, you're
Speaker:the— okay, let me put it in a better way.
Speaker:I am trying to articulate. So it is basically— Sorry, I know we went from
Speaker:like LLMs to like good old-fashioned ETL. Sorry about that context shift.
Speaker:But like, yeah, this seems to be where your center of gravity is in kind
Speaker:of that. And again, like, you know, it's been a while since I've
Speaker:like gone through that article, so I don't remember from the top of my
Speaker:head. Right. But eventually what I was talking about in that article is
Speaker:that, you know, your requirement changes over time
Speaker:and as like your reports change because the
Speaker:backend data changes for some reason, like, you know, somebody made a small
Speaker:change or like, you know, the business started interpreting a particular
Speaker:label differently, which started affecting your numbers. Right.
Speaker:And eventually now the KPI numbers
Speaker:represent— does not represent the accurate picture of
Speaker:what's going on in the background of your business.
Speaker:And do you think LLMs will help with that? Because one of the
Speaker:things that I've noticed is, you know, there's a, you know,
Speaker:there's a plugin for Claude for Excel, right? And it will go through and
Speaker:find— I would say even with Power BI, they have
Speaker:started integrating, for example,
Speaker:with the dashboard development because like we as data analytics professionals,
Speaker:we create a lot of dashboards for stakeholders.
Speaker:to get an accurate picture of the entire business. If you have a big
Speaker:business, that's the way to go about it. Excels are
Speaker:basically the analytical stuff that we do behind the scenes.
Speaker:Even there, we use a lot of Power BI and Tableau over here at
Speaker:CSM. Over there, you see a lot of AI
Speaker:integration and AI plugins. It is not as
Speaker:good as developing an actual report,
Speaker:but I What I feel, in my personal opinion, is that we will
Speaker:eventually get there. Right now, with these AI
Speaker:integration, you can create a very basic
Speaker:level of a dashboard. Given that you have connected your
Speaker:data with the dashboard and everything is in place,
Speaker:you can write in a natural language to
Speaker:the AI that you want the dashboard to look a
Speaker:particular way. These are the KPI metrics, these are their
Speaker:formulas, and they can
Speaker:build an initial draft of how your dashboard will look like,
Speaker:which will eventually give you a starting point of, you know, on how
Speaker:to develop your dashboard and, you know, better it with the number
Speaker:of iterations that you go through. Interesting.
Speaker:I also find it interesting that we went from you won't need dashboards
Speaker:This is, you know, the hype, the hype news cycle, right? To
Speaker:now you can design your own. I know Claude has an option now
Speaker:of a dashboard. It's like one of their products.
Speaker:It seems like dashboards are here to stay. They
Speaker:are. They're not going anywhere. They're not going— I, I never
Speaker:really thought they were going to go away. I thought that was far-fetched that they
Speaker:would go away. But what's your take on that? That, that flip, the
Speaker:180 that the, you know, trade press and all the, the
Speaker:talking heads have done on, on that.
Speaker:I think that's true with any new technology, right? Like, you know,
Speaker:there— I, I remember in the '90s there were, there were
Speaker:like trends that, you know, the, the old businesses
Speaker:before, before the, uh, like, you know, the, the dot-com bubble,
Speaker:that the that everything that is offline,
Speaker:you know, would fade away. But here we are, 20 years
Speaker:down the line. Like, you know, we have evolved over time wherein,
Speaker:like, you know, the offline businesses have also have an online presence.
Speaker:It's the similar situation, like with AI initially when what we
Speaker:thought was that we would not need data engineers.
Speaker:And to be honest, I was quite concerned by myself, like, you know,
Speaker:because with, with AI they can write code. they can
Speaker:analyze data. All of those things are like, you know, something
Speaker:that you can do pretty damn well with AI,
Speaker:even better than a human, a normal human. But
Speaker:eventually it comes down to a situation wherein
Speaker:you use that new technology to better the services.
Speaker:I think the similar thing will happen with dashboards, that the
Speaker:dashboards where, let's say, in In an
Speaker:era before the AI, I needed 2 or 3 analysts
Speaker:to work on a particular dashboard. Now, with the help
Speaker:of AI, that efficiency will improve, wherein, like, you know,
Speaker:I can just allocate one resource to
Speaker:handle multiple dashboards. That will happen,
Speaker:but dashboards as a techno— like, you know,
Speaker:as a tool will not go anywhere. That makes a
Speaker:lot of sense. You mentioned Power BI
Speaker:has built-in AI tools.
Speaker:I'm not as deeply embedded in the Microsoft ecosystem now.
Speaker:What sorts of tools does Power BI have now?
Speaker:I think there is a GitHub repo
Speaker:which you can use. I forget the name of it,
Speaker:but you can What you can basically do is you can
Speaker:integrate that with your Power BI
Speaker:ecosystem. And then it's a .md
Speaker:file wherein the Microsoft developers
Speaker:have written rules on how to
Speaker:design a dashboard, what are the things to consider.
Speaker:And it is not just related to Power BI, you
Speaker:can use with other Microsoft products. And then you
Speaker:can basically query the LLM and it will do
Speaker:the task for you. Interesting.
Speaker:That's amazing. It's just amazing how
Speaker:quickly this has gone from kind of a research project, you know,
Speaker:LLMs, to now a core principle
Speaker:or foundational tool for enterprise data
Speaker:systems, which is both scary and exciting at the
Speaker:same time. Yes.
Speaker:And yeah, it's called Skills. Yeah, go ahead. Go ahead. No, no, no, go ahead.
Speaker:So it's called Microsoft Skills, which is
Speaker:hosted. It's an AI agent skill wherein,
Speaker:like, you know, you can use those, you can
Speaker:use that particular project to integrate with your Microsoft ecosystem.
Speaker:Interesting. Exploratory analysis,
Speaker:is that still the right tool? Like, is that still a good thing for people
Speaker:to go through, or are we just going to offload that to the AI?
Speaker:I think that is already happening, wherein, like, you
Speaker:know, a lot of cognitive load that humans
Speaker:had when it came to, you know, analyzing
Speaker:some data from scratch has gone to AI. In
Speaker:fact, I myself, you know, when working with ERP
Speaker:systems or like, you know, a database system would
Speaker:not know what a particular
Speaker:keyword means, you know, in the database system. So for
Speaker:example, we at the institute, like as a
Speaker:higher-end institute, we use student management systems, and
Speaker:these student management systems have millions of
Speaker:tables inside, like, you know, the system,
Speaker:and these systems would
Speaker:contain n number of columns which I
Speaker:might have not seen before. So initially
Speaker:when I ran into some such columns wherein I had
Speaker:no idea what this column means, I used to go online,
Speaker:spend hours tracking like, you know, what this actually
Speaker:column means, and I'll find someone who has answered
Speaker:that question on You know, Stack Overflow. You remember those times?
Speaker:And then, and then you
Speaker:eventually figure out, okay, this is what is happening into, you
Speaker:know, the business, or like, you know, this is what is happening with the data.
Speaker:And you get in that entire picture. Then you analyze that data,
Speaker:present a report, all of those things. Now with AI,
Speaker:that has like, you know, reduced to a simple query.
Speaker:Like, you know, AI will go out scrapping the
Speaker:internet and finding the right answer for you. So like, you know, those
Speaker:hours of analysis has reduced to a single query
Speaker:that can be answered in 5 seconds.
Speaker:Yeah, I can't tell you how
Speaker:much I don't miss Stack Overflow or having to
Speaker:sift through the comments for the actual answer and all the
Speaker:smug, like, you should read the documentation.
Speaker:Stack Overflow, I think, was helpful for sure, but there was
Speaker:definitely a lot of cost, cognitive cost to
Speaker:using Stack Overflow. One of the things that's really intrigued me,
Speaker:and I don't use the term very often because
Speaker:once I said it in a room full of PhD researchers in Silicon
Speaker:Valley, real-time data, right? Okay.
Speaker:Because that is, real-time is in the eye of the
Speaker:beholder, I suppose. I got a whole lecture on the speed
Speaker:of light and causality. So I'm very cautious about using that, but
Speaker:the history of data engineering is largely around pipelines and
Speaker:batch. Um, are
Speaker:business users demanding something that's either
Speaker:real-time or very close to real-time? So
Speaker:from what I have seen, I've worked with, you know, the
Speaker:pharmaceutical industry as well as the hiring industry. In
Speaker:these systems, I do not have come across
Speaker:any scenarios wherein, like, you know, we need real-time data. Right.
Speaker:It is usually the data systems over here are
Speaker:that no matter if you report it late,
Speaker:but the data should be accurate. That is the primary
Speaker:concern. Whereas if you look at the financial sector,
Speaker:like, you know, banks or like, you know, trading firms
Speaker:over there, those systems, like, you know, those
Speaker:businesses would require real-time data. To answer
Speaker:your question, so far we have not had any
Speaker:scenarios or any business use case wherein you
Speaker:have to report the data as soon as it becomes
Speaker:available. It is usually the very next day after the refresh.
Speaker:The data refresh has taken place early in the morning or late night,
:00 AM. Now,
:I was just wondering, I guess it is industry-specific. Correct. What
:lessons do you think transfer or
:are universal, lessons that you've learned from pharmaceutical and
:higher ed? What
:lessons have you learned there do you think are universal that could apply to other
:domains? I would say
:that this field is ever-evolving, right? You
:know, you have to constantly, like, be in touch with the latest
:trends in the technology and constantly upskill
:yourself. That is one very important
:lesson that I've learned throughout my career. The second thing is that
:the basics of any, any
:skill remains the same. For example, when I was in
:pharmaceutical sector in my previous role, I— we used a
:lot of Power BI as our primary
:visualization tool. When I moved to my next role in
:higher ed, We over here use a lot of Tableau,
:but the data visualization concepts
:remain the same, although the tool changes, maybe the terminologies
:change, but the concept overall remains the same. So
:those are the 2 things I feel like are universal.
:Like, you know, the, you know, keep your basics right. Number 2,
:keep learning. I mean,
:that's very sound advice because this is a very fast-moving field. I mean,
:every week there's a new paper coming out. It
:feels like keeping up with what's going on requires a
:full-time job itself, right?
:It's interesting. What do you think the
:future of data engineering is going to look like?
:Even though I know that the future is very hard to predict, but yeah. Yeah,
:but I think we will see a lot of AI
:agents coming into the picture wherein, like, you know,
:as of now, if you need 5 data engineers to
:run an entire enterprise data engineering system,
:maybe that could be done by a single engineer who
:looks these AI agents. I think we are
:moving very fast towards that. Eventually we
:will see a lot of new self-healing
:healing systems. Some of the papers that I've read recently,
:they talk a lot about self-healing data
:pipelines wherein like if a pipeline breaks
:for some reason, and sometimes it could
:be something as small as a data schema change. So like,
:you know, the data that is coming in into your data
:pipeline, the schema of that data has changed, which is a
:very small issue whenever we as data engineers go
:into debugging these pipelines. failures, we find those
:errors and it essentially requires maybe to change
:the name of the column or remove a particular section
:of the code, which is a quick fix. These
:quick fixes can be done by these AI
:agents as they get smarter over time
:and they can rerun those pipelines in that
:very same moment instead of like, you know, we
:going into the system trying to debug what's wrong and
:then running it manually. I just find it
:interesting that what's possible, right? And
:even when it comes to data visualization, right? Like, right, you know,
:there's so much opportunity in what Claude,
:Code can do, Codex can do. It doesn't really matter, right? They're all
:roughly the same capacity. It's
:fascinating to me because one of the things I've recently discovered is
:you can get Claude Code to generate videos. Yes.
:And you could probably give it a
:dataset and tell it to animate a compelling story
:around your dataset, right? And it's not that much more of
:a stretch of the imagination to throw in
:a game engine and a VR headset so
:you can have visualizations that you would see only in movies, right? Like,
:it's not— Yeah. And that's the type of work that
:would just— it wouldn't have been practical in terms of timing and
:engineering-wise. But if you can offload a lot of that,
:right, to an AI, then it's not— it's just tokens at that point,
:right? Like, right, you know, obviously tokens are,
:are costly, but still, like, it goes from being
:impractical from the get-go to now it becomes
:something that's possible. Like, it just blows my mind that we're at
:that point. And, you know, I've experimented with some
:Claude-generated videos, and I'm
:suitably impressed about what's possible now. It's
:almost ridiculous, you know, in terms of what is
:possible.
:That's cool. Eventually— Or, um,
:no, go ahead. Yeah, go ahead. No, so what I was saying is like,
:you know, uh, the things that we used to see in the '90s
:and '60s as science fiction, we are living it right now at this
:point in time. We used to see in the, in those
:cartoonish movies that, you know, you can talk to your computer and
:the computer will reply to you in a natural language with your life problems.
:We are doing that. People are asking AI with
:you know, their life problems. And it
:is pretty good at suggesting you with some of the, like, you know,
:advice. It does give you a different perspective.
:Yes. Even if it's not always the right perspective, it does give you
:a different perspective. One of the things I've noticed is
:if you watch the Star Trek TV series, the older ones where they have the
:computer and they have a complete interactive
:conversation with it. And I remember
:seeing, thinking that and like, wow, that's, that's definitely way off in the
:future. And I even saw a demo video
:for Cortana, which was Microsoft's voice assistant at one point.
:And they showed kind of like this whole thing. And I was like, yeah, it's
:going to be a while before that happens. Turns out it was, you know, less
:than 3 years, you know, after that type of
:conversation is possible now. It's just, and when you have these
:ChatGPT dots and things like that, they have the ability to retain
:context. And like, I've had conversations, both voice
:and chat, where I've got like multiple threads on in my head, and
:it's able to keep up with that. I'm suitably impressed.
:And Mistral earlier, I think earlier this week, announced that they
:have a Voxtral as their conversational agent
:that can handle things like interruptions and stops in flow and can
:pick up on that. It's very we've
:gotten to the point where it's even better than the way it was presented
:on Star Trek, right? Like, and that's— this is just like the '90s.
:like, you know, in the early::Yeah, I mean, even till::uh, we had these— we were working with this machine learning
:system where we were trying to predict the next word in the sentence,
:and that was the frontier of the technology. And now
:we are trying— like, we are able to generate paragraphs of text
:based on what you ask.
:Yeah. There was an early version of
:a transformer. I think the website was called Talk to a Transformer.
:It was— this would have been
:2018, early
:,::remember you can go and you can talk to it, And it would kind of
:lose the context after a couple of sentences. Correct. But
:still, it was impressive. You can say, I need a packing
:list for a trip, and then give me 10 items. And the
:first 3 to 5 were usually on point. And then after that, it
:kind of lost the plot and got weird. And
:it was interesting to see that. Obviously, it's gotten really
:evolved since then. I am constantly impressed with
:what LLMs can do. I think calling back to something we
:early said in, you know, we talked about earlier, we've gone
:further with LLMs than I thought we would. The reasoning
:capabilities, the information like that, I
:didn't think LLMs alone would get us there, but they have.
:Yes. I used to joke that, you know, we're kind of on borrowed time with
:LLMs, but I've been Clearly I've been wrong with that because they keep
:surprising us with some of these things that they come up with.
:Yeah. Interestingly, if you, you know, I was just
:reading about like, you know, the history of LLMs, machine learning
:models. And what we found out is that the initial
:idea of these deep learning neural networks, the
:neurons and like, you know, the
:neuron architecture was developed back in
:1947. That's right. But we
:didn't— back in the time, we didn't have the computing power
:and the data to develop it. And now eventually we are here.
:Yeah, no, I mean, I remember when, you know, MNIST was like a
:huge big deal. Like, wow, you can actually tell numbers. You can run that
:on my computer. It just seems so
:science fictiony that you would have
:a computer could read, do OCR, just doing optical
:character recognition. OCR was science fiction. OCR is not far back.
:It's just::The whole idea of you can take a picture of a document
:and my phone does it
:now. Take a picture of a document and I can scan
:it where it does the— or it'll just read the text if I need to
:copy the text. I'm— maybe I'm old-fashioned, but
:I'm always like amazed that, you know, wow, that actually
:works. True. It's, it's, uh, it's one of those things
:where, you know, I'm— I guess I'm at that age where I'll
:say, oh, these kids don't understand, you know. Like
:my, my oldest, you know, my
:oldest, he's a teenager and he remembers when
:We had to put limits on like how much time you can stream
:on, watch videos on YouTube in the car. Mm-hmm. Because we would get, we'd get
:data overcharges. And my second
:child has no memory of that. And obviously my, my third child
:who's 4 has no, he's, my 2 younger kids
:are, we've always had an Alexa device in the home. Mm-hmm. Right? Like
:we've always, Right. Voice activated technology has always
:been a thing. And then for my youngest, for him, like AI is always
:gonna have been a thing. So it's kids today don't
:understand, you know, like, there's going to be a
:point when they'll say, what'd you do in your first job? And I was like,
:well, I programmed it. Like, you talk to an AI? No, you actually
:had to program things by hand. Even those punch card
:days, wherein you had to write a program by punching those
:cards? I'm not that old, but yes.
:Yeah. I mean, but those, those kind of things
:fascinate me, like how those systems function. Right? Like, you
:know, even when I— it's hard to, like, you know, I was, I was trying
:to develop a collection of, like, you know, my
:computer. You know, there's a thing called this computer
:collection that I have where I was trying to get a floppy disk. It's
:hard to get a floppy disk nowadays. It really is. It's
:kind of crazy when you think about it. Like, and I had my first
:digital camera, but my second digital camera was a Sony Mavica.
:And those were— those had floppy disks as a storage mechanism
:because at the time when I had it, you know, a floppy disk was
:like about a dollar per. Oh. And you can get them anywhere,
:right? So like, you know, you just— but the whole thought of that now is
:this like— and I get annoyed when I have to move photos off my phone.
:I have like 128 gig on this and it's just amazing how
:quickly this has evolved in that space.
:And I just,
:I'm just fascinated. But again, like when it comes to
:the data engineering side of things, right? Mm-hmm. I wanna get your opinion on this,
:right? Like how do you make data AI ready, right? So
:clearly a clean table normalized is no longer
:enough, right? You have to do, if you're doing RAG, Though it's a first
:start for sure. Yes. But it's not necessarily great training
:data if we're talking about predictive AI, which people seem to have
:forgotten exists. But when it comes to generative AI,
:and, you know, particularly I'm thinking RAG in particular, right? Or RAG-Raft, anything
:that involves retrieval. There's a whole
:art and science to the chunking methodology and like, how do you
:do that, right? And I just don't, I don't
:know, like what— that is also a kind of data engineering,
:certainly not classical data engineering. What's your take on that?
:So you are almost there. Like, you know,
:we have to classify different types of
:files. You know, the PDFs go into, you know, one section,
:the images, you have to classify them differently, the text data goes
:differently. You like, you know, you have to develop a processing
:architecture for each one of those in order to develop
:RAG. So I think,
:you know, there we already have started
:employing those, those mechanisms into the
:industry as of now. For example, you
:know, we had a chatbot developed in my
:previous role for training the
:factory workers who used to work in those pharmaceutical firms
:so that they can have direct access to their
:queries instead of going up to their managers. So,
:you know, we developed a RAG model which was
:eventually given to a chat, you know, a
:voice-commanded chat system so that you can just
:fire your query by asking it a question verbally and
:then it will reply you back. So yeah, I mean, there are a lot of
:use cases of RAG. You know, the way data engineering
:is going. That's my take. Yeah, it's
:interesting. I think data engineering, I think, is important to
:keep in mind that it's still here. It's going to be important.
:It may look a little different, but I think the enterprise data pipelines and all
:that sort of things have been going on for decades, are probably going to go
:on for a few decades more. Whether or not you'll
:need a team of data engineers of 50,
:The efficiency will increase. You can probably get away with 10,
:I would say, and maybe even less at some point. But
:what fascinates me is what AI can do for dashboards
:and data visualization, right? Like, you know, whether you want to add a 3D
:aspect to it or, you know, that's something
:that I'll be burning some tokens this weekend on
:trying that out. In the winter, I think I
:keep my home office warm just by running
:running jobs and things like that. And it's
:fascinating. Where can folks find out about— you seem to be a
:prolific writer. I've seen at least 2
:of your articles. Where can folks find out more about
:you and what you're up to? And I think you've written a book. Is that
:true? No, I haven't written a book, but yeah,
:I've published a few articles.
:related to data engineering, database management systems, and all of those.
:And like, people can find me on LinkedIn. Okay.
:That's my go-to, like, you know, as a professional. Awesome, I will do
:that. We'd love to have you back in the show, and
:you and Andy can geek out on some ADF and
:some SQL stuff. Because Andy is a
:legend in the SQL Server space. He will blush when he hears this, but like,
:when I was internal at Microsoft, there was a— we had a group group Teams
:chat of basically everybody in Northeast. So basically from
:DC all the way up to Boston. And, um,
:and maybe about 50 of us on the call. And somebody was like,
:oh, you know, you got to read this guy's blog. And it was
:Andy Leonard. And I was like, oh, that's cool. And, uh, and people
:like, oh yeah, Andy's great. Andy's great. Andy's great. And then I
:posted. I was like, oh, they even have a podcast. He has a podcast too.
:And then I said with a wink in the emoji, but his co-host
:is a bit of a jerk. That would be me.
:And you could tell who got the joke and who didn't
:right away. But yeah, but I've had people at Microsoft
:say like, whoa, you know Andy Leonard? So yeah.
:So, well, I really appreciate your time
:and really great interview. Any parting thoughts?
:No, I think thank you for having me. Anytime. For giving me the
:opportunity to, like, you know, share my thoughts.
:Awesome. Well, great having you, and we'll let the outro play.
:That was a great interview, man. I really appreciate it. Thank you, Frank.
:Awesome. We're slowly getting our infrastructure together.
