Tackling Data Engineering Challenges with Autonomous AI Agents

Something breaks and then you have to jump out of the bed in the middle

of the night to get those things fixed.

Hello and welcome back to Data Driven, the podcast where we explore the emerging

industry that is AI, data science, and of course, none of

it's all possible without data engineering. Now, unfortunately,

my favorite, most favorite data engineer in the world can't make it here

today. And I am actually enjoying the sunny but not too

ridiculously hot sunny day here in the suburbs of Baltimore,

Maryland. But I am excited here because I have Pranesh Patel,

who is the co-founder and CEO of a company that

does— makes data engineering a lot more palatable, it sounds like, Ultima

AI. Welcome to the show. Hey, um,

Frank, thanks for having me here. Super excited to chat with you

today. Yeah, so, so tell me about your company, Ultimate AI. Is, uh,

tell me about what it is and what led you to make it.

So Ultimate AI, we are a startup based out of the San Francisco Bay Area.

What we do is we use AI to

automate a bunch of data engineering tasks for the teams out there

who do data work, right? The tasks like building

ELT pipeline, extract and loading of the data, transforming

data, and not just building those pipelines but maintaining

those as well. It's always nightmares. Something breaks and then you have to

jump out of the bed in the middle of the night to get those things

fixed. So building and maintaining data pipelines or

managing your data infrastructure Like in the modern data

stack today, we have Snowflake, Databricks, BigQuery, those things, right?

We need to spend a bunch of effort to manage that infrastructure as well as

optimize that infrastructure, like making sure you have turned on the right knobs on the

infrastructure side as well as you have optimized those queries and pipelines,

etc. So we work on that as well. The whole idea is how we

can use the agents to automate a bunch of that work. so

we can scale our teams even further. I'm glad

you mentioned— I'm sorry, go ahead. No, you asked

me that, hey, how did it all start? So me and my co-founder,

we have worked in B2B enterprise space for a long time. You must have seen

all those data teams every company has. There's so much work involved. There is a

long backlog. We went through the same experiences, and we started

thinking there must be a better solution so that we can handle that

backlog. At the same time, a few years ago, this whole AI wave started

coming in. We jumped in headfirst and started building

agents first to sort of automate simple tasks like writing

documentation or writing some data quality tests. And through that,

the tool and the product evolved from there. And now

we automate bunch of things that are day-to-day things for

data engineering folks. That's a good point. You brought up a good point

because you mentioned Snowflake, you mentioned Databricks, you mentioned all of these

technology tools that the stack used to be a lot simpler in data,

in the data space, right? You, you know, you were either an Oracle shop

or a SQL Server shop and you had, you know, SQL

Server tooling, which those I'm way more familiar with, but also the

Oracle kind of stack too, right? So it went from being kind of a

The tools may have been limited, yes, but the tools also—

there only wasn't really that much of them, right? You picked one side

and stuck with it, right, for most organizations. And then we

had things like Hadoop and Pig and Hive and all of those things kind of

come out. And now this many decade and a half or so

or more, you can tell by my gray hair, that now we

have just an unlimited assortment, it seems, of these

tools, right? And each one of them has their own quirks, their own settings, as

you said, the knobs and dials. Is that something that your

solution offers a solution to? Yeah,

absolutely. Because as you mentioned, the use

cases exploded, and with that, the tool stack exploded as

well. You can't be exploiting 35 different tools to make sure

everything works perfectly and keep your tabs on everything.

So we help tremendously with that, like all those configurations in your

infrastructure as well as building out things. You can't be expert in so many tools.

And on the other side with AI, what has happened is

even these tools as a software are evolving so fast. In a

year, there are like 100 features come out. How can you keep tab on what's

the latest and greatest and what actually fits in my environment and to my use

cases? And, you know, AI for Rescue there, it can

do a bunch of those things for you. No, absolutely. And

I hadn't really thought about that angle of you have to be an expert.

It's like what happened in the software development world

where you had the notion of a full-stack developer where suddenly,

you know, it went from you would be a Visual Basic developer, right?

Or a developer of, you know, web developer. But

then it became, no, you had to be a full-stack developer. You had to know

the data side. You had to know the CSS. You had to know the HTML

and the JavaScript, right? Yeah. Data, I think, has followed a very similar trajectory

in that regard of no one person can do it all. At least

no one person can do it all without the assistance of some kind of

AI. And is that what your product enables? Like

somebody to like basically be a— in the DC area,

they love the term force multiplier. Is that kind of like what

your tool does? Exactly. Exactly. Because

as I think things exploded, use cases exploded, and even

the use of AI has exploded overall, your data

actually fuels all of it. And what's happening is the data projects

are growing exponentially, but our teams are not growing in that size. And

that's why there is a huge backlog that's happening. So the idea is then,

in your words, how we can have some sort of force multiplier that

can do a bunch of things for me automatically. And I focus on some of

the more complex things or something where AI needs more help

and guidance. So think of this as these bunch of agents

are my minions. They're getting the work done. I have like 30 of those and

I'm giving them directions, course correcting, telling them what to do. And

that's how basically the teams are scaling themselves today. But there are so many challenges

in doing that as well, in that whole process. And our

goal is how we can streamline and make that whole process smooth. So you have

teams of these minions which are getting a lot of work done for you.

Yeah, and I think you brought up a very real pain

point, right? They're not hiring tens of people in the data engineering

teams anymore, right? Data engineers are expected to do, to all

be 10x engineers, right? And so

I like the idea of you calling these AI agents minions, one, 'cause

I think the movies are really cute and I have small kids. But

how do you— what's the governance look like on that? Is that something that you—

how did you address that problem? 'Cause I'm sure that comes up quite a bit.

No, I think you touched on very important area, right? Because as a

human, we usually have general sense of understanding and

what needs to be done and what shouldn't be done. But for agents,

it's a piece of code, it's a machine, right? A lot of times it

doesn't have that, compass of common sense. So for example,

that's why you might have seen situations where agent ran a query and that cost

you thousands of dollars. Oh, and there's so many stories of agent

deleted my sensitive data or replicated it. So whether it's sensitive

data, access controls, cost guardrails,

all of those are very important factors around which

we need to develop a solution. Now we just talked about Lot

of tools in the data stack. Every tool has their access layer, RBAC

layer. Now, how are we going to do this, right? So what we have been

doing is, and that's the reason we launch the, our product as open

source project, we call it Ultimate Core. There is a governance

layer that's built in, which allows people to define these guardrails, rules,

and permissions. Lot of guardrails come inbuilt.

And the beauty of that is this is a layer that sits on top of

these bunch of different tools that you are using in your data stack. and gives

us the common ground around governance. So for example,

cost, it makes sure the agent doesn't spend more money than this

for your specific task. Or for agent

itself, you can create a separate view, separate tables so that they don't

inherit like service user permissions or user permissions directly. Because as

a user, I might have so many permissions, but I don't want my agent to

have the same permissions and use my same credentials, et cetera.

So that part also we have solved pretty well.

Yeah, I think, I think calling them minions works out pretty well too, because, you

know, the minions always— there's a whole sequence in the first movie about the dart

gun. Parents, if you know, you know. But

the minions misheard it and built something else. But

no, I think you're right. Making sure these things have guardrails on them,

I think, is— people are going to learn very quickly what happens if you don't

have guardrails, right? millions of dollars a query, or etc., etc.

Actually, recently in the news, and they haven't disclosed what was the core— what

was the core problem, but apparently AWS had—

was giving out bills that were like orders of magnitude higher.

And I can only— again, I have no inside information,

but I can only imagine that there was probably some kind of rogue AI doing

a math error or doing something crazy like that. What do you think the

future of these— this tool space is going to be? Do you think You know,

there'll be more MCP servers, right? What do you think,

um, what do you think that's going to look like, the ecosystem,

the data ecosystem? So how I

see it is based on what we want to do, the tools

get built, right? And as we sort of expand our

horizons to do more and more things, the tools get expanded also, right?

For example, I think MCP server is a great example. At one point

we realized, hey, we need to sort of feed in all this

information to MCP server, uh, to AI agents, right? And then

MCP servers were built as a solution to it. Now

we are hitting the limits of MCP servers themselves where,

you know, okay, governance— anybody can install any MCP server and I have

no control over who uses which MCP server in the organization. Right. That's

happening. Second is The MCB servers are putting out so

much output that my tokens are getting consumed like candy,

right? So then how do I control it where MCB tool outputs are

limited? And we have built some functionality around that, around context comparison

and tool output curtails, etc., especially for data engineering

tasks. Our third big thing is MCB server is like

API, is going to just pull the data from the other system and bring it

to you. But it is not going to do any intelligent filtering

or connecting those dots together. And then now people are looking at

semantic layers and those things, how I can do it also. So now we are

running into the limitations of NCP servers themselves, and then people are trying

to figure out, hey, what's the next thing we need to do to solve this

problem really well? And then people are talking about context graphs

or context store where all of your information is already there,

curated, filtered, smartly arranged, And then you use MCP

server to pull only right information instead of directly interfacing with the

tool, like for example Salesforce or HubSpot, and just dumping everything into your

AI agent as well. So how I see it is, I think

agents are going to become more and more autonomous, more and more ambient

as well, and the information we are going to feed them

is going to become much more curated as well. And there are so

many facets to this. There is a right information feeding angle around

context. There is a governance angle to it, and now. And now I think you

just touched on it. The one big angle that's coming into the play is cost

as well. There, there have been so many stories coming out where people spend

their entire year's worth of budget in like 3 months. I know some stories

which are not public where people's usage like 30x'd in

6 months and now they're like, oh my God, my AI bill is actually same

as my cloud bill now. I never planned for this.

And what is exactly the ROI that people are trying to measure also?

Yeah. So all these questions are coming up, and as the questions and use cases

come up, I believe we'll have more tooling and better solutions.

Is anything— is there anything in particular that your product addresses

to any of these problems? Yeah, so we talked about the governance

piece of it. On the cost side as well, what we started doing

is one of the features we have in the product is context compaction.

So I was talking about MCP tools. Dumping a lot of data, like

especially data-related MCP tools. So what we do is

we curtail the output correctly. We know all these MCP

servers, etc., so that you are not spending too much of a token cost. And

not just the cost, right? What they call is a context rot. If you

dump in too much information, LLM has too many directions to go in. So we

curtail and do that context compaction. Second part we

have introduced is Specifically for data tasks, we create

memories automatically. So not every time agent is starting from scratch,

and it's very curated for data-related tasks. So in that way,

next time when the agent does the same task, it can tap into those memories

that are shared across the organization and get the task done in

very less number of steps. That's an interesting

point because I noticed that, that was one of the When I started

poking around my OpenClaw instance, right, inside of there, there's the

soul, but there's also kind of this memory type of notion and managing

that memory so the context doesn't have to start from

zero every time. And does that really save on tokens? Like, what's

the rough order of magnitude in terms of what the token save is?

No, the memory will save you tremendously because, for example,

let's take an example of data pipelines. Right? So if your data

pipelines are failing, usually there is a pattern, same issues you're going

to see again and again. Hey, data hasn't landed, that's why this pipeline particularly

fails. Now if that thing gets stored in the memory, your agent

is going to check the first thing is has data already

landed? That will save you other 10 other things

that agent would normally try out before coming to that, right? Boom, your

multiple workflows are saved. Your token cost saved, your time is saved

also. We fixed it very quickly, right? So it helps us

tremendously in that way. And the beauty of that is even the memory

layer cannot be very generic, right? You need to understand

as tasks are happening, what kind of memories are important. It needs

to be curated. There needs to be, say for example, somebody's writing

agents and harness for finance, there needs to be a finance-specific memory. that

will remember finance-related things. Similarly, for data work, what we

have created is data agent-specific memories, which works amazingly

well. And since we are talking about memory, right, memory

is not just limited to one session. So for example, what I'm trying to say

is, now if you have data pipelines, probably you have a team of

engineers maintaining that data pipeline. So for example,

I fixed this data pipeline today, for that data not landing

issue, and it gets saved in my memory. Of course, next time

in the regular systems, I try to fix it. Next time it will read from

the memory stored in my, say, code editor or Cloud Code or something like that,

and I can draw to it. But what about somebody else on my team? They

are also going to work on that pipeline, and they might be on on-call. That's

when the pipeline failed. So the memory layer that we have

built, it's a layer that gets shared across the teams and

organization as well. So then it's the— I call this a

hive-like mind in which you are storing the memory, and anybody can

come in and use that hive-like mind and sort of use

that knowledge to do things better. And the big benefit of this is

even for newer people joining your team, right, they don't have to start from scratch.

All this tribal knowledge is stored in that hive mind, which

we call as a memory layer. I like that because then

your first few rounds of learning it, right, you're

literally onboarding this virtual employee, it sounds like, right? It sounds somewhere between a minion

and a virtual employee that can kind of capture that tribal knowledge,

capture that institutional kind of wisdom. Yeah. And so

it's not so much you're spending tokens, you're kind of investing tokens, right,

for the future and training this virtual employee. And

presumably you'll get that, you'll see dividends later on.

Exactly. And this system works with agentic

frameworks that are out there already. People use, say, Claude Code, GitHub

Copilot, Cursor. The system works with those

already. We don't build LLMs, we don't build—

give agentic frameworks, but we build this harness which makes

these tools extremely suitable or powerful when it

comes to data engineering work.

Interesting. Can— does it learn on its own or

can you edit those memories? Right. So like, what if— what,

let's just say we have a, somebody makes a mistake, right? You obviously

wanna mark that and kind of remove that from the memory. Is that, is that

like, obviously you probably edit, it probably adds its own. And,

and the reason why I mentioned this is because when I dove into my

my OpenCLAWS memory file. I thought it was funny what it read about me.

So if people are watching this, you'll see I'm kind of like winking my eyes.

Apparently allergies are really bad today, which I did not factor that in when

sitting outside. And one of the things it learned about me was I

always ask about the pollen levels for the day, which I think is kind of

funny. It said that, you know, I ask about the weather, I ask about stocks,

and I ask about, you know, AI innovations and pollen report,

right? Does this— but obviously I can go in, I can open up a terminal

and edit it myself. But how does your solution— is it built into

the UI or is it like just a config file? So

as you start using the solution, you install, it will start creating

memories automatically. You can, of course, in your prompt give a

specific instruction. As you give the instructions, those get saved also. But you

can say specifically create a memory also. And through our

MCP, it will get that memory created as well. Now the harder

part usually is what if there are bad memories and you want to change memories

or you want to erase those out? The good news is it comes with that

flash stick that I think I remember it from the movie Men in

Black, right? It comes with that. So you can go in, delete your

memories because as I was telling you, we store these memories in a SaaS. So

in that way, they're shared with your team members, they're shared with the rest

of the organization, and there is a granular control you can do who they get

shared with. But at the same time, if you want to update

it, you can go to the UI and get those updated or

deleted as well. Oh, interesting. So you

have that neuralyzer built in. I think that's what the thing is called in Men

in Black. But so you can go back

and you can remove like, hey, everything I did today was terrible, so don't remember

that. What about

security, right? Obviously, There's a lot of trade, you know,

there's a lot of sensitive information that are going to be floated around in the

data engineering space. How does your solution, how does Ultimate AI

kind of address that? So first and foremost,

we don't look at the data directly. Okay, so it's a

harness that comes in, and this harness people can install locally

as well. So if you're some sensitive industry, maybe healthcare

or something like that, You can use it completely

locally, and the LLM solution that you use in the

background, people can use their LLM solution. As I was talking about, Claude

Core subscription or Codex subscription, they can use

that. They can hook up their own models also. We support, say for example,

OpenRouter, all these different models people can use. Even if they want to use

on-premise LLM, we support that as well. On the other side,

as a company and as a platform, We are SOC 2 certified,

pen tested, a bunch of big Fortune 500 companies

use our product already. So even on that side, I'm

sure we can make people's security teams happy because we have done a bunch of

work around it already. Oh, that's interesting. That's good.

Because I mean, the security conversation

comes up in AI, but I don't think it comes up often enough or early

enough. And Obviously, I think that's going to change as more and

more systems get deployed. And obviously, the Fortune

500 companies obviously also take that into account as

well. What would be your advice to people who are data

engineers who are curious about how do I make— how do I get my own

minions, right? Like, how do I start thinking about— let's roll

that up. How do I start thinking about

my job as a data engineer in terms of

I'm managing a dozen potential virtual employees

as opposed to doing it myself, quote unquote, the old-fashioned way.

Yeah, yeah. No, I think that's how we should start

thinking about it if anybody hasn't started going on that

path. There are a bunch of tools out there, and I

believe it's easy to get lost also because there are just so many tools and

things that are happening. on the AI side of the things. And what I have

seen is people try to retrofit the tools for software engineers to data engineering,

and usually that leaves bad taste in their mouth. And sometimes

I heard, hey, AI doesn't work. It's because you're using the wrong tools for data

engineering most of the times. So you need to use the

harnesses or tools that are specifically done for data engineering.

On our side, we open source Ultimate Code Project, so definitely try it out.

It's MIT licensed, completely open source. But in addition to that, what

we have done is on our website, we have put this

course called AI Data Engineer. So there is a section on our

ultimate.ai website, AI Data Engineer, where we have put

curated articles and we keep spending time to make sure all the updated

information is there. So somebody might be like at day 1, somebody might

already be at day 50. You go through that material and it will coach

you. What are the latest tools and what are the best things you can possibly

use to get these things done? Oh, that's really cool. So you

have built-in training, and you did mention that your product is open source. We'll make

sure that the link to the GitHub repo and as well as the, um,

this onboarding, um, this kind of this ment— what would you call it,

a mentoring tool, an onboarding tool? Like, what would you call

that? I just call it training boot camp. So

join it. Even if you're brand new, there is a bunch of useful

information. Even if you have been using it for a while, you know, world

of AI and technology changes so fast. Definitely it will help you

keep tabs on what are the things changing as well.

Yeah, no, I— it's a, it's a very fast-moving space

and it, it's not gotten slower. It's— I think the pace of innovation is

accelerating for good or for bad. What

What would be your advice to someone? I started asking this in the question, like,

there's a lot of computer science students that are very still

in school and they're very worried about AI taking their jobs. I think you and

I have gotten to the point where AI doesn't really take away your job. I

think it changes the nature of your job.

Yeah. But what would be first? My advice would be stick it

through, kids. But like aside from that, what would be

your advice to particularly, so I think a lot of people

are getting started in data engineering and if they hear that there's AI tools

assisting with that or doing that, they may be a little bit afraid or concerned

about the future here. I think the future looks bright, but that's just my opinion

as a, I wouldn't say I'm a perpetual

optimist, but I've kind of been through that cycle of,

oh, you know, I, AI is taking away my job, but I really kind of

see the nature of it. It just changes the nature of my job.

Yeah, yeah. If I can summarize this, I have heard those

things, hey, is AI going to take my job? And I always tell

if anybody says this to me, in my opinion,

it's not AI that's going to take your job, but it's going to be somebody

who can use AI is going to take your job. So if you don't upscale

yourself, because I completely resonate with your point. Our

jobs and roles are changing rapidly in this new world, so

upskilling yourself for AI is very, very important. And being

on that cutting edge of the AI and data engineering and all that data

work, that's the most important part. How I'm feeling

is going to unfold is like how 100, almost 100 years

ago, Industrial Revolution unfolded, right? There were, for example,

there were people who are making clothes by hand, right?

Then the machines came and we got operators who would operate those machines

to produce clothes at a humongous scale and see how

many clothes we have today, right? Same thing is going to happen

with AI in general. It's going to give us those machines

which will increase the overall throughput of all the people

out there for different professions. It's going to bring in a lot of prosperity.

But at that time, people who were making clothes by hand, they went out of

job. Maybe some of them learned how to operate the machines, but that's

the important part. Learn to operate the machines. Like, don't just say that, hey,

AI is going to take my jobs. Learn to use AI, embrace it. Your job

is not going anywhere there. That's a really good way to put it.

One of the examples I like is, um, there's self-driving cars,

right? But there's also a lot of cars have adaptive cruise

control. So for those not familiar with it, you know, it's basically cruise

control, which will keep your speed, but it also has proximity sensors. So

it'll apply the brake, right? And kind of keep you within a certain distance of

the car ahead of you, et cetera, et cetera, et cetera. And also can keep

you inside your lane. When I got that in, in

my car for the first time, I felt like— It's a good analogy for AI

because I'm still controlling the car. Right. I'm not fancy enough

to have a full-on, you know, self-driving car, but, um, but I

did notice that I became more like a captain of a ship, so to speak,

where I would— I felt like I was guiding the car which direction to

go. Now, in that case, I'm not a big fan of driving around

all the time, but so it would be nice to have a full driving car.

But I think that's kind of a good analogy for how,

how jobs are going to look like, right? And if you're a software engineer, you're

not going to be writing every line of code by hand anymore. I think,

to your example, I think, you know, when humans were sewing, doing the

sewing, right, individual stitch by stitch, clothes were made. And accordingly,

clothes were expensive. And I think we're going to see that kind of happen with

AI, right? Whether it's code, whether it's data engineering tasks.

Because certainly I think anyone out there who is in a professional

position where they're doing data engineering, There's plenty of

things that are just kind of on the back burner that they would do to

make their jobs more efficient, right? And I just think that,

I think by having AI enabling,

you have a lot more cognitive free room. And all those things

are on the back burner, on the whiteboard somewhere, or kind of on a Post-it

note somewhere about, hey, you know, I bet I can improve this process by

X number of percent, but they're too busy keeping the machines

working, right, keeping everything working. So with AI, I think you can have a

bit of cognitive surplus where you can tackle those. Sorry, I cut you off. You

were about to say something. No, nothing to add there. I think

that's so spot on. This is where the world is going.

That's cool. That's cool. So what's next for your company? Like, what,

what's, like, what other big challenges do you think you

all are going to take on? So one thing is, of course,

right now we have been automating few of the data engineering use cases,

but we are planning to expand to more and more use cases and at

the same time improve the support of the data stacks

we support as well. For example, you're talking about SQL Server,

maybe older Hadoop, Hive stacks. There's so much information

out there, and being early-stage company, we support certain

stacks, certain stacks we don't support. So the plan is to expand

into more ecosystems and at the same time expand for more

use cases as well. Yeah, that's cool. Yeah, and it's funny

because I know there's a lot of Hadoop legacy solutions out

there. Somebody told me, and I don't want to pick on Hadoop, right? But

Hadoop was one of the first, you know, big data solutions out there. And

I would imagine there's a lot of— somebody told me that it was moved to

the— what's Apache call it? The attic? the attic or the basement where it's basically

considered a legacy product now. And, you know, obviously

there's a lot of engineering— data engineering projects historically

have not turned over that quickly, right? Like, you know, there were— there are probably

And that's not that unusual. So I think that

because data modernization projects even if they're

needed, they may not happen because people are too busy keeping

the machines running, right? So if you can kind of— I think people are missing

the point about AI taking their jobs. I know we're back on this again. You

can always tell when the coffee hits me, right? Like, my mind goes in like

3 different places at once. But the release

of cognitive service, cognitive surplus,

I think is going to do a lot of good things for The IT

industry, and ultimately, I think, for the broader economy as a whole, right?

I think you, you mentioned prosperity before, and I know it's a, a lot of

people are very anxious about what AI can do. And you live, you know,

you're in the Bay Area, you're probably at the epicenter of a lot of this

back and forth, right? But I think that

by releasing some of this, the human minds

from the drudgery of some of the, the machinery that we have today,

I think has enormous potential for improving IT, right?

You know, take any kind of like regular person out there who, who has to

interact with their IT department. Would they describe it as a wonderful experience, right?

Would they describe it as a, you know, or is it more like going to

the DMV, right? Probably more like the DMV. Although

I will say credit where credit is due. In Maryland, I had to spend— I

recently got a new car and it wasn't as bad as I feared

it to be. You know, maybe they're using AI, I don't

know. Yeah, yeah. And as you mentioned, right, a lot

of this work we sort of parked it for later. Like you were talking

about batch jobs or Hadoop jobs, etc. There's still a bunch of those things running

because migrations had been nightmares, right?

Like people spend millions for years. But now I think, now

if I look at it with the angle of AI, AI can automate

90-95% of those migrations as well. Now, who knows, like, even

that whole modernization and digitization

in the data world, that will start accelerating much faster because of

this. And a lot faster, different companies can be on the

most modern technology. And they can, they can take advantage of

the newer features, the optimization technology. Yeah, I mean,

because no, no company in their right mind is going to suddenly, you know, say

we're going to hire, you know, we have this system, we can either

A, keep throwing a little bit of money at

it per year, right? And keep it going indefinitely. Or

B, we're going to hire 100 new data engineers and all these project

managers and things like that. We're going to spend billions on upgrading it. They're not

going to do that, right? But they do need to upgrade, right? So then if

it becomes a, well, you know, if you just spend just a little bit

more than what you do have to do to keep the lights on, and you

kind of have AI take a lot of the brunt work of that, right? Instead

of hiring 400 people, you can probably hire, you know, say

40, right, to do the migration. It starts to become more

palatable, right? It starts to become more justifiable in a, in

a balance sheet. And, and, you know, I think a lot of the breaches

we've seen have been from improperly

configured systems, right? Or legacy software

that hasn't been patched or legacy software that has no business still running

in this day and age. And I think you're right. I think AI is really—

I think if you put the right type of glasses on, AI

is a way to solve the problems that we've just

been pushing back for years and years. And technical

debt, that's the word I'm looking for. I think AI has the potential to

be a great way to pay down technical debt. Yeah.

But So while we're talking about

that, like, what is the thing that your customers who are successful on your

platform, what is the thing that they're most happy with at the end

of the day? What are they like, they call you up and they say, wow,

Pranesh, I'm so glad we did this because

what is it, increased productivity? Is it something else?

A few things, right? So for example, one example I gave was

we can manage people's infrastructure using AI. Right? And

we can optimize it. Now AI does it at scale

at which humans can't even do it. For example, it can change the configurations

300 times in a day. We can't do it, right? Like, how can we

change the configuration of single machine 300 times a day? You need 150

people doing just that. Yeah. And

especially in the space of data, your workloads are always changing, right?

As your data changes, your workload changes. So one of the things we have done

is that infrastructure optimization, where we have built agents which can

automatically analyze your infrastructure, change the configurations. We have built agents

which can analyze the data pipelines and automatically optimize them. And

of course, that saves tons of time for engineers, but at the same

time, your infrastructure runs at 90 to 100%

utilization, and those pipelines are always top-notch when it comes to

optimization because people run millions of SQL queries and hundreds of thousands of

pipelines. Now AI does that. The result is not just the

engineering hours saving, but real dollar savings on

the infrastructure cost also. So some of our biggest customers, we

have saved them millions of dollars in their Snowflake and

Databricks environment by optimizing this.

And yeah, that's super happy about it. They're like, hey, tool pays for

itself multiple times over. We are getting all these engineering productivity also. We're saving

tons of engineering hours. But hey, right there and then you're saving me

infrastructure costs also by automating that process by a huge margin.

No, it's a great way to— that's a great way to look at it. And

I really think there's so much opportunity. I know there's a lot of people, there

are a lot of naysayers now about what AI looks like and,

and, you know, will we, will we realize the gains that were been promised?

I think we will. I think it's going to be in places where we may

not think of, right? No one You know, and data engineers will be the first

people to tell you, like, they're not usually first of mind for a lot of

people in the C-suite, right? Data is like air. You don't think

about it. Good data engineering is like air. You don't think about it

until you don't have any, right? Like, it, you know, if you, you know, it's

kind of lost in, in, in the haze, so to speak. And

I think that, I think if people,

organizations that get their data estates kind of sorted out,

are at a competitive advantage to anyone that doesn't, right? And

it's very often— that's why I always make a big deal in the intro is

that people don't think about data engineering. I've been in hundreds of meetings where

the data engineering aspect is often neglected, right? One

story in particular was this guy had this great idea for this,

that for his organization, and he was going to hire

He was going to do all this sort of stuff, integrating all these different data

sources, you know, from public, private, enterprise, you name it,

right? You name it, he mentioned it, right? It was that type of guy. And

he goes, I'm going to need 20 data scientists. And I'm looking at it, I'm

like, you're going to need 18

data engineers and 2 data scientists, right? Like,

you know, like, you know, because like the data engineers, Quote unquote,

the old school kind of proper data engineers, I

mean, data scientists, they don't want to deal with SQL, right? They just want to

get kind of their data. But like, in terms of the sheer

amount of systems this guy wanted to integrate and do all the

translation, I mean, he's going to need— and he's probably going to need,

you know, 50 data engineers realistically, right? Or 20 of them doing

the work of 50. But yeah, it just goes to show you

that no one appreciates data engineering, right?

You know, it's almost invisible. It's certainly invisible when it works

well, and it's certainly visible when it

doesn't work. But it's also when it's working kind of mediocre,

it's also kind of invisible too, right? Yeah, yeah.

A lot of times it gets underestimated how much time

the work is required to put, I think, right amount of data

on the table, right? And that's why a lot of times people don't

understand why there is a huge backlog. Hey, I'm asking for simple data, why you

are telling me it's too much? So there are 50 people ahead of you and

every project is going to take 4 days, right? Takes time to find that data

curated and put it somewhere where it can be consumed by, say,

analytics use cases or data science use cases.

You know, one of the things that comes up a lot is the joke is

that DBAs used to stand for don't bother

asking, right? But it— but I think that

we had an earlier guest a long time ago now basically talk about that

one of the important shifts, I think AI really puts the acceleration

on the shift of mentality, is that data

DBAs, Data engineering managers have to think of

themselves less as gatekeepers and more like shopkeepers,

right? They're not the bouncers at the door anymore, right? They're the people

behind the counter that say, how can I help you? Bonus points if they can

set up a system that is— again, I'll go back to the DMV, right? If

you go to the Maryland DMV, I'm sure it's even more so where you

live, given it's the Valley, right? There's plenty of self-service kiosks

where if you just need a new XYZ, You can just scan

your driver's license, enter some information, of course,

enter some payment information as well, and you, you know, they'll,

they'll print up or it's almost fully self-service. And I

think that the data engineering organization in the enterprise in the future

is going to look a lot more like that than in the way it's looked

like historically. Yeah, a lot of it is going to

be self-service also, right? People just come in if it's simple tasks.

Hey, as an organization, data organization, will

serve it for you. You don't need to bother us, or we don't need to

bother you. Like, you can just get what you need. If it's complex stuff,

then we will put our minions to work. Otherwise, our minions will work

completely independently with you. Yeah. So do you

have— do you track metrics in your product that says,

hey, you know, this saves X number of hours, or is that something

that Will come later? No,

absolutely. We track it today. So it tracks what kind of tasks

have been done already by AI for you, and we

try to make sort of a guesstimate also. If this task was

done by you manually without using AI, this is how much

timewise it would have costed you. So that's one part of it. Second

is we do a bunch of benchmarking. against some other

tools out there. For example, if you use— if you try to do this

task using just vanilla Claude Code or GitHub Copilot, or

if you use some other AI solutions that Snowflake and Databricks has

put out also, like Codex Code, or dbt has put out

Visit, how much better we do against that also. So

we do the benchmarking, we publish those benchmarks. That's great. There are open-source

benchmarks around this. One is called Agentic Data Engineering,

ADE, and the other one is Data Agent Benchmark, DAB.

So we do that testing, we put out those benchmarks, and those keep continuously changing

as the space itself is so fast evolving. Wow.

It's not a real technology until there's benchmarks, right?

Yes. In the data world, in infrastructure, we love our

benchmarks, right? So why not do it? It's the easiest way to compare different

tools and different options. Well, awesome. Uh,

and the website is Ultima— Ultima—

I'm sorry, an Ultima just drove by. I'm sorry. Let me

help you there. It's called Ultimate, so

ultimate.ai, and our open source

project is called Ultimate Code, so ultimate

and code. Just search on it and you'll find the repository.

You should find our website also right there. Excellent. Thank you very

much. And I appreciate— I know we had some issues rescheduling, so

I appreciate your patience with that. And I'm really looking forward to

checking this out. You have my curiosity because there's all these like little ideas that

I have in terms of, you know, I have datasets and things like that, personal

and otherwise, that yes, I would love to go through and

organize those. However, I don't have time. Right?

Just as a personal project. I'm telling you, when you start running your own home

lab, you really start to feel the pain of what IT organizations

feel, right? I have an LLM server, and when it's down, my

kids tell me, and it just becomes like this, like, nightmare of

situations. So, like, the best way to learn is to do. And

I encourage everyone out there to build out a home lab with whatever you got

and just start messing around with the AI tools to make things easier.

And, uh, with that, I'll let the outro music play.