Human Oversight in AI Agents – Addressing Trust, Authorization, and Enterprise Governance

Human verification has the risk of becoming just a rubber stamp

because the AI can be engineered to a point

where they, where they are just so good at what

they do is they're 97% of the time. You would need a human to

be able to go through and just pay attention well enough. So for the 3

errors you get out of 100, that they won't just rubber stamp it out of

habit. Yeah. And I think another problem is if we continue to build on

this habit, then That also is going to backfire badly. I read

a lot of articles and I see a lot of people writing on Reddit also

that majority of the companies are pushing that, okay, use AI to

build your code, move fast. We have to deliver it fast.

70-80% of the code is now built through AI. Welcome to Data Driven,

the podcast exploring AI, data science, and data

engineering. Today's guest is Nitin Gupta, who's writing a book

on enterprise AI. His question is one the hype skips,

not what an agent can do, but what it should be allowed to do. Hello

and welcome back to Data Driven, the podcast where we explore the emerging

industry of artificial intelligence, data science, and of course,

without it all, data engineering. However, my favoritest data

engineer in the world can't make it today, but over from our sister

show Impact Quantum, I brought one of the most technology-curious

people I know, not just quantum curious, but she's curious about a lot of things.

She's sporting really cool glasses. Welcome to the show, Candice.

Thank you, Frank. It's really great to be here. It's been a little while, so

I'm really happy to join today. Awesome. Awesome. It has been a wild

fall start to fall and end summer for everyone on the show.

But I'm excited today to have a special guest, Nitin Gupta.

And he is going to talk about a question that gets often gets

overlooked in the excitement and hype around AI. Not just what

an agent can do, it's what an agent should be allowed to do.

Welcome to the show, Nitin. You are in New Jersey.

started in the virtual green room with what exit. But so you say

enterprise AI fails on authorization rather than the

models. What does, what does that look like in an actual real business

workflow? Sure. So thanks for having me here, Frank, and glad to meet you as

well, Candice. So I'm sure like everyone is trying to build

agents, AI agents, left and right. Everyone wants to try out that. Now,

especially when it comes to working on enterprise AI agents,

it's important that we understand what should an agent be authorized

to do? And it should not be given access to anything, to everything which

sits under your data. So usually what I've seen is if the

harness is not done the way, if the agent's boundaries are not

set in a way that, okay, it is doing a specific task that it is

supposed to do, majority of the times that leads to failures. Something

which works primarily quite well in terms of the pilots

doesn't work when you actually go to production. Or they get cut off,

especially from a legal or compliance perspective, because the agent's

tasks have not been clearly defined, its boundaries are not clearly

cut off, and it operates when— because it's going to move from

one kind of data to another. Usually the authority is something which gets

mixed up because if it fails, the ownership goes to one team

and they're like, no, this doesn't come under our purview, it belongs to some other

team. And that's where we need to have clear ownership and understanding as

to who should actually own it and how it needs to do the decisioning.

So that's what I would say from an authority perspective would help in

defining it clearly. And this is very timely because I

just heard on the news that a number of large banks

have basically have said that there's a lot of bots on their network. And I

think that's interesting. Yeah. And they're trying to

stop it. And I think this also comes for the last couple weeks.

ChatGPT has released GPT Dots. Yep.

Muse by Meta, I think is another one. I haven't used Muse yet. And

then GrokBot. GrokBot, I think might've been around a little longer. Yeah. I just

got access to GrokBot yesterday because Candace and I were talking with,

shout out to Barb. Okay. And it's another friend who's kind of works

behind the scenes in the show, other ventures. She, she was

all gaga about GrokBot, so I'm like, all right, I was Yeah. I was like,

go ahead, let me install it. Let me see what this is about. Right. So

in fact, I was reading about that and I'm using Muse a

lot because I'm literally doing a lot of my tasks and I just hand it

off to it. And I'm mostly chatting to Muse more than I was

doing it on Claude earlier. And prior to that, of course, on ChatGPT.

And I think when I was reading online, I saw, okay, people are divided between

GrokBot and then Muse. They're like, oh, Muse is nowhere compared to what

GrokBot does. So I think there are like 2 kinds of audiences, one who are

like pro specifically looking out what GrokBot can actually do.

And of course Muse, I mostly treat it as a personal agent,

which is just helps you fill with all kinds of forms, replying to your emails

and also all basic stuff. If you want it to be automated and you are

bored with those kinds of tasks, at least I feel Muse is pretty good. Maybe

you should give that also a try. And yeah, GrokBot is something which I'm

also thinking, okay, I should probably get a hang of that as well. Since probably

people are a lot gaga about it. And I think for the first month, we're

not sponsored by them, although if you're listening, if you're listening,

Elon, the, I think they give you a 50% discount for the first month. It's

just $15 a month. And I'm like, I want Meta Glasses. But yeah, of course

those are also not sponsored. There you go. No, because it's very real because I,

so I have a number of things. I have my own home lab and stuff

like that. And she's not powered on now, but I have a Reachy Mini robot.

for those listening. And I actually have her hooked up to an

LLM and a voice transcription service and all that, all

inside of my network. Okay. And so I'm trying to set up that. I have

a couple other bots running Hermes and things like that. I

also, I don't know, a lot of our listeners may know who Network Chuck is,

but Network Chuck is kind of this infrastructure guy who does a lot of AI

content now. And long story short, he always does these things

about, he did this thing about Paperclip recently, which is a way you can

orchestrate your agents together. So I'm trying to, at this point now I have have

3 or 4 agents. 2 are on-prem, 2 are the cloud ones,

GPT Dot as well as GrokBot. So, but I'm

feeling that pain too. I give it access to my email. Do I really trust

it? Like, yeah. And I know Candace is probably, I can't believe you gave it

access to your email. But it was useful because this morning I had a

crazy morning this morning. So I'd taken my kid to the doctor and all that,

and Andy told me he couldn't make it. So I just basically called

Candace, hey, if you wanna be on the show, you can. She's like, send me

all the information. So I just basically got on the phone with GPT Dot and

said, send Candice all the information. Nice. And she got it.

So it was just kind of, it's almost, and I think this is very real,

not just for home labbers and super nerds like us, but also

for the general enterprise. They're putting these

agents in. Consumers now are putting agents and giving it

access to their bank accounts. Whether or not that's a good idea, I think is

another thing. But it's interesting how you see these banks Now, oh

my God, we have bots everywhere on our network, like from customers. And I think

they're trying to block them, which again, I don't know if that's a great idea,

but everyone is all hyped about AI. But I think it comes

down to how do you own AI? Like how do you manage AI? The

most successful companies so far, and there's not a lot of them I've

heard, is that they treat it like an employee. Yep. It's like an AI employee.

It's gonna make mistakes. To give it access, you have to treat the RBAC and

all that stuff seriously. Walk us through what you've seen, because I'm,

I'm always curious about this because everyone always has a story for the

keynote, whether it's Jensen, whether it's Sam Altman.

But like when reality comes, when somebody comes back from,

say, any conference and they come back to their office on Monday and say, I

want to do this, how do we do it? And that's when reality kind of

sets in. Absolutely. So that's a great point that you brought in. Yeah. And

Jensen's recent, that duck show was impressive that, okay, he

was trying to use AI and then those ducks were like jumping and dancing. So

it was good to see. And those are Like attractive pieces, right? So those are

all flashy versions of AI that you see and which people are— when especially it

comes to enterprise AI. And by the way, I'm writing a book now on enterprise

AI and how decisions and authority should be done around that, especially when it

comes to working in large conglomerates and companies. Most of the

pieces fall when we try to force-fit AI

to, or wrap it around to our existing processes and see

how AI can benefit. In my opinion, it should be sitting within the

processes which are there. Because ultimately what happens is if whatever

futuristic product you build, AI product you build, if the adoption is not

happening, if people do not trust what it is, the problem that it

is solving, it's not going to give out the results that we are looking out

for. So you can keep on improving the models. You can keep on improving the

latency, make it like as grounded as possible, as realistic as

possible. But if people don't use it, people don't see the benefit

or what pain points it is solving for them, it's of no use. Right? And

that is one of the reasons why something that looks very good in terms of

pilot, because there you have all your champions who are going to be

totally evangelists about it. When it comes to

on-ground folks whose lives you are actually impacting or

making it efficient for them, they don't see it the same way. They feel

threatened that AI is taking away their jobs. They don't feel

empowered by using the AI. And hence what I feel is we should

not try to do a big bang in terms of AI. I see lots of

posts where people say, okay, we have built an army of AI agents and we

have a complete agency of where one person is having 10 AI

agent employees reporting, AI employees reporting to them. That is all okay. But when

you work on large enterprises where your field forces are different, your on-ground

folks are different, who may not even be comfortable in using Excel, for example,

but they are busy in selling your products. What should happen is that,

okay, we solve one pain point. Let it be a narrow problem that you're solving.

But it should be something that, okay, the agent completely owns it. It

understands that this is its boundary and we solve for that. Make it

replicable, deploy it, let the end users or ultimate

users to whom it's going to benefit, they see it in actuals

and then you start expanding its boundary. And unless and until you do that,

it's everything that you build is going to just stay. It's going to stay in

your own garage. It's not going to see the fruits of what it should.

So potential is there. I'm not saying that this potential is not there. Potential was

always there and it will continue to be there. But then on-ground reality is

that if people fear to use it, if people don't have the confidence,

or if the results, like you said, right, should you trust with a bot

reading your emails? That confidence would only come when you give it a narrow task,

right? That just focus on this aspect, right? I just want you to summarize, give

the entire podcast or details to Candice, and she's comfortable about that. You

don't want the agent to manipulate the text and provide

information in some other way. And if you are not having a human oversight, then

that's going to be disastrous. You're not going to trust it again. So building that

trust is important. And hence, in my opinion, we should go small

section by section, prove the value, gain the trust, and then

deploy it in a large scale. I think that makes a lot of sense. I

was thinking, as Frank was mentioning, I was playing around on chat.

GPT, and I was putting together the instructions to create my

first Grok bot, right? And it wanted to connect to

my LinkedIn. It wanted to connect to my email.

And then I'm thinking, I got a little bit like, I don't know. I don't

know what I want to allow you to connect to so you can just go

off and do whatever it is that you're going to do. So that's my question.

Where should we draw the line? Are there certain decisions,

particularly in maybe healthcare, Or finance

or hiring that AI agents should never be allowed to make on their

own. Correct. And that boundary is like interesting. I'm trying to put in,

in my current company as well that, okay, certain things which are

like, can be done in an autonomous way, let it go through the agent.

There you don't require a human oversight. Something which leads to changing, for

example, should we increase the dose of a medicine? Should we

increase the number of orders which are getting placed? AI can recommend, but then

actually it has to be a human who should like approve or adjust or make

changes on it. And something where completely there should not be

any intervention from AI and it has to be properly governed through a

human. Those decisions should be completely given to as a handoff

that, okay, here is where the agent should stop and say that, okay, this is,

these are certain decisions which are very critical or like crucial. For example,

healthcare, like you said, accessing bank account or information. Those are certain

things which are very sensitive topics. Which might have an

enormous impact. And that's where the boundary is important that

we completely hand it off to the agent that, okay, do not try to

action on that and route it to a human to act upon that.

So that's how we are also breaking things out that, okay, at certain places it's

all agents, certain places human in the loop, certain places

let LLM act as a judge and hand off

strategically how many, what portion should a human handle. And yeah, of

course, within the industry also, since different cases are there, supply

chain, order placement recommendations. So at different aspects,

we are giving different levels of handoffs to humans where required.

So can we trust AI agents to know their

own limitations? So how I see

that is we should probably add one. You can do

it with a human to look into it, but then there's a limitation in terms

of how many agents or what actions they are taking, a human can look

into it. Having an agent govern, and I see recently an eval

framework completely, and I think yesterday or day before Google launched where

they are completely letting the agents governed by another set of

agents who are supervising them and looking into what actions those agents

are actually taking. So I think probably we may have to move away from humans

looking at or just start certifying the samples and seeing

certain extracts of it. While other things, there has to be another set

of agents who are more governed, who are more rule-centric and deterministic,

and they supervise and see how these agents are behaving. So we may have to

go that. Like you said, there are so many bots which are coming onto the

banking system and they are trying to block it. You cannot block how many of

it you will you block. There will be some workarounds which people will figure out

and do. I also try to do some workarounds with by using Chrome as

an extension of Claude. And we, though it keeps on

giving you messages that, okay, it might treat you as a bot or as an

automation, there are ways to circumvent those aspects as well. And people will

figure it out very soon as to how to circumvent that piece. So we may

have to have more efficient way of looking into whether what actions

can be trusted and how do we build a trust system around that or a

governance around these agents. So who is responsible

when an AI agent makes a mistake? So in my opinion, making

mistake is a later part. Even before building an agent,

we should define an owner. So owner of the agent, we should, every agent should

have an owner. And ideally, in my opinion, that should not be a tech person.

It should be a person from business who should actually own the agent's

outcome. And then when you start developing that, the business person

should be able to see that, okay, this is the intended outcome that they

are looking at from the business perspective. Let's define the boundaries around that

and let the tech team then deliver or build on top of that, especially from

an enterprise perspective. So every agent which is there needs to have traceability

and accountability should stay with the business owner.

And then of course you can build a RACI around what the responsibilities should be

of the technical team, but then ultimate accountability should be owned by business.

And that's when, if a mistake happens or committed by an

agent, we know that this person would be accountable to see the outcomes, whether

the intended outcomes are getting met or not. And it's easier that way, right? Because

there are so many handoffs happening between data, right? Certain data can be in supply

chain, something, certain things can be in inventory, something can be in commercials

or consumer or customer side. Data can reside in different buckets and

handoffs would happen. Now, one, one team's responsibility, other team will

not take. So hence it's better that you assign responsibility

or ownership or accountability to a business person. Let that

person define it. And then that way it's easier to manage. And

future would be much lesser debating and discussing as to who

owns it or why did the agent go wrong or who should try to

fix it, but rather having a clear sense of responsibility from a business

side would make it more apt. And I think convincing and

for everyone. I guess it's also context-specific too, right?

So having, I really sat on it for a couple of days.

since getting Dot, like, I need to give it access to my calendar. I need

to give it access to my email, right? You need this. What? And I was

like, let me try it. And then now I'm totally hooked on it, right? But

I wouldn't give it access to my bank account just because I don't think that's

a good idea at all, right? I think there's also context too. And what's

interesting is, and I don't know, I don't know if this is part of the

system prompt as part of OpenAI or whatever, but when I,

whenever I have my GPT Dot, I call her Dottie.

very creative. When I have her send something on in my

email or through Teams, she's always very

insistent to say, this is not Frank, this is Dottie

acting on Frank's behalf, which I think is interesting. And Candace got to see that

firsthand. My wife got an email from Dottie the other day and she was like,

who's Dottie? She had problems. I was like, yeah, no, we

were, we're in the middle of some property shopping and

Story for another day. But I basically get on the phone and I talk to

the bot and I say, I need information about this. Is there broadband here? What's

the schools? And that sort of thing. And it'll go out and give me a

whole thing and I can— it's just very interesting. But you're right. I think there's

also context. I would not go to— I would be very leery

of a medical diagnosis on that. Right. And certainly financial

transactions or things like that. Right. For information retrieval, Which

I think as an information worker or just a person living in

2026 where so much information is thrown at us, right?

I remember when email used to be exciting. Oh, I got an email. Now it's

like, oh God, I have how many unread emails, right? Let alone instant messaging,

text messages, the whole shebang, like everything,

right? I think there's a real opportunity for it to act like a

virtual assistant that you would have that would filter through. Yeah. Now, yes,

are there chances it would mess something up? Yeah, absolutely. But the

chances of me missing something because I have 100 emails came in

today and I can only read 30 of them or enact on 10 of them,

right? There's a certain scale that we need to

get as people trying to live their lives in today's

world. But one of the things that makes me wonder, like, where do you put

human— where do humans sit in the loop, right? So it

becomes an actual additive. a value add rather than just a rubber

stamp, right? Like I said, probably the areas

where information can become a disaster, we have to set

that boundary as to where exactly that decisioning boundary has to be.

Something which is really impacting the bottom line of your business and that,

okay, a mistake can significantly impact your business.

That's where we should definitely have humans sitting and taking those actions.

Anything which is, like you said, a virtual assistant or something which is a

summarizer or just helping you understand your

emails and sorting things out or calendar, calendarizing things. Those are

certain things which are low impact things, right? Especially it's not impacting the

bottom line of your business. Those things can very well be handed off to

AI. And since you brought about your example for context and you would not be

comfortable in sharing the banking piece of it, I wanted to share yesterday I

was speaking to one of the founders and then he's handling payments for

a quick commerce. He's handling a quick commerce company and assisting multiple

quick commerce companies to look into and the payments of the

blue-collar workers. They have automated their entire payment system

and it hits the banking softwares without even hitting the banking softwares

with the help of AI. They have literally automated payments. The day they get it

from all these big commerce companies, the very next day it gets

reconciled and split across to all the gig workers who are there.

So at one places, and again, it's by context, at certain

places, business owners are more comfortable not giving such

sensitive information and details to AI to let it handle, whereas

it's acting as a powerful medium for them in the other place,

right? Where it was too much of a human effort, which was like

involved in the past to give out payments to all these workers, because it's like

a weekly payments which have to be given for the hours that they have put

in, the amount of miles that they have driven and the number of orders that

they have completed. That has completely been automated with the help of AI and with

a click of a button. And there are no complaints. There are in fact lesser

complaints because people are not reaching out that my payment did not arrive on time.

There are issues in the payment and calculations. All those

have been handed off and given to an agent and it is doing

the work seamlessly. So I would say again, as

confidence increases, as we see more repetitive tasks which are

there, even such sensitive information also, people will start

trusting AI to handle it. Of course, you'll have to do your due diligence that

it clearly separates. It should not transfer a million dollars when

it is supposed to be sending out a few hundred bucks. So yeah, of course

you'll have to do your due diligence. But I do see that trend also where

people are trusting in AI making such kind of decisions

as well. It's almost like the AI needs to learn common sense. Yeah.

If you get a call, let's just say, I don't know, you bought me concert

tickets or you bought me tickets to the NVIDIA show. Right. And let's

just say that's $2,000. Then suddenly my agent sends you

$20,000. Right. Like, clearly that's common sense

would know I wouldn't say no human, but most humans would be like,

wait a minute, how much does that shit cost again? Yeah. I just feel like

common sense would go a long way. Actually, humans are not—

could use some more common sense, but that's a whole other thing. But some listeners

are going to say, and I go back and forth on this myself, is that

this is just good old-fashioned identity and access management

with an AI label attached to it. But that pessimistic

view doesn't seem like it covers it. Something's different. What is that

difference? I can't put my— I can't put it into words. What's your take on

that? So I think because the roles are evolving, because

the boundaries are increasing and we are seeing a number of

dimensions through the data, those limits are not just roles,

role-based or access-based, right? And because you are seeing different dimensions

with the help of same data, that probably requires different levels

of roles and accesses. I think it is something beyond probably worth, and that is

the reason I think companies are spending time in setting up the governance practice

or AI governance standards and approvals in place because

they themselves are seeing with every new use case, we have to work

around as to what should be the guardrails that should be implemented. So there's a

change and shift which everyone is observing. And I think the

guidelines are still getting set, or at least they are still work in

progress and they are evolving as we see more and more changes

or improvements in which systems are getting created. So if you were

designing an AI agent for your own company,

what's one thing you would absolutely refuse to let it do

without your permission? I would say, again, it will depend on the use case

that I'm working on because there are like multiple products on which like I'm

thinking of deploying the agents. But then something which I'm not going to

let an agent decide is if it is something which is directly impacting my

consumers or customers. If that decisioning layer is something which

I'll still let be driven through a human. Let all the

preparation work, let all the extraction work from the data, from the

documents, everything, let that happen through the agent. But where the

decisioning of actually giving out what should be given

to the ultimate consumer, the approval should still sit with

the human. And I'll not let that at least someday. Somewhere which is a

lower priority task or something which is not impacting the

end consumer deliverable, that I'm going to let the agent decide, but

anything which is from a decisioning perspective where we definitely

require human supervision, like placing an order or accepting

or deciding the amount of returns that should be done, that is something,

or labels like checked, that is something I'll not let an agent do that. Otherwise

for extraction, pulling in the information, summarizing,

getting all the, even getting the decisioning framework ready, I'll let that agent

do that. So let it even create the decision. I'm trying out Jev right now.

If you, have you heard of Jev by TypeScript? Yeah, I heard of it. I

have play with it yet, but yeah. That's pretty good. So it's good if you

are creating certain things on like deterministic way of happening or

something which is, can be like answered in a yes or no. So if you

want something to do a quick filtering or quick classification, you don't need an

LLM to do that. And Jeff can do it like pretty fast without requiring

so much of an effort. So those are the decisions which probably I'll let

these kind of models brought in place because I know LLMs may hallucinate as

well. So if a capability like Jeff can be brought into, then

probably it's going to further increase my confidence in the decisioning. And

maybe there might be some more aspects of it, which I can give off to

the agents rather than doing it by a human way. So I think this

also probably what we are discussing today may evolve to more depending on the

capabilities and improvements which would happen. So I think this is probably

worth watching as to how this space improves and grows. So what

happens then when an AI agent makes the right decision,

For the wrong reasons. If the outcome is successful, but we can't

explain how the agent arrived there, should we trust it to make that decision again?

Usually whenever I work on anything related to AI

agents, I ensure that there is proper auditability and traceability.

The data lineage has to be tracked. If we don't

have that, then we are calling for our own disaster. I've worked in regulatory

industry also in the past, and that's where it's coming from. You can very well

be sued any given day and people can question the decisioning which has happened. So

it's very important that anything which the agent decides or governs is

properly traceable. So even though right answer has arrived, if

you are not able to trace it back as to how did it arrive to

that answer, then it's of no use. And I wouldn't let that go into production

if we don't have traceability. Interesting. I do worry about the

traceability and the audit trail of this. Clearly that has to be

part of any serious implementation of an agentic system. Yep.

What's your take on that? It should be there. And even within

that also, there should be a checker and a corrector. An

agent can be a checker and a second agent can be a corrector that, okay,

whatever work an agent has done, one another agent should check it. And

then the third agent should actually correct it. Someone who is a creator should

not be an approver of the same task. So it has to be handoff and

given to another agent to be approver. You cannot be an author

and approver of the same tasks. So that's how I'm going to frame it. And

that way it becomes easier to understand and trace things also

and understand that it's not biased in terms of the results that it is

giving out. It's almost like LLM as a judge. Yeah. You can think of

it like an LLM as a judge, but LLM as a judge would apply to

the entire agent's output in the end. This is more when actually that

agent is taking that decisioning. So at that time also, we have to check

whether the output that it is giving out is actually correct or needs to be

corrected first. Interesting. So could AI agents

eventually have too much influence over human

decisions? Personally, I would say yes. Right now,

whenever I try to present or I create any decks for my leadership,

or if I'm presenting internally to any of my

stakeholders or to my team, I run it through an agent to see

and take its perspective, right? And then of course, I provide the personality

of the persons also to whom I'm about to present. It takes the role of

that particular personality and gives me feedback. And a lot of times what I've

seen is it is giving me that actual blunt feedback that I'm looking out for.

I'm not looking for sugar-coated feedback. So I try to give it that this is

the personality of a person to whom I'm going to speak to, run it through,

run my take as if you are acting that person. Majority of the times I've

seen that whatever feedback it has given or the questions that I'm taking to my

leadership, I ask it, okay, here are the questions which I'm about to ask. Do

Can I think of certain answers already or what my leader

might respond to? Usually it has given me very accurate or near accurate

answers. And when I present, mostly I get an answer in a yes or no,

agree or disagree mode when I'm speaking with my stakeholders. So I see

that trust building, that increasing further in my opinion. And at least I've been

relying more on that. And if we are giving it the right prompt and

asking it to stay in the right role. So that's how I know I see

it. Interesting. So do you think the biggest risk with AI

agents is that they'll become too independent or that

humans will become too comfortable letting them think for us?

I think we are becoming too comfortable in letting agents think.

It's like what you sow is what you reap, right? So probably we are progressing

towards that side, in my opinion. Interesting. It's fascinating because

that's a huge question. There's research now being done that some

mental cognition of certain types have

declined in people who are heavy AI users. I also think

too, when's the last time you actually stopped and read the popup on a

website, right? Accept all cookies, reject all cookies, right? Like, Yeah. I

think there's a definite risk of click fatigue. Okay,

fine. There's definitely a risk of, oh, popup approved, right? The

human verification has the risk of becoming just a rubber

stamp, right? Because the AI can be engineered to a point,

the agents can be engineered to a point where they are just so good at

They, what they do is that they're right 97% of the time, right?

You would need a human to be able to go through and just pay attention

well enough so that for the 3 errors you get out of 100, that

they won't just rubber stamp it out of habit. Yeah. And I think another problem

is if we continue to build on this habit, then that also is going to

backfire badly. I read a lot of articles and I see a lot of

people writing on Reddit. All the companies are pushing that, okay, use AI to

build your code. Move fast, we have to deliver it fast. 70-80%

of the code is right now built through AI. Remaining 30%, I'm not talking

about my current company. I'm talking in general as existing as a, as what

we are seeing as a trend. The code that is written by AI, people

are not even finding time to verify what it has written. They are just seeing

the outputs. It is writing extra lines of code, which

no one has seen, or no one is trying to see whether it was

required at all or not. And later on when it's the people are trying to

fix those things, I think people have no clue. The companies are running

short of that. Okay. We can, 10 people of work can be done with 2

or 3 folks. You use AI to code all that stuff. Later on when disaster

happens, your team will not be able to figure out what they have done. So

I think too much reliance on AI also should not be done.

If even if you are doing that, that peer review concept, which was earlier there,

right? That a developer, one developer develops the code and second

developer does a peer review of the work and then it goes to production.

Probably something like that should definitely be done where you have AI engineers, of

course, who will validate and the code or the developers would

validate the code, what has been written through AI as well. Because if you

don't have these guardrails, probably for wipe coding, for

prototyping is fine, but at an enterprise level, these kinds of things may not

work or it's going to lead to a significant loss if you don't have

right governance in place for whatever code has been integrated. Interesting.

So what happens when AI agents develop conflicting

priorities? When they develop conflicting priorities,

again, should be given to a human to decide.

Whenever there is a conflicting priority which arises, it should be routed to a human

to make a decision and move forward. So then should AI

agents be allowed to learn and change their behavior

without asking permission? Again, it will depend on what task we are giving

to them and what data access they have. If it is on sensitive

data, then I would not let that agent learn or adapt or change its

behavior. We have to limit the access and try it out, and they should

not be given access to any sensitive data. Interesting. I think about the people who

are using these AI agents, and is that creating a new

kind of digital inequality? It is. Those who are

using it, probably it's like you can just think like a booster, right?

Like that a car is having this on NOS where you just press the button

and then you're literally flying your car. That's the kind of, I think,

difference which is it is creating. It may not be done the

same way at an enterprise level, but at least at a small level or small

or medium scale businesses, for sure it's giving

significant advantage to the people who are using it. If done the right way,

I would say, and relying on it is fine, but then having a

governance and then properly seeing what it is doing is very important. So I do

see that, okay, people who are using it are also getting significant

benefit of using and adopting it the right way. And then productivity has

increased. I'm sure like in at least 10x is what everybody

else says, that their productivity has increased. They're able to focus on more

valuable work and even the development work also has significantly

increased or improved. Yeah, that's interesting. No, because that does seem to be

what you hear when things go right. Sorry. Yeah, go ahead. I

was saying, do you think that we have the right to know when an AI

agent is acting on someone's behalf? Sorry, I didn't get it. Could you

repeat? I'll repeat that. Should we have the right to

know when an AI agent is acting on someone's

behalf? So that's a tricky question. And like Frank

was saying, when he was trying to do property shopping, there was some AI agent

which was getting all the questions. I think there will be times where things

will get too blurry when we don't even know whether we are talking to an

AI agent or it's an actual person who is talking on the other side of

the table. It's funny. I was actually talking— a friend of ours

was talking on the phone, on the speakerphone. He was ordering takeout and

the lady on the other side sounded like an AI.

So when he hung up, I said, was that a person or an AI? And

he looked at me like I was crazy. And he goes, no, it was a

person. I was like, okay. But no, I think you're right. I think there should

be— again, I'm always a fan for transparency. I think you should

know that an AI is acting on your behalf or

something like that, right? Like on somebody's behalf. I think there should be some kind

of transparency, right? Where there's no guessing, right?

Yeah. People are creating their own digital twins or digital AI and then they're

launching it. Okay, this is my digital twin. You can speak with it and

try to get that information that you would otherwise need from me. So I do

see people or famous personalities, they are creating their twins and they are calling

it that this is their digital version. Which can be accessed 24/7.

But yeah, at least from video creations and all, I see people are making so

realistic videos that it's difficult to understand whether those are fake

videos or like deepfake videos or actual videos. So I would

say impersonating probably, or an AI acting or impersonating an

actual person, probably that is something which should be disclosed that,

okay, this is an AI acting on behalf of a person. But to what

level people would actually do it, this is something which time probably would tell

because it's getting more and more blurrier these days. For sure,

technology will improve and it will become closer towards an actual human.

Interesting. What do you think about it? What do you think of how is this

taking forward? I think there's so many ethical questions and

they range from— there's so many questions for this. This is what makes it exciting,

right? Because very rarely do you get to have deep philosophical

questions around technology and enterprise technology in

particular, right? I just think that there's so many unanswered questions

here from the mundane to the profound, right? Obviously you

don't— there was, I didn't see the whole thing, but apparently one of the

healthcare companies can reject the claim in 1.2 seconds or something like

that. And something like, which obviously there are

multiple ethical, probably even some legal implications

to that. Yeah. I'm not a lawyer. I just have spent some money on lawyers,

so I wouldn't dare guess what that's going to look like. But what

happens— and this gets to the old self-driving car thing— if a self-driving

car algorithm hits somebody or gets into an accident, who's

responsible, right? Who— or who— if somebody gets killed,

God forbid, what— who gets charged for the crime? Is it the person in the

car, the owner of the car, the company, the software maker? There's going

to be— I predict in the not too distant future, there's going to be some

very aggressive district attorneys that are going to stake their

career on arresting some Some kid in a cubicle or something

like that, right? Like, I don't know. I don't think we're ready for this. I

think as a society it's here anyway. I think this is going to be one

of those things where it's going to change

so many things that we won't look at the world quite the

same way again, right? A big historical event, right? Like a bit like the

Industrial Revolution, right? I think it's not a stretch to compare

this to that, right? Because instead of— the Industrial Revolution took human

muscle power. And mechanized it, right? The

AI revolution, or whatever historians will end up calling this time,

is mechanizing cognitive human cognition,

right? Not always perfectly, not always well, but it is

doing it in a way, right? I can generate an image or I can

generate an outline or something, right? Before, I'd have to think

about it. I had to be in a certain state of mind. I experienced this

personally. Personally, Candace was there for it when I had a concussion

earlier this year, right? And just the context switching

for about a month or so was very difficult, right? To switch

from one task to another. And I found myself using AI as

like a crutch, so to speak, where— and not in a negative way, but I

need to be able to just do this context and do that context, right? And

I was able to kind of get AI to automate some things for me.

So my productive output, both in the podcast, the blog, and

at work, was about 80, 90%, even though

cognitively I was at 50% on a good day. So I

don't know. I really don't know. I think it's— I'm really looking forward

to your book. When your book is done, you can come out and we can

talk about it. But I think it's a— there's so many questions

to ask that are unanswered, and there's so many questions we don't have

yet. Right? Can you imagine 10 years ago? So 10 years ago, I was at

a Microsoft Research event, and it was the—

I've talked about this conference a lot, so a lot of our listeners will know

this— MLADS, Machine Learning and Data Science Summit, right? And this was

2016, and I asked the guy, I was like, do you think AI will be

able to write code? And he's like, eventually, right? Are

software engineers out of work? And he's like, eventually, probably for not

another 10 to 20 years. And sure enough,

just around the 10-year mark, will they actually replace them? I don't think so. I

think there's a lot more to that. I think the nature of the job will

change. Yeah. Just like power tools, right? If you talk to

carpenters in the Middle Ages, had a very different workload than they do

today, right? I just think power tools and things like that. I think

these are power tools for information workers and what that

means and how things change. I don't think we've really figured it all out yet.

And I'd be very cautious of anyone that says they've got all the answers because

there's no answers yet. Absolutely agree with you. And to your point, I

absolutely agree that for sure, what are cognitive— with our cognitive

power or what are with our mindset, we probably can do limited. But then with

AI in our hand, we can probably expedite the work and

let it do certain things which in a normal given day we may not be

able to think of. I like to— every night I hand off certain

tasks to Claude, to ChatGPT, and to Muse. I just let them

run. And when I come back during my morning, it's already done and it's ready

for me to consume. And when I read what it has created, it gives me

new ideas and thoughts, which I could not think of because I'm

limited to my work or my area of expertise. I did not think from someone

else's persona. And I asked it to behave like that particular person and

create it in a different way. And it gives so many ideas that every

single day I try to improve that. Okay. I need to improve or work on

this aspect because my cognitive limitations did not allow me to think from those

perspectives. So I see it like as an interesting way that, okay, it's

actually helping humans also improve and the

capabilities or skills that we should possess. We actually need to upskill

our game as well. Rather than just purely relying on AI, we

should— it's actually helping you improve your own skills too. And

by the way, on a funny side, just to say, so my wife actually cloned

my voice and then she's given it to AI to use it on my hands.

And she's like, I am not feeling well. So she expects me to pacify and

comfort her. And I said, okay, just take some rest and lie down. And she's

going to put on her phone and see, this is what digital Nitin says to

me to comfort her, to make her think that, okay, I'll get something for you,

et cetera, et cetera. She's like, this is the behavior I expect from you, not

from the AI to do. So you learn so many things, right? Especially

from these kinds of instances also with our limitations, human limitations, we

behave in a certain way or do things in a way. So yeah, all good

use cases too, which we need to look into. We live in the future. Yeah.

One last thought for a team that's going to start Monday or they're

listening to this and what's one useful agent, agentic

workflow that they can deploy safely in about a month? And

what's something they absolutely shouldn't? I think we touched on this. What's something they shouldn't

even automate yet? What's a good, safe place to start to experiment?

I think it, again, it would depend on the kind of company or the kind

of team which is working or the the problems of businesses

that they are working on. If I were to give recommendation, what I'll do is

I'll ask the team to identify top 3 pressing problems which

can be automated or which can be handoff where decisioning

cannot be impacting the bottom line of your business. Something which can be

automated, something which is repetitive, something which is taking away significant amount of

your— it is mundane. And there are deterministic steps to achieve

that piece straight away. Those are low-hanging fruits which should be tackled first.

As a low-hanging fruit to develop an agent for that. I would not touch something

where crucial decisions require human discussions,

be it discussions with your leadership, be it discussions with your management. If

those decisions are required or those are sensitive, if you're

working in a regulated industry like healthcare, finance, then to

completely limit your exposure towards those kinds of agents, not to give them

any kind of sensitive information. So keep your agents grounded towards those

datasets first. Which you feel will not let or

impact or let someone else sue you. Okay. If I were to say that it's

not be impacting negatively your industry or your work.

So pick those grounding use cases first, which can be

low-hanging use cases. Learn from the mistakes that you get

from your first deployment and then retrospect from those learnings and

then move to the next agent. Don't try to build 10 agents in your very

first month. Make one thing right, pick one small use case, have

accountability. That's for sure. Have a business user who's accountable for it

from an outcomes perspective, and then you build on it, right? Because that's very

important. And I think probably keep building and keep exploring. You have to

fail fast. The faster you fail, the faster you're going to learn and reiterate. So

always learn, relearn, undo your learnings, what you have done in the past,

and then try the new ways of this world and be futuristic. Interesting.

Any parting thoughts? I would say there are some frameworks which every company should have,

especially when you are working on an enterprise strategy. So I've listed some of

those frameworks in my upcoming book as well, but then have frameworks

also ready when you're working on agency and on, always look out for these

frontier model companies and these Microsofts and Googles of the world. They,

on a regular basis, they release articles and frameworks on

agentic AI, on responsible AI governance. So even though

if someone is a small startup or a medium company or smaller medium

company or a large enterprise, always look out for these kind of frameworks and

learn from what others are doing. That way, what not to commit

a mistake which others might be committing. So that's a good way to start with.

Rather than we committing mistakes, we can learn from others' mistakes

when we are deploying and working on these agents. Interesting. All right. Well, with

that, we'll let the outro play. Awesome, man. That was great. That was fascinating.