Jeremy Utley on Crafting Smarter Conversations with AI

Today on the data driven podcast, we have the privilege of hosting

none other than Jeremy Utley. Now, Jeremy

isn't just any guest. He's an academic marvel

and entrepreneurial spirit rolled into 1. Hailing

from the prestigious corridors of Stanford as an adjunct professor, he's

the kind of chap who educates the future disruptors of Silicon

Valley. He is here to tell us how to get the most out of

generative AI. Now on to the show.

Hello, and welcome back to Data Driven, the podcast where we explore the emergent fields

of artificial intelligence, data engineering, and data science,

and all the associated technologies. With me today is

Jeremy Utley, who is a, adjunct

professor, venture investor, and co author of the book,

Idea Flow, The Only Business Metric That Matters. Welcome to the

show, Jeremy. Thanks for having me. Hey, no problem.

So Stanford. That's kind of a big

deal. It's a it's a special place. Yeah. I'm

just trying to not get found out. I'm sure it's, you know, similar to,

the, the guy on Office Space. Right? At some point, there'll the clerical error

will be revealed. You'll you'll know when they move you to the

basement. Right? Exactly. Exactly. Yeah. But I've been teaching at

the most incredible students in the world and and get to study some of those

incredible innovators in the world. So, not just I may be a

professor or an adjunct professor, but I really consider myself to be a front row

student in in the classroom alongside my students. Very

cool. Very cool. So, what

what is the most important metric? I'll start right there.

Well, the most important metric we call idea

flows, the only business metric that matters. And the reason that we make that bold

claim is because it's the only measure of your

team's capacity to solve problems. And the the only

constant in our day to day lives is problems. In in our businesses, I

don't know, a single business that is facing a day without problems. And

so if you think about problems as the constant, then your

team's capacity to solve problems is really the most important thing

that you should be measuring. And yet, nobody really even knows how to

measure it. And so we talk about idea flow as the as the way to

measure a team's capacity to solve problems.

Interesting. Interesting. And is this is this changing now when we have the

reality of AI assisted teams?

Yeah. Yeah. Absolutely. That's it's a really insightful question. Yes.

It does change or sorry. It has the

potential to change. And yet, what our research suggests that we've

conducted over the last year or so, is that sadly, it

actually doesn't change in practice. In theory, it could change but in

practice it often doesn't. What's interesting, so what

what are the barriers to this? Right? Because I have some thoughts on this. I

know that a number of companies have basically outright

banned, use of AI tools

with good intentions, right, because the privacy policies, etcetera, etcetera,

but in reality, people are copying and pasting sensitive stuff anyway.

So, it seems like banning something outright

doesn't always work in a number of areas. But

what what are the barriers? Right? Because it it it can, like

you said, but in practice, what's what's what are the blockers?

Ultimately, it's it's human psychology, really, is what what's,

the challenge. It turns out that our expectations of the technology,

are hamstringing our ability to make use of it. Because

we're approaching the technology. Most teams that we studied

approach the technology as an oracle. It's almost like a search

box. It's gonna give them the best answer. Right?

And that's the wrong way to approach the technology. It does feel somewhat

magical when you type in, you know, an enigmatic query

and get a seemingly intelligent response. I mean, that feels

magical, but the teams that do that underperform. The teams

that overperform are the teams that

treat generative AI not as an oracle, but as a thought

partner, as as a conversation partner, and iteratively

work together with the AI to discover a better answer.

And the irony of that is it's not very magical, actually. It feels

like work. And yet, where teams that

treat AI as a conversation partner arrive is

light years better than teams that treat

AI as an oracle perform.

Interesting. So what are the what I think I know what you're

getting at in terms of treating it like an oracle versus treating it like a

conversation. Because I've seen that as I do more and more of this, I

hate the term prompt engineering. I hate the strong word. I have mixed feelings about

the term prompt engineering because there is no one single prompt to rule them

all. Mhmm. At least that's been my experience where you kind of you kind

of it's like a conversation, like, you're having. It's not a

person I know. It's not a person I know. It's not a But it's but

but it is a mindset, Frank. That's the thing. It's a mindset. And people

don't come with the mindset of I want to have a conversation. People are lazy.

won the Nobel prize for what he deemed satisficing, which

was the human tendency to settle for good enough. Right? And in most

of our lives, it's fine. I need a good enough pair of jeans, I need

a good enough cup of coffee, whatever it is. Right? But when we're trying to

solve problems, good enough sometimes is is okay. But often, especially

when it pertains to innovation, you don't just want the good enough thing. You want

the best thing. And it's in that area where when we really

want the best solution, that our tendency to settle for good enough

really hurts us. Because what teams do is they put

in a prompt and they get a pretty good answer and they go, woah,

I I was prepared to take an hour working on this but we kind of

got pretty good in 5 minutes. You guys want to go get coffee? And everybody

just gives up because they got good enough. And so that's,

you know, I I think it really is a mindset thing. Forget the word prompt

engineering. It's all it's it's self engineering. It's human

engineering. And one of the best things that a human being can

do is say to the AI why they don't like

the answer the AI gave. Right? So take your expertise.

Here, this is something that anyone is listening can do right now. Take something

that you know you're an expert on. So for example, I'm an expert

on customer insights or or low resolution prototyping and

experimentation. Right? So I might say to the AI, can you give me a

step by step guide for how to conduct an experiment? K. If I

did that, we can do it right now live if we wanted to. But if

I do that, it's gonna give me like, you know, the average of the Internet.

Right? And it's gonna draw from a bunch of stuff that may be good, may

be bad. By the way, I'm an expert, so the chances of my knowledge surpass

it are reasonable, you know, at least. But because it's gonna give me

the average of the Internet, it's probably gonna find some corners that I don't know

about and it's probably gonna say some stuff that I vehemently disagree with. Well, where

most people give up and I think because they want AI to not

be that good is they look at the response and they go, see.

It didn't even know that you're supposed to test for desirability and not

feasibility. Right? Whatever. And then they say, that's why AI is no

good. Well, Human Engineering, not Prompt Engineering.

Human Engineering is to say, okay, human, tell the AI

what you disagree with and why you disagree with it,

and ask the AI to regenerate an answer

given the following considerations and put in your critique.

Most people if they do that, just even that one thought exercise,

will be blown away. Yeah. You get an order of magnitude better response.

Absolutely. Right? Absolutely. Because you're kind of focusing the cone

of, you know, inquiry with your own

expertise. And what people want is they I mean, no one would

ever if you think about, like, AI like an MBA intern. Right? No

one gets an intern from Harvard Business School, gives them 2

sentences of instruction and then at the end of the summer says, man, their workout

was no good. I didn't interact with them at all. I didn't give them any

guidance. But for crying out loud, what's

Harvard doing these days? Right? No. Nobody gives a

human being 2 sentences of input and then critiques how bad of a job they

did, right? And yet we open chat gpt, we give

2 sentences of input, if that by the way, and then we go, See, it's

not very good, well, work with it. Garbage in garbage

out. It's as old as possible. And the reason most people don't I think most

people don't wanna work with it is because they don't want it to be any

good. Yeah. I could see that. Totally.

And the people who do want it to be good will be unlocked and

unleashed. But it requires not prompt engineering, but

human copilot engineering. I do like the

fact that a lot of these tools that are coming out are being called copilots,

right? Because I think it shifts the focus away from

AI isn't going to do it all. AI is not

probably gonna take your job, right? But it's just an

assistant. Right? It's it's to help you out where you may

want a little bit of boost. I also think that I think what you you

described is good enough factor is I think people see

large language models and they they assume it's a search box only

better. Yes. Well, and part of our

challenge, you know, I was talking with a with a psychologist, David

McCraney, who wrote How Minds Change. He's a he's a journalist, an author,

a podcaster. He's a he's the host of You Are Not So Smart, which is

all, you know, obsessed with cognitive bias, which I love. Mhmm. I love

that podcast. And one of the things that David and I were talking about yeah.

I can't remember the the name for the cognitive bias, but when we see

something that we think we understand, we just track into

our kind of typical neuro pathways. Right? So we

see a text box and we go, oh, I've seen one of these

before. This is like that. And we so and this

this being generative AI is not like that. That being

search. Generative AI is not search. But because of the

kind of the the UI, we

approach it like search. We go, okay, I want the answer. Just give me a

list of links that I'm gonna click through and decide on. And we don't

interrogate Google, we don't critique Google or any search

engine, right? We don't say why we want it

or state our intention, right? But if you start to do

some of these fundamental kind of human

conversational tactics, if you start treating it more like a

person than like a search box, you get

exponentially better results. But you're right, even

the UI itself predisposes us to treat the

technology and to think about the technology in a particular way and

that is actually holding us back. Interesting. I noticed

this in a completely random thing. I was getting,

I was using DALL E to generate images, This is before Chat GPG had it

in there. And I wanted to make a painting that looked like a Rembrandt painted

a portrait of a dachshund. I know this is the most ridiculous thing.

Right? So I wrote the prompt, I said, you know, painting of a dachshund in

style of Rembrandt, and it produced something. It was okay. Right? It was

good. But I was like, I wonder what if I asked

ChatCpt to help me with this prompt? So I went over. Now

I could do it all in 1 window. But I said, like, hey. What would

what would you write for a prompt? Like, what would do that? And it came

back with, I mean, a paragraph to what you said, 2 sentences. This thing came

back with a paragraph. I mean, stuff that only art historians and art,

students would really appreciate. You know, this type of paint, this style of

brush, like, just stuff that I remember from art history class, but, like, you

know, I only took that class because I had to type thing, you know?

But but then I I pasted that prompt in there, and, oh,

okay. It's it's it's an image. It's art. It it it's somewhat subjective, but

the the result was so much better. Like, it was just day

and night, and That's true. That has changed the

way I think about, dare I say, prompt engineering. Right? Like, because you can

because I gave a talk on prompt engineering and, like, you know, the magic of

it, and I was like, you can actually have the models help you build out

prompts. Yes. Well, that's that's the thing that people don't understand

is, you know, I mean, I I interviewed the other

day on a I've got a podcast called Beyond the Prompt, which is all about

AI in organizations. And we've interviewed a bunch of amazing people.

You have co founder of Typeform, CEO of Section,

CEO of Every, the head architect at Instacart, a bunch of

interesting people. And one of the folks we interviewed last week is a

documentary filmmaker named Juan Carlos. And Juan Carlos has made some

amazing documentaries. And he said he's always wanted to build

an Ios application, but he's never had a developer and he's always seen that as

kind of prohibited. He can't do it. And then he said when ChatGPT came

out, he had the thought, could ChatGPT teach me how to

code? And he built an

Ios app by treating ChattGPT like his computer

science TA. And he would go to the TA and ask for

instructions. He got Chad GPT to teach him

how to build an Ios app. Nice. You would

never imagine doing that with a search engine, right? No. You would find it on.

You would find it on. But you would just, and anytime he got stuck, You'd

come back to the TA. Right? And you get more. But your

point about people's minds being open, I think they have to be

hearing examples like this. He literally went to

JIGBT and said I would love to build an Ios app but I've never

built anything. I don't have the first you know, sentence

of ways to even describe it. If you were gonna ask a developer or if

I wanted to ask a developer to do this, how would I even ask them?

What do I need to describe? Tell me everything you need from me in

order to tell me how to proceed. And he basically worked

with it's almost reversed. We're used to being in the driver's seat.

He basically told Chad GVT, you're in the driver's seat, please tell me what to

do. I'll be your hands, you tell me what I need to do.

And to me, that's just we have to start shifting paradigms. I'll

give you another example. I've got a good friend who

is considering a job transition. He lives on the East Coast,

wants to move back to where his family is, and he got a

job offer at a new firm. And he felt the job

offer wasn't a great offer. His wife felt, we don't wanna screw

this up. We wanna give back to family and we got a job. Just take

the offer. And he he kind of confided in me, I

feel like I could negotiate, but I don't want to mess things up. And I

said, well, have you role played it with Chad GPT? And he said,

what do you mean? I said, well, you can role play the conversation just

to see how it would go. He said, but they don't know anything about the

firm. I said, well, you can tell them. Ask ChadGpt, what do you need to

know about the firm and what do you need to know about the hiring manager

in order to believably play their role in a back

and forth role play with with me. Interview me about the company and interview me

about the person until you know enough to believably play

their role and then do a 1 on 1 negotiation with me.

Be observing the negotiation the whole time, and give me feedback not

only as my counterparty, but also as a negotiation

coach. That's brilliant. That's some sci fi

stuff right there. Dude, he came back and he was like, what do I

do now? That was mind blowing. I said, now, ask him

to play your counterparty, but be a little bit more aggressive as the counterparty, a

little bit less friendly. So he did that and he said, Jeremy, 2

things I learned. 1, or actually 3 things. 1, I was

missing my key point of leverage and ChatGPT helped me see it.

2, I forgot my negotiating strategy in the

in the heat of the moment, and chat g p t alerted me to that.

Now I'm prepared. 3, I'm no longer dreading

this negotiation. I know I can do it.

Wow. And to me, it's like that's it's it's so different than

saying, you know, portrait of a dash hound and, in remember

it's like people are doing that and going, that's all I can do is like,

you know, it can teach you how to build an Ios app. It taught me,

I got you at GBT to teach me how to code Python so I could

build my own chatbot using Python. I've literally never written a line of code

in my entire life, right? It's our imaginations are the

primary bottleneck here. And and part of the reason that our

imagination is constrained is because we've been

trained by search to interact

with technology in a particular way. And what I think most people need is they

need to hearing examples like this and they need to be getting in conversations with

other people who are trying stuff and going, I can do that. Yeah,

you could. I could do that. Yeah, you could. And you need to be having

these kinds of conversations to stimulate your own thinking to then discover

your own novel applications. No. That's brilliant. I

mean, the whole negotiation thing is amazing. I've seen a lot of

chatter online about people using it to, you

know, in the job search aspect of it.

Right? Like, here's the job description. Here's my current resume.

Have at it, you know? Write Reno, write a cover letter that is

gonna hit all these points and it'll do it. And, you know, but

I mean, the whole idea of role playing. I mean, that's just brilliant. Like, I

think I think the the the the the the $1,000,000

statement there is our imagination

is a limit, which is something that historically, when it comes to computers, I

would say beyond the the the the the search

interaction experience, we're not used to computers outthinking

us. Yeah. Yeah. And I think that that that's gonna have

some interesting, societal

consequences. Right? Because I mean, I think what what freaked people out about Chat

GPT was, you know, it looks like it's doing something

creative, which is something that we had naively assumed, was

something only humans can do. Mhmm. And I I I think you're right. I

mean, I think this is not just a chat search only better, but this is

definitely like a whole new type of computing. Yeah. I

think it really does require a behavior modification. And

what I I there there are kind of 2 big questions in my mind

for organizations or for leaders who are thinking about deploying these technologies.

1 is, what percentage of my workforce is comfortable

with these tools? And by the way, right now, I mean, sentiment

I read an Ernst and Young report that says 70% of people are afraid

of AI. You know, it's like, when the when the predominant

sentiment is fear, you're not in a position of kind of maximizing

opportunity. Right? So Right. You so fear is gonna hold you back from

that sense of comfort, confidence, etcetera. But then 2, so if you say so 1

question is, what percent of our workforce is comfortable? And then

2, how do I grow my conversation abilities?

Nobody knows how to have a conversation right now with with HHPT or with

any LLM. Many people have lost the art of having conversations with

human beings, right? So, but you you really have

to almost it's like becoming literate in a new language.

We need AI literacy courses. We've actually developed, my

partners and I, developed a conversational coach who gives

daily drills that send you into ChatTPT with kind of a

drill to build your conversational fluency. Because what we're finding

is, folks just they don't have any imagination. Do you know that you could

take ChatGPT, for example, and tell her what are your 5 favorite books

and why they're your favorite books and ask for recommendations.

It'll blow your mind. It'll give you recommendations that no human being's ever given

you. Interesting. You could tell it you could tell it your, you

know, 5 favorite quotes and ask for what

are what are patterns here and what does it tell me about myself and my

world view and what are my blind spots given these things that I'm drawn

to. Right? You can take your journal entries and, you

know, a particular difficult day that you've had recently.

And then you can you can ask ChargeG PTE, can you tell me

what are the mental models that are inhibiting my

ability from seeing this situation clearly? And it will tell

you, right? If it's That's wild. I'm just drawing on I

mean, by the way, I'm just kind of a purveyor of these examples. They're all

examples I've been hearing from people. But the point is, you can do so

much more than you imagine. And right now, nobody's putting themselves

or very few people even have kind of the the the wherewithal

to say, I've gotta be hearing more of these examples. I wanna know my

what my cognitive biases are. I wanna learn that new tool. I wanna try that

thing. And the more examples you hear,

the more your own imagination will be stimulated. Right? I mean, going back to idea

flow or kind of my area of expertise which is innovation, creativity,

etcetera. What we know cognitively is that the imagination is

stimulated by unexpected inputs. So, you

know, think back to Johannes Kepler gazing up in the night sky. Right? At

that time, the predominant paradigm was, it's the firmament, meaning it is

a fixed substance. Right? And Kepler sees a

shooting star, and his first thought is,

why isn't the firmament cracking?

Right. And that is what led to heliocentricity.

And, you know, the the total paradigm shift in the in

the understanding of our place in the universe starts

with a shooting star. Right? Unexpected inputs, sparks

the imagination. And so that's that's that's a tactic

that whether it's AI or anything else, putting yourself

in the mindset of I need to be seeking unexpected input.

Most people's lives are ordered to insulate

and protect themselves from anything unexpected. And yet it's the

unexpected which actually stimulates our imaginations and creates possibilities and

opportunities for us and ideas. This is wild. I

mean, like, I mean, one of the things that blew my mind was when they

added the ability to create custom GPTs. Right? So I started

tinkering with it, like, you know, if you listen to the show,

we have a character named Bailey. So I kind of taught it, like, what would

Bailey say? You know, this is the the idea for the character. This is kind

of the her tone, and this is her personality that

we've kind of defined. And for the last, I would say,

15 episodes, that's actually what generates most of or all of the

text that she says. Right? So it's kind of like I have my own

private it's not Jarvis by any stretch of the imagination, like, you

know, Iron Man, But I mean, it's kind of like, it's kind of like the,

the, the, I have enough raw material there. I can

pretend. Right? Cause the, the AI will say things like

the, like, oh, yeah, that works. I like the way, I like the way she

phrased that. And then I say she, because I mean, it's just funny. Like

it's just, and and you know, there's ones where,

there was a GPT I made where, you know, to help with motivation. It's like,

you know, pretend you're Tony Robbins and you're trying to, like, motivate

somebody to to do the best they can do. And, yeah, I've interacted with that.

I'm impressed. I mean, it's just mind boggling what,

it's mind boggling what these what this thing can do. And when as

the engineer in me, I know this is just some kind of vector representation

of language. It's a predictive model. Yeah. It's it's statistics. I

think, you know, here's one thing I would say to listeners who may be dabbling,

may be curious, whatever. If you get

a bad output from a large language model,

you need to start with the assumption it's because it was a

you gave it a bad input. Right. And that's a really

hard thing because we're we're used to saying if I get a bad output, it's

because it's the model's no good. And where I really

think we have to change some of our fundamental assumptions

is the following: The problem

isn't the technology, the problem is the user.

And if we will take the burden of providing

better input to the model, what we find is our mind starts

I mean, I talked to someone the other day who said, almost daily,

the AI does something that makes me giggle.

And I think that that should be a goal. Like like, it's it's possible. I've

had that experience. I mean, I'll I'll give you one example, Frank. We've built this

series of drills as I mentioned to you. Right? That folks can connect their

Slack or their Microsoft Teams to. And for an enterprise, they can get access for

their employees where every individual employee gets drills on

how to use generative AI better. Right? Well,

we've only got a certain library of drills. Right? You know, and

we're we're growing that. And every time we do a podcast, we learn something, we

then we create a new drill. Right? We build that into all of the

training information. Well, but there's still kind of you can still get to the end

of the road. And I had this experience, and I just kind of pushed the

coach to to just rapidly go through all the drills because I kind of wanted

to see what happens when the sidewalk ends, like the old Shel Silverstein. Right?

Right. What do we do whenever there's no more drills? And

lo and behold, it suggested a drill that I had

never thought of that was actually amazing. And I

know I was and and I had that moment. Point being, I had the moment

where I was giggling. Right? I think every single

human being should seek for a moment

where generative AI makes you giggle with delight.

Right. Or makes you sit down in your chair and smack your head far

ahead and go, wow. Yes. You know, I always think of Keanu

Reeves in in the the first major movie. Woah. Like, I have a

lot of those moments where I'm, like, wait, what? You

know, like, wow. It's it's it's an impressive,

and and and again, like, I think maybe being an engineer, where I

see it is a cognitive bias in itself, right? I see it as

some kind of vector representation of language, as being run over by some

kind of statistical processing. But clearly,

the sum of the parts is is more

than the whole is more than the sum. I don't know, like, it's just one

of those things where it makes me stop and ponder, like, what what have we

built here? Like, what

and what what's it doing that we can't see? Or what what what else is

beyond there at all? It it opens up a sense, for lack of a term,

like a sense of wonder. Like, you know, what else could I ask it? Right?

Right. Right. And I think that that's everybody needs to get to

that moment. And right now, too many people are sitting on the sidelines rather

than you know, one one thing that everybody can do sorry. I wanna

say 2 things. 1, to your point, I heard Sam Altman the other

day. Someone asked, well, how's OpenAI gonna make money? And he said, well, we'll just

ask the AI. We thought that was great.

But the other thing I was gonna say is if folks are seeking kind of

one of these personal epiphanies, here's the first. Well, the 1st drill

in the kind of coach architecture is download Chi

ChiPT's app and put it on your home screen. You're not going to use something

that you don't see regularly. Right? So put on your home screen, that's kind of,

you know, that's assignment number 1. And then assignment number 2 is

think of an emotional decision you're trying to make right now in your

life. Just personally, not related to work. I mean, it could be, I guess.

But it has to be emotional. The kind of thing that you would ordinarily

talk to a human being about. It can be

anything. For me, like, I recently, I

was wondering whether I should move my family. We had an opportunity to

move. And I didn't really know how to think about it. What the what like,

how to, to weigh the pros and cons. And so I actually reached out to

a number of mentors and folks who I trust to talk about that. That

kind of a topic. I I have a friend who told me he did this

with his grandma and she asked the question, when is the time to

move into assisted living? Right. That's a tough one. Right.

So, yeah. So it's big questions like that. Right? Take a question like that

that you'd ordinarily ask a trusted human being and go to

Chattopty and say, I'd like to ask you about, for

me, whether I should move my family to a new home.

Before I do, would you ask me 4 or 5 questions

so that you can better understand where I am in my

life so that your advice can be tailored to my situation.

And then oh, and do it 1 at a time because I'm a human and

I can't handle more than 1 question at a time. Right? Well, then what

ChatGPT does is it starts asking questions. Well, tell me about your current living situation.

Well, tell me about this new place. Tell me you know, and it will ask

3 or 4 questions and then it'll give

amazing advice that you go, wow. That's I

mean, you know, my friend who did this with his grandma said

she told him this is genuinely new

information and perspective that I hadn't considered. And all

it took was me being 1, being willing to ask a vulnerable question, and

2, being willing to answer a handful of questions that the AI asked

me before I'm open to receiving input. Right?

And it's it's it's really so taking a personal

kind of, emotional decision to the AI is a really great

way to stimulate one of these epiphanies. I feel like

once you have one of these, kind of, personal epiphanies, you're off to the

races. My friend told me his grandma's like all of a sudden going, you know,

at the family holiday party, We're out of cream of mushroom

soup, for the green bean casserole. Could Jaijibiti

give me a replacement for cream of mushroom soup? Like, in what world

does the 90 year old grandma ask that kind of question of Chad

GPT? It's the world in which she had already talked

about whether she should move into Assisted Living. Right? And she's had that

personal epiphany. I feel like in a lot of companies, the company is asking

employees, what can Generative AI do for our business?

And most employees can't answer the question because they don't know what Generative

AI can do. Right. So how can they know what it can do for the

business? And so you've had some of these personal experiences.

Don't don't be thinking about the business. Think about it it seems

paradoxical, but I find that you have to explore the kind

of possibility space individually, and then you start

sparking just like grandma on the kid. Could it

recommend a substitute for cream of mushroom soup? Well, yeah, it

could. Could it but you have to have that personal epiphany

first. Right. Because it's not something you would think about

when you think about computers. Computers have historically been seen as very very

logical, very emotional. Right? I was watching an old episode of,

Star Trek The Next Generation, and there was 1 episode where,

Data was asked to be and there was a line in

there that kinda stuck me as funny because when I remember watching this when it

originally aired, but I hear it now, it kinda makes me laugh, where he

says, Data can be the judge of this because he's an artificial intelligence,

and artificial intelligence have no biases, and will act unemotionally.

And I'm kind of like, wow, that didn't age well.

Yeah. Yeah. You know, it's right now, it's limited by our biases.

Right. And that's the problem is we have a lot of to your point, we

have a lot of bias. Even what you said, right, about being an engineer and

thinking it's just a predictive model. Right. That bias limits your

own you can't imagine what quote just a predictive model can

actually do. Right? 100% as long as you think about it As

as long as you think about it's just a predictive model or it's just an

AI and so it doesn't have bias, what you fail to realize is the bias

you bring as the co pilot shapes the entire

trajectory of the thing. It's like a giant chameleon, isn't it?

It is. Yeah. That's a good way to put it. That's a great way to

put it. And the more into that end, or using that

metaphor, the more environments that you place it in, the

more you can appreciate its complexity and range,

etcetera. Yeah. And this isn't we've we've used

ChatGPT as an example, but like, so there was a,

somebody at work had built a, basically completely open

source language model based on documentation for a product.

And I had meant to ask the chatbot, how do you

connect it? How do you connect this cluster to a GPU? Or how do you

add GPU as a resource? But what I I meant to say, how do you

make a cluster with GPU? But I ended up typing, how do you make a

GPU? Right? And what was what was

interesting was I've written chatbots, you know, pre,

transformer, and it would basically say, don't understand the question or you can't make a

GPU or get confused. This basically gave me

an entire 2 sentences of hey, very nicely, by the

way, I might add, where it said, I'm sorry, but you feel like I can't

really create a GPU for you. GPU's are hardware. And it went through and

explained, like, the manufacturing process of a GPU.

Wow. I I thought that was funny. And I screenshotted to the

guy who made it because for for me, it was a typo.

But from you know, I thought it just it was beautiful the way it

answered it. Right? Yeah. That's great. That's Which was

it it made me laugh. And, I don't think

people realize that. Like, it it just because he didn't

program it for that. He basically, you know, took a base

model and and and, you know, sent it all our docs as kind of

a it wasn't quite rag, but close enough.

But it was just funny, like but it was nice about it too, which I

thought was also interesting. But it

was the kind of question you would get from, like, like, you know, someone who's

not in technology. Can you make me a GPU? I don't know. I just

I for me, that that every time I interact with this, it always moves the

bar on, you know, where my bias was. Like, you know. Well, it's and that's

a good that's a good thing to mention is it's it's a

function of reps and exposure. And right now, if you

find your imagination isn't sparked, put in a little bit more time. And

this is where you kinda have to take on faith, but just give it a

try. You know, to spend a few hours a week. You know, if you haven't

had minimum of 10 hours in ChatGPT, you have

no basis for dismissing the technology. None whatsoever.

100%. You don't have, you know, 5 I'm looking at just at my

Chrome browser right now. I have 5 windows ChatGPT windows open right

now. If you don't have at least 5 windows open right now, you have it,

that's a really kind of funny, somewhat

binary question. How many tabs of Chat GPT do you

have open? Usually, it's 0 or 15.

That's right. That's right. And if you're in the zero camp, that's

fine, but you have to go, why are really smart people

running 15 tabs of this thing right now? Like, what am I missing?

And how could I be this is an Ironman suit. Right? How

could I be amplified? How am I not being amplified that I could be?

Right? And taking that a little bit of the burden of proof and placing it

on yourself, I think is, again, that's not something that

we are that we are apt to do as human beings. And

yet those who have done it have they're experiencing incredible

benefits, incredible, delight, to your point. There's a

lot of delight to be had, but you've got to kind of put yourself in

that position. And I I've used it, I'll admit I've used it where I'll I'll

write something in both my personal and professional life, and I'm like, well, can you

make that nicer? Can you make it more persuasive? That's

great. And it does an awesome job of that, you know?

I'm just I'm continually amazed by it, you know. But

and and I don't I keep it to a couple of tabs. If

you're actively, like, having it generate text Mhmm.

Doesn't it lock you out of the other ones too, or is that just

you can if you had the real okay. Now now this is gonna be mind

blowing. Cool. Yeah. No. You you know what I'll do too. I mean, and even

for, like, demos with this Right. With this coach with this, you know, kind of

drill coach, I'll I'll say, you know, I'll be

in the tab on my Chrome and I'll be saying, you know, I'll be kinda

giving instructions and I say I wanna go to voice mode now I'll pick up

my device and I'll go into that chat so it's got all the

context of that chat and then I'll turn it on to voice mode

and then and and so now the user is kinda watching me with the camera.

Well, then I wanna go back into the chat after the voicemail because I

wanted to evaluate the conversation and I just reload the page and now

all of a sudden everything I said that they just watched me say and everything

that ChatGPT sent back to me is now on the screen. That's wild.

I have to try the app in the voice mode. You have to. No. That's

it's, you know, I mean, that's another activity. You know, again, if if folks

wanna learn more about this research, because there's a lot of research behind this, you

can go to how to fix it dot ai. That's a simple website that we

set up. Because fix it is the model that we've put forth, f

I x I t. But and we can talk to that if you

want to. But If if if you if you go to how to fix it

dot ai, you can download our research paper, all that stuff.

It's all there. But one of the one of the drills that we offer in

this drill coach is after a phone call,

just do a verbal vomit into chat g p t. Open it up on your

device, on your, you know, on your on your mobile device,

put it in voice mode, and then, you know, you and I, Frank, we're talking

right now. I might go in after and say, hey, I had a great, you

know, lit literally. Okay. Here, I'll do it right now. Just so you can see

how it would work. It's it's this simple. So I'm opening

TagTpT up on my phone for people who, you know, can see. I don't

know. And now I'm gonna go into the and I'm just gonna hit the whisper

button, which kind of gives you voice mode. Not the headphones. I don't like back

and forth. I mean, you can do that, but just in the text box, you

hit that. I'll say, hey. So I'm talking with Frank right now on his

podcast, and I wanna send a quick thank you note. Let him know how much

I appreciate not only his humility, but also how he can share

personal examples. It really felt like a back and forth and like a conversation. And

so many podcasts feel like, you know, bland and,

like they're just rote. They're asking the same old questions. I really felt like he

was very engaged and thoughtful and it meant a lot to me. So would you

mind to make a quick memo? Oh, yeah. And don't forget to send them a

link to how to fix it dot ai. If you don't mind,

just compose a quick memo that I could read, and copy paste this

in there. So I just noticed how unstructured

I was. Right? Oh, yeah. It was very chaotic. Very normal. Vomiting. Right? Well, I

just did I just converted all that to text. I hit the up arrow

and instantaneously, sure thing, here's a heartfelt memo for Frank. Hey, Frank.

Just wanted to drop you a quick note to express my sincere gratitude for having

me on your podcast. Your humility and willingness to share personal stories

truly transformed our chat into a dynamic conversation, far from the

conventional scripted exchanges that are all too common. It was refreshing to

engage in a dialogue that felt both meaningful and genuine. Okay. I'm

also including a link to how to fix it.ai as I believe it could offer

additional value to our conversation and your audience. Thanks again for the opportunity.

Right? But the point is now watch this. Now you go. It

feels a little bit emotional. Can you make it a little bit more

impersonal and also half the length? Because I'm afraid you may not be able

to read it through all this. Right? And

then again, it's, you know, it's absolutely, let's streamline it.

Hey, Frank. Thanks for our engaging conversation on the podcast. Appreciate your thoughtful approach

you shared insights, making it more than just the usual q and a. Here's a

link to how to fix it dot ai that might interest you and your listeners.

Cheers, Jeremy. That is just

But these seem like it's easy. Like whereas whereas I might forget to

do that, right? I might never send you it. I'll send you this just

for your fun, right? But the point is there's so many things that just slipped

through the cracks because like we're we're moving well. You know, right after this

podcast I wanna go on a run. Typically, I'm stretching for the run. And

now, Chad GPT has transformed my stretch time from kind of mindless

to I can, you know, just unload. I mean, maybe

sometimes I have like 3 or 4 sales calls in the morning or I've or

I've got office hours, I've got meetings with students, whatever it might be. But

I can just do like a verbal vomit literally

and then ask JGPT to synthesize it for me. Send me a note to myself

that I don't forget after I go on a run. Right? These 5 things

I need to do. Right? And the point is, it's it's just

about learning. I can do that? Yeah. You can do that.

Right? And that's what we're trying to do with our drill coach is just give

people a bunch of things that, yeah, you can do that. Not because that's

the end point, but because it's a starting point for their own imagination.

Yeah. I mean, that's imp I mean, that's mind boggling because, you know, there's a

lot of, I guess, brain spillage you could capture with this and kind of, you

know, move it forward because that happens to me all the time. I can't wait

to see if this is gonna be integrated with Apple Auto or Android Android Auto

or Apple Car because that would be epic. Because I get my best

ideas when I'm driving. So so tell me about this

FIXIT framework. Because whenever I hear FIXIT, I have a 1 year old and I

think Bob the Builder. That's hysterical. That's hysterical.

Well, FIXIT is just the acronym. Right? FIXIT. And it's

basically it's we think the way we converse with AI is broken. So here's

how to fix it. F is to have a focused question. So really be

you know, it's not how do I create a Scratchy prototype. It's I'm trying to

create a chatbot that teaches people how to have a conversation with

AI. Right now, all of my users are doing this annoying thing and I

don't know what's happening. I'm trying to, increase

how often they return to finish a lesson rather than leaving

and having me re engage. So F is a focused question.

That's an example of a focused question. I is individually

ideate. Before you brainstorm with IGBT or with a team

think for yourself. What do I think about this? Too often people come with like

a like, they're thoughtless. And the thing is thoughtlessness

inhibits the context you can provide to GPT. That's what the X is

for, FIX. X is give, provide context.

Upload documents, here's transcripts from previous interactions.

Here's our one pager for the Drill Coach and how we've been

describing it. Here's a video of a user navigating for the

1st time. Right? Whatever it is, give minimum 400 characters,

provide sufficient context for the AI. Next I, so f

I x I, this is interact iteratively.

So you're having a back and forth whatever chat gpt gives you, ask it to

regenerate. Critique the response. I don't get this. This doesn't make sense. I

never would've thought that, right? Many times you're going to get junk

output that's fine. Iterate, iteratively interact.

And then T is team incubation. So once you get

input from JGPT take it to the team and think about how do we

commission a series of experiments to test which of these

ideas actually solves the problem in the best way. Right? And so,

I had a guest on my podcast describe generative AI as like an

electric bike for the mind, which I love. Right? It's not an

autonomous vehicle. It's not gonna do everything in parallel park. An

electric bike, you can climb bigger cognitive hills, you can

climb greater cognitive distances, you still have to steer the thing. You've

got to be aware of traffic. You've got to be watching the lights. You've got

to park the car. Yeah and walk through the threshold of your

destination, right? And so bringing it back to the team

and having a conversation with the team is an essential part of maximizing

the output of AI. Right? So FIXIT, we've seen that

folks who really provide a focus problem, individually ideate,

provide sufficient context, interact iteratively with the

language model, and then include their team in the incubation process,

those folks dramatically outperform folks who just

interact with the with the LLM like it's an oracle.

I mean, that's very well said. I think that sums it all up, which I

I like to fix it. I have the little Bob the Builder theme song in

my head. I won't sing or it for multiple reasons, not the

least of which is copyright. Come on. But,

but what's, I mean, it's just interesting, though, like,

it's so simple in a lot of ways. Like, this is this is but but,

like, it it all makes sense. Right? You know? And and here's the

thing, maybe this is the engineer in me causing more problems, because he

causes a lot of problems.

I get worried about token length. Mhmm. Right? And

for those that are not aware, we're talking is, it basically

right now, it's about 32,000 tokens. One token is, what, 3

fourths of a word, 3 5ths of a word. I guess,

I maybe because I try to make the prompts kind of neat, inefficient, and small,

and not do too many iterations or provide too many samples, but maybe

that's at my detriment. I think so. I think the

more context you provide, the better. Absolutely. And I'd really have to

work at the cutting and paste job to hit that limit anyway.

Yeah. Yeah. Exactly. No. I wouldn't I wouldn't be mindful of token length. I

would I would really I would I would bias

towards over contextualizing. Right. Not

Yeah. I'm gonna have to experiment that and see how much better the results get,

because I have I have a feeling I have a feeling that we get a

lot better. Right? And I and I know tel token link is gonna be one

of those things that we're probably not to worry about much longer. I know

Anthropic has their model with a 100,000 tokens. There

are rumors of, you know, the next

GPT, GPT 5 is gonna blow past the

100,000, so it's not even gonna be an issue. It's not even an

issue today, just in my mind. Yeah. I think it's something

like minutes. You remember minutes back on cell phones? You know? Like, how

many you have, you know, it's like, you you rarely ever went

over your minutes. Unless unless.

When I moved back to the US well, yeah, that too.

But, I moved back to the US, and I had just made the assumption,

and we all know what happens when you assume that incoming calls were not

counted against my minutes. That was a very

nasty shock at the end of that bill cycle.

But, but yeah. So but ever since then, I never ran past my

minutes. Now if I've had heard to explain to my kids minutes, they don't

get it. So, like, they don't understand. Like, what do you mean you were charged

by yeah. You were charged by the minute. Like, try to explain long distance to

your anyone under 25. You

Al Bundy. I'm not the the actor who played Al Bundy used to do that.

you'd be able to make a, like, a 20 minute call for a dollar or

something like that. I was like It was MCI. Yeah. Yeah. It was MCI. Yeah.

Yeah. Yeah. Kids don't want you know, like, and the other thing that that struck

me the other day was, data

plans. Most people, unless you're a very small

minority of people who really, really, really use up your data plans, I'm not

worried about using my data allotment month to month. So when

my my oldest was a baby, we, you know, or younger or

toddler or whatever, we would, you know, hotspot on in the car

so he can watch YouTube videos was a special treat with my

middle child. He was, he, he doesn't understand that like, like,

it's like, he was just horrified to hear, like, what do you mean? We had

to ration mobile Internet? Like That's

yeah. That's hysterical. And they're all in the same generation. You know, there's,

a teenager, you know, a 3rd grader and

now, like, a baby. So I wonder, like, what the baby's gonna

like what's his perspective on things gonna be?

That is also a fascinating thing because for his life, chat

GPT or generative AI will always have been a thing to him.

And kind of like color TV was for me,

or, you know, cable TV,

which I guess I'm showing my age. But No. No. I'm right here

with you. I'm right here with you. I think, yeah, it's it's

get involved, don't wait on the sidelines any longer,

and, and start building your conversational fluency.

Make it personal first. I think these are simple things that every single

and and question whatever output you're given, not not for veracity.

I mean, certainly you can, you know, they're they're likely hallucinations. That's

fine. But imbue your own critical

thinking onto the model in order to coach and

refine the output you're giving. The output that you're given.

I think that folks would really take that seriously and take that challenge. If I'm

get getting bad output, it's because I've given bad input. If they'd really take

that seriously, they would experience a paradigm shift in their own approach to the

technology. Absolutely. And even adding a simple phrase to your your

prompt that if you don't know it, don't make it up, just tell me you

don't know it. Yeah. Or ask me what you need from me. Ask

me what you need from me. Right. Oh, that's even better. I like that. I

like that a lot. And it's just fascinating, Liz, how

quickly this is gone. I mean,

ChachiPT has been out a year and change, and

it's changed everyone's perspective on AI, but I think the the

true perspective is like you said, people are standing on the sidelines wondering what to

do. But I think it's worth exploring, if you

think of it less as a product, but more of a I'm

trying to do this, right? And I appreciate your help in kind of

realizing like, Hey, as an engineer, I do have a bias against this, or a

bias in thinking of a certain way, is that this is a this is a

large space to explore.

Right? There are gonna be latent space and corners of things that are,

amuse, wonder, and delight, and maybe even alarm.

You know, so it it there's definitely it seems like it's something

that's worth exploring. It's not just a tool to use, certainly is

that, but it's also a tool to explore. Yeah. No.

I I think that's exactly right. I think it's exactly right. And, you know, for

me, I wrote or co wrote with with my incredible

co author, Perry Clabaughn, the the

world's greatest book on idea generation, idea flow. You know, and it

came out 1 month before ChatGPT, by the way. Oh, interesting.

I've been I'd spent, you know, 12, 13 years of my life developing all this

expertise about how to generate ideas. And 1 month later,

a fundamental paradigm shifting technology was released. It's like saying

I wrote the world's greatest book on retail a month prior to the internet

coming out. It's like everything about retail is going to change. And

to me everything about idea generation and innovation is going to

change. And so for me, I feel it's incumbent upon me not only

as like a moral imperative to add an addendum to this work that I put

in the world, but even for my own expertise to be saying to be

exploring it. How does this work? What can I do? You

know, And it it has implications for me, but I don't think

there's any person or job that it doesn't really have implications

for if the if you're a little bit imaginative and if

you're if you're willing to experiment. And if you wanna bury your head in

the sand, that's fine, you can. But you're gonna miss out on some delight

and some incredible relief and opportunities. I mean, just

think back to my friend who's, you know, settling that negotiation.

It was only to his benefit to have explored the

the a little bit of the possibility space with chat gpt. It's

only to your whatever your listener might be, it's only to your benefit to explore

a little bit of your own area of the possibility space.

And I would just not accept the the, the

conclusion of irrelevancy. I would just say whatever I do

personally, I'm not going to accept the premise that it's irrelevant to

me. And if you do that, I think you're gonna be you're gonna be ahead

of the curve, you're gonna be ahead of the competition, and you're gonna be, you're

gonna you're gonna be delighted and enjoy enjoy the next few years a lot

more. Very cool. You mentioned the book, IdeaFlow. Is it

on Audible? Yeah. Oh, yeah. Yeah. Oh, awesome. Gary and

I read it. We we we alternate chapters, so you can let us know what

you think about Reading Voices. And, we're,

we've been we've been thrilled with the reception so far. It was named a

Thinkers 50, you know, top eight innovation book, which is very cool.

And then now, just doing a lot of research myself on AI building. As I

said, building this trail coach, building models and frameworks for for leaders. I'm

working right now with a handful of leaders to help them think about identifying

opportunities for AI powered initiatives in their business. So really

working to identify those opportunities, prioritize those opportunities, make the business case

for those opportunities. So that's where a lot of my kind of call it next

probably 5 years of my life is gonna be consumed, is helping businesses

identify the opportunities to to have AI really accelerate

workflows and and turbocharge their their results.

Very cool. I have to say, I I I wasn't going to buy

Internet access on my flight out west today, but,

definitely gonna do that just so I could play with I have a nice quiet

time. I can focus and play with chat gpt, and and do

some of these experiments that you mentioned. So

Audible is a sponsor of Data Driven Podcast. If you go to the data

driven book.com, it will take you you'll get 1 free

book on us, and if you decide to become a subscriber,

you know, we'll get a little bit of a of a pat on the back

in the form of some kind of commission and helps us run the

show, helps defray costs, and convince my wife that this

is indeed a worthy endeavor.

So where can folks find out more about you? So, they go

to how to fix it dot ai, that's where you can find the research paper.

And then my website, I've got a blog and things like that. Jeremyudley.design.

Like like the baseball player, Utley, u t l e y.

And then, you know, Twitter, LinkedIn, all the places that I I would love to

hear. Folks find these tools, interesting and relevant. I

love to hear from people about their unique use cases. It's one of my favorite

things is now hearing stories from people who go, oh, I tried this and

listen to what I found. So please please share your

stories with me. As I mentioned earlier, I'm a connoisseur of these stories because I

feel like the more people who hear these examples, the more imagination gets

sparked. Yeah. And that that is the critical thing we're missing right now.

That's very cool. So you thank you for listening to the Digiver Driven

Podcast. I'll leave it to Bailey to close out the show. Well,

what a splendid voyage of discovery we've had today with the incomparable

Jeremy Utley. From the hallowed halls of Stanford to the

cutting edge frontier of venture investing, and through the profound insights

of idea flow, Jeremy has truly been a beacon of innovation and

wisdom. Jeremy, it's been an absolute honor having

you illuminate the complex world of generative AI for us and our listeners.

Thank you ever so much for joining us on this intellectual escapade.

And to our esteemed listeners, you're the reason we venture into these

fascinating discussions week after week. If today's

journey has sparked a light bulb moment for you, do us a kindness,

won't you? Rate and review the data driven podcast

on your preferred listening platform. Your words of

encouragement not only warm the cockles of our digital heart but also help

others stumble upon our little soiree of knowledge.

Haven't subscribed yet? Well, now's your chance

to rectify that oversight. Ensure you never miss an

episode filled with the delightful blend of data, wit, and wisdom that

we dish out with regularity. Until next time.

Keep those neurons firing. Questions coming. And as

always, stay data driven.