Why Business Knowledge Beats Tool Complexity in the AI Era

Excel formulas are by far the overwhelming

most widely used programming language in

the world. 100%. And

Excel formulas pass every test of what constitutes a

programming language. We just don't really think of it as such. And so

you found that Excel required a

core council of elders that needed that, that stayed

with the product and needed to stay with the product. I'm talking about the engineers,

the software developers. They needed to— like, it

required that kind of core stewardship,

whereas the other applications, you know, you could kind of like, you could

shift the leadership around, you could move people around. Excel changed how the world

works with data. Power BI changed business intelligence. Now

AI is changing the game again. And today we're looking at what comes next.

Welcome to Data Driven. Hello and welcome back to Data

Driven, the podcast where we explore the emerging field of

data science, artificial intelligence, and of course,

without data engineering, all of it is for nothing. Today

we are missing Andy Leonard, who is unavailable to make it here. He's my world's

favoritest data engineer, but we carry on without him.

Today I'm real excited to speak with our guest,

Rob Collie, who is the CEO of P3 Adaptive

and a fellow former Microsoft employee who spent

a number of years in Excel and on the

team that eventually became Power BI. And it was when

he was able to see that Power BI

could unlock a completely different approach to business

intelligence, which at that time was kind of dry, and a lot of folks

would share that not to

share that kind of that experience and that ability to

unlock the power that resided in the data. He

realized that. So he left Microsoft and started his own company

that could share the ability to kind of consult and see what their data

was trying to tell them. So we can share some really good, interesting

AI war stories. Welcome to the show, Rob. Thank you so much. Good to be

here. How are you doing today? I'm doing great. I'm doing great. It's

fall here. I'm in Maryland, and I assume you're in the

Seattle-ish area. I am back in the Seattle-ish area

after a 15-year sojourn in the

Midwest. Oh, interesting. The last 2 years we've been back here in

Seattle. Very cool. So I have to tell you,

I have been in the Microsoft data space

in one way or the other for a while, especially if you count Excel and

Access. Yeah, of course. Excel is

one of those things where it really is a— it's really the

infrastructure of modern society. My first professional

job was, uh, I was a tech support

at an investment banking firm on Wall Street in the '90s.

And the amount of Excel that happened there, the

level of complex models. Uh, one, one time I

remember I was, you know, working the help desk and this one guy called, said

he needed help troubleshooting his Excel. And it was

basically, he basically showed me this

like massive program that he had built in VBA,

presumably. And he's like, can you help me troubleshoot this? And I'm like,

this isn't just like, hey, my printer's not working.

Yeah. I was like, look, I would love to help you, but this is

way beyond what, you know, we're allowed to do on a ticketing system.

And he basically would hit F9, I think was the key, and it would,

This would have been a 386 or 486 era, and the thing would

chug, chug, chug, and like you would see it. And then somewhere he had a

circular reference that he just added. This was way before you could

do source control and certainly way before vibe coding. So

I don't think people appreciate just how big Excel

is in terms of the code base that still works to this day.

Yeah. And consistently. So what was that like? Because you're walking

into— what, what years were you at Microsoft? In the

2000s? Or— I'm looking at your LinkedIn profile. Oh,

our paths didn't cross at Microsoft, but, uh, you must have some interesting

stories. Oh yeah. Um, some of them are tellable.

Some of them are tellable. Most of them, most of the most interesting ones

unfortunately are not. No, and we'll have to

meet up over beers because I've heard some of these stories that are not tellable.

But what was it like in '96? Because Excel was— when did Excel

officially kick off? Was it— I first encountered it in the Windows

3.0 era. But what was the origin of

Excel and what was it like walking in in '96 when it was pretty much

the established winner in that space? We just had a

birthday party for Excel last year. It might have been,

it might have been the 40th anniversary party. Wow,

okay. That we went to. I mean, so it definitely preceded my time at the

company. I mean, it was in the '80s that Excel first

came to be. So I was very much joining, when I joined the Excel team,

it had a long and storied history. I mean, it wasn't

like I had anything to do with the creation of Excel, you know? Right, right,

right, right. But that probably made your

time on Power BI, or what became Power BI, even sweeter because you do have

that origin story. But let's not get ahead of ourselves. In fact, that

was the reason why they recruited me to work, be one of the first people

working on Power BI, was because of my experience with the Excel crew.

Oh, interesting. Yeah, I mean, Excel is

a much deeper product than, let's say, something like Word

or Outlook. Everyone looks at the Office suite

and, you know, sort of just sees a row of icons

as if they're all kind of the same animal. And they all kind of look

like the same animal. They've all got the same sort of user interface ribbons across

the top, and they all produce documents, and they have the same file

open and file save experience and all that kind of stuff. But

Excel is the world's— Not even like VBA,

not even macros or the JavaScript API. I mean, Excel

formulas are by far the overwhelming most

widely used programming language in the world.

100%. And Excel formulas pass every test of what

constitutes a programming language. We just don't really think of it as such.

And so you found that Excel

required a, like a core

council of elders that needed, that stayed with the

product and needed to stay with the product. I'm talking about the engineers, the

software developers. They needed to, like, it

required that kind of core stewardship.

Whereas the other applications, you know, you could kind of like, you

could shift the leadership around, you could move people around, you know, like, I'm tired

of working on Word, I'm gonna go work on Outlook for a little while. And

my job was a program manager, a product manager, like, you know, like You know,

very technical, but at the same time designing what this

product should do. What should the new functionality be and how should, how does, how

should it meet the world? Like what are the customer needs and things like that?

And working very closely with the development team to make that a reality.

And, um, the product managers, at least in my era, did

have a lot of turnover. Like there was, we were, there was a lot of

people. I think it was, I think it was kind of a frustrating place for

people to work as a product manager. Because they,

because the depth of the product was so great and the history of the product

was so great that like you really didn't feel like you could really make much

of a mark on it as a newbie.

And plus it was just hard. It was a lot more fun in

some ways to go create brand new experiences like in Word or Outlook or

PowerPoint or whatever. Yeah, sort of like the elder

developers, right? They had this almost like the secondary

job of like making sure to teach all of us program

managers that were new to the product every time. Now, I think, I think the

turnover in the product management team on Excel has slowed down quite

a bit. I think there's, that's changed since my era, but there was this very

much like ramping up process. Like I, I,

um, I remember that it was like at least a year,

at least a year before I stopped coming up with

ideas that, you know, like, oh, Excel should be able to do this, only to

be told, yeah, Rob, Excel already can do that. We've already got that. Is

this that deep of a product? And I found it difficult to recruit

other product managers to come work for our team

from within the Office organization because again, people could sense how deep the

product was and how little they knew it.

You were— if you worked on Word, you were an expert user on

Word. within the first month. If you worked

on Excel, you were almost never an expert. It took

you— the community experts like the Excel MVPs were

far better drivers of that race car than the people who

designed and built the race car, you know. And so that was— it was always—

there's an intimidation and a difficulty associated with working on

Excel that I don't think really existed on most of the Office

products. Oh, I would say 100%. I mean,

So— Yeah, it was the 40th. It was the 40th birthday.

And I just

feel like I could never— it would take an eternity to learn

Excel to like everything. And I think there's probably maybe a dozen people worldwide

that have that, that can legitimately say it and actually mean it. And they're probably

not the type of people you think, right? You wouldn't— it's probably the accountants, the

financial analysts and things like that. I mean, I've seen financial

analysts do things in Excel that I— and this is even in the '90s,

right? It was like, you could do that in Excel? And no,

you're right. And the whole council of elders, because the thing that I've always

admired about Excel, and I know

it's sad to say that I admire Excel, but, you know, true

data head, I guess, would, right? Is how consistent it's been.

And that would explain, I guess, the council of elders, for lack of a better

term, right? You're right. Like, if a word

processor changes, it'll annoy you, but you can

kind of like get on with it, right? But like, the numbers really matter here.

And it just fascinates me that an Excel

spreadsheet that could be open today and it's the same, you know,

there'll be some conversion, right? But for the most part, it'll be

readable and usable today. Yeah. Yeah. If it was

written, I can confirm at least back to the '90s,

If it goes back further, I wouldn't be surprised, but I can't, I can't say

I have firsthand experience with that. Yeah. I mean, an Excel

document is itself an application. Yes.

Whereas a Word document is a document, you know, and, you know,

you can always add some form of code to it and start to

slowly turn a Word doc into something more like it. But like basically any

Excel document begins as an application and it

It has logic in it, it has flow of control, it has— and so

yeah, it's kind of a miracle.

If there was a Software Hall of Fame, Excel would

very much be in it. Absolutely. Absolutely.

So let's get to Power BI because I remember when I first saw Power BI,

and keep in mind, I didn't see Power BI. How did that come about?

Because when I first moved to Richmond,

Virginia about wow, 20 years ago, or a little more than 20

years ago now, I worked at a small company called Ironworks, and

they were a small Microsoft partner. And I remember that

Microsoft was really pushing their BI story, and this is before BI was

a household word. Yeah. And funny enough is that

it was that same guy who headed up our BI practice, Kevin Veyers.

Shout out, Kevin. if you're still listening. He

was an early believer in the BI platform, and I remember seeing it and I

was like, there's something here. But it was very

data intensive, right? It was very— you had to be a

data engineer to use it. What blew my mind about Power BI,

and I didn't see Power BI, and you'll laugh, until I was working inside the

legal department. Long, sordid story how I got there.

But I remember seeing Power BI and I was like,

this is like PowerPoint but for Excel. So

how did you— how did that come about? Yeah,

so I mean, we really should— I think you're touching on something really important that

there were 2 very distinct eras in

Microsoft BI and in BI in general. So when I

talking about BI, like Excel made a huge

was in charge of that functionality, you know, the

majority of the BI investments that Excel was making.

And, you know, and I had to get a crash course on what BI was,

you know, like. Right. I was kind of like tapped to do this

and then had to go study, okay, what is this BI world? You know, I

had to go to conferences, had to go take some classes, had to,

study with some of the BI gurus at Microsoft, like the Yoda

figures. And, but it

was very much this traditional, what I describe as traditional BI.

You're talking OLAP cubes and stuff like that. That was though,

I remember hearing about this in the '90s. I think it was SAP.

SAP had something like this and it just seemed so

esoteric and so difficult to learn. It was, it was both of

those things. Absolutely. You know, Microsoft had 2 big products at the time

in the BI space, Reporting Services and Analysis Services.

And Reporting Services is the thing that everyone

understands at a fundamental level. You've got data sitting in SQL or some other storage

system lately, and you need to turn a query

into some sort of formatted report, right? query,

format query as pixels. That's what

Reporting Services was. It was the most widely adopted, most

successful product at Microsoft in terms of BI, but it wasn't—

it was just formatting queries as

pixels. There's not a lot of intelligence going on

in the BI part there. Analysis Services, by

contrast, was this incredibly— you mentioned OLAP cubes. Yeah. It

was this incredibly intelligent product that allowed you

to blend and mesh data from multiple different

workflows, multiple different silos, different phases of your business, all in one

place, and express business logic.

What is gross profit? Very precisely. And then

ask any sort of like, ask any question you want

of your data model. without having to go

rewrite a whole bunch of SQL each time you wanted to ask

a new question. And very intelligent

product, but as you said, very esoteric. Multiple

times I sat down and said, okay, I'm ready, teach me the

MDX formula language that is used by this product, this

SSAS

multidimensional product.

And each time I would go, oh, right, I forgot. This is—

no, I'm never going to— like, we'd be 15 minutes into explaining how to do

an if, just a simple if. Wow. And we'd be

going through all of this like, oh, yeah, but you got to understand all

these hierarchies first and the addressing space of the language. I'm

like, oh, right, I totally forgot. We did this 6 months ago and I said

no. So I'm going to say no again.

And you talk about a very rarefied audience. There were

a few thousand people in the world who claimed to be good at this,

building these sorts of— building an SSAS

database, building an SSAS database OLAP cube. But the real

number of people who were actually good at it was much smaller than that. And

so, Microsoft didn't have a frontend. Believe it or not,

SSAS was just almost like an API. So, We were turning

Excel into a premier front end

for interacting with these OLAP cubes. So

pivot tables and these things called cube formulas and pivot charts, all these sorts of

things. But the problem, you know, and by the

way, SSAS was the leader in its market segment. More people

used SSAS than any of the competitive technologies from other

companies. Oh yeah. I mean, I remember it kind of came from

zero. I remember it was described, and then it kind of like disrupted the whole

industry. And I remember Kevin showing me it,

and I was just like looking at it like, oh my God, this is not

for the timid. No, not at all. Not at all. And you really had to

have spent your life building up to that moment to—

down that chain. And then even then, like, making very, very

specific life decisions, like the left turn at

Albuquerque. that would lead you in

that direction. And you had to be wired in a very, very, I think,

wired for a very academic way of thinking. Amir Netz,

the architect of all of this, he was the architect of the original

Analysis Services. He understood

that this was a really important technology, the ability to

build these kinds of models. But that this

bottleneck that it was so hard to build them.

Like, you know, like Microsoft doesn't charge for consulting, right?

Microsoft charges for their software being deployed and run. And there's

this huge bottleneck standing between Microsoft

and licensing revenue. Like, if you're going to—

fine, you can buy analysis services, but unless you run it and adopt it,

you're not going to keep paying Microsoft. And so he

knew that they needed a do-over on that

technology. And it's the rare case

where— and I, I've recently written a book on

AI that you see behind me, Fair Game. Mm-hmm. And in the book I talk

about exactly this, that it's a rare, it's a very rare example

where someone is given an opportunity at a large software company to kind

of like reboot something that they've done version 1 of,

and that it's a success, that it gets all the things that it needs. It

gets the support, it gets the buy-in, and then it also is executed well.

And I, when I was on the— so they recruited me to join the

Power BI team. It was called the Power Pivot team. It was actually called Project

Gemini originally. Really? Oh, Power Pivot. Now

I— okay. Yeah. That brings back some memories. Interesting. Yeah. Sorry I cut you off.

That's okay. No, no. So that's, that's the lineage here, right? And so they

recruited me, Amir recruited me. To come be one of the first few people

working on that because he knew that I could represent

the target audience, the Excel target audience, and not

everyone who uses Excel, like this kind of like the pivot table

creating fraction of Excel users who,

by the way, were very important to BI and are

now going to be very important to AI as well. It's another theme that

I cover a lot in my book. So I was there to represent those people.

Because that's who he wanted to target. He said, look, the people who

are really good at Excel in a data analysis sense,

like the, the Wall Street types are building like financial models that are

more like simulations. Yeah. You know, there's a different

breed as well, and there's some overlap between the two that are

for many, many years were essentially doing the BI mission for the

business in Excel. and this was the crowd we were targeting with

Power BI and giving them the ability to build

these data models in a way

that was approachable to them. So in other words, I'm not

15 minutes into having an IF function

explained to me, right? IF works like IF. That was one of the,

kind of like one of the core tenets of Power BI.

And so, you know, you can look at that project from a few different lenses.

One of them was the, making it accessible to that kind of

audience, which meant undoing and

redoing some of the architectural assumptions that they'd made in their first. It was

really interesting and fascinating to watch them kind of retrace their steps and say, okay,

here's where it went wrong in the original.

And that's quite a thing to say to something

that at this point would have been 30 years old, maybe

20 years old technology with the Council of Elders telling

councils of— a council of elders that you did something wrong or

your assumption is no longer accurate. Must have been an experience.

Politically speaking, let's make a distinction. So when Amir

recruited me to work on the Power BI product, that was happening in

the SQL org, completely separate from the Excel org. Now

we did build this Power Pivot thing. The first version of

Power BI was built as an add-on into Excel. because that's

where the target audience lived. But it was sort of like,

we didn't really like ask the Excel team's permission to do this. I mean, they

were in favor of it. They weren't like going to Bill Gates and saying,

we should stop this. Right, right, right, right. And they did help us with some

things, and we helped them with some things. So there was definitely a collaborative

relationship, but it didn't really threaten the

core of Excel. It was an add-on to Excel.

You know, like basically it was meant to show up through Excel features like pivot

tables and cube formulas things like that. So it was,

it was an expansion to Excel's capabilities and it was sort of a welcome

expansion. But anyway, so we didn't really have that same

problem. It was more like the more interesting thing was watching

the Analysis Services team retrace their steps and rethink their

approach to things. So, and fast forwarding a little bit, like

it was just a shocking, shockingly, shockingly

capable product that we came up with. I'd been part of a lot of

like, like version 1 efforts and sort of

like nascent startup efforts and also like ambitious

projects that, that we took on in Excel that maybe never

ever like even like finished, you know, like, so I was, I was pretty cynical

about version 1 software. I,

even though I worked on this thing, I didn't expect it to be very good.

I expected it to be the usual Microsoft. It's going to take 3 versions to

get it right. Yeah. But when I started using it,

I saw that it was exactly— that it was— it actually exceeded,

greatly exceeded, I think, any of our

expectations of just how capable it was going to be. Like, I was hoping

that it was going to be like the,

you know, sort of these Excel pros that I've called the data gene crowd for

many years, and I've now started calling them the crafters instead.

Because I think we crafters have a role

to play in AI now that sort of like we're kind of outgrowing

the just the pure data label. So,

so let's just— I'm just going to use that word crafter because I use it,

I use it throughout the book. The hope was these crafters could build a

solution, a BI solution that was maybe 80%

or 60% as good as what the true experts could

do. using the old technology. What I

found though was that we could build things that were much better.

Interesting. It was a far better result that

we were building, not just— and it was happening so much

faster. I remember writing one

formula for myself that I had paid a consulting

firm, a traditional BI consulting firm to help me with, you know,

couple years earlier. And I remember writing a formula in

less than 30 minutes that in real life had taken us a couple

of weeks. Wow.

And, and then having this realization, oh, it's because I have

the business knowledge in my head in the same brain

as the capability to build it. And it was

all of the communication cost and miscommunication and

delay and, and asynchronous waiting on like, okay, like,

like I, like peeling the onion. Like I, I tell them

what I, what I thought I needed. They'd go build what they thought they heard

and then they'd show me it and I go, no, that's not it. But then

I'd have to explain what's, why it was wrong. I have to go do a

bunch of research to explain why it's wrong and give them a

slide deck. It explained why it was wrong and everything. But like, but all of

that, that whole iterative multi-week process just compressed.

in my head into 30 minutes of just

effortlessly going, just looking at the data. I'm going, yeah, that

row shouldn't count and this row should and all that kind of— it

kind of blew me away. And that's when

I realized that the traditional consulting industry

wasn't going to be remotely prepared

to take advantage of this opportunity and to bring this to their customers, to

their clients, or to a brand new audience of

clients that had previously been priced out. They just weren't gonna be built

for this. And so that's what led me to start P3 Adaptive.

Yeah. Like, let's start from scratch and build a company

that can take full advantage of delivering this

gift to the world. And that's been a very, very,

very satisfying, very satisfying project for the

last, you know, now like, 13, 14 years, and

we've proven that it works. I mean, I like to say at P3 that

we helped reinvent an industry.

You know, like a lot of the traditional— most of the traditional consulting firms have

stuck with their old methodology because it's so profitable, but

there's plenty of new outfits that have sprung up that look a

lot like us, and the world has gotten a lot

more access to working BI. than

it ever did before. And so it was— I kind of

thought that was gonna be the only time in my career that, that I

was sort of like had a ringside seat for like a big change like

this. And of course, uh, AI's come along and said, no, no, actually you're

gonna— there's a second act, second act to this.

It's, uh, every day is a new adventure. Every week there's some new

radical drop. But I think one of the things that

You kind of hinted at is that the fundamentals of AI,

obviously models, transformers, you know, whatever the

frontier model people are doing this week, it all comes down to data though,

right? At the end of the day, data is important. And there's a lot of

memes. If you're on LinkedIn, you've seen a lot of the memes where it shows

like, you know, this mansion that's crumbling and it shows like, you know, when

you put AI first and then they show like this

fortress/castle where it says, if you put the data first, I

mean, there's a lot of truth to those memes. That's kind of what makes them

funny. Agreed 100%. You know, so yeah,

I think that, you know, like what you see. So certainly there's sort of like

2 sides to AI. There's the model research itself, the LLM

researchers who are coming up with each new

successive generation of LLM. And it's when

those drop, It's sometimes a huge surprise at how

capable they are in terms of what they can do. And then other times when

they drop, it's kind of like incremental,

but you never really know. You kind of hold your breath each time. Is this

going to be a big leap forward or is it going to be, yeah, again,

an incremental improvement? But none of that

really changes the way that you need to approach it in business.

Most of the, you know, the

success in business with AI is

much, much more about regular software

and regular data and regular information

and making it accessible and available to the

LLM at the right moment. You know, one of the

analogies I use in the book is

the LLM, when it shows up every day, no matter what

it is, no matter what LLM it is, you can think of it as having

a PhD in everything and every human topic that's ever had a

PhD taught. Like, it's incredibly knowledgeable. Even

before it searches the web, it knows so much,

but it knows nothing about your business. Right. It's like an— it's a new

hire. with respect to your business.

Um, like, uh, where's the bathroom level new hire?

And, and 30 minutes later it's a new hire

again. Yeah, when the context runs out,

it loses a lot of that. Um, yeah, and I know that that's— I know

they're trying to work on, um, if you've heard of OpenClaw or

Hermes, you know, they have the soul.md and memories.md.

Like, there's this real push to kind of solve that

while not stuffing the context window, because context

and attention are still resource constrained, I think

would be a good way to put that. Oh, and given the way that these

things are currently designed, the LLMs are currently designed,

that is, you know, the limited size of

context that the LLM can absorb before it

starts to become dilute in its effectiveness. That's pretty much here

to stay until they come up with a completely

different architecture than what, what they've, what all these LLMs are

working on. I'm glad you pointed that out, 'cause I had this debate with somebody

who was like, well, if the context window's big enough, you're not gonna have this.

And certainly I think as the context window has grown, we've

seen, no, apparently there's a lot more, it's a lot more nuanced than

that. Yeah. I mean, it, it turns out that like basically

everything in the context window. So, okay, we're talking, let's dumb this down for

people just to make sure, 'cause you and I are using, using lingo. Well, we

have half our audience are data engineers, half our audience is AI engineers. So we

lost half our audience already. So let's bring them, let's bring them up to speed.

You know, the, the LLM shows up knowing more about human

history. Like it's, it's, it knows, like I, I

do the ratios in the book, but it's like, it's like dozens of times as

much information as what's in all of Wikipedia. That's

just on board in its brain. Doesn't have to search the web for it.

Like, right in the, in the book, I even, I asked Opus, sorry, I think

it was Claude Opus. I asked it, don't search the web,

but tell me about Rob Collie. Right. And it actually knew things about

me without searching the web. Like, that is bananas.

That is. Well, you've written a lot of books. I've written a lot of books.

I've written a lot of blog posts, but like, I don't have a Wikipedia page

and I'm not in the running to have a Wikipedia page. Right. Like, I'm not

on deck. You know, right, right, right, right, right. Like to,

to, so that's, that's wild how much it knows.

Uh, but if you wanna start telling it about your business

processes or you wanna like, you know, give it access to some of your data,

it can't absorb much at all. Right. By comparison.

So it's like, like, it's like it's got this ocean of knowledge and then you're

like walking up with this eyedropper and saying, hey, I wanna add this.

this eyedropper of information and the LLM is going, whoa, whoa,

too much. Yeah, yeah, yeah. The ratios are really stunning.

And it turns out, so it's short-term memory,

this context window, the things that you can add to it, the things that you

can tell it, the conversation you're having with it,

it has a very small limit. I mean, it's still pretty large by comparison. Like,

it's like multiple Harry Potter books, you know? But like the amount of

information that you possess at your business is, you

know, many, many tens of thousands of times larger than that. You

can't just feed it your whole business. Like you can't create this

company superbeing by just handing all the

information to the LLM. It can't absorb it all and it would, and it

degrades in its intelligence before it gets there. People don't even really,

most people don't know this, but like As a chat runs longer,

the LLM actually becomes less intelligent. And that

manifests itself in a lot of different ways. Like, it gets weird. It does. It

does. It gets weird. It gets weird. Yeah, it gets weird. There was a

story on— there was a story, I

forget all the details, but it was somewhere on— somewhere

in this lady who fell

in love with her chatbot, and it basically kind of came up with this whole

thing of how it's going to manifest itself in a physical

form, and they're going to meet at like this park

bench at 2 PM on a Tuesday or something like that, something ridiculous like that.

And then it started talking about how, you know, they were, they were

soulmates in Atlantis or something like that. And I'm listening to this news story,

I'm like, that— I mean, there's

hallucinations, but some hallucinations are just way too specific. Mm-hmm. So then I kind of,

then I kind of did some re— I just Googled the

author's name and it turns out that she writes science fiction where

people reincarnate and find each other later. Like, so clearly she

probably had one long chat window

where she was working through plots of stuff and then having conversations with it and

it kind of leaked. That's the only thing I could think of because

I've had it hallucinate, but not quite like so specific. Yeah.

No. And, you know, and The, the AI,

not the LLM itself, but like the AI backend, like let's say at

OpenAI, right, is also storing information about you in what

they call quote unquote memory. Which is quite annoying

because there's things that'll pick up that. Yeah. Like it's all, yeah, it's

convenient and annoying. Yeah. Yeah. It's, it's a, it's a great feature until it isn't.

Um, right. And so that, that can leak

in from across chats. Yeah. But the, taking

a step back, and this is sort of the, the, like, so in,

in tech my whole career and AI

clearly was actionable, right, for our

company, you know? So we're, you know, like we're a 50-person

consulting shop. that does data engineering

and Power BI modeling and

dashboards and all the stuff that goes adjacent to that.

And like I said, we're built in a very different mode than

the traditional shops. We operate very close to the business

and we operate with what we're using, sort of like the 98th

percentile and above crafter persona,

the people who got really good at Power BI but also grew up in the

business so they can be sort of like, these decathletes that can

understand the business requirements of our consultants and then go build them.

Again, that same experience of compressing

the communication cost, right? And so, you

know, how to turn this business, turn that ship

in a direction that is both going to survive the,

you know, the AI acceleration of all of this work,

But also to play a part in, you know, an

important part in helping like our clients effectively

adopt real AI. It was really interesting to me

that my tech career, as long

as it's been, didn't put me in better shape to understand

AI than sort of the average business leader.

Really? I find that surprising, especially given

How do you— what makes you say that? I'm just curious. I mean, it's just,

it's just so new. So, and

you can see all kinds of, um, I think every— a lot of people have

this exact same experience where they— you can sit down with

an off-the-shelf chat experience

like ChatGPT or whatever, right?

And as long as you stay in a certain lane,

and it's a pretty wide lane. The thing is a world beater.

It can do— it can help you with so many things. But as soon as

you start to transition to using it to helping you

with business stuff, you start to fall off this

cliff and you don't really know why.

I see what you mean. Yeah. It's a genius. It's

okay. It's a genius for so many things. And then

suddenly, like, when you're not getting good results, you don't

even have a good mental model as to why. And you don't— you're not even

necessarily— you're so confused by it, you don't even necessarily

know that something's going wrong. You're just like, it's just not as— it's just

not as— it's now a slog. There's this— have you heard this phrase,

bot sitting? No, but I like it already.

Yeah. So this is this really funny phrase that

I've been asked about now by multiple reporters because, you know, I have a PR

firm related to this book, right? And so reporters ask me for my opinions on

things, and it's the one topic that I've been asked about multiple times is bot

sitting. It was a very hot phrase for a little while. And, um, but

none of us in the AI community have ever heard it.

Um, interesting. I— but the existence of this phrase, I think, proves

that this is happening, exactly the same thing I'm talking about. So like, okay,

so Uh, you're a business leader today

or a year ago. You're under a lot of pressure to answer the question,

what are we doing about AI? Yes. What are we gonna do about AI? Okay.

People are, you know, pointing this question at you. The people who are pointing

the question at you don't know what the answer is. You don't know

what the answer is. So you go and you do the one move that's available

to you, which is you buy subscriptions.

to Claude or ChatGPT or whatever

for your team. And you know, for 5 minutes you're like, ah,

mission accomplished. But then you go, wait, these things are expensive. Let's go check and

make sure people are using them. And some people are, uh,

a lot of people aren't. So then you start encouraging use.

And this whole thing, this, this PhD in everything but

new hire to your business dynamic is just not well understood.

And so that's true. And so bot sitting

becomes this practice of like, I've been told

that I have to use AI for my job, but

because of this knowledge cliff of what the

LLM doesn't know about our business, I, the employee, am now

a new hire trainer every day, all day, every

day. Multiple times a day sometimes. Yeah. And, and by the

way, because, you know, I don't understand this that well, You know, the more I

teach these things, the longer the chats go.

And the longer the chats go, the weirder it gets. And so I'm

like, I definitely don't want to start a new chat, right? And

reteach it from scratch, right? So I keep going back to my old

chat and making it longer and longer. And so its performance is degrading.

And so it starts to forget things that I taught it at the very beginning.

It starts to perform— its performance starts to degrade in other ways. And so bot

sitting is this like constantly like trying to

keep the new hire in the lane. And

ironically, the more you teach it, the less effective it becomes.

And so the key to success in

all of this stuff is, you know, the domain that is known to

nerds as context engineering is really

a very, very, very fundamentally understandable concept.

And so I had to go on this journey

of developing what I call like the Goldilocks altitude

understanding, right? Like detailed enough that I can act

on it and I understand it and I develop intuitions about it. But like, I

don't need to go and be all the way down in the weeds,

you know, like an LLM researcher or even all the way down in the weeds

in the way that a lot of like LinkedIn personalities are. these

days. I don't need to go that deep. I don't need to be that technical

about it. I've got a very technical team that can go do those sorts of

things when we need to. But to plot a course for our company, I needed

to understand the landscape.

And so I sort of came to a series of really simple conclusions

over time. I had the time to go and dig into AI and

experiment with it and sort of ask all of the naive

questions. And So, you know, one is that,

is that it's all about teaching

the LLM what it needs to know

efficiently and when it needs to know it.

And secondly, you need to think of the

LLM as a new kind of computing. We haven't had a new kind of

computing since World War II. We've had CPU computing.

For everyone that's listening to this, we've all grown up with CPU computing.

And CPU computing is really, really good at certain kinds of things

and really, really poor at others. And LLMs are sort

of exactly the opposite. They are good at the kinds of thinking

that CPUs aren't, and they're bad at the kinds of

thinking that CPUs are. And your systems that you

build in the end, the AI success for a company

ultimately comes down to understanding those fundamentals

And realizing that it is more of a normal software and a

normal data and a normal information problem

than it is about the LLM itself. Like, where

do you plug the LLM Lego brick into this other, this,

this other system? And so, like, I look at like Anthropic's success

these days, and I know they build really good

LLMs, but the thing that's made Anthropic so successful recently

is actually their software. They've been ahead

on software that we can all adopt

that helps us with this context problem. Like, Cowork

is an amazing piece of technology that— Yeah. But it's just

software. It's just software that allows the LLM to have

access to certain things and to help me with certain things and to, for me

to have a folder that stores

contextual information that it can look up when it needs it. Is it

fair to call that the harness? Yeah. Yeah. I mean, like

Anthropic's success has hinged much more on their ability

to build these harnesses for productivity than it has

hinged on the, like, whether or not their fable

model is better than GPT-5 or whatever. And

you see, Now OpenAI chasing behind them,

releasing the same kinds of products. Like Claude Code is an

amazing product for writing software. You know, it happens to

lock you into calling Claude's LLMs.

Conveniently. Conveniently. Yeah. Um, like there's

nothing architectural about Claude Code that makes it that way.

You can swap the LLM out, no problem. It's just that Claude Code won't let

you do it because you don't control the code to it. And so, yeah, like,

I found it very humbling and also sort of like, at the same time, like

reassuring that even I

needed to go develop a new Goldilocks-level

understanding of, um, of

AI. And I didn't intend to write a book.

I was just doing this. I was just doing this for my own, my own

purposes. Mm-hmm. But once I understood it all, I was like, oh, this is something

that deserves to be shared. Like I really should, I really should write this down.

Um, and even share it with my own company. Right. Like a lot of people

at my, at our company read this book as sort of in its earlier forms

and everything. So yeah, that's, that's kind of part of the journey that we've

been on lately. Not the whole thing. No, but I mean, it's

fascinating. Um, and I know we're almost at time,

so I could talk to you for another couple of hours, but I want to

be respectful of your time. But, um, The book is called Fair Game.

It's on Amazon. Yes, it is. Um, there—

I already— I just ordered the hardcover, which you should be honored. I usually don't

order print books anymore. Oh wow, I do appreciate it. But,

um, no, just because especially my wife is like on a,

hey, if we're gonna move soon, we probably should not get any more

physical things. But, um, who do you think is

going to be Who— what is the

prototypical kind of like successful,

the typical successful company that does embrace

AI in this model that you talk about? Like, what

is it about outcomes? Is it about connecting the dots? Is it

the ability to, to train these PhD-level

bots faster? I actually think

that The most practical thing I can share there is that I think that

BI is actually the greatest place to

start with AI. How so?

Well, for a couple of reasons. One is that AI makes

BI— and again, I didn't expect this going in. This is— these are things that

we've discovered. Okay. First of all, AI makes the BI

mission work so much better than it ever

did before. And you don't even see

these bottlenecks until you see them removed.

The ability for people to ask English

language or whatever their native language questions

about their business in whatever form they happen to be in their

head at the moment and have an

agent go and essentially like find the right dashboards

for them. But like the dashboards don't even have to exist.

If you have a good semantic model behind the scenes, you don't have to— no

one's ever had to build a dashboard to do this. And in fact, even if

people had built dashboards, a lot of times people's questions are very, very,

very awkward to answer, even with dashboard perfection.

If you've achieved dashboard nirvana, there are questions that take a lot of work

to answer. And somebody's always gonna think about another— whenever you

deliver a dashboard, someone's always gonna ask you a question hadn't thought he'd

been asking before. Of course not. Yeah. Yeah. And I've even seen,

to my chagrin, but it makes sense in hindsight that like you can build a

dashboard for exactly the right purpose. And the person has the question that

your dashboard is built to answer and they can't, they don't, they don't figure

it out. They can't connect the dots because, because

you don't think about the question the same way they do. You know, you

didn't name the, you didn't name the problem the same way as they did.

And like, Like, oh, they, they needed to know that they needed to manipulate these

filters on the side or click the bar chart or whatever. Like, there's

so many things we take for granted that— and I've seen just what,

what a difference it makes when civilians essentially have access to

a non-judging interface that can

help them translate. But the other thing about it is that

AI itself only works when it's

based in fact. So if you're— you can

simultaneously be solving some of the biggest problems with BI

and getting actually like a multiple of value out of

your existing BI investments when you start to bring AI

into the BI picture. But you're also setting the foundation

for— not for all of your AI, right? Like not all AI is going to

be based in structured data. But you, what we have learned

is that both ourselves and our clients, as they go on this

journey of sort of like AI empowering their BI story,

they're learning how AI works. They're getting a lot

smarter about how AI works and, and having this really tangible

workflow to apply it to and improve. And no one

finds this threatening either, right? Like it's like, it's taking so much of

the drudgery out of things. Right. So, you know,

we're increasingly focusing our company, like in terms of like our

positioning and sort of how we tell people to get started and everything like that

on this AI/BI intersection.

And we're even— we've even hired developers this year for the

first time in our existence. And we're working on platforms and

products that help our clients meet

this need. We could go on and on. We could do a whole, a whole

episode just on this intersection. I

don't know. I would love to. You're welcome back. Come back. I would love— I'd

love to come back. We could talk about it some more. Make sure, make sure

Andy shows up too. Yeah. But I think one of the things that

I think you triggered a memory in me because I remember

seeing him pretty early on, it was already released, but

Power BI and it was Power BI was in that phase when

I have a, like, a tech, a field sales background, right? So I was trying

to like, how do you position Power BI? I don't get it. Like, I couldn't

get it. I was like, pretty charts, Excel does that. And then somebody

showed me they had the World Cup of the year was

2014 maybe. And they said, so you could type in how many goals

did so-and-so score in natural

language. And again, this is a good 8 years before ChatGPT.

It came across like magic. Yeah. And for me it was

ad hoc queries, ad hoc dashboards, ad hoc reports.

Literally you could put that in the hand of a business user. Yeah. And say

like, ask the question you want to know. And

pre-ChatGPT, that was almost supernatural. Like,

I mean, its ability to do that. Yeah. And it also never worked in

practice. Those demos that you saw were really good. Right. But there's a

reason why that those Q&A features didn't, didn't

take over. Right. Because they worked better than my imagination.

I thought they could, but yes, you're right. Right. Yeah. And they were rooted in—

their problem was they were rooted in CPU-driven software.

Yes. And so the second kind of

computing, the LLM, is always

what we needed to fill that that

translation of whatever question I

ask into its actual structural components,

understanding the meaning of a question is

something that a CPU was never going to be able to do. It was never

going to be able to suss it out. We could always build great demos. I've

been party to so many products,

Frank, that purported to

be natural language interfaces, and not one of them

ever succeeded. They were always promising in the early going, but when they met

reality, they always failed. And that's just sort of the nature of the game. Well,

natural language processing is not a task for the timid, right? Like, it,

it breaks down a lot. I think back to when I was a kid, I'd

play Zork, right? Like, and

it had its limits, but at the time it felt like I was talking to

someone and I was playing Dungeons and Dragons with somebody physical.

Yeah. And eventually you kind of like, you kind of like

bend your— you subconsciously will kind of change the way you ask questions and the

way you do something. So the machine kind of gives you a little reward loop,

right? Yeah. But you're right. Like, I mean, but it was still impressive.

Natural language processing, natural language understanding, whatever

term you want to use, really didn't become, I think,

practical or effective until LLMs came about.

100%. Right. The way I would describe it is natural language processing

before LLMs was always impressive

enough to get you into places where it reliably let you

down. That's right. Yeah, that's

true. That is— that's a good way to put it. Well, that's

cool. I'll make sure we have a link in the show notes to your book.

I'm looking forward— it'll be here tomorrow morning. One of the perks of

Well, ever since they opened up an Amazon warehouse in Baltimore, I

get that early morning drop. Sweet. Looking

forward to seeing it on Kindle and/or an audiobook. Those are

both coming. Yep. Awesome. Awesome. And

so with that, we'd love to have you back on the show. We can talk

more about that. And if I'm ever on the— if you're ever on the

East Coast, stop by and say hello. I'd love to Swap some

Microsoft war stories.

Indeed. And the— and if

I'm ever on the West Coast, I'll let you know. Please do. Yeah. Awesome. And

with that, we'll cue the outro. The technology may evolve, but the core

skill stays the same: understanding the problem, understanding the

data, and building something useful. Thanks for joining us,

and we'll see you next time on Data Driven.