Steve Orrin on the Importance of Hardware in AI Development

On this episode of data driven Frank and Andy interview

stephen Oren, the CTO of Intel Federal

yes. Intel, the computer chip company. Because if you want

to train your AI models in a reasonable amount of time, you need better

hardware. Well, it turns out that intel has developed new

CPU instructions to accelerate AI workloads

FPGAs allow for faster development in custom

applications with specific needs. Speaking of intel,

you have to check out an upcoming intel and Red Hat webinar

link in the show notes. Tell them Bailey sent you.

Now on with the show.

Hello and welcome to Data Driven, the podcast where we explore the emergent fields

of data science, data engineering, and of course,

artificial intelligence. As with me, I always have Andy

Leonard, my most favorite data engineer in the world.

And today we have a special guest, Steve Oren, who is the federal

CTO of intel. Yes, that's right, intel, the chip

company. And although they do a lot more stuff

now. So welcome to the show, Steve.

Thank you and glad to be here. Frank and Andy cool.

So one of the things that I think people have not realized, people

think that AI is a software story, right?

Primarily. But quickly, once you get into it,

everyone goes gaga for things like Chat GPT or

well, no one's really gone gaga for Barred just yet. We're going to give that

a few more time for the paint to dry on that.

But quickly, I think when people start

becoming builders of AI tools, the

number one restriction, aside from kind of what your data engineering pipeline looks

like, is how quick you can train these models. And

obviously, I'm pretty sure intel has a thing or two to say about

hardware. Absolutely. And as you've as you've

alluded to AI, and all the things that make up

AI rely heavily on the infrastructure that you're

training you're inferencing. But even before you get to the fun stuff, how do you

do the data curation? How do you suck in the data? The ingestion get the

large multi node data sets that these large language models are

trained against. There's a lot of hardware and infrastructure that has to make

that happen. And then when you get to the important phase with how do

you train those in a timely fashion, hardware is

the answer. And what we're seeing in a lot of these spaces,

especially we start looking at things like large language models and transformers

as well as looking at other approaches that are coming out,

is that not only does the hardware matter, but the type of hardware

matters. If you think about it, it's not a one size

fits all. It's a heterogeneous architecture to make sure you have the right

hardware for your workload. One great example. So

large language models in graph analytics requires not just

heavy duty hardware but the right memory architecture to keep those nodes

in place while you're training. And what you find is that often

doesn't fit well. Intel just a classic GPU only kind of mode,

which is what the classic AIS leveraged, just the sheer number

of cores that you would have in a GPU. And so what we're seeing is

optimizing the hardware for the kind of workload is the answer to getting

timely training. And especially when you start doing more. That sort of iterative. And

feedback training, it's not a one and done, it's an ongoing process. So you need

that to be quick enough and powerful enough and robust enough to handle those

workloads. And then the other side where hardware really starts to matter is on the

inferencing, you want to be able to ask the question and get a response fairly

quickly, if not near real time. If you're in a car and it's

autonomous driving, you want it real time. You want to know that's a tree and

not a shadow. If you're talking about online and doing some

fun stuff with chat GBT, you still don't want to wait 20 minutes for your

response. And so inferencing matters, training matters, and

the kind of hardware and infrastructure that support it. And that's why intel

and our ecosystem are looking at providing a

heterogeneous set of architectures. So our classic CPU, so the Xeon and

the server and CPU and the client core, but also FPGA

based logic AI accelerators like our Habana chips in

the cloud and our targeted edge AI

chips like Movidius for video processing and the like. But then

really, besides the hardware, it's that software infrastructure layer. How do you

optimize your code? Because most AI developers are not hardware

experts, nor do I want them necessarily to be. So a lot of it is

about building out those abstraction layers that optimize your code, that's

doing your hugging face or whatever, to take full advantage of the

hardware underneath you, without you having to know what hardware is underneath you so

that you can provision your workload where it needs to go and not have to

worry about the hardware infrastructure. And that's part of our overall strategy. And working

with the broader ecosystem, the open source community, the

commercial providers, and the software frameworks to give them the

tools to get the best performance out of their AI and their

data science, right? And I think you hit the nail on the head. I

think we're at an inflection point. Not so much in engineering,

right, but more in the perception, right? Because whenever you think, oh,

we have a large workload we got to do, let's throw some GPU at

it, right? And it's a little more nuanced than that. I think

people are finding out that you need more than just a

bunch of GPU. And I was on a call

and I want to get your thoughts on this, because he said something very similar

to what you said. You ever have these moments

when you're on a call and somebody smart says something, you're like, I don't know

about that, right? And it's kind of like what they did

in World War Z and where there was like the 10th Man Rule, where no

matter how ridiculous it sounds at first, you kind of want to

investigate it. And that's why I was glad when your

name popped up in the feed because I'm like, yeah, I want to talk to

you about this. Because he was basically saying that

GPU usage is

overrated and that where the real advantage is going to be

is going to be in software acceleration and on

CPU kind of optimization too, which sounds

a lot like what you said. And when I first heard that, my first thought

was, I don't know about that, but this guy's plugged in. He's a

big shot at Red Hat, right? He's plugged in, he knows a lot. And I

was like, I didn't want to just dismiss that. Like, if my cousin said that,

I'd be like, yeah, okay, but if this guy says it,

whether or not he's right, maybe yet to be determined,

but the fact that he believes it means that there's a trail there to follow.

So I've been kind of poking around at stuff. Tell me

about that. It sounds like there's some weight

behind that opinion. So Frankie, you hit it on

the head there. It's not that GPUs aren't important, it's just GPUs

aren't the only and best solution for all aspects of AI. And there are

certain vendors that want, again, for a variety of reasons, want GPU to be the

foundation for all of your AI activities. Like if you're a GPU based

hardware company. Exactly makes sense. But

when you actually go look at the benchmarks across multiple and here's the key thing,

across multiple AI types. So different

algorithmic models as well as the flow, so there's different stages. So the

inference versus training, ingestion and curation

versus the training, versus the feedback training, what you'll find is

that GPUs will rock for certain things and they are important for certain things,

both from that vendor as well as from a variety of other vendors. GPUs do

play a key role, but when you look at the breadth of AI activities

and the benchmarks associated, you actually find that a lot of really

good work just happens on standard commercial off the shelf CPU. And

actually most of the inferencing, I mean, we're talking in the 70% to 80%

of inferencing happens best on CPU

and areas like large language model and graph analytic based

approaches. The numbers really show very

clearly that it's not a core bound problem,

it's a memory bound problem. And so having efficient in

and out of memory, which is what you get from a CPU or an accelerator

with ample memory on board, is actually much more powerful

for training those types of data sets because the GPU you're dealing with that

latency across the bus. And that actually starts to matter when you're

talking about billions or trillions of node graph

analytics. So I wouldn't say that GPUs

are a dying breed. That is absolutely not the case. And there's going to be

a huge market for GPUs or GPU like

functionality. I want to be careful about that because you don't have to have a

discrete card. The reality is you can have GPU capabilities embedded in your

processor. We've already seen from intel and from other

architectures. The real interesting thing is making sure that

whatever your workload is can be optimized, like your friend said, optimized

through software to that hardware. So that if you are

running a large language model, that you're actually

running it on the right hardware, and that the hardware and your software know how

to work together to give you the best performance if you're working

on. I'm seeing a lot of really cool things right now around graph based

approaches in the memory intensive side of that

and the switching back and forth between that. Those

latencies can really come to bear when you're talking about cross bus

kind of communication. So having high amount of memory available directly to

the CPU to be able to do those training, keep all that data in flight

so you can train, is going to be one of the key

differentiators of how you can take those large angle models, apply them to

more than just writing cool essays by Shakespeare.

I think what we're going to see is things like chat, GPT, and that whole

category of transformer based approaches applied to just about everything, not

just chat, but prediction

approaches. And it's really about getting it the training sets to become

smart on those very vertical domains.

That's going to be a resource intensive process and it's not going to be throwing

a bunch of GPU or it's going to be a lot of cloud scaling and

it's going to be a lot of memory intensive activities. And like your friend

highlighted, the software is going to really matter, that it's taking full advantage

of the hardware to get you those performance report. Well, this reminds me a

lot of just patterns I've seen over the decades of

being in computing as a hobbyist and then a profession

is you see a lot of things come into the

fore as being very monolithic, and then people

realize, wait, that's really a team effort.

And I think about it as a baseball team, right? You don't want to put

the pitcher, the person who's skilled at pitching in center field, can they

perform there? Well, gosh, yeah, but you're wasting them,

right? They are tuned their whole body, their

desires, their motivations. They love being pitchers.

So put that person on the pitchers mound and you see this

happen. And it's in all sorts of places. We saw it, frank and I have

seen it over the years when the unicorns were the big

deal, the data science unicorns who could do data engineering and everything

that we've kind of broken out now into other fields.

And we're seeing it now in the hardware

and in the distribution of the separation of

concerns and the distribution of concerns, getting every component to do

what it's best at. And along with that, and I'll shut up after

this, is this whole idea that it's moving so

fast that the hardware that's going to perform

the task first sometimes isn't even identified

yet because some new approach popped into the equation. If

somebody tested something and went, this is great. Now whether I run it

and you just see that and it's on a scale now where it

used to be measured in years and moved to months, it's now weeks

and sometimes days. It's just amazing how fast this

is going. And not that long ago, people were predicting

an AI intel. Right.

I think Dolly kind of and the whole generative artwork

stuff, I think kind of like, wait a minute, there's something here. Then Dolly came

out and then OpenAI did the one two punch of here's

Dolly a couple of months later, here's Chachi BT. Now you're just seeing like

it's on fire. Like it's not just AI summer, it's an AI heat

wave. Yeah, exactly. It is. It's a full El

Nino. I like that. That's the

quotable, for sure.

I think one of the things I think people realized is,

and a lot of the thinking was that AI

winter was coming because we're hitting processor or

hardware kind of upper barriers. And I think we're

finding out, I think much to what you said is that it's not just about

throw this many GPUs at it. It's right. The entire story, the entire

bus matters. Right. So the shortstop matters using the

baseball analogy. Right. The outfielders. Right. You can't really win

a lot of baseball games if not everybody on the team is

playing at their best. Absolutely. And just to take that metaphor

all the way, the turf matters, too. The infrastructure that you're running

those specialists on, you're going to play better in different

fields. That's true. That's a good point.

I love that you took the metaphor to the next level. That's awesome.

I think you mentioned whether it was in the virtual green room or here something

called habanero. And I know you're not talking about just

cooking. Right. Spicy habana. Yes, habana. I'm

sorry. I had food on my mind, as is

often. What is habana? Because I've

heard whispers of it. I know we're recording this middle of

May. There's going to be some announcements at the Red Hat Summit. Well, they'll

probably already happen by the time this goes live, but what is

it? So Havana is an architecture, an AI

accelerator, and it's a specialty chips specifically

designed for accelerating AI. And it's actually two

chips. And the reason it's two chips is that you want, again, going back

to what we were talking about, you want the right hardware for the AI workload.

So you want to be able to have the right hardware to opt optimized for

training flows and a separate set of hardware

for cloud scale and hyperscale inferencing

workloads. And so that's actually what Habana is. It's a two

chip strategy. So habana gowdy which is out available.

V two is available. V One has been out for some time. If

you go to the Amazon cloud, you can get it today. It's also available in

data centers, and a lot of universities have them in their high performance computing

environments. And it's geared to doing that sort of scale,

large data set training that you would find

whether it be in a cloud kind of environment, a chat GPT level

of analytic, or in the case of high performance computing.

Whether you're doing climate modeling or flow dynamics, those kind of big

training model sets that you want to be able to do at scale. And

what's nice about it is that like your cloud scale, it scales with your architecture.

So it allows you to be able to scale up your training based on

the compute needs with an AI accelerator specifically tuned to

that. The other chip, the Goya chip, is an inferencing

chip. So it's again tuned for that inference. But the reason,

again, this is for high end cloud scale hyperscale or things like high

speed training, where you want to be able to do large amount of inference in

as near or close to real time as possible against really

complex kind of data flows that you're trying to do

the analysis of. And again, looking at the right

hardware, we wanted to make sure to not just meet what we call the sort

of the normal scale. So the kind of things you would interact with when

you're going to do fraud detection, but you also want to be able to handle

really large scale inferencing because you're dealing with ingestion of multi data

sets across multiple different domains and having to be able to do that

inferencing in a streaming kind of mode. And that's really where the Goya chip

shines, is an inferencing platform that can scale

with the cloud. And that's really the Habana strategy is about giving you the

hyperscalers and high performance computing, the equivalent of

an AI custom chips. And that's really where Habana

sits. And then when you look at sort of the majority of what most

people will leverage in a cloud or on prem, what we've been

doing there is adding new instructions to the CPU. So

VNNI was the first really big one in AVX 512,

which really accelerates the math that you're doing behind

inferencing and training and give you those

instructions. That software, whether it be Intel's OpenVINO software

or TensorFlow or other frameworks can take advantage of

that math to use hardware offload to accelerate the math that you're

doing in your training and your inferencing workloads for most of your normal

kind of AI. A lot of the AI we deal with, not the high performance

computing style. And so you get the balance. And again, it goes back to what

we talked about in the beginning, the right compute for the right AI. We've also

introduced data center graphics because again, there are workloads that absolutely

make sense for a GPU besides fun gaming. And

that's really where you'll see GPU shine on, those kind of specialty

workloads that take full advantage. And a lot of the deep learning object

recognition ones work well on GPUs. They actually work well on other

kind of platforms as well. And one of the things we're seeing in the Edge

is a shift towards more customized approaches, whether that be using

an FPGA as sort of a hardware platform that you can code

in your algorithms to do inline inferencing, do feedback loop

training. And you see this a lot of times in the image processing, video

processing side, also in the signals processing. So whether it's five

G and being able to do signal quality testing or signal acquisition

and being able to do RF signal analysis, FPGAs

actually really shine for that kind of workload. Where you want to put in your

custom algorithm that you're going to actually test against or

use as part of your conditioning. And then we get to the idea

of what we call an ASIC. And that's where you know your workload, you

know you're going to be doing this kind of inference. You can actually code that

into a custom chip that will do just

audio AI inferencing or

do certain aspects of video coded. And this way you get the most

performance in a low swap. And that's the idea here

is you want to be able to handle everything from the pointy end of the

spear, the Edge sensor and give it the ability to do AI as

opposed to waiting for it to send the data to the cloud and get a

decision. You want to be able to give it something, but it also has to

operate at the size, weight and power that

you'd expect from an Edge sensor. You obviously don't have a data center power

system for your car, for your drone, or for

your camera on the streetlight. Right. That would be a very heavy to

fly that drone. That's okay.

I'm curious how you kind of manage what

I'm just going to make up words here, but like an innovation chain,

I'm thinking about like supply chain management. And I know

I've got experience in electronics engineering, and I

know some of how much it takes to go into mind you my

work was decades old, but this whole idea of getting

ahead of the curve or at least being able to predict where the

curve is going and how steep and when. That

sounds like a huge challenge for figuring out what

will be needed next. So what you're talking

about is how does a company that's building out both the hardware and the infrastructure,

stay ahead of, like you said, the week to week turnaround

in the AI world. Part of that is having a diverse team

of specialists. So the Intel Labs,

people who full time looking at process node technology,

security, AI data science. They're across multiple domains

and within each domain we have specialists in different areas.

One of the really I'll give you a great example. Before Chat GPT blew up,

I had two different of my AI specialists, one on the

government side and one on the performance side. Start talking to me about this thing

called Transformer. Like, oh, there's this really cool thing that we're seeing here, it's called

a Transformer. And I'm like, okay, that's interesting, and tell me more. And they explain

sort of how it worked. And then fast forward, six months later,

Chat chips BT shows up and I'm like, I know what that is because that

has the word Transformer. I've seen this. And again, it's about giving

your people the ability to go out and look. I think one of the

advantages of being at intel, and it's really why I've been here so long,

is everyone knows intel inside.

But there's something to that. Our chips are inside the

edge. Clients are inside the financial services, healthcare,

manufacturing, oil and gas. They're in the government system, they're in the cloud,

we're in the network. Which means we see workloads both current

and coming from all those different domains. So in some

respects we're on the cutting edge because we're seeing what people do because they come

to us, say, hey, I've got this software, I want to optimize on your hardware.

What does it do? Well, it does blah, blah blah blah. I'm like, okay, let's

help you. And then eventually that becomes open AI.

That's the kind of thing because ultimately every startup, every big company

wants to get the most out of their software and our teams. And one of

the things people don't realize is intel has over 19,000 software engineers

and a large majority of those do you know, they really divide up into three

areas sort of research and pathfinding, ecosystem

enabling, and then software development for

compilers, software services, software tools. That ecosystem enabling team

is a very robust team, it's been around for a very long time. Whose job

is to make Microsoft Windows rock on intel, make Oracle

rock on intel, make red hat rock on intel, make open source. We have

to open source. We're actually one of the largest committers to open source

community and a lot of what they do is build the optimized

version of those Linux kernel libraries or to

that AI model running on intel and give it away and open source

it. We've created whole model zoos optimized for the variety of intel

architecture because we know if you can run it best on intel, you will run

it, and that consumes resources. We like that. But ultimately

it gives us they call them bell cows, if you will.

We're seeing those bell cows of what's coming next because they come to us and

they say, hey, help us. And very few see us as competition because

we're not going to go build the Chat GPT. We're not going to build a

new operating system or a new sort of predictive maintenance

solution. We're going to give you the architecture for you to run it

best. And even our OEM, whether you buy from Dell or

HP or from Lenovo, we don't care. You're buying intel hardware

inside. And so let's help you take the best advantage of those platforms. And that's

really been the approach from intel, is we want everyone's software

to work. And even with the GPU vendors, they still run on a CPU

platform. And so we want to make sure that that code runs best. So that,

again, you're driving the overall consumption. We raise the bar for everybody. We

raise the bar for everybody. Nice. Yeah. I

think there's a lot to unpack there. Right. And I think one of the things

you brought out, which is something that people don't, I don't think people have

widely realized yet that Edge is probably going to be the next

frontier in just

computing. Right. Obviously the last ten years have all been about cloud. Right.

But I think we're swifting as companies kind of take a look at the bills

and realize that lift and shift was not a

financially great decision. Right. Whether or not cloud is a good

thing or not, I think it always goes back to those two

words that every consultant and every It person always says it depends.

Whereas previously it was last ten years was

oh, definitely was the two words. But I think now we're realizing it depends.

And I think one of the drivers for this are things like autonomous systems

or drones or self driving cars, right. No matter how good

5G is, and I can tell you I know all the dead spots

in the DC area, but

if you're driving along at 60 miles an hour, 100

miles, 100 km/hour for our friends overseas,

and like you said, is that a tree? Is that a shadow? Is that

a person? Is that a grandma? Right. You don't want to wait on

the latency to come back. You want the inference or the decision to

be made on device. So you're really bumping up against the

speed of light and you're talking nanoseconds, not

milliseconds. Right.

What do you see? Because you mentioned you want there to be

sensors, but obviously these things have to be relatively low power. I guess in

a car it doesn't matter as much, but certainly on a drone that

matters.

What sorts of challenges does intel see in that regard in terms

of you want the most performance, but you want the most

energy efficiency. That seems like two

opposing forces. You would think that way, but if you

look at Moore's Law and you look at what's really behind that, it's about

reducing the size. And really that means the

power and increasing the performance, increasing the amount of

transistors. And that's really been what's driving compute all along, is how do we get

to lower power per density. Now, where it

becomes interesting is in the cloud. It's a cost measure. It's about getting

more for your dollar in a car or in a

drone or even in a factory floor. It's about being able to

operate closer to where the decision needs to be made

without having to, again, to have to power it and have that immense

cost. Or in the case of a drone, the weight of the battery pack and

so forth. So lower swap actually enables those edge use

cases. And again, one of the things that people realize is that Edge can mean

different things to different people. You talk to the cloud providers and Edge is just

a couple of racks closer out of the cloud. On

Prem, you look at Azure Stack or Snowball or these kind of

approaches. It's really about pushing pieces of the cloud closer to the edge through like

the core or they called it the

fog back in the day. You look at the edge and

you take a look at a Tesla, it's like a driving data center.

There's compute capabilities in there. A plane is a flying data

center. Your drones are getting to be more

computing. And when you move from a

discrete mode to a logical mode, and I've seen these already, where you have a

drone who actually has one processor but multiple containers, so actually running

multiple functions that could be thought of as different

applications on different nodes, but now they've all been collapsed with either virtualization

or container. So you can have navigation being one, you can be

doing object detection and mapping with another, and then be able to do sort

of other kinds of sensing like temperature

or barometer and things like that and doing analysis in

real time. One of the best examples that we demonstrated

at our last year's Fed summit was a set of drones out

mapping a region. They were going about their business, but they had a policy that

if somebody walked into a specific area of interest, let's say in front of an

embassy or in front of Lloyd or too long, that one of the drones would

be retasked and go over and investigate and do facial

recognition. All the things you want to do to make sure, hey, is this person

up to no good? And it didn't require a reprogramming

of a drone. It didn't require a special drone that was just the investigator. It

would basically retask itself with a new. Mission in real time

and go investigate. And when the person left that zone, it go back to its

day job of mapping the environment. That's just sort of the tip of

that simple prototype to show that even a very

small autonomous system and these were like sort of my mini drones

here, is capable of the compute necessary to

do multimission kind of use cases. So the edge absolutely is

that new frontier. And it's again similar to the cloud. When you say cloud,

everyone thinks, oh, public cloud, really? Cloud is all those architectures

all the way down to the edge. It's the way we develop those cloud native

apps that can flow back and forth. So from a cloud provider, it's moving

more of their cloud infrastructure closer to the edge. And what the

edge, folks, whether it be the actual device or sensor manufacturers

are looking at, is bringing some of those cloud

capabilities to their device to operate

independently. And there's a reason for that is that, number one, latency, like you

mentioned, Frank, but also the cost of shipping all that

data. No one wants to ship Raw 4K video feeds to the

cloud just to be able to tell me, is that a tree?

You want to be able to send the results that I saw a tree

here with the longitudinal latitude, which is a small data

packet, and let the sensor do the AI, do the inference

at the edge. Right. And then you have the case

where you're talking about planes or vehicles, right?

Like the whole time it's tracking, did the wheel fall off? Did the wheel fall

off? Did the wheel fall off? Right, but at one point when you get to

your destination, the wheel either fell off or it didn't. Right.

So you collapse that entire thing

to one integer level or really not even an

integer. Like a bit. Right, a bit. And then if the wheel does

fall off, I'm sure there's plenty of other stuff you can pick up too,

but hopefully nobody gets hurt. But I mean,

ultimately you're right. The problem with data is so much

that there's value, but there's a certain

amount of we've gotten to the point where

just because we can, we've done it. Right. Yeah, sure. Bring up that

4K. If I'm a salesperson for one of those cloud

providers. Yeah, man, bring in all that 4K data you want,

we'll take it all. We'll be happy to charge you for it too. Right,

but I think as we get to the point where

there might just be too much data, I think people organizations are going to start

thinking like, where can we scale back on the storage? Because

we don't really need it unless there's some kind of regulatory reason for

it. Now, one thing I want to double click on,

because this is a fascinating conversation, we'd love to have you back

on the show at some point. What's the

deal with FPGA because you mentioned

that and this was a huge deal. So a couple of things that are

interesting is that I first heard about Transformers at

the Microsoft has this internal data science conference

MLADS, and they first talked about Transformers. I went into

the talk and ten minutes, my head went

boom, right? I didn't quite follow it. Somebody later on in the

day in the reception area was kind enough to explain it, how it

works. And one of the other things that came out of that conference was talking

about the importance of FPGAs and what they're going to be like in the future.

Now, again, I'm a data scientist. I really don't focus on

hardware so much until when I need to buy new

hardware, like a new desktop or laptop.

What are FPGAs? And I remember hearing a lot about them and then

they kind of went dark for a while and then now they're kind of coming

back into vogue. Can you talk to us about, one, what they are and then

two where you see they're going? Sure. So Ed and FPGA are a field

programmable gate array. They've been around for forever. I mean, computer

science engineers going back, electrical engineers going back to the

80s played with FPGA. They were very early FPGA, but

basically they're programmable hardware. That's really the way to think about it.

You think about a CPU or an Ace or any chip it's

laid down with its transistors, and the flow of those transit is

fixed. CPU can do multiple software

flows, but the instruction flow is the instruction

flow. What makes FPGAs interesting is that you

can create new RTL, new layouts of flows, what

they call netlist of those instructions going across those transistors

each time. You can go in and customize it after. So the

manufacturing builds you a clean slate of a bunch of think about a bunch of

rows, and then you program them to your specific need

at a hardware style abstraction layer. So it gives you a much

faster capability because you're now really writing in hardware. It's a lot more

complex of a coding. It's not like doing Python,

but what you get is a very optimized piece of

hardware for your specific use case. And what's nice about that

is one of the great examples is in signals conditioning. When

you're doing like 5G research or testing signal amplitudes and

things like that, as you put in your algorithm actually into hardware, you go out

and test it. It works sort of here. I need to tweak it well, instead

of going and spinning a new piece of hardware, you just upload new code and

you go right in. So it's a much faster time of development for doing

those custom things. What people have found when we start looking at sort of

AI use cases and machine learning and pattern matching

is that FPGA really lend themselves well

to be able to create different kinds of architectural approaches to how

you process that data flow. If you think about a GPU

or CPU or even an ASIC, it's a fixed data flow. It's good for the

things it was designed for. What FPGA allows you to do is to customize

your flows based on what the data is or based on what your algorithm are.

And so a lot of the FPGA work they were seeing in AI is people

coding their AI algorithms or the machine learning algorithms right into

hardware and then deploying it. And so it allows you to be able to deploy

your thing quicker and you get pretty good performance. It's not as

good as say, as a custom ASIC for your algorithm. And it's not as

scalable really as like a software abstraction on running on a

cloud set of CPUs. But for a lot of these training and

inferencing use cases, one of the areas where it shines is in the whole

area of neuromorphic processing. So a whole part of the AI machine learning

space is modeling after brain activity or how our

brains process. It's a whole field. FPGAs are actually

well designed for those kind of algorithms that X 86 and

other CPU style Arctic just aren't yet.

And that's why FPGAs really shine in those environments, because you can create

these linear sort of permutation flows that you find in neuromorphic

algorithms. You just code those into the path for the

FPGA. They're really good. You'll see, FPGAs are very often used

in cellular and RF communications that are really good at those sort of

channelizer and signal optimization and

be able to do those kind of algorithms that you do on RF and

Comps, again, really good for those kind of workflows. And so why we

see the resurgence of FPGAs, although they've never gone away, you find them

everywhere. Open up your big screen flat screen TV, you'll find a couple of

FPGA in there. Where they're shining is because it

allows you to do some rapid prototyping on AI. And because we're seeing

now FPGAs come to the cloud. So you go to Azure has an FPGA

cloud. You can now deploy those algorithms at cloud scale,

or you can deploy an FPGA into your edge sensor and be able

to do that real time, sort of. Let's go try this inferencing model. Oh, we're

going to change the inferencing model. Let's go do that one. And where this becomes

really interesting in those low slop environments is a modern FPGA is

reprogrammable in milliseconds, which means you can go from one

program to another by just pushing a firmware, if you will,

update. And now you go from a 5G communications

system to LTE or to a six G

without actually going and swapping out the hardware. That's wild.

That's wild. Yeah, it's exciting times. So

with that, the updatable part of it,

how do you secure that? Because I can easily see that being like particularly

you work in the in the federal space, right? Like security

is top of mind in that work. It should be top of mind everywhere,

but in the near term it's top of mind, at

least in the federal spaces. FPGA

sounds like awesome, but it also sounds like that just seems

dangerous in a lot of ways. You can reprogram it in milliseconds.

There's got to be some kind of security story there. Oh absolutely. And

Fpjs have actually in many cases led as far as the kind of security

mechanisms built into the hardware for that very reason.

At its core, at the core level, it's the same kind of approach you do

for verifying your firmware on your system. It's signed

by hardware so that basically you're verifying

your load and if you're going to do an update, you're going to verify a

signature against a hardware rooted key so that you make sure that only

legitimate folks can do the update and that it's only be able to be done

by someone who's got the permission. From a cryptographic

perspective, what we find in the current FPGA that are out in the market

is that they've built in a whole suite of security

capabilities. Things like Puff Provably, unclonable

functions, which is basically a hardware root key that is

really secure as that hardware route of trust, signing in

cryptography functions, anti tamper functions to make sure someone can't go

pop open the lid or put in a jumper and try to try to change

the code. So those kind of mechanisms have been in place for a long time

because FPGAs have been used in such critical places. We find them in

radar stations, we find them in systems and so they've been building security

in for a very long time. And it's part of the workflow that when you

build your code you're going to take advantage of these implicit, let's call them IP

blocks that do security for your RTL, for your code that you're putting

in place. The other important thing is that the way that the code works

is once you lay it out, once you translate your software into that

layout, the layout is you can't just sort of go and reverse engineer

back. And so it's really a very powerful

mechanism as opposed to say firmware. When you're it's software.

If you think about the BIOS update, it's software that you're loading just deeper in

your platform and if anyone wants to go inspect, you'll find

there's a lot of software in the hardware that you don't realize is actually

software. The same kind of security mechanism we did there. You verify it against a

hardware of trust, you make sure it's signed before you run it

and then you apply cryptography to make sure that it can't be changed or it's

integrity protected. You find those same capabilities built into the

hardware of an FPGA and the software development tools, the

dialogue, the cordis and so forth have the mechanisms to take advantage.

So again, programmers don't have to be security gurus. They basically say,

I'm going to push this, and it's auto going to take advantage of those features.

It's good because programmers historically are very bad security

people. I say that. It says, yeah,

it's its own specialty. And yeah, you can't be

good at everything these days. There's too much. So I'm going

to echo what Frank said earlier. Steve, we got to have you back.

I really appreciate you being here. We could talk and geek out on

hardware stuff forever, but we want to

pivot and go to our questions and if that's

okay, we want to start with unless Frank, unless you had anything else you wanted

to do before. Let me

rephrase. No.

In the virtual green room, you talked about some things that are going on and

kind of operationally and

wow, we didn't even get there. I mean, I

think the important thing I took from this conversation is

that one, GPUs, they are important, but

they're not the whole story. And two,

at the end of the day, chat

GPT, any of these magical looking AI

models, magical seeming, right. They're all mass,

right? Yeah. And being beneath the math are electrons

bouncing around inside these microscopic chips. And

there's all sorts of things you could do to tweak and improve that, even if

it's like a billionth of a second, right? A billionth of a second times

a billion adds up.

And that adds up in terms of whether you're driving a car

or you're flying a plane or

you're a company like AWS or Microsoft,

where, hey, if I save one compute second per

transaction, I do trillions of those a day. And that's real

money. Exactly. And that's the thing that blew my mind. But yeah,

let's switch because we could geek out for hours. Because this is very

true. Yeah. Amazing.

It really is. So how did you find your

way into not so much data, but it how did you find your way into

data? Did you find it or did

it find you or hardware specifically? So ring. It's a really good

question and going back to the very beginning, actually, I started

out in the molecular biology

bioresearch side of the camp, going all the way back. I was going to be

a research biologist and probably still be there today,

except for a couple of key life events early in

the early ninety s, I was a hacker as a kid.

I loved seeing how things fell apart and how to code and break code

and things like that. But in the late 80s, there really wasn't a

career other than a COBOL programmer, which

wasn't an exciting career at the time. So I went the bio route,

which was my, the love. And right after I graduated and was going to start

med school, I had a year off and

someone had some money, wanted to do a startupy thing and they knew I was

a hacker and say, hey, why don't you help me get this thing running? And

I'm thinking, well, med school is expensive. This would be a good way to help

pay for it. And so I started my first company in

95 and after three months just fell in love with everything that was

going on. It was the exciting time to be in the internet. Got to apply

some of my security hacker background in an interesting way

and had some really good mentors. People like Bruce Schneier,

the writer of Applied Cryptography sort of took Zebru Schneider.

Zebrus Schneider was one of my mentors and took me under his wing.

And like I say, I sucked his brain dry as best as I could. But

really it just sort of got the opportunity to get on the ground floor

right before Netscape went public. So really early days on

a startup in the email encryption space and then one thing led to another and

I just felt this was what I was going to do. And for the next

sort of several years, I did multiple security startups throughout

by intel. I like to joke, I'm still trying to figure out

how I ended up here for 18 years. But I think what intel

has provided me and provides a lot of our folks is the ability to sort

of innovate in an environment where a, you've got a big company

behind you helping you do that. But one of the best

reasons why I think intel has been fun for me, my most

successful startup, we had 500 of Fortune Thousand companies using

our product. The first project I worked on in intel went to 40 million

PCs. So the impact is just

unbelievable. Now from the data

side again, at the end of the day, like you mentioned earlier, underneath the data,

underneath the machine learning, underneath the AI, and even before we were talking about AI

was machine learning and advanced pattern matching. There's electrons

moving around it's running on hardware. And so a lot of what my

job has been before I came to the federal team was looking for ways to

innovate or take advantage of new use cases in software, to

take advantage of hardware in interesting ways. And so we call that

pathfinding. So you think about our labs or thinking about the next generation

hardware five to ten years out, I ran the team, the security

pathfinding team that was looking at the two to five year horizon. I

knew this was the hardware platform that was going to be there next year. What

would be some interesting things I could do with it to either advance security or

increase security, that was my area domain. And so things like

antimalware technologies, cloud security, before they knew how to spell

cloud. We called it virtualization security first and things like that.

Web security, that was the fluffy stuff. That was Steve's world while

the hardware engineers are figuring out low level cryptography and hardware

roots of trust. And we sort of worked in tandem to innovate.

And so as things like data science started to take off, it was like,

this is a key area both from a security and perspective. How do I secure

that data? How do I secure the algorithms? How do I use that? I mean,

one of the really cool things is being able to use machine learning and AI

and apply it to the cyber problem.

And when you start doing things like that, you immediately run to, well, we've

got too much data flowing in. I mean, the classic example is streaming

analytics on network at network speed. Well, how do you do

deep packet inspection at gigabit or higher

speeds without losing data? That's a big problem. That's where hardware can

help save you, that you just can't do in software.

And then when I transitioned to the federal team and took over and

drove our federal technology practice, you really opened the door to

all the different use cases. And one of the things I like about the federal

government is that it's a macrocosm of all verticals. You want to

talk finance, you've got IRS and CMS, some of the largest

processing of financial data. You want to talk healthcare, the VA is the

largest provider of healthcare, the largest insurer in the world. You want to talk

logistics, DoD logistics is huge. So

you sort of look at it, every kind of use case you'll find in government.

So it's really a good way of looking at all the different verticals. And they

all have unique or interesting data problems. There's some

commonality. And one of the things I really like about the federal government is that

you get that commonality across the divisions. They all are having trouble doing data

ingestion. That is just fundamental. It doesn't matter if you're the federal government or

Citibank or startup in Silicon Valley. Data ingestion is hard

and doing it at scale and being able to then do something

once you've got the data. And I like to use the analogy

of an iceberg. So AI, Chat, GPU, all these are the tip of the

iceberg. That's the cool, sexy stuff you can do, the hard work,

the data curation, data wrangling is all the work that has to be done before

you ever get there. And that's data ingestion, it's labeling, it's curation,

it's data set management, it's all that stuff. And then layer in things like

removing bias or dealing with bias and securing and integrity, protecting your

data. Like all those things have to happen before you ever start having

the fun math that happens towards the end of that curve.

That's where you find that coming out. Everyone is challenged with those things, and I

think that's where the excitement is today. No, you definitely hear in your

voice, sorry, Andy. Yeah, definitely. No, it's okay. We refer to

that as kind of a joke that's been going on

for seven years now. We say, first you get the data,

and that's 90% of the work. We know

that and your iceberg analogy fits that, Frank.

We need a shirt that has a picture of an iceberg against us. First you

get the data under the I like that. I'm definitely going to do that.

We launched a magazine, actually, yesterday as we record this, and

the cartoon segment is called First You Get the Data. And it

kind of like cringy things that you'll hear about data, and one

of them was like, yeah, first we get the data. My

favorite was how

to prep and clean the data. And they were like, oh, no, our data is

already in the normalized database. We don't need to clean it or prep it. It's

already ready. Like, oh, boy.

You need you need a picture of someone throwing data into a washing machine.

That's a good shirt. We could do that. Yeah,

no, that's cool. And I think you bring up something that I think,

folks, we don't know our exact age demographic. We have a rough

idea, but if there's anyone, let's say, under the age of 30,

right in the car with the parents

or they're listening, it's hard to imagine the time because we're about the same age.

I think you're a little older.

If this was not seen as a good career path, like, coding was not the

whole learn to code movement is a modern

phenomenon. I started my college career to be a

chemical engineer because

I had to convince my parents that software engineering was a

viable career path. And my mom, God rest her

souls, was like, I don't want my baby to be one of those weird

people in the basement. Right?

And then my dad, God rest his soul, was like because when

they came to visit me, I had a Sunday print out of the New York

Times, which of course had the job section, which was

at one point like a book. Right. And look at all these

jobs for computer programming. This is a thing. And my

dad looked through it, and he saw all the starting salaries, and it was like

seven or eight pages of near six figure

salaries in the early 90s, which was a lot of money back then, right?

Yeah. Like, looking through, like, on Wall Street stuff. And

he's like, I'm sold. And it's like

and my mom was like, no.

That is literally, like, my experience as well. When I told my parents that I

was going to not go to the research biology route and do the MD

PhD, I was going to go into the security thing. They wanted to do an

intervention. They thought something was wrong.

About two years. In 96, after I'd done the start, for about

a year and a half, there was an article in the New York Times, Paul

Cotcher, had done the timing attacks against RSA, and it

was front page news. And when you read down the first blurb, it says, 22

year old bio student from Stanford cracks RSA encryption. So

I cut that out and faxed it to my parents because they have an email

yet and said, look, another bio student doing security. It can

happen. Right? That's funny. One of

the best web developers I ever worked with, his degree was in biology

as well. And I think there's something to be said about understanding natural

systems, and I think there's some pattern matching gifts

that go along with that. I know my friend was that way as well. And

Frank, when your mom said she didn't want you to be one of those

weirdos in the basement that flew through my head, but I

maintained discipline was too late.

And I could say the same for me as well. Too late.

In her defense, my mom stayed with us in a house that my

wife also works in technology too.

She had an entire suite in our basement of our

house, which was not

windows, walk out yard, everything.

It worked out well. Sometimes

your parents my mother encouraged it without realizing. She allowed me to buy

the haze modem and connect it to our phone. And I did get

disciplined when I had that $1,000 phone bill from dialing into BBS's overnight.

But they should have seen it coming. Yeah,

my mom freaked out when I wanted a modem. She's like, no, absolutely

not. And my dad was like, yeah, you probably should stay out of trouble.

It's easy to stay out of trouble. Then. I think I was lucky

that my parents didn't know what a modem was, so I didn't know what

they were getting me. Right. This

is awesome. But I want to jump to question too sure. And ask, what's your

favorite part of your current gig? Favorite part of my good

gig? I think honestly, I thrive on being challenged,

on trying to solve big hairy problems. I think that's what has always

excited me is present to me with something that isn't being done well today and

trying to figure out how to do it. And I think one of the things

that I love about my job is meeting with government customers who

have big hairy problems and looking at a variety

of technologies. And I think what makes my role somewhat unique at intel, so we

have like a CTO for memory and a CTO for various

architectures is my role is pan intel so I can look

across FPGAs server parts,

networking, and sort of see that collective of where do the bits can

come together to solve big hairy problems. And that's really, I find

keeps me very excited is that every day I could be talking about an

IoT problem today with an edge sensor, and they're

talking about petabytes of data being processed in the cloud tomorrow.

It's looking across the technology domains and again, coming

from a background of cybersecurity, which again looking at various different domains from a security

perspective, but then adding to that AI, high performance computing,

it's a technology playground, right? And the federal

government, when I first joined Microsoft,

I was in the public sector, part of doing basically

technology developer evangelism for the federal government. And a lot

of my commercial sector colleagues were like, wow, it must be really boring

there. I might be like, you know,

we see things that you don't see

and what it is, is like there's interesting work going on, but the folks doing

interesting work for many reasons do not want

a lot of attention. Indeed. So you see

some things that like, wow, see, I hadn't really

thought of that type moments. Well, decades

ago I spent just a little bit of time in a really odd shaped

building up that way. Just a touch of

time. So I can have five it did. So

I can go yes and amen everything

you both have shared about. So now we have three. Complete

the sentences. When I'm not working, I enjoy blank.

Spending time with my kids. I have two small children and they keep me young

and full of fun and keep

me trying to stay in shape to keep up with them.

Very cool. Both Frank and I have

children as well. Frank has the younger kids. I'm

probably the old guy in this conversation now that I think about it.

But number two, complete this sentences. I think the

coolest thing in technology today is blank.

One thing that is a tough question,

I would have to say. So the two things that I think are really cool.

Number one, again, not the chat GPT, but

what the future will do with that capability is one

area. And then again, because I'm a security geek at heart, post quantum

crypto is going to be fun. Figuring out the next generation of algorithms

and how robust they'll be once quantum computing comes online.

I think that's an exciting area of math that is going to

spurn a lot of mathematic. Academia is

excited because it's a renewed interest in that space

and the algorithms are really interesting. The lattice

space structures are fun area of math to look at. Nice.

Interesting. The third and

final, complete the sentence. I look forward

to the day when I can use technology to

blank. So I'm going to give you two answers. I look

forward to the day when I can draw something on a

whiteboard and it turns into code. That's one thing I'm looking forward

to. Oh, nice. I can totally and that's not that

far off. It's not, I think a little bit of sort of the

image to text, image to code. I think

building box, you have to be able to read my horrible handwriting. That's going to

take an AI in its own right. But I would love a day. When I

can start draw my design like I like to do I'm a whiteboard kind of

guy, and then have it create a prototype. I think that's one thing

I'm looking forward to. And then I think

the other thing is I'm looking forward to the day when

augmented reality becomes reality, where it's not just

a cool toy, but where we actually see it integrated

into our daily lives. And I'm not talking to glasses and all that. I'm talking

about having the digital world and our physical world actually start to make

sense instead of it being a throwaway toy and I think we're seeing

pockets of it, but I think that the future is going to hold a lot

more of that immersive experience that we only see in movies today. I think

those are the two things from a technology perspective, I'm looking forward to.

Although I have to say, if I can get that, the code from the whiteboard

is going to make me a lot more efficient. No, that's true. And

it's funny because things that once seemed impossible

are now possible and even mundane. So I remember

when I was a kid, there was a story, there was like a story we

read about a kid who wrote a built a homework machine, right? And this was

like first or second grade and a bunch of us kids were like, yeah, how

do we do this? We got to make one of those. Now you look at

Chat GPT, obviously we abandoned the effort

because it just wasn't possible at the time. But you look at how kids

are using Chat GPU today, that machine exists

not in the way or the shape or form we could have imagined, but

it's definitely here. So to have that whiteboard to code

thing, it's totally

within sight. Whether it'll be within reach, only time will

tell. Probably a few weeks. If there are VCs out there listening, this is an

idea to invest in, for sure. I would love to see

especially for you, Steve. I'd love to see whiteboard

two FPGA code. That'd be even

better. We're just combining ideas. There you go.

I know that would make some of my engineers happy. There you go. Really

cool stuff. So we ask all of our guests to

share something different about yourself. But we caution

everyone to be fair that remember, we're trying to keep

our clean rating at itunes, so please keep that in

mind. So something different about me.

Well, I guess one thing we've already talked about that I have a bio

background, but the other thing I like to do is I play

tournament poker. I am an avid

poker player when not in COVID Lockdowns and things like

Really? That's something I like to do as a

past. It's a different use of my skills, of sort of social

engineering, if you will. And I like the tournament play

because it's sort of a long game. Right? Well, I have a

stack of money and I'd love to learn more about

is that the joke? All you need is you're always

welcome to my table. I'm lying about the

money. My wife is

actually a pretty good poker player, and when she was pregnant with our second,

she's short and she would carry a stool with her because she would have

to set up and her feet didn't reach the floor. And I think I gave

her like $100 in seed money and said, go knock yourself out.

And she came back like she was spending money. I think she turned that into

something like two grand before she had to quit and go have

Emma. I

would love to see you, because I don't think she's

your level by any stretch, but she did okay. We should have

a data driven poker tournament. We should. There we

go. That's an idea, Frank. The other time we had an

idea of somebody on the live stream said we should do like an ATV

race or something because we always go off track. That's kind of the joke.

Very true. But no, that's cool. Audible is a sponsor

of data driven can you recommend a good book? Ideally

audiobook if you do, audiobooks if not. Sure. Absolutely. Actually, I just

finished one that I think would be perfect sort of summation of this. So

Chips is an excellent book.

You think it's talking about today, but it gives you the history of how we

got here. And even one of the things I thought was really interesting is

some of the decisions that were made early on from the

policy, the government policies that we've seen and how it

affects where we are today. Fascinating reading. So, yes, absolutely.

Chips wars, it's available on Audible because I literally just finished reading

listening to it on Audible. So that would definitely be a book I would

recommend. Cool. I watched a show called Halt and Catch

Fire a few years ago when it was at, and it was similar. It was

in that vein of when things were developing and trying basically

the laptop development story. And of course it was

fiction, but I know enough about it to

know there were some true parallels in there. So this

would be very appealing to me. I'm going to get it. I hadn't heard of

it. Thank you for recommending and our listeners can go to

thedatadedrivenbook.com I didn't test it today, Frank.

Some days it's moody, but if you go there, it should

redirect you to Audible. And if you decide you get a free book on us.

And if you decide later to sign up, then it buys

Frank a cup of coffee. So when

you do that, we get a little bit out of it. It's a great way

to support the show and we really appreciate it.

Awesome. And where can people find out more about you and

what the federal team at intel is doing. So find out more about

me, go to my LinkedIn page. That's S-O-R-R-I-N on

LinkedIn. And then to find out more of what intel is doing in public sector,

just go to Intel.com public sector and it will redirect you to our

Government Solutions page. It covers everything from AI

data science to Cybersecurity to Edge, with lots of white

papers. Use cases podcasts with folks like myself and

others that are recording content on how intel is helping our

ecosystem. So definitely come check us out. Awesome.

And with that, I'll let Bailey finish the show. Now that was some

show. Is it me or are the shows getting better? It could be my

bias that leads me to say that, but I figured I would ask to get

more input. After all, what's an AI without good

input and a feedback loop? Speaking of feedback, have you

checked out Data Driven magazine yet? We are looking for writers