Arjun Patel on Vector Databases and the Future of Semantic Search

Welcome back to Data Driven, the podcast where we chart the thrilling

terrains of data science, AI, and everything in between.

I'm Bailey, your semiscient host with a pangshang for

sarcasm and a wit sharper than a histogram spike.

Today's episode promises a delightful mix of the analytical and the

artistic as we dive into the fascinating world of vector databases,

retrieval augmented generation, and origami. Yes.

You heard that right. Origami, the ancient art of

folding paper, somehow finds itself intersecting with AI,

proving that the future really does have layers or should I say folds.

Our guest, Arjun Patel, is a developer advocate at Pinecone

who's on a mission to demystify vector databases and semantic

search, turning complex AI concepts into snackable bits of

brilliance. He's also a self taught origami artist and a

former statistics student who actually enjoyed it. So if

you're ready to unravel the secrets of modern AI and maybe pick up a trick

or two about folding life into geometric perfection, you're in the

right place.

Hello, and welcome back to Data Driven, the podcast where we explore the emergent

fields of data science, AI, data engineering.

Now today, due to a scheduling conflict, my most favorite is data engineer

in the world will not be able to make it. But I will

continue on, despite the recent snowstorms that we've had here in

the DC Baltimore area. With me today, I have

Arjun Patel, a developer advocate at Pinecone,

who aims to make vector databases retrieval augmented generation,

also known as RAG, and semantic search accessible by

creating engaging YouTube videos, code notebooks, and blog

posts that transform complex AI concepts

into easily understandable content. After graduating with

a BA in statistics from the University of Chicago, his journey through

tech world stands spans from making speech coaching

accessible with AI at Speeko to tackling AI

generated content detection at Appen. Arjun's

interest spans traditional natural language processing into modern

large language model development and applications.

Behind beyond his technical prowess, Arjun has been designing and folding his

own origami creations for over a decade. Interesting.

Seamlessly blending analytical thinking with artistic expression and his

professional and personal pursuits. Welcome to the show, Arjun.

Hey. Nice to meet you, Frank. Thanks for having me on. Excited to be here.

Awesome. Awesome. There's a lot to unpack from there, but I think it's interesting to

note that you have a BA in statistics. Yes. So you were probably

studying, this sort of stuff before it was cool?

Yeah. Yeah. A lot of the old school ways of analyzing

data, understanding what's going on, so on and so forth.

It was kind of, like, made clear to me pretty early that

understanding how to work with data at small scale and at large scale is gonna

be very important going to the future. So I kinda just took that and ran

with it with my education. Very cool. It was

definitely, you know, one of those things where I don't

think people realized how important statistics would be until,

you know, until the revolution happens, so to speak. So and it's also

interesting to see because there's a lot of people that I think could benefit from,

you know, picking up that old picking up a, an old statistics book and

reading through it and understanding, like, a lot of the fundamentals. Obviously, there's a lot

of new things, but a lot of the fundamentals are largely the

same. You know, just I'll

use this example. You know, McDonald's can add a Mc McRib sandwich,

but it's still a McDonald's. Right? Like, it's This

is what happens when you're shoveling snow. Like, your

brain gets I absolutely agree. And, like,

another proof on that point is that Anthropic just released a

blog recently kind of recapping how to do statistical analysis when you're

comparing different large language models. And when you read the paper in the blog,

it's basically just like 2 sample t tests and kind of going over really,

like, not introductory, but still statistics that's easily accessible for people to

learn and understand. So it's still relevant, and it's still important.

Interesting. One of the things that that that stood out in your in your bio

was, people tend to forget that there

was a natural language processing field prior

to chat gpt launching.

How do you, you know,

we wanna talk about the difference between those 2? Sure.

So the one of the first and probably only

course I took in college related to natural language processing was

called geometric models of meaning. And everything I learned in that

course was like everything before, what we now would

consider, like, modern embedding models. So bag of

word methods, understanding how to represent documents and text purely

based on, like, the frequency of the words that exist in the text,

and then trying to understand, like, okay. Based on that information, how can

we learn about the concepts that exist in text from the words that are being

used? Like, what is the framework we can use to understand what these

words mean based on their, co occurrences with the other words and

texts that you're working with and based on, what those

words mean as well. So, like, what the words' neighbors are and what their meaning

helps and also what those words are doing. And I think a lot of traditional

natural language processing, methodologies kinda stem from that, and

there's a there's a lot of mileage you can get out of just thinking about

approaching problems there before you step into these more complicated methods,

like, these embed modern embedding models that exist. So that's kind of, like, what I

would consider, like, traditional NLP, like, doing named entity recognition,

trying to understand how to, find keywords really

quickly. And then once you get really good at that, there's a whole host of

problems that you encounter afterward that kind of modern techniques try to

solve. Right. That's interesting. So so

what was it, what was your thoughts

when you first, like given that you were an NLP practitioner

prior to the release of transformers and things like that, what was your initial thought?

Because I'm curious because there's not a lot of people there are a

lot of experts today that really kind of started a couple of years ago. No

fault on them. They see where the industry is going. Totally understand it. But what

was your thoughts? What was your thoughts when

you when you first saw the attention all you need? The

attention is all you need paper. So that would have been

probably around the time I graduated college, around

maybe a year or 2 after I took the course that I was just describing.

So I I just started learning about, like, okay. Like, this is

how, like, old school, quote unquote, like, embedding

methodologies work. And the biggest takeaway that I got from those is that they work

pretty well. They work pretty well for, like, a lots of different kinds of

queries. And I think what the attention all you need paper did

was it kinda helped you, understand how

to rigorously create representations of text that

generalize way better than, any sort of, like,

normal, keyword based, bag of word based search methodology.

And I think that at the time, I probably didn't

grasp as much what impact the attention all you need paper would have on the

field until we started getting embedding models that people could use really

easily, like Roberta or Bert. And we're like, okay. Now we can do, like,

multilingual search without any issue. Now we can represent,

like, any sentence without keyword overlap when we

wanna find some document that's interesting, without doing any

additional work. Like, once those papers started hitting the scene, I think now we start

seeing, like, okay, this is what attention is doing for us. This is what the

ability to, like, contextualize our vector embeddings is doing for us.

And now we can see what's kind of getting benefited there. But I think I

think my, understanding of how beneficial that

was kind of lagged until we started seeing these other models kind of hit. And

I'm like, okay. Now I can kinda see why this is important and why, like,

future and future models are gonna get better and better based on this architecture.

Interesting. So so for those that don't know kind of and even I'm rusty on

this. Right? Yeah. One of the things that was interesting about this was the in

on this. Right? Yeah. One of the things that was interesting about this was the

in first, appearance. What was it? You you just described it a

minute ago, but it was something like the the prevalence of a word

in a bit of text versus the lack of prevalence and how that

metric becomes was very important in in

I'll call it classical natural language processing.

Right. So this is the idea that if you have words that co

occur together in some document space, the meaning of those words are gonna be

more similar than words that don't co occur in some other given document

space. This is rooted in something called the

distributional hypothesis, which is basically this idea and the other

idea that, concepts cluster in in this type of

space. So what what does that mean actually? Right? So if you have the word

like hot dog, it's probably gonna be seen in a corpus that's

near other food related words than it would be if you picked some

other word like space or moon. And there's something we can

learn from that relationship to infer the meaning of what that word

is and how we can use that meaning of that word to learn about what

other words are doing. So So this is kind of, like, the theoretical

basis of, like, why we can represent words geometrically,

with with a little bit of hand waving. But that's kind of the core idea.

And attention kind of takes this a little further by allowing the

representation of these tokens or words to be altered based

on the words that occur in a given sentence. So you might have a

word like does, like, does this mean something?

You might say something like that. Or you might say, I saw some

does in the forest. Both spelled exactly the same, but have

completely different meanings based on their context. And if you used a

traditional, maybe, bag of words model where you're just counting the

words that occur in a given document and kind of creating a representation of what

that document looks like based on the words that are composed in there, you're gonna

overlap and conflict with the meaning of those of of the word

does and does because they're spelled exactly the same. They might look

exactly the same with this type of representation. But if you have a way of

informing what that word means with its context, which is what attention

allows us to do, then you can completely change how that's being

represented in your downstream system, which allows you to do interesting things

with with search. So that's kind of, like, the biggest benefit that's coming out of

that type of methodology, and that kinda enables what is now known as

semantic search and retrieval augmented generation and so on and so forth. I was gonna

say, that sounds very it's almost like it was, like, the old pre

that error, the vectorization of this and the distance in

that vector in that geometric space. I guess

we've been doing that for a lot longer than most people realize in in a

sense. Yeah. I mean,

looking through, indexes or document stores with some sort of

vectorization has has has been,

something that people have done, except instead of being dense vectors, which is, like,

you have some fixed size representation that isn't necessarily interpretable

to the human eye for some given query or document, it would

be, like, the size of your vocabulary. So you think of, like, Wikipedia. You

can find, like, every unique word on Wikipedia, and, like, that is gonna be how

big your vector's gonna be. And every time you have a new document come in,

a new article, somebody's kind of, like, wrote up and published to Wikipedia, like, you're

representing that in terms of its vocabulary. But now instead of doing that, we

have, like, this magical fixed sized box that allows us

to represent chunks of text in a way that is

extremely fascinating and abstract. And every time I think about it, it just, like, blows

my mind, but that's kind of, like, the main kind of difference is the way

we're representing that information and how compact compact that is and

generalizable it has become. Yeah. That is, like, it it's almost

like you're, you know correct me if I'm wrong, but, you know,

creating these vectors, these large vector databases, right, with, you

know, 10, 12,000 dimensions, right, of how these words

are measured in relationship to others.

It's almost as a consequence of training a large language

model, you create a knowledge graph. Is that is that true? Is that really the

case where, you know, like, you know, dog is most likely to be

next to, you know, the word pet, you know, or

it has the same distance. Is that I'm not

explaining it right. No. No. No. You're you're on you're on the right track exactly.

And I think this is, like, one of the most fascinating qualities

of even, like, what people would consider, like, older

embedding models is this idea that you can take, like, a training test that

seems completely unrelated to the quality that you want in a downstream model,

and it turns out that that actually achieves that quality. So, what you were referring

to, Frank, is this idea that you might have, like, a sentence. You

might have, like, I took my dog out on a walk, and you might say,

okay. I'm gonna remove the word, walk, and I'm gonna have

I'm gonna train some model that tries to predict what that word

where I removed was. This is masked language modeling, which is this idea that you're

kind of getting at of, like, okay, what are the words and how are they

in relation to the other words in that sentence? And it turns out that if

you, like, do this with, like, 100 of 1,000 of millions of sentences and

words, in some corpus that is somewhat representative of

how people, use human language, you can

act you will get really good at this task, number 1, because you're training the

model on that task exactly. But if you are training a neural

network on that model, some intermediate layer representation

in that model so somewhere in that set of matrix

multiplications where you're turning this input sentence into some fixed size

vector representation is gonna be a good representation

of what that word or that token or that sentence is going to be.

And the fact that that works is not intuitive. Right?

The the fact that that works has been shown empirically, and it turns out that

we can kind of do that and kind of have these models work really well.

And nowadays, in addition to kind of doing that, which is what we would consider

pretraining on some large corpus, we now fine tune those

embedding models on specific tasks that are important to us

for retrieval. Like, okay, we have this query or question we're

asking. We have the set of documents that might answer this question or might

not. We want a model that makes it so that the query's embedding and the

document relevance embeddings are in the same vector space. So you're on the right track.

That's, like, basically how these models are able to learn these things. I don't know

if I would call them, graph representation, maybe a little bit

of, being being pandactic on, like, use of words there because that can

be a little bit, different how how you're organizing that information.

But you can make the argument that the way that these large language models are

representing information is a compressed form of, like, the giant dataset that they're

trained on. And we don't actually know exactly, like, where that

information lies inside that neural network. There's some research that's,

like, trying to get at answering that question, But you could, for the sake of

argument, be like, yeah. There's probably, like, a a a dog

node somewhere in this neural network that knows a ton about dogs, and that's how

we're able to kind of learn this information. That is the stuff that we don't

exactly know. Interesting. Because, there was a really good

video by 3 blue one brown, which you probably are I love that

channel. Where he gives examples where, you know, famous historical

leaders from Britain have the same distance

from you change the country to Italy

or the United States have the same kind of distance. So you can kind

of infer I'm not saying that the AI it

almost seems like this knowledge graph is also is also a byproduct

of of of building this out. Like, the there's some

type of encoding or semantic, I guess, is this is really what it is. Right?

Like, that that you get with it. And, I wanna get

your thoughts because yesterday, I I caught the part the

first half of the Jetson Juan keynote at c s CES,

which this you know, we're recording this on January 8th. Right? And one of the

things that the video starts off with is, you know, the idea

that tokens are kind of fundamental elements of

knowledge. And I did a live stream where I'm like, well, I never really thought

about it this way. Right? They're they're building blocks of knowledge or the pixels, if

you will, of of of of knowledge. And I wanted to get your

thoughts on that because, like, that kind of blew my mind and maybe I'm simple.

I don't know. Maybe I'm not. But it all it seems like we've been kinda

dancing around this idea where and now NVIDIA is really

fully, you know, going all in on this, the idea that, you know,

these are not, this isn't an AI system. It's a token factory

or a token score. What are your what are your thoughts on that? I'm curious.

So when I started learning about how, like, tokenization works

and how we're able to kind of, like, basically build these

models without having massive, massive vocabularies,

it is it is pretty it it is pretty

interesting to be, like, okay. Like, maybe maybe there's some,

abstract notion of information that each token has that

is being that is what the model is learning during training time. And then

we're just combining these sets of information in order to kind of, like, understand

what words mean or what documents mean, so on and so forth. Because when you

look at how, tokenizers work and the size of the number of

tokens for, like, maybe the English language or maybe, like, a really multilingual

model like Roberta or multilingual e five large, they're a lot

So it is kind of

odd to think about whether those tokens

themselves hold information that's readily interpretable for us. But I

think that we've gotten so far with using

systems that are just combining, the operations on top of

these tokens in order to retrieve the information that these systems have learned, that there's

definitely something important there. And I would love to, like, know

exactly, like, what is happening when we're able to do that. The the

heuristic that I like to use is, large

language models are generally reflections of the training datasets that they've been trained on,

and they're basically creating, like, really efficient indexes over that

information. And sometimes those indices hallucinate. And the reason

why is because we are when we ask, quote, unquote, what

a question to a large language model or query a large language model, we

are kind of conditioning that model, on a probability

space where every token being generated after is

likely to exist given the query or the context or whatever we're passing to

it. And once you think about it that way, then it just feels like

instead of thinking about what each of the tokens are doing, you're kind of just

querying what the model has been trained on and what it will tell you

based on what it, quote unquote, learned or knows.

And then you can kind of run with that metaphor a lot and build systems

on on top of that. That seems, much more actionable than thinking about,

like, what each of the tokens are doing individually. Does that kinda make sense? No.

That makes a lot of sense. I think the whole gestalt of it is what

really makes it magical. Right? Like Yeah. You know, you can you

can obviously, I I don't this is not this is not, like, the newest iPhone

or whatever. But, you know, if you go through the the text auto complete,

you can maybe make a sentence that sounds like

something you would write. But much beyond that, it starts getting weird. In

early generative AI was very much like that, particularly the images.

Well, you know Don't like, yes. A 100%

understand. I started learning about generative, text

generation before we had instruction fine tune model. So are you

familiar with, like, the concept of instruction fine tuning, Frank? I think I am,

but I IBM slash Red Hat defines it one way. I would like to get

your opinion. Yeah. So, this is the idea that

you can train or fine tune large language models to follow

instructions to complete tasks. So, before we had,

like, models that could that we could just, like, ask questions of and just, like,

receive answers directly, you had to craft text

that would increase the probability that the document that you want to

generate would happen. So if you wanted a story about, like, unicorns or something,

you would have to start your query to the LLM as there

once was, like, a set of unicorns living in the forest. Blah blah blah blah.

And then it would just, like, complete sentence, just like a fancy version of autocomplete.

Right. And that that's kind of, like, what we used to have, and that was

pretty hard to work with. And then once researchers kinda cracked, like, wait a second.

We can create a dataset of, like, instruction pairs and, like, document

sets and fine tune models on them. And it turns out now we can just,

like, ask models to do things, and they will do them. Whether or not

those are correct is kind of the next part of the story. But getting to

that point, it was, like, pretty interesting and pretty significant.

Interesting. Interesting. When I think of

fine tuning, I think of I think of

primarily InstruqtLab, where you basically kinda have a

LoRa layer on top of the base LLM doing

that. Is that the same thing? Or is it kind of slightly

it sounds like it's slightly nuanced. So the nuance there

is that, one, though this the methodology that I'm

describing is mostly dataset driven. So you have, like, your original LLM,

and then you have, like, a new dataset that allows the LLM to learn a

specific task. Or in this case, like, a generalized form of tasks,

which is you have instruction, answer, user query,

give it an instruction. Whereas in your case, you're kind of, like, adding another layer

to the LLM and, like, forcing the LLM to learn all the new

methodology inside that layer in order to accomplish a specific

task. So that's kind of like what client cleaning ends up doing. So the other

way there's multiple ways to do this, it seems. Right? Like, there there's that way

we add the layer, but there's also kind of I hate the term prompt engineering

because it's just so over overblown. But, like, giving it

more context and samples. And now that the the token context

window is large enough that you don't have to be well, if you wanna

save money, you have to be very mindful of that. But if you're running it

locally, like, doesn't really matter. Well, you could give it an example of

let's just say you had I'm trying to think of a short story or a

novel. I don't know. Let's pretend,

Moby Dick was only a 100 pages. Right? I

could give it that as the part of the prompt. Let's say write a sequel

to this book based on what happens in this one. Is that what you're talking

about? Were you kinda giving an example as part of the prompt? Or is there

some and not part of the layer? Or some combination thereof? Or was some third

thing entirely? So this would be like, what what

you're describing is more like few shot learning, which is you gave kind of an

example, and then you're, like, okay. Like, given these examples, can you do this other

task this test that I've described on this unseen example? What I'm describing is

kind of, like, slightly before that. So, like, before we had the ability to, like,

give models examples, we had to, like, give them we have to

create the ability to follow instructions. And then once you have the ability to

follow instructions, you can be like, okay. Here are the instructions. Here's

examples of correctly completing the instruction, now do the instruction.

And that is the reason why that happens in that order is

because first, you have, like, just, like, sequence completion, like,

autocomplete. Then you have, like, okay, given this

task given this set of instructions, just follow the instruction instead of,

like, trying to do autocomplete. And then you have, okay, now you know how to

follow instructions. I'm gonna give you a few data points in order to

learn a new task. Now do this new task. So you're kind of,

like, moving from a situation where you need tons and tons

of data just to get the, sequence completion. And then you need

a smaller set of data to, like, get the capability to follow instructions.

And then you need a very, very, very small amount of data, like,

maybe 3 points or 10 examples or 15 examples to complete kind of, like,

a new task. So there's a lot of kind of nuance in, like, how

modern LLMs are being used and how they're kind of trained and fine tuned, so

on and so forth. And I think there's a lot of, like,

important importance in, like, learning what what happened kind of

before because the advancements have happened so quickly. It can be really hard to kind

of differentiate, or, like, oh, why is why do models perform like this? Why

do things kind of happen like that? And even though, prompt

engineering has kind of, like, let's say, traveled through the

hype cycle where people were, like, really excited about it, and then we're, like, this

is not actually that interesting. Right. What's interesting is that,

doing building a good RAG system or trivial augmented generation system,

you really need to be good at prompt engineering in a sense

because you're assembling the correct context for this model

to answer some downstream question, And it's not

intuitive how to assemble that context. So understanding, like, how are these

models are trained, like, whether they can follow instructions, how good they are at

doing so, how many examples of information they need in order to accomplish some task

really affects how you build that knowledge base in order to help the

model do some sort of new thing. Interesting.

So RAG is obviously all the rage now.

Yep. But there's also a relatively new because this this

space changes rapidly. Like, I mean, I took 2 weeks off in December, and

I feel completely disconnected from the cutting edge, you know.

Because when I was watching the keynote from CES, and I'm like, wow. That's

really cool. And I was texting, you know, slacking with a coworker, and he goes,

oh, no. This is a retread of their, like, last keynote they did. Like

and I'm like, okay. Wow. Blink and you missed

something. So what

you're describing the fine tuning, is that really what Raft is, where the

idea that you have kind of retrieval augmented fine tuning, which I think is what

the acronym stands for. Is that not I'm

not familiar with how Raft works. So I don't wanna, like, kind of venture

and guess without without knowing what it is. But do you remember, like, what context

you encountered this in? Basically, it's the idea that

it's the idea that you can fine tune the results. Sounds very

similar to what you're doing, and I've haven't read the paper in a while.

Back when I was a Microsoft MVP, like, you know,

they had a Microsoft Research had the thing for their calls, and they

were all raving about it. The paper had just come out and things like that.

It's the idea that you can kind of give it pretrained examples.

You start with a base LLM, and you give it pre trained examples, and then

you add on top of just to retrieve an

augmented portion of it. It's very similar, not to

plug my you know, for my day job. I work at Red Hat. That's why

there's a fedora there. We have a product called Rel

AI, which is based on an upstream open source project called instruct

lab. And it's the idea similar idea in that you you you

basically give it a set of data.

And then you we there's a there's a little more to it because there's a

teacher model. And basically what it'll do is it will and synthetic data generation.

So you can start with a modest document set.

And based on how the questions and answers that you

form and the the the,

the taxonomy that you attach to it, it will

create a LoRa layer on top of an existing LLM.

And it it could be that it's it's it's not quite exactly the same as

Raft, but it's definitely in the same direction. Same same thing as, like, Bert, Elmo,

and, you know, Roberta, which, I think

I think I understand. So it's kind of like you so the I think the

problem that might be addressing is kind of just really similar to the problem that

traditional RAG tries to address, except in a more kind of deliberate fashion

Exactly. Yeah. Where you have some document store internally. Like, let's say we

both work at some company, and we have a giant customer support document store.

You take some LLM off the shelf. It's not necessarily gonna know the

contents of your internal kind of documents. So how can you get

it to, like, successfully help answer tickets or triage tickets that

you're trying to build, so that you can answer, like, most difficult tickets and

kind of work toward that. In this situation, maybe you

want to, inject some of the knowledge of

the documents in addition to having the

model being able to search over the document store. So maybe, like, the what this

lower layer is doing is, like, absorbing Yeah. Some of the knowledge from the

document store so that you can kind of more

efficiently query, the database and so

that you don't have to, like, query it all the time. The only,

issue, quote, unquote, I'd have with that method is that you'd have to, like, keep

that updated from time to time, and that's, like, not that's nontrivial. Whereas

if you just do, like, traditional RAG, you just need to

update your, Vector Store, and then you can just have the model

query that new information when you need to. But, you know, it's always best to

use whatever solution works best for your, given use case.

And experimenting with different use cases is always really important. But I imagine that's, like,

kind of what that is trying to address, which is the That is basically it.

The I, you know, I don't wanna go down that rabbit hole of that. But

but, basically, the idea is that, if

you train an LLM or you have a layer on top of an

LLM that not only does retrieval from a source document

store. Right? I think that's a pretty set pattern. But it also has a

better understanding of your business, your industry, the jargon.

Right. Right. Blah blah blah. Right? The idea is that the retrieval success

rate will be higher. Now we're not publishing the numbers yet,

but the research is still ongoing. But basically, it's a

pretty substantial from what I've seen well, I haven't

seen the actual numbers yet, but from what I've been told those numbers are by

the researcher, that it is a it is a substantial improvement

that is worth the, the juice is worth the squeeze in that in that regard.

You're not and it's also computationally, you're not quite training the

whole thing again. You're just kinda putting a new Instagram filter, so to

speak, together on top of the base. So it definitely

does it definitely does some things. Now when we get the hard

numbers, then, you know, I mean, I can

say them publicly, then I think we'll we'll know is the juice how

much does the the the the squeeze to juice ratio is?

But, I can confidently say publicly now, like, there's a there

there. Yeah. And, you know, we'll have those numbers soon

enough. But it's it's interesting because you're right. I mean, this paper

explosion of these different mechanisms. You mentioned Bert. You mentioned Roberta.

Fun fact, my wife's name is Roberta. So that was kind of fun.

There was Elmo. There was Ernie. There was a whole Sesame

Street themed zoo of of model

types. That seems to have kind of that branching out of

those different directions has seemed to have stalled, and we're going into more of

these retrieval augmented generation systems. So for those who because

not everybody on our listeners know exactly what retrieval

augmented systems are. Could you give kind of a a

level 200 elevator explanation? Sure.

So, when you speak to a modern chatbot,

what's happening is that they've learned information through their pre

training processes, the large corpus of basically the entire Internet,

and are generating information based on the query that you're passing in.

The problem that often occurs is that

these AI models might error, and the error could

be making, inform making information up that doesn't

exist. For example, if a model is trained before a period of time,

like, it might not know about that period of time, which is which happens more

often than you think. The information could be false, untruthful, or it could

just be incorrect in a way that's not, like, bad, but still not

helpful. And the reason for this is the way that these

models are accessing that information. The idea behind retrieval

augmented generation is that instead of having the model try

to, generate the correct document or the correct

response given its pretraining process, you instead

add factual content to the query that you're asking

the model for. You first search for that content, which is where

the retrieval part comes, and then you augment the generation of what that

model is going to create based on that content, hence

retrieval augmented generation. There's usually, a querying

step. So you take in a user query, you hit it against some sort

of database, usually a vector database. In our case, it could be Pinecone.

You find a set of relevant documents. You pass that to the generating LLM.

The generating LLM uses those documents to generate a final

response. And it turns out that if you do this, you can reduce the right

hallucinations. And that makes sense because if the model was given true

information and then conditioned its generation on that information, it

follows that the probability of generating information that is

correct could be higher. That's a good exam that's a good

explanation. So you're basically giving it a

crash course in what documents you care about. Right? Like

Exactly. Interesting. And that's a good segue

because you work for Pinecone. So so tell me about Pinecone. What is Pinecone?

Yeah. So Pinecone is a, knowledge layer for AI. It's

kind of like the way we like to describe it. We the main product that

we provide is a vector database. So this is a way of storing

information, information that has been vectorized, in a really

efficient manner. And it turns out that if you have the ability to store information

in this manner, you can search against it really quickly, with

low latency and to find the things that you need to find really interesting for

these types of semantic search and rag systems. Pinecone has a few other

offerings now that kind of help people build these systems a lot easier. There's

Pinecone Inference, which lets you embed data in order to do that querying

step. Pinecone Assistant, which lets you just build a RAG

system immediately just by upsurting documents into our vector database,

so on and so forth. But the reason why, like, you

need a vector database is because all of this advance of

semantic search of embedding models. People have gotten really, really

good at representing chunks of information using these dense sized

vectors. But once you have 1,000, millions,

even billions of vectors across tons of different users, you need a way

of indexing this information to access it really quickly at

scale, especially if your chatbot's gonna be querying this vector database really

often. And so having a specialized data store that can handle that type

of search becomes really useful. That's why Pinecone is here, and that's

why we exist. Interesting. Interesting.

One of the other interesting things from your bio, aside from

the the the origami,

Tell me about this. So so you

your crew does your do you create the YouTube videos, or do you use your

tools, or is it something completely it's just part of your job as a developer

advocate? So it is just part of my job as a

developer advocate. Oh, okay. Like, often that, you

know, I do that because we are interviewing people or because there's a new

concept we wanna teach people, so on and so forth. Or we do a webinar,

and we just upload it to YouTube. Oh, very cool. Very cool.

Yeah. I started my career in developer

advocacy. One was called evangelism. So I was a a Microsoft

evangelist for a while. So yeah. Yeah. Cool. YouTube

is very important. Yep. But it's

also it's also, I think, speaks to how people learn,

but, how people learn. YouTube University is very

real. Right? And Yep. You know, not not a knock on

traditional schools, not a knock on traditional publishing, but this space

is moving so fast that if it weren't for YouTubers like 3blueonebrown

I think his real name is, Grant Sanderson. I think that's his real name.

Somebody will send me hate mail if I get it wrong. But,

he he is, like, really good at explaining these

really abstract mathematical concepts. And

unlike you, I didn't study math undergrad. I didn't I mean, I had to. I

only took the requirements. Right? But I have comp sci degrees. So, like, for me

to kind of fall in love with math again or for the first time, depending

on depending on how you wanna say that, for me, that

was very helpful. And under having an understanding of this, if you're a data engineer

and, you know, or wanna get into this space, it's

definitely vector databases for traditional kinda SQL kinda

RDBMS person will look very awkward at first. But

I know a lot of people that have made the transition, and they kinda love

it. Right? Because in a lot of ways, it's way more efficient,

than, I dare say, traditional data stores. But when you're

processing the large blocks of text, it's really good for kind of

parsing through that. But

that's that's really cool. So, we do have the preset

questions if you're good for doing those. I'll put them in the chat in case

you don't have them. Sure. They're not brain teasers

or anything like that. They are pretty basic of,

questions, and I will paste them in the chat.

So the first question is, how did you find your way into

AI? Did you did you find AI, or did

AI find you? So this is a little bit of a

crazy story, but AI definitely found me.

So when I was in college, when I was looking for my 1st

internship, I couldn't find any internships, basically, because I had, like, no

previous experience in working at tech or anything like that. And,

the first company I worked for, Speeko, took a chance on me because they were

building public speaking, tools to kind of help people learn how to do

public speaking better, for an iOS app. And I had some

public speaking experience. They were, like, close enough. We'll have you come on and kind

of help us, like, work work things out. And while I was there, it was

made very obvious to me how important building

very basic deep learning systems and AI systems to kind

of accomplish really specific tasks that could help serve an

ultimate goal. Like, what we were trying to do is just, like, see how many

filler words people are using or how quickly or slowly you were speaking.

And that requires a lot of, complicated

processing because you have to do transcription and because you have to figure out what

words are being said, so on and so forth. So kind of experiencing that and

seeing that firsthand really opened my eyes to how powerful

since then, I started learning more and more and more about statistics,

AI, natural language processing through my internships,

learning more complicated problems, reading research papers, so on and so forth.

And I got to where I am now. A lot of where I learned is

just out of pure curiosity. Just like, okay. There's this new thing. I wanna learn

about it. That's where I wanna be. And that's kind of how I fell into

large language models and AI, just by wanting to learn about what was going to

happen and then eventually being there. So it definitely found me. I was

not looking for it. Didn't even know I liked statistics until I started doing

statistical modeling. And I was like, wait. This is really fun. I wanna do a

lot more of this. I wanna learn a lot more of this. And I knew

that, once I was in college and I bought a statistics book for fun, and

I was like, okay. I'm I'm past the point of no return. Like, this is

definitely Right. Right. Right. Right. That that might be one of the first times in

history that that's been said. Right. Because I I learned statistics for

fun. I I took stats in college.

I hated it. Hated every minute of it. But

when I got into data science,

I the first two weeks were not fun. I'm not gonna lie. Yep. But

just like the VI editor, once you stick with it,

Stockholm syndrome kicks in, And you start loving

it. That's cool. 2, what's your favorite

part of your current gig? The favorite part of my

current job is being able to learn interesting,

fun, even complicated things in data science and AI,

and figuring out how to communicate them to a wide

audience. It's a really fun challenge. It's really similar to, like,

what, 3 blue one brown does all the time on the YouTube channel, and it's

something that I get to learn and practice and keep keep doing. That's the best

part of the job. I love learning things and, like, teaching other people about them

and learning even more things. And the fact that I have an opportunity to do

that every single day is, like, the best. That's cool. That's

cool. We have 3 complete sentences. When I'm

not working, I enjoy blank. When I'm

not working, I enjoy, baking sweet treats and

goods. I can't have any dairy. So very often, I had to kind

of give up a lot of the cakes and desserts that I loved eating when

I was younger. So now I, like, spend my time trying to figure out how

I can make them again without dairy so they taste really good. So that's that's

something I enjoy I really enjoy doing. Very cool.

Next, complete the sentence. I think the coolest thing in technology

today is blank. I

thought really hard about this question because we're living in a

crazy time of technological development. But the thing that really

stuck out to me and the thing that was also the moment for me

when I started working with, like, chatbots and LLMs was code

generation models. The first time I learned how to

use, GitHub Copilot specifically, I

was I was completing some function, and it completed it before I was done typing

it. And I was like, what the heck? This is amazing. Like, this this this

actually figured out exactly what I needed. And because I was still, like,

a budding developer, it was extremely helpful because I could learn

faster rather than having already a huge kind of store knowledge already in my

brain and kind of pulling from that. So I could see it benefiting my workflow.

So I think the development of those tools and modern tools like

Cursor, so on and so forth, extremely cool. And I can't wait to

see, like, what the next generation of those technologies will look like. Yeah. I

mean, that's a that's a great example. It's almost like you don't

that. It's almost like you can leverage the AI to take on the

have to put in some reps, but not to the degree that you used to.

No. I think that's gonna be very transformative. I mean, I mean, I'm

learning, JavaScript and Next. Js on the side because it's something I have no

experience in. Right. And I was able to build my personal website

entirely through using Cursor and Progression. Nice. I

often check that out. Which is insane. Right? Which is, like, really, really

fascinating. And and I'm not gonna claim to, like, suddenly be an expert in

NextGen or anything like that. Right? Right. Right. Right. I still wanna learn, like, exactly

what's going on under the hood, But having a project that you can kind of,

like, tinker on that's, like, pretty small in scale and that you can kind of

afford to make a few mistakes on and having, like, an expert system kind of

help you go through that, expert, quote, unquote, being close enough, really cool

learning experience. No. That's a great way to put it because, like, I I

I don't have any apps on the modern devices. Right? Like,

so, it would be nice if I

had an Android app that could kick off some automation process that I have.

Right? Or do some kind of tie in with, you know, Copilot

into that or things like that. Like, where, you know, I

originally wrote a content automation system I wrote. I originally wrote in

dotnet, but I ported it to Python with the help of

the help of AI. And I could well, that's just it. Right?

It really the true valuable resource in in life is

time. Right? Yes. It's not Yes. I mean, I could have done it by hand.

I could have done it by myself, but it was one of those things where

am I gonna do it because it's gonna take x number of hours or whatever?

But if I can just kinda here's the dot net version that I, you know,

I posted. This is before there was Copilot, so I pasted it into chat g

p t. And it basically spit out a Python

version, had some errors. You know, this was a while ago. But I

was able to, inside of a day, get it done as opposed to

before. Like, I know how my ADD works. Right? Like, I'll start it.

First 3 days, working on it, grinding on it, and then

I don't touch it again for 2 weeks. And it never gets built. But

with this, I'm able to kinda harness the the spark of

inspiration and and execute much faster. Now I think I don't think

people fully realize, like, you know, it's not all doom and gloom. Nobody's

gonna have any programming jobs. There's a lot of upside too. And I

guess that's just where we are in the hype cycle. As you said.

Yeah. Yeah. Yeah. Exactly. That's a good segue into I look forward to

the day when I can use technology to blank. I look

forward to the day where I can use technology to get a high quality

education on any subject for free. So Nice.

Free education is really important to me. A lot of

what I learned about large language models, deep learning, all that

stuff was online courses that I took for free on places like

EDX, Coursera, so on and so forth. Or people sharing

articles and kind of learning from them, or YouTube videos, or all that sort of

things, in addition to my education. But there's a lot of things you kinda have

to learn after that. Right? And I think that especially with, like,

cogeneration models, it's, like, very easy to be, like, okay. Build me this app

and, like, just make it work. And you can sit there for a couple hours,

and it'll, like, work. But I think the missing piece is

creating a structured kind of learning path that's, like,

personalized to whoever you are for the

thing that you're really interested in with the context of

having, like, these tools that can help you do that thing. And I'm not sure

if we have anybody or any offering that can

kind of do that technologically, because you need a lot of information about what the

user knows or doesn't know. You need to be able to create ability, and then

you need to be able to kind of create, like, an entire mini course that's

personalized to whatever that person needs. But if we can do that, we can solve

so many wonderful problems. Absolutely. I'm

thinking about special education needs and things like that. I don't think we're that

far off from this. No. But I

the biggest issue, is going to be just hallucinations. Right? And,

hopefully, people can build, like, rag systems using tools like PineCone to kind

of produce those hallucinations. But we will also for for something like

that specific use case, we probably need, like, another breakthrough in

indexing information or kind of presenting it, or we need a process that

really allows people to create this information quickly

and verifiably in order to kind of make that happen. But if if that is

a future that we can live in, where technology can can kind of, like, help

people learn, like, really important things really well, that would be

wonderful. And I think that would be, like, amazing for for humanity.

Oh, absolutely. Share something different

about yourself, but remember as a family podcast.

One of my favorite hobbies for about a decade is

designing and folding origami. And it's really fun.

It's very easy, but it's also very hard. There's a lot

of comp complexity inside it as well. One thing people

don't know about that is that there's a lot of mathematical complexity.

So once you get to a point where you wanna design a model with

really specific qualities, really specific features, it suddenly

becomes a paper optimization problem where you

have, like, a fixed size square, and you have different

regions of that paper that you're allocating to portions of the model you're

designing. And it turns out that there are entire mathematical

principles and procedures to solve this problem. So much

so that one of the leading, like, practitioners in the

field is, like, this physicist who wrote a textbook on how to do origami design,

and that's, like, the textbook everyone looks at. So, like, learn how to solve it.

Yeah. I'm not surprised. There's definitely there's definitely a a correlation

between the mathematics of that. And I look at origami creations, and I

just fascinated that could be done from a single sheet. Like, it's

just how is that I mean, that's just mind bending. Now it's

and and makes sense that there's a mathematical because you have a certain type of

constraint, And there's obviously

folds factor into it and things like that. And, yeah, that's that's

interesting. I I should what's the name of that book? I should pick it up.

It's called Origami Design Secrets. Got it. Alright. I will check

it out. So where can people learn more about

you and Pinecone? Of course. You wanna learn more about Pinecone? The

best place is our website, pinecone. Io. You can also find

us on LinkedIn and on x and other social media platforms.

You wanna learn more about me? You can go to my LinkedIn, which you can

find at Arjun Girthi Patel, or you can go to my website, which is also

my name, arjun, k I r t I p

a t e l.com. Cool. And we can also check out your

Next JS skills there too. Exactly. Hopefully, nothing is

broken, but, you can you can see you can see how well I've gotten by

with the Awesome. Trust me.

JavaScript alone is is a is a frustration

creation device.

Audible sponsors the podcast. Do you do audio books? Is there a book that you

would recommend? I do do audiobooks, but I've just

started recently, so I don't have a huge, audiobook library. But

there is I I am a huge fan of short story collections, and

kind of the one that comes to mind is really anything by Ted

Chiang, who does a lot of kind of sci fi short stories. If you've seen

the movie Arrival, the short story based on that is story of your life,

and it's wonderfully written. It's one of my favorite short stories ever.

Yep. So highly recommend that. I believe the collection is

called, story of your life and others, something like that. So

Oh, interesting. Careful with audiobooks. They are very

addictive. So,

with Audible is a sponsor of the show. So if you go to the data

driven book.com, you'll get routed to Audible and

you'll get a free book on us. And if you

choose to subscribe, we'll get a little bit of kickback. It helps run the show

and helps, helps us bring, bring some good stuff to to

the masses. So any any parting thoughts?

No. But thank you so much for having me on, Frank. This was a ton

of fun. I learned a lot from you, and I hope I I helped you

learn one one small thing as well. Absolutely. It was it was

a great conversation, and, we'll let the nice British lady finish the

show. And that's a wrap for this episode of Data Driven, where we

journeyed from the intricacies of vector databases to the surprising

elegance of origami. A huge thank you to Arjun Patel for

sharing his insights on retrieval augmented generation and his passion

for making AI accessible to all. From turning raw data

into actionable knowledge to turning paper into art, Arjun

proves there's beauty in both precision and creativity. If today's

episode left you curious, inspired, or just itching to fold a

piece of paper into something meaningful, be sure to check out

Arjun's work and Pinecones innovative tools. Remember,

knowledge might be power, but sharing it makes you a force to be reckoned

with. As always, I'm Bailey, your semi sentient guide to

all things data. Reminding you that while AI might shape our

future, it's the human touch or sometimes the paper fold that

gives it meaning. Until next time, stay curious,

stay analytical, and don't forget to back up your data.

Cheerio.