Your SQL Is Correct. Why Is Your Report Wrong?

Hey, Frank here. I just want to let everyone know real quick that I will

be going to KubeCon this week. So if you're there, be sure to say

hello. And if you're not there, you can live vicariously through

me and check out the coverage that I'm going

to do. All right. And with that, let's get on to the show.

Well, hello and welcome back to Data Driven, the podcast where we explore the emerging

industry of artificial intelligence, data science, and of course,

none of this is all possible without data engineers.

Unfortunately, my favorite data engineer in the world could not make it

today, so I'm going solo today. And

I am speaking today with Praparna

Moharana, and hopefully I said that right, who is a reporting

analyst at Tempus Unlimited, and she develops healthcare analytics

and compliance reporting solutions She's a senior consultant at Accenture, where she develops and implements

data solutions that transform complex operational data into

actionable business insights. She specializes in

SQL, Power BI, data modeling, ETL, and

healthcare reporting with a focus on improving transparency,

compliance monitoring, and decision-making within Medicaid and healthcare

programs. And just because she's here doesn't mean any

statements she makes is a— Her

statements are not matching, may not reflect the opinions of

her employer. Is that a good safety, uh,

thing there? Yes. Awesome, awesome.

So very interesting. First off, tell me

what EVV is, because that's a new term I had not heard

of. And, um, for those who are longtime fans of the

show, and dare I say readers of my blog, I have a

special place in my heart for electronic medical records. That's how I got into the

tablet PC. platform. So what is EVV?

So EVV stands for Electronic Visit Verification.

So it is a technology which is used to

track home-based care services. For

example, if a caregiver is scheduled to visit a

patient, so it will verify the clock-in and

clock-out time of the caregiver and track

Using the technology called EVV.

Okay, interesting. So it's kind of like home healthcare visits and things like that?

Yes. Okay. What—

I'm unfamiliar with this, and I wonder how many of our listeners are too. What

do you capture? And what is— why does

getting it right matter? So mostly each

of the patients would be provided with different kind of

caregivers. And each day, each of the patients— and

every day, the EVV system will be tracking the

clock-in, clock-out of all the caregivers, and we

collect the timesheet data. So based on that, the

caregivers are being paid. So we are collecting

all the timesheet data basically using all the EVV

systems. So we need to make sure that the

.evv systems are documenting correct reports.

There should be no negative discrepancy during

duration, or that we are getting the

capture duration in a proper document manner.

And you have to follow the sort of healthcare compliance

requirements based on the particular

organization. So this way, uh, we analyze the

EDV data And document it properly and do the

validation. Interesting. How did

you find your way into data analytics and data engineering?

So it is a bit of both, I would say. It's a long

journey. I didn't plan from the beginning or think about it,

and I'm really grateful that I came across data in

my journey while exploring my career options. So basically it began

from 11th and 12th grade when I had elected

for computer science. And I have other subjects also, but

I was so, uh, happy that like you study a small

concept in computer science, you are done, and you just apply

across various problem statements. It's so easy. So

that's why I did my undergrad in computer

science. But in computer science, I was a little

bit overwhelmed because of exploring different

areas of computer science and different

programming languages and web applications, I would

overrun. But later I was quite determined I

will go for a higher level degree. And during my gap

year, COVID happened, and during my gap year I took data science

course and I learned

about exploratory data analysis, data quality,

data cleaning. Machine learning methods I

learned. I, and I was really interested in,

uh, that, and it, and it, uh, applies, um, naturally

with my instincts because I am good in observing and noticing

things. So I was really find it very

fascinating. So instead, when I saw my curriculum

for master's in computer science, I didn't saw any data

analysis or data science course. I thought it's better

to switch for master's in data analytics previously,

and that way data found me and I found

data. It's a mutual relationship and it started.

That's cool. Like, that's the question I always like to ask is like, did you

find data or did data find you? And it sounds like

it was a match made in heaven, I suppose, because you— yeah, no.

And data can be very— from the first glance of it, for me, it wasn't

love at first sight. It was kind of like, It

was kind of like, I kind of saw it again in a fresh light.

And I realized like, there's a really good

engineering discipline behind this. And there's so many

problems you can solve with advanced analytics and things like that.

So my notes say that you do independent research. What, what questions are

you— what are you exploring in your independent research? So mostly in

my independent research, I mostly deal

with collaborating with other industrial people. For

example, I collaborated in how to build

and like whether it is beneficial to

go for to discuss in our research and help people

across industries in topics like it is, is

it better to build an in-house BI tools in place

of vendor-based. We discussed that in our

collaboration and when to go for

in-house BI development and what kind of

customization you can have with in-house BI

developments and what kind of— for example, I've got

someone who is, who has knowledge of Salesforce,

somehow who has knowledge of AI intelligence, and we would

collaborate that how with integration of

Salesforce and the data architecture knowledge I

have, we can collaborate and make the architecture and

people can use that architecture and

Salesforce collaboration across industries and

AI intelligence tools. So this way we try

to help, like even in my speaking opportunities, I

extensively talked about how we can

automate like all the reports using tools like

SQL, Python, and Power BI,

and why it is taking exact various

examples. For example, healthcare compliance, for

example, and how to handle voluminous

amount of data, why it is really important to have an end-to-end

architecture, and based on your data source, what

kind of tools you have to decide. For

example, if you are collecting data

from API calls, why it's better to choose

Python if you have diverse data source, because it

would provide a large source of diversity and flexibility

while doing data monitoring or data log, like

detecting the errors in the pipeline. It will be

easier, and even API validation, authentication,

tokenization. If you choose Python and if you have 2

databases, it's best to go with linked servers and

open queries because if you use Python for that, what happens

if you have 2 open cursors from different databases?

If cursor 1 is running a select statement,

so what happens? The cursor 2 open, so

it— meanwhile the cursor 1 is executing the

query, the cursor 2 is dead. And because of that, you

would have data latency issues

and you won't get timely reports. So it's better to

go with SQL Server. So you have to be smart around

deciding the tools, whether you should go with Python or

SQL. And because mostly I talk about data

automation, means automation of

reports which are manually generated into Excel,

So whenever people think about reports, they mostly

go around a basic idea or think about in generalized way

about dashboards, summary reports, charts, but it is

not about that. It's entire lifecycle of data from

data extraction, making of reports, applying, knowing the

business rules the stakeholders want,

and providing reports to using Power BI, and

even you can even automate reports further using

Power BI if the stakeholders need email

like more than 8 refreshes in Power BI

overnight. Like Power BI can do 8 refreshes, but if

you want more than 8 refreshes, you should use Power

Automate or various things. If you want to split the file

into various categories and you can't provide

like Power BI Pro licenses, licenses to

various users, you can sure put that in a

SharePoint, pen drive, or OneDrive, whatever you use, and

the Power Automate can split it. And even you can email it.

So Power Automate can further automate the work which we

manually do in Excel, like splitting of files

or like emailing the report. So

mostly I am more interested to the automation parts of

reporting and Making things easier.

Interesting. And when you say cursor, you don't— because we live in the AI

age that's dominated by AI tools, you don't mean cursor the

AI tool, you mean the cursor of the old-school SQL

process, right? Mostly it is

being processed sequentially, so it's better to go

with— but I won't say because of AI, because many

organizations still refrain from using AI because of

high sensitivity and confidential data. refrain from using

AI tools and really against Copilots, and,

uh, they totally don't use that. So the thing is, I was

talking about cursors because in Python we sequentially

would be running a line of code. So it's better if

we use open query, it won't run sequentially. Like both, we would have,

uh, both the active connection. In cursors, one cursor

is running, so other cursor would be dead when

first cursor is running. That That's the issue I

had faced while troubleshooting a lot, which I

discovered from my own when I used 2. Yeah, no one—

every time I mentioned doing a— using a

cursor, they always— everyone's told me do

something else, right? It's really, you know,

I think I've only ever used it professionally twice in my

career. And once I got yelled at for

doing it, and then the other time was someone who was far

more experienced than me said, no, that really is the only use

case. Like, that really was the only solution. So I think that's— it's

very telling kind of how that worked out. But

in practical terms, what does good data

quality look like?

So mostly good data quality will look like

that each— if you look at a single

record in your database, it would have all the

accurate, no missing values in a single record,

and there wouldn't be any kind of

data discrepancy like negative, for example,

duration is negative 15 minutes, you need to validate the

data. So I would I would say missing— no

missing values. All the fields should have values in structured data.

And I would say that's the most important part of data

quality. And when you go through the

entire lifecycle of data, there should be no duplicate records.

You should have each unique record.

That's the most important thing about data quality,

I would say. And like while building architecture,

like the entire lifecycle I was talking about, when you pull data from the raw

source, you're moving from SQL layer to Python

layer to the reporting layer like Power BI. You have to

see that the data which you have collected, if

it is 1,000 rows, it should be

1,000 rows till the end of the reporting

when you reporting layer, taking into

consideration how you validate the record.

Why do teams struggle so often with data quality? Because data quality

has been a problem that since I

was a young man, I had way less gray hair, had been an issue. It's

like, so why are we struggling with it still?

I remember when the discussion was,

Does data quality matter? Clearly it does, and I think

we found out the hard way that it does. But why is this so hard

to not have missing values and things like that? What

is it, industry use case specific? What's your thoughts on that?

For me, it depends how we— how the source is collecting the

data, right? Uh, maybe we could have

inaccurate time zones. Because the, for

example, system would be in a different time zone in UTC and we

are, we want in EST time zone where we are.

So mostly time zone issues and

it could be like we would have negative,

like missing values as I mentioned earlier, and

we need values in standardized date format. We

would always face issues because we are trying to collect data from

different sources, not from a single

centralized unit. And we are trying to

gather information to get an

answer to a single question which we

want, as the stakeholder

wants. I think mostly because we are

trying to collect it from different systems. From one would be coming from

API call, one would be coming from a different

database system. So mostly because of different systems, we

would always face data quality issues. You

are on mute. I'm sorry.

Do you think AI can help with data quality issues?

Yes, they can help with data quality issues.

But as far as I'm aware, I'm reading through

other people's experience, and from what

I've read about AI is that when you try to

solve something complex, problem, it's really

difficult for AI to keep track of the data

quality and what we want. So

we really need the human judgment to ensure that

what kind of standardization and normalization we need to

apply to our data for improving the data

quality. We can take help of AI to

improve the process or take suggestion, but it

can't hopefully help with automating the data quality,

like, by any human

interaction or human reviewing it manually.

Interesting.

What's your take on the use of AI in

data systems, like, in the data engineering space? Is that—

I think data engineers have a pretty bright

future in terms of career-wise, but

how do you think the rise of AI agents are gonna

impact people's careers?

Um, it would— it wouldn't—

I think from my point of view, it can't replace—

like, it is mostly said that AI will replace

human jobs. But I feel it

won't actually completely replace. It could

make data engineers' life easier

because in certain directions

we could use agentic AIs and AI

prompts, or to make our— to

make— to get suggestions how to make process easier.

But because industries want to maintain the

confidentiality and maintain AI governance,

so mostly AI won't affect that

much. What I have seen, because mostly

I have seen organizations who are in

healthcare industries, they refrain from

using AIs. So It

really, like, I know AI can help you to

optimize SQL queries, AI can help you

build metadata and ETL pipelines and suggest a better

architecture, but there is still a

lot of issues with AI governance and data

issues with the confidential and sensitive data which

organizations own. That's a great

segue because we talked about AI, we talked about that, but what

parts of— how do you decide which parts of an ETL

or ELT workflow are worth automating first?

Like, how do you— is it about how often the job has to run?

Is it about criticality? Is it something else, some mixture

of the above? So mostly the most

repetitive task which we perform should be

automated using the tools we want.

For example, every time transport— I would

say, uh, whatever reports we need to

generate, we first need to understand

first the business requirements of the stakeholders,

because, and because it is, every

organization would have different business rules, and it becomes quite

difficult for us to understand which

records we should include, exclude while building a

metric. So mostly, first is the

business definition and the requirements of

stakeholders, and secondly, on the

base voluminous amount of data being generated, so you

have to come with efficient design of a data architecture

such that the data is maintainable and

recoverable. For example, if the one of the data

pipelines fails and you, you

won't get the report being loaded in the table which

we require for the reporting, it's better to show the

last refresh historical data to the

stakeholders instead of showing no data because

the data failed, so you can gain

stakeholders' trust. And most importantly, it

is important to maintain a control log,

like when the data is inserted or deleted,

in what amount of duration data is getting inserted, and how

many records were inserted in the database. So

this is— and you even have to see that if there is any

failure in the system, You have to monitor when you rerun the

data pipeline, you have to see the last

log point, like last timestamp when it

did stop running, and you have to— when

you start try to rerun the data, you have to see there is no kind

of duplication happening in the data. So you have to

maintain the idempotency of the data, so we

don't have duplicate records while rerunning the data.

And mostly, so this

way, uh, you have to, you have to even

maintain the valid or invalid record

history because in future if business requirements change,

it is really important to know, figure

out in the data that what is,

it is really important about what kind of data records are getting affected.

Right. It's just interesting. Obviously, reports

have to be reliable and accurate.

What do you— how do you

balance moving quickly with reliable reports?

It seems like, you know, the old, you can't have it good— you can have

it good, fast, or cheap, and you have to pick 2. Right.

What's your take on that?

For having reliable reports, first I would say

I will provide an example. For example, I'm trying to

calculate the percentage of

completion rate of a visit. I will take the

completion rate and a denominator as the total number of

visits. So I don't know in total number of

visits whether should I consider the invalid records

and incomplete records of visits which were never done.

So it is something which could be defined by a

business. So we need to have the answers

documented from the stakeholders, what kind of— what

we need to include in the total visit denominator so we

can get accurate number

in our Percentage of completion rate.

So this way we could have a reliable report which could

show us accurate values, because

sometimes the— our report to be technically correct, we

have built a good SQL, we are doing the good

data source extraction and

validation and maintaining all the control logs,

but Still technically the process is correct, but

our reports will still be showing inaccurate values.

So it is really important to have clear

business requirements set. And secondly, you should have—

you need to build the stakeholder trust by showing the

last refresh rate in your reports. So

it is a— this is where you can help their health building

a reliable report. So it is

like they can see the last refresh date, date, and if

it is, it is past midnight and today it is

daytime because the pipeline failed, but

they would be like, yes, we have accurate data for which

we want to make the decision. So this way we could make

reliable reports. Taken

into consideration these practices.

Interesting. I often find

it fascinating that

people think that AI can do all the— all sorts of things, and

I think ultimately human judgment, and people rely on these

reports, right? Data engineers, data types,

so to speak. When I was doing consulting

back in the day, people hated doing reports,

right? Probably because Crystal Reports was this awful software. And

if you've never heard of it, you're very thankful to be born when you were.

But I remember just having to work with it.

Again, it wasn't really a fun experience, but the fact

that it was able to— the fact that you

You could. What really matters with reports is

reports are what people who write the checks see,

right? Whether that's a KPI dashboard or anything like that, the people who

are the business users are the ones that see the reports.

And that is crucial in

business decisions are made. And plus they're also the people signing the checks usually.

So if you keep them happy, that that's good for your

budget. What would be your advice to someone starting out? If you

had to go back and tell, you know, your 11th, 12th

year student, what would you say to yourself or someone

else in that similar position? I would say

first to believe in yourself, because when I

was doing my master's, I was quite keen

that I would get an internship. To get

real hands-on experience before moving

into a full-time opportunity after my

graduation. So when I did my

internship at a manufacturing company,

for— at Balmain Richardson Foods, so

mostly I was— my work was to

automate Power BI reports for the manufacturing

flow. So

mostly I would say that I

was doing a lot of things out of the blue, like from

a beginner Power BI knowledge and foundational knowledge,

I was jumped into for advanced Power BI

things and real-life problem requirements. For

example, I need the data was

imported into Power BI using API calls.

So it does not like that if you are using any other data

source, it would collect that time zone. Because of the API calls,

it was collecting a different time zone, the

default UTC. And in manufacturing, you

need the data to be visible to the operators and the

supervisors, which is 24/7, and the data should be visible

at midnight also. So out of the blue,

the report was blank and there was nothing showing

up in the Power BI report. Wow. So it was kind

of— yeah, so it was like,

I'm doing everything right, I'm not doing— I'm doing— not

applying any kind of formula on the time zone, what's

happening? And later I figured out that it

is because Power BI defaults to UTC time zone and you

don't have any settings in Power BI So you can change the time

zone. So I have to develop

that formula to change from UTC to

EST or EDT time zone. So because of

daylight savings, it was a bit difficult. And you have to

work and you can't find a solution anywhere on the internet

and you have to use your own knowledge to come up with the

idea. So it was a big troubleshooting

ability I gained in my internship. And secondly,

I need to— like, most of the reports need to be deployed

on it. Some of the reports need to deploy on TV so the

workers on the floor can

see the TV, what the Power BI reports, and know

the required business, what is happening. But like, they could

know about downtime of a machine, or they could know

about overall hourly

performance of what is happening in the

manufacturing flow. So when the

report is deployed on TV, you can't have user interaction.

And I was like, the challenge was like how to

make the Power BI refresh every—

to refresh every 5 or 10 minutes or 15

minutes without any user interaction. So it

was a challenge for me too, because if you

notice, when a Power BI report refreshes, there

would be any kind of human interaction, or you would

refresh the browser. So that was a major challenge,

and I have to come up with some unique solution from my own way.

Yeah, interesting.

There's just no substitute for doing to learn, is there?

No, those horrible experiences

tend to stick with you and you learn the most from. I haven't

figured out a way to learn with, uh, and still avoid them.

If anyone else, you know, who's listening to this has found that, I would love

to know about it. Audible is a sponsor. Do you listen to audiobooks

or— Oh yes, I do

listen to audiobooks. Any recommendations for our audience?

I would say Storytelling with Data

by Paul Nuttlick. Okay.

Interesting. Very cool. Audible is a sponsor of Data Driven,

and you can get yourself a free book if you're listening and you go to

thedatadrivenbook.com. That routes you to Audible and you'll get a free audiobook

on us, and we get a little tap on the shoulder that'll probably be enough

to buy a cup of coffee. Maybe not, depending if it's Starbucks or

gas station coffee. What's your favorite part of your current gig?

My favorite part is when I have already built

existing projects which are already deployed,

and when somebody comes out, there is an error or there's

some issue, I don't know, I enjoy to notice

the discrepancy and Troubleshoot

the issues. So that's my favorite

part of gig. And the challenges I, I told you

which I faced during my internship for the things

which are out of the box and you can't find a solution anywhere

on the internet, uh, that's nice.

And I really like— I learned a lot of things from my job because

First, I was a data analyst, and from

beginner I jumped to advanced because of the

things which were— because of the requirements which were given to me.

And then later I became a data engineer in my

job, so I learned various things— how

to do link server and how to

interact with multiple databases.

And how to use OpenQuery, how to build

effective data architecture based on a volume of data, how to

manage 100 million amount of records. And

because in Power BI you can't import 100

million number of rows, you need to optimize before importing

the data in Power BI. So you have to be creative

around getting a good data architecture

and streamline the process.

So that's my favorite part of the work I do,

what I learn every day in my job. Very cool.

What do you think the coolest thing in technology is today?

For me, I know many people would say AI,

but I was working in a health industry You

help mostly organization refrain from using AI

for the sensitivity and confidential data. So for me,

automating the reports using Power BI,

Python, and SQL is the most interesting

part in the technology than AI. I can just

get— use AI prompts to get suggestions how to make

the system more effective and how to make

each of the reporting

lifecycle of data easier.

But I would say reporting

automation is the most— using SQL,

Power BI, and Python is the

most— is the thing that fascinates me in the technology.

Interesting. What are you looking forward to technology doing one day?

So I, I look

forward that we don't have to do repetitive

tasks and have all the

reports automated and just work

on the maintenance of the reports rather than

still a lot of— I see a lot of

reports are being generated and people being very

like, they are very, uh, like used to, or like for

example, when I was doing reporting automation in

manufacturing for manufacturing flow, though the reports

were automated, people were still using paper timesheets

to validate it. They didn't rely on it. So

I would say mostly

build a more reliable reporting automation

where people can trust. Interesting.

Where can folks find out more about you and what you're

up to? People can find more about

me on LinkedIn. So you can see

articles written by me and the

podcast I participated in, and

the— in the roundups I

participated in. And you can

know more, you can get the articles and roundups and the

opportunities I go for and learn

more about me mostly through LinkedIn.

Interesting. Very cool. Any parting thoughts?

What do you say? So any, any final thoughts that you'd like to share with

the audience? The final thought I would share

with the audience is that stay curious and

don't stop learning. And if you want to learn something,

try to, try to participate in, uh, if

you're a beginner, try to participate in hackathon because it

could change the way you look at things and you will

network with fascinating people early in your

career. And if you are a beginner, definitely

go for internship because during internships employers are a

bit lenient and they just want to know how

excited and what— you have good amount of knowledge

before applying the hands-on skills. But if you are going for a

full-time opportunity, it becomes really important that

you have close industry experience because people don't want to train

you. So always keep learning. And while, uh, in my

job mostly I learned a lot of things. And when

you look at a legacy architecture or listen

to people who are already in the industry,

you— in— when I go for my speaking opportunities, you

come up with different solutions. It is not that only that

non-STEM people only have to come with creative

ideas. STEM people even have to be

creative with their problems which we are trying to solve. And it's

quite fascinating to see that even

in STEM, each problem would have different solution

and effective way in doing it. So always

try to learn and be curious. Very cool. What a great

note to end on. Thanks for joining. Thanks for being on the show. We'd love

to have you back again. You can Go really deep into the nerdy

Power BI and SQL Server stuff with Andy. And for this, we'll let the

credits roll. Awesome. Thank you very much.