Learn T-SQL With Erik: Aligning Queries and Indexes Part 4
Chapters
- 00:00:00 – Introduction
- 00:00:21 – Consulting Services
- 00:01:10 – Free Tools Mention
- 00:02:18 – Upcoming Event
- 00:03:16 – Tipping Point Queries
- 00:04:00 – Query Execution Time
- 00:05:35 – Query Plan Analysis
- 00:07:01 – Optimizing the Query
- 00:08:20 – EXISTS vs IN Subquery
- 00:09:10 – Adaptive Join
- 00:10:12 – Nested Loops Join
- 00:11:00 – Original Query Performance
- 00:12:24 – Conclusion
Full Transcript
Erik Darling here with Darling Data and today’s video we are going to carry on in our task which is learning how to better align our queries and our indexes. If you need help aligning your queries and your indexes, boy do I have options for you. You can hire me for, aside from watching these videos, you can hire me for consulting, do this stuff all day.
You can also purchase my training. The videos that you’re watching here are just tiny little snippets from the full course material in the Learn T-SQL with Erik course. The link to buy that for a hundred bucks off is down in the video description if you feel like doing that sort of thing and watching more videos of me. It’s crazy. You can also become a supporting member of the channel, ask me office hours questions, and I guess outside of the downstairs links you can also do other things that would make me think of you as a more useful human being.
Such as liking this video, subscribing to this channel, and forcing all of your friends. Hijack their browsers and force them to love me as well. If you need SQL Server performance monitoring, I got you covered.
There’s nothing Erik Darling won’t do for you. Maybe a couple of things. But this thing I’ll do for you. I would do anything for you but I won’t do that.
Anyway, I don’t like that song. Totally free, open source. You can see everything it’s doing. It’s free. It’s right out there on GitHub. It’s a bunch of T-SQL collectors.
They all run on a schedule. They collect important performance information from your SQL Server, put it into pretty charts and graphs, and allow you to talk via your robot companions using MCP servers to do that analysis on your performance data. The MCP stuff is all opt-in.
It is not on by default if you don’t want it broadcasting that it’s there. But it’s just, you know, gives you a different way of… figuring out what’s up with your SQL Server aside from just looking at the pretty charts and graphs and doing your own form of analysis.
So, all that good stuff. If you want to see me live out in the world and you happen to be in the Croatian area, I also got you covered. June 12th and 13th.
I will be at Data Saturday Croatia. I have an advanced T-SQL pre-con. It’ll be the material that you’re seeing here and more. If you come to the class, you get all of the T-SQL stuff. All of the T-SQL videos that I publish as part of the full course.
So you show up, you hang out with me for a day, and then you get like 100 hours of videos to go watch at home. But until then, let’s continue our maddening descent into heat brain leaking hell. I guess that’s what this is.
Maybe it’s just allergies. I get those too. The databases are just allergic as hell to everything. Especially users and developers. Just like me.
Anyway. We’re going to look at some interesting sort of tipping point queries. And this video is going to explore both rewriting queries to get better performance and tweaking an index to get better performance. So you get a twofer on this one.
Don’t say I never did nothing for you aside from all this stuff I already do for you. Anyway. We’re going to start by running this query. And we are going to use drop clean buffers.
Not because this one ends up terribly. Because the next one will end up terribly. So we’re just like this worst case scenario. This has a little go to after it.
So it executes twice. Even if you look in the messages tab, you will see this handy little message here. Beginning execution loop. Batch execution completed two times. Thank you. But the first query, it is a little bit slower. It does take about 1.2 seconds to run. And the second query takes about half that time.
And this is just the effective cache data. Right? And what’s kind of funny is it’s like when you look at these things, it’s like almost hard to spot where they really go astray. Like sure, this takes 60 milliseconds.
This takes 237 milliseconds. Somehow we end up at 922 milliseconds in the nested loops join. So the nested loops join did spend some extra time in there. I’ll talk about why in a minute.
But if you look down here, really the big difference in time. It’s not this, right? That’s about like 12 seconds different. That’s actually 60 milliseconds slower somehow, right? 237 to 295.
But this is at 460 milliseconds. Now part of that is because the nested loops join is responsible for a little bit more work than it lets on if you are just looking at the graphical execution plan. If you right click and you go into the properties, you will see this prefetch attribute assigned to your nested loops join.
This one just happens to be unordered. The same thing would happen if it were ordered. But this is just essentially telling SQL Server to go out and read a bunch of data ahead of time and get some extra stuff that we might need to make this query run and return stuff.
So the nested loops join here doing a little bit more work than in this one. We’ll forgive it though. But this isn’t really like the crappy one.
The crappy one comes. So this is looking through 2013-03-18. This is looking through 2013-03-19. And if we run this one, this is where things get demonstrably worse, right?
Because we have hit a tipping point when SQL Server is no longer willing to give us the query plan that we had before. It is no longer willing to do that key lookup. It just goes ahead and scans the clustered index.
Scanning the clustered index on the POST table for me takes about 8 seconds when I’m reading from disk. When I’m not reading from disk, it takes about 10 seconds. When I’m not reading from disk, it takes about 618 milliseconds.
I know which one I prefer. I also know that I’m pretty sure that I would prefer if SQL Server chose that lookup plan a little bit more reliably. How can we do that?
Great question. If we wanted to influence the optimizer to avoid the clustered index, we might rewrite the query like this, right? So what we’ll say is, again, sort of almost doing the same sort of self-join technique.
But we can just use an answer. We’ll say, just give me the top 1000 rows that would qualify for our original query. And just say where the ID from the outer POST table is in this list of IDs.
And this will influence SQL Server to use that same fast query. Use our nonclustered index instead of the clustered index, right? We’re going to go seek right into that bad boy over here.
Find the rows that we care about. And narrow it down to just the 1000 that we need to satisfy our query. And then go get the columns from the POST table via the self-join here.
And we return all that out. And that’s even a bit faster than either of the ones that we did before at 147 milliseconds. Now, IN and EXISTS often behave as far as the execution plan goes identically.
Often, right? But not in this case. When you have a top 1000 in an IN subquery, you look at this.
Again, the query plan, it looks like this. You see a top operator in it, right? SQL Server is like, oh, I need to limit this to a top 1000. If you do that with EXISTS, though, and I’m just going to get the estimated plan here.
Because if I run this query, things will not go as maybe they look here. The top 1000 is not, there is no top operator present in this. SQL Server will go and find all of the top 1000.
The rows and figure out which ones exist. The top is just ignored inside of EXISTS. SQL Server just throws that away.
It’s not valid to use top in there. So this does not turn out probably as you might expect or as you might have planned on it turning out. This would run for a long time and return a lot of rows.
Because we’re just essentially asking for everything from the POST table where the IDs exist. Even the top 1000 here and all of the rows that this would match. So we could do this, right?
But even this won’t turn out so great. What we’ll do is, no, I’m in the right place. There we go. We’ll say, we’ll put the top 1000 on the outer part of the query where SQL Server can no longer just dispose of it and throw it away and say, you’re not valid.
But if we run this, it’s still a little bit clunky, right? We’re back up to like a second on this. We had this tuned nicely with that in sub query.
If we’re not in a place where SQL Server might use, I should probably stop here for a moment. We get a batch mode adaptive join for this query, right? So good for us, right?
We’re on developer edition. So we’re getting that enterprise edition class for free. That’s cool. But we get a batch mode adaptive join here. SQL Server has chosen batch mode for the query.
And it said, well, I’m going to figure out. The best join strategy based on, at runtime, how many rows come out of one thing or the other. And then I will choose the correct join type based on how many rows leave here.
Great. You may not always get that. If you don’t always get that, you will most likely end up with a hash join here. And the hash join takes, on its own, just about the same amount of time.
Most of the stuff in here does still run in batch mode on rowstore. So you’re still getting just about the same improvement. Just without the join choice at runtime.
The join choice at runtime doesn’t add anything bad here. But it doesn’t add anything good here either. Batch mode makes this thing, like, still okay, but not where it was before. We did a much better job.
We could also force a nested loops join here if we wanted. And we could get down to an okay amount of time. But still 678 milliseconds.
That’s not really what we had before. If you recall. It was several queries ago with our beautiful in clause query with the top 1000 in it. This all ran in 135 milliseconds.
So that’s really more the time to beat. Everything is 600, 800 milliseconds. That’s a regression. It’s not a huge one. But, you know, it’s not really one.
We don’t tune queries to make them regress, do we? We tune queries to make them faster. It’s a crazy concept, I know.
Now, one thing that I want to point out is kind of funny about the array. The original query is… And all of the other ones are ordered by elements. Yeah.
Mouthful of marbles. Are creation date and then score descending. If we just run this query ordered by creation date and score, no longer descending on the score column, our original query still runs really quickly.
Actually, it runs faster than ever. Interesting. Well, we spent a lot of time rewriting this query to sort of have it suit the index that we had available better. But sometimes, every once in a while, you might be able to change an index.
And if we change our index definition, or rather we’re going to create a new index, I guess, to creation date and then score descending, so this fits the query that we were writing, better suits the query that we had originally, then we get the same fast execution as we did when we changed our query.
So, sometimes there are ways to rewrite your query to better suit the indexes that you have. Other times, if you have options and choices, you might choose to change your indexes up a little bit so that they better suit the queries that you have.
All right. I reached the end of the file. Thank you for watching. I hope you enjoyed yourselves. I hope you learned something. And I will see you next week on Tuesday for Office Hours.
All right. Have a great weekend, everybody.
Going Further
If this is the kind of SQL Server stuff you love learning about, you’ll love my training. Blog readers get 25% off the Everything Bundle — over 100 hours of performance tuning content. Need hands-on help? I offer consulting engagements from targeted investigations to ongoing retainers. Want a quick sanity check before committing to a full engagement? Schedule a call — no commitment required.
The post Learn T-SQL With Erik: Aligning Queries and Indexes Part 4 appeared first on Darling Data.
The series