Intuiting Latency and Throughput

Although they may seem highly technical, you've already experienced both concepts - and why they matter - if you've ever done a load of laundry.

This is the twelfth video in Part 3 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated weekly.

We've reached the point in the Performance-Aware Programming Course where we will be talking about latency and throughput a lot. These are concepts we've seen a in passing already, but we've never really talked in detail about what they mean — more importantly, we haven't had a chance to build an intuition for what they mean.

So now that we're getting into the nitty-gritty of how CPUs process instructions, I wanted to give you an intuitive explanation of latency and throughput so that anyone who's not familiar with those concepts, or hasn't had a chance to internalize them, has a solid foundation to start with.

The way I'd like to do this is by talking about laundry. Hopefully you have at least some experience with doing laundry?

Specifically, I want to talk about the kind of laundry setup some homes have where there is a washer and a dryer — two separate machines, one that washes the clothes, one that dries them.

The idea here is that you put a load of laundry into the washer, it swirls around for an hour, and then your clothes are washed, but damp. You take that load and you move it to the dryer, it tumbles around for an hour, and then your clothes are dry.

That’s a finished load of laundry, and if it takes one hour in each machine, then we would expect one load to take two hours:

Now I'd like you to consider another configuration for the laundry room. Instead of two machines — a washer and a dryer — what if there was just one machine that washed and dried the clothes?

These machines do exist, they’re just less common. You put the laundry in a single cylindrical compartment. The machine fills with water and spins the clothes around for an hour, just like it was a washer. But then, it drains the water out, and heats the compartment while it tumbles the clothes around for an hour — just like a dryer.

When we think about this “combo” machine, it doesn’t seem much different than the split washer and dryer. It also takes two hours to do a load of laundry, right?

We might even think that the combo machine is more efficient than the split machines — after all, it probably takes some small amount of time to transfer the clothes from a washer to a dryer. The combo machine doesn’t require that transfer.

So even if both cases spend one hour washing and one hour drying, we might naively expect the combo machine to wash clothes slightly faster than the split machines. We certainly wouldn’t expect the split machine to somehow be significantly faster at washing clothes somehow, right?

Well, if you’ve ever done a lot of laundry before, you might intuitively sense something’s wrong with this picture. We’re only considering what happens when we do one load of laundry. Yes, with only one load, both setups seem equivalent. But what happens if we have a lot of laundry to do?

For example, suppose we had three loads of laundry to do — A, B and C. In the combo machine, we put in load A, and wait two hours. Then we take load A out, put in load B, and wait another two hours. Then we do the same with load C:

It takes use six hours to do three loads of laundry. Looking at the pattern, it’s intuitively obvious that to do any number of loads of laundry, it takes twice that number of hours: for n loads, it’s 2n hours:

What happens when we use the split machines?

Well, we put load A into the washer, and we wait one hour. This is essentially the same as what happened in the combo case.

But then, when the hour is up, we transfer load A to the dryer:

The washer is now empty. So, instead of waiting another hour to put in load B, we just put it in right away — before load A has even started drying, let alone finished:

We didn’t need a degree in CPU architecture to know to do this. Everyone who does laundry with machines like these does this intuitively.

But this unremarkable process leads to a rather remarkable result. This laundry pipeline allows us to constantly overlap the washing work with the drying work for the entire time we have more laundry to do:

Suddenly, the expected six hours it would take to do three two-hour loads of laundry only takes four hours:

Two of the hours have vanished. Where did they go? They didn’t go anywhere — they just happened at the same time:

Specifically, in the combo case, each hour of laundry only resulted in one hour of work. In the split case, two of the hours did double. The first and fourth hours did only one hour each, since the dryer and washer were left empty, respectively.

But in the second hour of laundry, the machines did two hours of work: the washing of load B and the drying of load A. Similarly, the third hour of laundry was another two hours of work: washing C and drying B.

Instead of taking 2n hours to do n loads of laundry, the split case actually takes just 1+n hours thanks to the washing and drying happening simultaneously for all but the first and last hours.

While this may be a completely unsurprising result for anyone who does a lot of laundry, the principle here is profound: when we split an operation into a sequence of smaller steps, even though the total time it takes to perform the operation stays the same, the time it takes to do that operation repeatedly drops dramatically because steps can now be performed simultaneously.

Which brings us to latency and throughput:

Latency and throughput are measurements we use in performance analysis to summarize the behavior of a CPU pipeline — and, if we’d like, our laundry as well!

Latency is the amount of time it takes to do one entire operation to completion. In our laundry case, this was always two hours. In the combo machine, it took two hours to wash and dry one load of laundry. In the split case, even though it could overlap multiple loads of laundry, that did not change the latency. It still took two hours to wash and dry a load.

Another way to say this is that latency is measuring the time from when a particular load goes into the washer to when it comes out of the dryer. It’s the total time a load spends in the laundry, all steps included. Since it doesn’t matter how much overlap there might be with other loads of laundry, latency does not improve when you can do unrelated work simultaneously.

In other words, if your favorite shirt is dirty, and you want to wear it, it’s going to take two hours to get it washed and dried. Even though we clearly see that the split machines are faster at doing laundry in bulk, there would be no difference between them and the combo machine in the two hour wait time for your favorite shirt. Both configurations have two-hour latency.

Throughput, on the other hand, is a measure of the rate at which you expect operations to complete when the pipeline is operating at full capacity. In a typical hour, how many finished loads of laundry do you expect to receive from the machine? Alternately, you could think of it the other way around, and ask how many loads do you expect to be able to start in a given hour?

In the case of the combo machine, it’s clearly it takes two hours for each load of laundry, no matter how many loads we do. So the number of loads started, or finished, per hour is 1/2, or 0.5 loads/hour.

For the split machines, once the pipeline is going, we expect to get one finished load out every hour. Equivalently, we expect to be able to put one new load in every hour. Therefore, the throughput is 1 load/hour.

But even this simple laundry example illustrates an annoying aspect of latency and throughput: the senses are inverted between the two.

Latency is measured in time per operation. It’s the amount of hours it takes to do a load of laundry, the amount of cycles it takes to do a CPU instruction, and so on. This means it goes up as it gets worse, and down as it gets better — we want our laundry to take less hours to do, we want out CPU instructions to take less cycles. We want lower latency.

Throughput, on the other hand, is measured in operations per unit time. It’s the number of loads per hour, or the number of instructions per cycle:

This is the inverse of latency! It goes down as it gets worse, and up as it gets better. We want our laundry to do more loads per hour, and our CPU to do more instructions per cycle.

To fix this mismatch, in performance analysis, we often talk about throughput as reciprocal throughput instead of regular throughput:

Reciprocal throughput is exactly what it sounds like: it’s 1/throughput, the inverse of the throughput. This turns operations per unit time into time per operation, which aligns it with latency. Now, we can talk about latency and reciprocal throughput together, and keep the same sense — since the units are the same, lower is now better in both, and higher is worse.

Furthermore, it makes the numbers match when the performance matches, which makes the numbers easier to understand at a glance. The combo case is a good example: we know it doesn’t have any work overlap. It takes two hours to do each load always, no matter how full its “pipeline” is. So we would like the two numbers to be equal in this case, so it’s easy to see what’s happening.

With latency and regular throughput, this doesn’t happen. The latency is 2 hours per load, but the throughput is 1/2 loads per hour. These look completely different, even though they’re representing the same performance.

With latency and reciprocal throughput, on the other hand, everything lines up. The latency is still 2 hours per load, and the reciprocal throughput is 1/(1/2 loads per hour), which simplifies to 2 hours per load — exactly the same as the latency.

Now, unfortunately the regular-vs-reciprocal throughput situation gets more confusing when you're reading things on the internet. Often, blog posts or reference materials will omit the “reciprocal” and just say “throughput”. But they don’t really mean throughput, they mean reciprocal throughput. And if you’re not already fluent in these concepts, that can throw you for a loop. So, it’s best to always keep an eye out for this omission! If the numbers being discussed sound like reciprocal throughput, the often are, even if the author says keeps saying “throughput” alone.

So that's latency and throughput, but before we conclude, I’d like to mention another concept implicit in everything we just did: dependency chains.

Even in the simple laundry example, we already have dependency chains, they’re just very short. Where are they? They’re right here:

Each drying step we did was dependent on its corresponding washing step. Intuitively we knew this: we never tried to dry a load of laundry before washing it. A dependency chain is just the formalization of this concept: that some operations are dependent on others, and these dependencies form “chains” of operations where each operation in the chain can only happen when the previous operations have completed.

I didn’t mention it when we first drew this table, but now that we’re talking about dependency chains, there’s a crucial observation here: if it weren’t for the dependency chains, we could have done this work even faster.

How? Well, in the first hour, the dryer is empty. If we didn’t have to wait for clothes to be washed before drying them, we could have moved the drying step of load C up to the first hour, and done it while load A was being washed! That would have reduced the time from four hours to three hours.

So here we see the true nature of latency, and why it is usually slower than throughput: when we have to wait for some earlier work to be done before we can start some later work, we reduce the amount of work than can be done simultaneously. This leads to idle work units — like our dryer in hour one — which leads to slower performance.

In fact, we could also say that the whole reason our split machines had lower (better) reciprocal throughput than latency was because loads of laundry aren’t dependent, so they can be done in simultaneously. If we lived in some weird universe where there was a law of physics requiring you to include a piece of clothing from your most recent load of laundry in each new load of laundry, we wouldn’t be able to use the split machines to do laundry faster! Since we’d have to wait for load A to be washed and dried completely, so we could take an article of clothing from it and put it in load B before we started washing load B, there would be no way to overlap the operations. Our split machines would be relegated to the same poor performance as the combo machine: two hour per load of laundry, period.

Now, this is a lot of concepts to internalize — but hopefully you can understand now that these aren't weird or imaginary things that only occur inside computer hardware. They're all just basic concepts of work and time that arise naturally in the real world. They work the exact same way, whether you’re trying to optimize a piece of code, or streamline your daily routine.

If you have any questions on these concepts, and you're taking my Performance-Aware Programming Course, please be sure to put them in the comment section of the most recent Q&A post. As always, I answer all the new questions every Monday, and the most recent Q&A post is where I look for new questions.

If you're not taking the course, well, this kind of intuition-building is one of the things we do here! We try to take the complexities of modern computer hardware and software, and make it intuitive through exercises, analogies, and explorations. If that sounds interesting to you, you can subscribe here:

The next video in the series will be for subscribers only, and it will expand on the simple example we worked through here. We’ll take this idea of a “dependency chain” and we’ll apply it to a more complex task to see how we would analyze the more complex dependency chains that result. We’ll do it for an everyday task, just like the laundry — but then we’ll translate that into the CPU equivalent, to prepare us for doing CPU performance analysis of real code.