The first night's queue: concurrency and parallelism

Ten o'clock on opening night. The doors open and the guests pour in. You put the first order's soup on the heat and stand in front of the pot until it boils. Three minutes later the first bowl goes out, and you start on the second order.

Chef Yoon Seoyeon glances at her timer. "While that soup was boiling, what were your hands doing? Nothing. That's tonight's problem."

1-1. 0.3 seconds per order. What about eight? (6 min)

Let's fake the wait for a pot to boil in Python. time.sleep(0.3) means "wait 0.3 seconds and do nothing." Think of it as three minutes squeezed into 0.3 seconds.

import time

def make_order(n):
    time.sleep(0.3)   # wait for the soup to boil
    return f"#{n}"

start = time.perf_counter()
done = [make_order(n) for n in range(1, 9)]
print(f"{time.perf_counter() - start:.2f}s", done)
# 2.40s ['#1', '#2', '#3', '#4', '#5', '#6', '#7', '#8']

• time.perf_counter() is the clock you use for timing. Read it twice and subtract to get the seconds that passed in between.
• Eight 0.3-second orders done one by one take 0.3 × 8 = 2.4 seconds. Of those 2.4 seconds, the sous-chef spent almost none actually working. It was all waiting.

The last digit may change a little from run to run. Timing results are always approximate.

1-2. Waiting work and hands-on work (8 min)

Kitchen work comes in two kinds.

Concurrency saves a lot of time when there is plenty of waiting work: while one soup boils, you put another pot on. Hands-on work only gets faster with more actual people, and Python has a hidden rule about that. We'll get to it in Lesson 8.

1-3. Concurrency is not parallelism (8 min)

The two words often get mixed up, but they mean different things.

• Concurrency: making progress on several tasks by taking turns, so they appear to move forward together. One cook puts a soup on and stirs another pot in the meantime.
• Parallelism: running several tasks at literally the same instant. Four cooks each knead their own dough at the same time.

Parallelism is one way to get concurrency. In a kitchen full of waiting work, concurrency alone makes things much faster, even with only one pair of hands. On the timeline below, switch between "Alone, one by one" and "Several cooks." Each bar is one order, and the horizontal axis is time.

• With "Alone, one by one," eight bars line up end to end in a single row and take about 2.4 seconds.
• With "Several cooks" set to 3 cooks, three rows overlap and it takes about 0.9 seconds; with 8 cooks, about 0.3 seconds.
• This lab is not an animation. Real threads on the server call time.sleep, and each bar is drawn from the moment that order actually started and finished.

1-4. How much faster do more cooks make it? (5 min)

If every order takes the same time, the total is "the most orders any one cook handles × the time per order."

import math

def estimate(orders, cooks, each=0.3):
    rounds = math.ceil(orders / cooks)   # the most orders a single cook handles
    return round(rounds * each, 2)

for cooks in (1, 3, 4, 8, 16):
    print(cooks, estimate(8, cooks))
# 1 2.4
# 3 0.9
# 4 0.6
# 8 0.3
# 16 0.3

• With 3 cooks, 8 ÷ 3 rounded up is 3 rounds, so 0.9 seconds. That's the same number you saw on the timeline above.
• Once there are more cooks than orders, it stops getting faster. The time for one order (0.3 seconds) is the floor.
• If orders vary in length, the longest order is the floor. However you split the work, you can't finish before the longest single piece does.

1-5. A preview of the three tools (8 min)

Python has three tools for doing several things at once. Tonight is just a taste; from the next lesson on we dig into them one at a time.

import threading

threads = [threading.Thread(target=make_order, args=(n,)) for n in range(1, 9)]
for th in threads:
    th.start()        # eight cooks start at once
for th in threads:
    th.join()         # wait until everyone is done
# about 0.30 s

import asyncio

async def make_order_async(n):
    await asyncio.sleep(0.3)   # look after other orders while waiting
    return f"#{n}"

async def main():
    return await asyncio.gather(*(make_order_async(n) for n in range(1, 9)))

print(asyncio.run(main()))
# ['#1', '#2', '#3', '#4', '#5', '#6', '#7', '#8']  (about 0.31 s)

1-6. Concurrency isn't free (5 min)

More cooks come at a cost.

• Overhead: creating threads and switching between them takes time too. If orders are very short, it can even get slower.
• New bugs: when two people touch the same ingredient at once, the result comes out wrong (Lesson 3). When each waits for a tool the other one holds, the kitchen freezes (Lesson 5).
• Thinking cost: the assumption that code runs one line after another breaks down. That's why this course always checks "what ran when" with your own eyes on a timeline.

Closing time (10 min)

Tonight's goal was to get eight orders out in under a second. Every order is waiting work, so eight cooks taking one each will do it.

start = time.perf_counter()
threads = [threading.Thread(target=make_order, args=(n,)) for n in range(1, 9)]
for th in threads:
    th.start()
for th in threads:
    th.join()
print(f"{time.perf_counter() - start:.2f}s")
# 0.30s

The chef stops her timer. "From 2.4 seconds to 0.3. Not bad for a first night. But once there are eight cooks, who keeps count of how many each one made?"

Exercises

1. On the timeline, set 12 orders and 4 cooks. Work out how long it should take first, then run it.
2. Replace time.sleep(0.3) in make_order with a loop that adds up numbers (hands-on work), run it on threads, and time whether it gets faster. If the result looks odd, Lesson 8 explains why.
3. Write down three pieces of "waiting work" and three of "hands-on work" from your own daily life.