The critic's night: putting it all together

Friday, ten o'clock. As promised, the guest by the window walks in. Always alone, always one bowl of noodles, always the wristwatch and the small notebook. Tonight, as soon as she sits down, she lays a business card on the table. Food critic.

"I've been watching this kitchen for a few weeks now. Tonight I'll order properly. The twelve-course menu." The critic opens her notebook and takes off her wristwatch.

The kitchen goes quiet. Minho drops his knife, Sua straightens the ticket rail all over again, and Mr. Tak just wipes down the burners. Lou walks in with the ticket. Chef Yoon Seoyeon turns the timer around her neck back to zero. "A kitchen is a program. Tonight we run everything we've built."

12-1. Which tool to reach for (8 min)

Eleven nights have left us with a lot of tools. When a guest walks in, the first question is always the same: "Is this waiting work, or hands-on work?"

Choose in this order:

1. Do you really need concurrency? If doing things one by one is fast enough, leave it. As Lesson 1 showed, concurrency isn't free.
2. Waiting work or computation? Waiting work means asyncio or threads; computation means processes (or 3.14t). On a build with the GIL, splitting computation across threads doesn't make it faster (Lesson 8).
3. Do the libraries you already use support async? If they do, go with asyncio. If not, start with threads, or wrap them with to_thread inside asyncio.
4. What is shared? If there's shared state, add a lock or a queue; if a resource is limited, a semaphore; if things must finish together, a TaskGroup.

12-2. The critic's course handler (14 min)

Here is the critic's ticket. Twelve courses, 0.5 seconds each. But courses 3, 7 and 11 take three times as long (1.5 seconds) the first time, because the ingredients haven't fully thawed. On a second try they take 0.5 seconds. There are only three burners.

You need three parts: a semaphore to enforce the burner count, a time limit so no course drags on, and retries to redo a course that got cut off. All of it goes into one TaskGroup.

import asyncio, time

SLOW = {3, 7, 11}                      # courses that take three times as long the first time

async def plate(n, attempt):
    await asyncio.sleep(0.5 * (3 if n in SLOW and attempt == 0 else 1))

async def order(n, stoves, limit_s, retries):
    for attempt in range(retries + 1):
        async with stoves:             # cook only while holding a burner
            try:
                async with asyncio.timeout(limit_s):
                    await plate(n, attempt)
                return True
            except TimeoutError:
                pass                   # put the burner down, go to the back of the line, try again
    return False

async def service(stoves_n, limit_s, retries):
    stoves = asyncio.Semaphore(stoves_n)
    start = time.perf_counter()
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(order(n, stoves, limit_s, retries)) for n in range(1, 13)]
    served = sum(t.result() for t in tasks)
    print(f"stoves={stoves_n} limit={limit_s} retries={retries}: "
          f"served {served}/12, {time.perf_counter() - start:.1f}s")

asyncio.run(service(3, None, 0))       # no time limit
asyncio.run(service(3, 0.6, 0))        # limit only
asyncio.run(service(3, 0.6, 1))        # limit + one retry
# stoves=3 limit=None retries=0: served 12/12, 3.5s
# stoves=3 limit=0.6 retries=0: served 9/12, 2.2s
# stoves=3 limit=0.6 retries=1: served 12/12, 2.7s
# (times vary by about 0.1 s between runs; the first line shows 3.5s or 3.6s)

• asyncio.timeout(None) means no limit. That's why the same function can run both "no limit" and "0.6-second limit."
• With no limit, the three slow courses each hold a burner for 1.5 seconds, so even though all twelve go out, it takes a little over 3.5 seconds.
• With only a limit, the slow courses are cut off at 0.6 seconds and the burners free up sooner. But those three courses never go out at all.
• With a limit and a retry together, a course that gets cut off puts its burner down and gets back in line. The second time it's done in 0.5 seconds, and all twelve courses go out.

Two points about the design are worth noting.

• A retry puts the burner down first. Because async with stoves sits inside the for, a course that gets cut off releases its burner and lines up again. Retry while still holding the burner, and another course sits idle in the meantime.
• Failure is returned as False, not raised as an exception. When an exception happens in a TaskGroup, all the others get cancelled (Lesson 10). That's right for a group booking, but in a tasting menu the other courses still have to go out even if one fails. For the same reason, collecting them with asyncio.gather would also work.

In practice, you'll often rest a little longer between each retry, as in await asyncio.sleep(0.05 * 2 ** attempt). That keeps everyone from rushing back at the same moment (the same reason as Lesson 5's livelock).

The timeline below runs the same handler on the server. Each row is one course; an attempt cut off by the time limit is gray, and retries are tagged r1, r2. The starting settings are deliberately set so you miss the goal.

• With the starting settings (2 burners, 0.4-second time limit, 0 retries), after about 2.4 seconds you get Served 0 · Failed 12. Even an ordinary course takes 0.5 seconds, but the limit is 0.4, so every course gets cut off.
• Raise the burners to 3 and only the retries to 2, and it gets worse. With the same 0.4-second limit, each course just gets cut off three times, so it's still Served 0, and the time stretches to about 4.9 seconds. Retries don't fix a time limit that's wrong. Even setting the limit to exactly 0.5 seconds, the same as an ordinary course, cuts them off by a hair and they all fail.
• Turn the time limit off and everything is served, but even with 3 burners it takes a little over 3.5 seconds. Four burners bring it to about 3.0 seconds, which is cutting it close, and anyway, there are only three burners.
• The page waits only a little over 5 seconds for the experiment. A slow setup like 2 burners with 2 retries may look cut off partway, so change the settings and run it again.

12-3. Everything in one kitchen (8 min)

The critic's course has dough (computation), bread from the old oven (blocking) and burner dishes (waiting) all together. Hand each one to the right person, and Lou on the floor waits for them all. It uses processes, so save it to a file and run it.

import asyncio, threading, time
from concurrent.futures import ProcessPoolExecutor

def knead(n):                          # dough: hands-on work → branch kitchen (process)
    return sum(i * i for i in range(n)) % 97

tally_lock = threading.Lock()          # a ledger both the oven thread and the loop update
tally = {"plates": 0}

def count_plate():
    with tally_lock:                   # no await and nothing long inside the lock
        tally["plates"] += 1

def old_oven(item):                    # the old oven: blocking → thread
    time.sleep(0.4)
    count_plate()
    return f"{item} baked"

async def stove_dish(n, stoves):       # burner dishes: semaphore
    async with stoves:
        await asyncio.sleep(0.2)
    count_plate()
    return f"#{n}"

async def main():
    loop = asyncio.get_running_loop()
    stoves = asyncio.Semaphore(3)
    with ProcessPoolExecutor(max_workers=2) as dough_room:
        async with asyncio.timeout(5):         # the deadline for the whole course
            results = await asyncio.gather(
                loop.run_in_executor(dough_room, knead, 3_000_000),
                asyncio.to_thread(old_oven, "bread"),
                *(stove_dish(n, stoves) for n in range(1, 7)),
            )
    print(results)
    print(tally)

if __name__ == "__main__":
    asyncio.run(main())
# [56, 'bread baked', '#1', '#2', '#3', '#4', '#5', '#6']
# {'plates': 7}

The ledger is updated from both the oven thread and the loop, so it uses a threading.Lock. It's fine for the loop side to take this lock because there's no await inside it: it changes one number and lets go right away. The accident from Lesson 11 happens when you yield with await while holding the lock.

12-4. The concurrency bug checklist (6 min)

So the kitchen doesn't freeze in front of the critic, check these six things before service.

Here is the fix for "tasks with no reference": keep the tasks in a set so they stay referenced, and have each one remove itself when it's done.

import asyncio

running = set()                        # where the tasks are held

async def wipe(table):
    await asyncio.sleep(0.05)
    print("wiped", table)

async def main():
    for t in ("T1", "T2"):
        task = asyncio.create_task(wipe(t))
        running.add(task)
        task.add_done_callback(running.discard)   # removes itself when done
    await asyncio.sleep(0.1)
    print(len(running))

asyncio.run(main())
# wiped T1
# wiped T2
# 0

"A queue that never ends" happens if you miss even one task_done(). When you suspect it, put a time limit on join() to check.

import asyncio

async def cook(q):
    while True:
        await q.get()
        await asyncio.sleep(0.01)
        # q.task_done()  ← forgot this

async def main():
    q = asyncio.Queue()
    for n in range(3):
        q.put_nowait(n)
    worker = asyncio.create_task(cook(q))
    try:
        async with asyncio.timeout(0.3):
            await q.join()
    except TimeoutError:
        print("join never finished:", q.qsize(), "in queue")
    worker.cancel()

asyncio.run(main())
# join never finished: 0 in queue

The queue is empty (0 in queue), yet join() never finishes. The three tickets that were taken out never got marked "done."

12-5. Running eleven nights' kitchens again (4 min)

The kitchens from past nights are gathered into one lab. Run the same eight orders, changing only the mode. The cooks and burners sliders only have an effect in the modes that use them.

• Alone, one by one: a single row, about 2.4 seconds (Lesson 1).
• 4 cooks: four overlapping rows, about 0.6 seconds (Lesson 2). Limit the burners to 2 and yellow waiting appears, about 1.2 seconds (Lesson 4).
• Agency cooks: the later the order number, the shorter the order, so the returned order gets jumbled; about 0.4 seconds (Lesson 7).
• gather: all eight tables overlap, about 0.3 seconds (Lesson 9). Whatever the number of cooks, with Lou alone.

Closing time (10 min)

The settings that nail the critic's course are 3 burners, a 0.6-second time limit, 1 retry. These values were checked by running them on the server's lab engine.

# with the service function from 12-2
asyncio.run(service(3, 0.6, 1))
# stoves=3 limit=0.6 retries=1: served 12/12, 2.7s

• A 0.6-second limit lets ordinary courses (0.5 seconds) through and cuts off only the courses that drag the first time (1.5 seconds), at 0.6 seconds.
• The three courses that got cut off put their burners down, try once more from the back of the line, and finish in 0.5 seconds.
• Tighten the limit a little to 0.55 seconds and you still hit the goal (about 2.65 seconds). Loosen it to 0.7 and you're at about 2.9 seconds, right up against 3; at 0.8 you go over.

The twelfth course goes out at 2.7 seconds. The chef's timer stops. The critic writes a few lines in her notebook, finishes her bowl of noodles and stands up. At the counter she tears a page out of the notebook and hands it to the chef.

"All twelve courses came out without a single one going cold. There was waiting, but nobody stood around waiting. I'll be back. As a guest, this time."

The critic leaves, and the kitchen lights go off one by one. The chef takes the last recipe card from her apron pocket and hands it to the sous-chef. "That was twelve nights. Tomorrow at ten the doors open again. This time it's your turn to write the ticket. What goes on the first line?"

Looking back on twelve nights

Practice

1. Once you've hit the goal in the lab, cut the orders to 8 and, by changing the time limit and retries, find out whether 2 burners are enough to finish in under 3 seconds.
2. Add a rest between retries (await asyncio.sleep(0.05 * 2 ** attempt)) to the handler from 12-2. Should the rest come before grabbing the burner, or after? Run both and compare the elapsed time.
3. Pick a program you'd like to build and design it with the table from 12-1 and the checklist from 12-4. Just write one line per tool: which tool, and why.