Carry-on bags: closures

Right after the morning announcements, three passengers who should be at three different gates are all standing in front of C3. Seo says three guide functions were created, so there should be three different announcements.

The supervisor holds up a carry-on bag. "Functions carry bags too. The question is whether what's inside is a value or a name tag." Today we open the little bag a function carries around: the closure.

6-1. Functions that return functions (7 min)

As we saw in Lesson 1, functions are objects. You can bind them to names, put them in lists, pass them to other functions and return them.

def make_tag(owner):
    def tag():
        return "owner: " + owner
    return tag

t = make_tag("kim")
print(t())
# owner: kim
print(t.__name__, t.__qualname__)
# tag make_tag.<locals>.tag

• Every call to make_tag runs def tag(): and creates a new function object.
• By the time t() is called, make_tag has already finished. By the rules of Lesson 5, the local name owner in make_tag should have disappeared with it.
• Yet t() remembers "kim". How does the inner function hold on to a name of a function that has finished?

6-2. The carry-on bag: cells and __closure__ (9 min)

When an inner function uses a name from the outer function, Python puts that name into a small bag called a cell. The outer function's box and the inner function point to the same cell. Even after the outer function ends and its box disappears, the inner function holds the cell, so the cell and the value inside it survive. A function that carries such a bundle of cells is a closure.

def make_tag(owner):
    def tag():
        return "owner: " + owner
    return tag

t = make_tag("kim")
print(t.__code__.co_freevars)
# ('owner',)
print(type(t.__closure__[0]).__name__)
# cell
print(t.__closure__[0].cell_contents)
# kim
print(make_tag.__code__.co_cellvars)
# ('owner',)

On the tag board, while make_tag is running, owner in its box is shown with the value inside the cell. After the box disappears, check that the owner row of function tag still holds on to 'kim' through the cell.

• The carry-on bag contains only the names the inner function actually uses. The outer function's other local names are cleaned up when it ends.
• Conversely, whatever goes in stays as long as that function lives. If the inner function uses a big list even once, that list stays along with it.

def make_tag(owner):
    notes = ["memo"] * 1000
    def tag():
        return owner
    return tag

print(make_tag("kim").__code__.co_freevars)
# ('owner',)

6-3. Late binding: why all three passengers went to C3 (10 min)

Here is the reporter's code.

def make_guides(gates):
    guides = []
    for gate in gates:
        guides.append(lambda: gate)
    return guides

gs = make_guides(["A1", "B7", "C3"])
print([g() for g in gs])
# ['C3', 'C3', 'C3']
print(gs[0].__closure__[0] is gs[2].__closure__[0])
# True

• Loops do not create zones (Lesson 5). Through all three rounds, gate is one name inside make_guides, and there is one cell.
• All three lambdas carry that one cell. When the loop ends, the value in the cell is "C3".
• The gate in the body of the lambda opens the cell when it is called. Nothing is written down when it is created. This is called late binding.

Writing it as a list comprehension gives the same result.

fs = [lambda: i for i in range(3)]
print([f() for f in fs])
# [2, 2, 2]
print(fs[0].__closure__[0] is fs[1].__closure__[0])
# True

It happens even without a loop. If you move the name outside after creating the function, the value the function sees changes too.

def make():
    gate = "A1"
    def guide():
        return gate
    gate = "Z9"
    return guide

print(make()())
# Z9

On the tag board below, watch the arrows from the three functions converge on one cell. In the fixed version, each function has its own defaults tuple.

The fix is to store "the value at the moment of creation" in the function itself.

fs = [lambda i=i: i for i in range(3)]
print([f() for f in fs])
# [0, 1, 2]

from functools import partial

def guide(gate):
    return "go to " + gate

guides = [partial(guide, gate) for gate in ["A1", "B7", "C3"]]
print([g() for g in guides])
# ['go to A1', 'go to B7', 'go to C3']

• lambda i=i: i: default arguments are evaluated once, at definition time (Lesson 4). The i of that moment goes into the function's __defaults__.
• functools.partial(guide, gate): creates a new callable that bundles the function with the argument of that moment. Its intent is clearer than the default-argument trick.
• The default-argument approach has a downside: callers can overwrite the value, as in f(5).

6-4. Functions with state: a nonlocal counter (7 min)

Cells are not only for reading. Declared with nonlocal (Lesson 5), the inner function can move the tag inside the cell. Then the function has state that carries over from call to call.

def counter():
    n = 0
    def tick():
        nonlocal n
        n += 1
        return n
    return tick

c = counter()
c()
c()
print(c(), c.__closure__[0].cell_contents)
# 3 3
c2 = counter()
print(c2(), c())
# 1 4

• Every call to counter() creates a new cell. c and c2 know nothing about each other's numbers.
• Without nonlocal n, n += 1 makes n a local name and raises UnboundLocalError (the same cause as in Lesson 5).

You can do the same thing with a class. The state lives in the instance's attributes (Lesson 7).

class Counter:
    def __init__(self):
        self.n = 0
    def tick(self):
        self.n += 1
        return self.n

k = Counter()
k.tick()
print(k.tick(), k.n)
# 2 2

6-5. A taste of decorators (7 min)

A closure that takes a function and returns a new function is called a decorator. The original function is stored in a cell of the inner function.

def log(func):
    def wrapper(*args, **kwargs):
        print("call", func.__name__)
        return func(*args, **kwargs)
    return wrapper

@log
def board(seat):
    return "seat " + seat

print(board("12A"))
# call board
# seat 12A
print(board.__name__)
# wrapper

• @log is the same as board = log(board). The name board now points to the wrapper function, and the original board sits in wrapper's cell func.
• That is why board.__name__ becomes wrapper. That is awkward for debugging and documentation.

One line of functools.wraps copies the original function's name, docstring and so on onto the wrapper.

import functools

def log(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print("call", func.__name__)
        return func(*args, **kwargs)
    return wrapper

@log
def board(seat):
    return "seat " + seat

print(board.__name__, board.__wrapped__.__name__)
# board board

Deeper topics, such as giving a decorator arguments or writing decorators as classes, are covered in the generators and decorators course.

Case closed (10 min)

All it takes is giving each function the gate from the moment it was created.

from functools import partial

def announce(gate):
    return "go to " + gate

def make_guides(gates):
    return [partial(announce, gate) for gate in gates]

gs = make_guides(["A1", "B7", "C3"])
print([g() for g in gs])
# ['go to A1', 'go to B7', 'go to C3']

def make_guides_v2(gates):
    return [lambda gate=gate: "go to " + gate for gate in gates]

print([g() for g in make_guides_v2(["A1", "B7", "C3"])])
# ['go to A1', 'go to B7', 'go to C3']

From the next announcement on, each of the three passengers goes to their own gate. After Seo leaves, the supervisor leafs through the conveyor 13 records and mutters to herself. "Even things from a finished function don't get thrown away if someone keeps them in a carry-on bag. Maybe someone's holding on to those two bags too."

Exercises

1. What happens if you change [lambda: i for i in range(3)] to [lambda: i * 10 for i in range(3)]? Use __closure__ to check how many cells there are.
2. Change counter() so it returns two functions together, tick() and reset(). Check with is whether both use the same cell.
3. Write a decorator count_calls that counts how many times a function has been called. Should the count live in an attribute of wrapper or in a cell? Try both.