Handing over a bag: functions and arguments

At lunchtime, a startup developer rushes in with a laptop. The first customer ordered just a coffee, but the second customer's receipt printed both the coffee and a bagel.

Supervisor Do Miran puts her finger on one line of the function definition. "When you hand over a bag, you're not handing over the bag. You're adding one more tag. And this empty bag here was bought exactly once, when the function was made."

4-1. A function hangs one more tag on the bag (11 min)

When you call a function, the objects you pass as arguments are not copied. The parameter name inside the function is attached as one more tag to the same object. This is called call by object reference, or call by sharing.

mine = ["laptop"]
def show(bag):
    print(bag is mine)
show(mine)
# True

Since it is just one more tag, the two actions from Lesson 1 apply exactly.

• Changing the bag: if the function changes the contents, as in bag.append(...), the change is visible through the outside name too. It is the same bag.
• Moving the tag: if the function rebinds with bag = ..., only the tag inside the function moves. The outside name stays on the original bag.

def pack(bag):
    bag.append("charger")

def swap(bag):
    bag = ["new"]
    bag.append("charger")

mine = ["laptop"]
pack(mine)
print(mine)
# ['laptop', 'charger']
swap(mine)
print(mine)
# ['laptop', 'charger']

The tag board below shows a call box (frame) for the first time. When the function is called, a pack box appears on the left, and the tag bag inside it points to the same bag as mine outside. When the function finishes, the box disappears.

The += from Lesson 2 comes back here. Depending on the type, the same += is either visible outside or not.

def bump(n):
    n += 1          # int: new object, only the tag inside the function moves

def extend(lst):
    lst += [1]      # list: __iadd__, changes the same bag

count, items = 0, []
bump(count)
extend(items)
print(count, items)
# 0 [1]

Python has no "argument that changes an outside variable". The only ways a function affects the outside are changing a mutable object it was given, or returning a value.

4-2. Default arguments are created once, at definition time (11 min)

Boiled down, the reporter's function looks like this.

def f(x=[]):
    x.append(1)
    return x

print(f())
# [1]
print(f())
# [1, 1]
print(f.__defaults__)
# ([1, 1],)

The default value [] is evaluated once, when the def statement runs, not when the function is called. The list created then hangs on the function object's __defaults__ tuple and stays there as long as the function lives. Every call that leaves out the argument attaches the parameter x to that one and only list.

print(f() is f.__defaults__[0])
# True

Follow the two calls on the tag board below. The call boxes appear and disappear, but the list hanging on defaults stays put and the orders pile up.

The fix is standard. Use None as the default, and create a new list inside the function each time.

def add_order(item, orders=None):
    if orders is None:
        orders = []
    orders.append(item)
    return orders

print(add_order("coffee"), add_order("bagel"))
# ['coffee'] ['bagel']
print(add_order.__defaults__)
# (None,)

If None is a meaningful value and you need to tell it apart from "no argument given", create an object nobody else uses and use it as a sentinel. It is one of a kind, so compare it with is.

_MISSING = object()

def lookup(key, default=_MISSING):
    if default is _MISSING:
        return f"no default for {key}"
    return default

print(lookup("gate"), lookup("gate", None))
# no default for gate None

• [], {} and set() are not the only risky defaults. The rule "evaluated once at def time" applies to every default value alike.
• Defaults for keyword-only arguments hang separately on f.__kwdefaults__.

4-3. Functions that change vs. functions that return something new (7 min)

It helps to think of functions in two groups: functions that change the bag they were given in place, and functions that leave the original alone and return a new bag.

nums = [3, 1, 2]
print(nums.sort(), nums)
# None [1, 2, 3]

nums = [3, 1, 2]
out = sorted(nums)
print(out, nums, out is nums)
# [1, 2, 3] [3, 1, 2] False

Python has a convention: functions and methods that change something in place return None. list.sort(), list.append(), list.reverse(), dict.update() and random.shuffle() all do. It is a promise not to hide the fact that the original was changed.

That is where this mistake comes from.

nums = [3, 1, 2]
nums = nums.sort()
print(nums)
# None

When you write your own functions, pick one of the two. If a function both changes its argument and returns it, callers can easily use the result without realizing the original was changed.

4-4. *args and **kwargs, and defensive copies (7 min)

*args collects the leftover positional arguments into a new tuple, and **kwargs collects the leftover keyword arguments into a new dict. But what goes into them are tags for the objects you passed.

def g(*args, **kwargs):
    return args, kwargs

items = ["a"]
opts = {"tag": items}
a, k = g(items, **opts)
print(type(a), type(k))
# <class 'tuple'> <class 'dict'>
print(a[0] is items, k is opts, k["tag"] is items)
# True False True

Even when you unpack with **opts, the kwargs the function receives is a new dict. So if the function adds a key to kwargs, opts stays the same. The list stored as a value, however, is the same object.

Be especially careful with functions that keep a mutable object they were given. If the caller changes that object later, what was kept changes with it.

def make_flight(code, passengers):
    return {"code": code, "passengers": passengers}

names = ["Ari", "Bo"]
f1 = make_flight("HS1", names)
names.append("Cy")
print(f1["passengers"])
# ['Ari', 'Bo', 'Cy']

In that case, make your own copy the moment you receive it. This is called a defensive copy.

def make_flight(code, passengers):
    return {"code": code, "passengers": list(passengers)}

names = ["Ari", "Bo"]
f1 = make_flight("HS1", names)
names.append("Cy")
print(f1["passengers"])
# ['Ari', 'Bo']

list(passengers) is a shallow copy (Lesson 3). The contents are strings, so that is enough. On the other hand, a function that only reads what it receives for a moment has no need to copy.

4-5. Return values move tags too (4 min)

return does not copy objects either. It hands the bag built inside the function to the caller, and the caller's name hangs a tag on that bag. Even when the function's box disappears, the bag survives because the outside tag holds it.

def build():
    bag = ["boarding pass"]
    return bag

b = build()
print(b)
# ['boarding pass']

return a, b, which looks like returning several values, returns a single tuple. When the caller unpacks it, a tag is attached to each item.

def stats(xs):
    return min(xs), max(xs)

res = stats([4, 9, 2])
print(res, type(res))
# (2, 9) <class 'tuple'>
lo, hi = stats([4, 9, 2])
print(lo, hi)
# 2 9

A function without return returns None. That is also why the "functions that change" in 4-3 return None.

Case closed (10 min)

Here is the reporter's function and the fixed version.

def new_order(item, cart=[]):
    cart.append(item)
    return cart

print(new_order("coffee"))
# ['coffee']
print(new_order("bagel"))
# ['coffee', 'bagel']
print(new_order.__defaults__)
# (['coffee', 'bagel'],)

def new_order(item, cart=None):
    if cart is None:
        cart = []
    cart.append(item)
    return cart

print(new_order("coffee"), new_order("bagel"))
# ['coffee'] ['bagel']

The old orders were inside the list hanging on the function object's __defaults__. The server defined the function once and kept calling it, so that list collected every customer's orders until the server shut down.

After the developer leaves, the supervisor adds one more thing. "Tags fall off when a function ends, but the bags a function object holds on to don't. A function is a bag too."

Exercises

1. If def add(x, seen=set()) does seen.add(x) and then returns len(seen), predict the results of add(1), add(2), add(1), and check add.__defaults__.
2. What effect do def clear(lst): lst = [] and def clear(lst): lst.clear() each have on the list outside?
3. Write a function that takes a list and returns a new sorted list without touching the original, in two ways (sorted, and copy then sort).