Standard airport procedures: dunder methods

Nam manages exhibit crates with a hand-written class. The crates look fine, but the conveyor stops the moment one is loaded. Supervisor Do Miran turns a crate all the way around. "The conveyor doesn't look at the shape of a crate. It grabs the handles. This crate doesn't have a single handle."

Python's syntax and built-in functions work by grabbing methods with fixed names on an object. Their names start and end with two underscores, so they are called "dunder (double underscore) methods". Today we fit those handles one by one.

8-1. Syntax calls dunder methods (9 min)

Here is the reporter's crate. Let's ask the same crate to do four things.

class Crate:
    def __init__(self, items):
        self.items = list(items)

c = Crate(["vase", "lamp"])
for expr in ["len(c)", "c[0]", "c + c", "iter(c)"]:
    try:
        eval(expr)
    except TypeError as e:
        print(f"{expr:7} → {e}")
# len(c)  → object of type 'Crate' has no len()
# c[0]    → 'Crate' object is not subscriptable
# c + c   → unsupported operand type(s) for +: 'Crate' and 'Crate'
# iter(c) → 'Crate' object is not iterable

len(c) is not magic. Python calls type(c).__len__(c). That method does not exist, so you get the "has no len()" error. Add __len__ and it works right away.

class Crate:
    def __init__(self, items):
        self.items = list(items)
    def __len__(self):
        return len(self.items)

c = Crate(["vase", "lamp"])
print(len(c), type(c).__len__(c))
# 2 2
c.__len__ = lambda: 99
print(len(c), c.__len__())
# 2 99

The last two lines matter. Even with __len__ attached to the instance, len(c) is 2. That is because the dunder methods called by syntax are looked up on the type (class), not the instance. The conveyor follows only the procedures written in the spec.

A class with these handles fitted well is often called a "Pythonic class". Nobody needs to memorize separate size() or merge() methods; it works with the len and + everyone already knows.

The switchboard below feeds one line of syntax to a Crate with handles fitted, and shows the methods Python actually called, in order.

• bool(c) has no __bool__, so it asks __len__ instead. A nonzero length means true.
• Why for x in c calls __getitem__ is covered in 8-3.

8-2. __repr__ and __str__ (7 min)

When the reporter printed a crate to debug it, all that came out was this.

class Crate:
    def __init__(self, items):
        self.items = list(items)

print(Crate(["vase"]))
# e.g.: <__main__.Crate object at 0x000001F4C2A8B2C0>

The default __repr__ inherited from object gives only the type name and an address. You cannot tell what is in the crate. There are two handles for turning an object into text.

class Crate:
    def __init__(self, items):
        self.items = list(items)
    def __repr__(self):
        return f"Crate({self.items})"
    def __str__(self):
        return f"crate of {len(self.items)}"

c = Crate(["vase", "lamp"])
print(c)
# crate of 2
print(repr(c))
# Crate(['vase', 'lamp'])
print([c])
# [Crate(['vase', 'lamp'])]
print(f"{c} / {c!r}")
# crate of 2 / Crate(['vase', 'lamp'])

• print([c]) used __repr__, not __str__. When a list is turned into text, its elements are always written with repr.
• Without __str__, str() and print() use __repr__. Not the other way around. So if you add only one, start with __repr__.
• The variables pane of a debugger, lists printed in logs, values inside error messages: they are all repr. What you see while chasing a bug is __repr__.

8-3. The container protocol (9 min)

Five handles are enough for a crate to behave like a container: __len__, __getitem__, __iter__, __contains__ and __bool__. Yet in 8-1, for worked even without __iter__.

class OldCrate:
    def __init__(self, items):
        self.items = list(items)
    def __getitem__(self, i):
        print("getitem", i)
        return self.items[i]

for x in OldCrate(["vase", "lamp"]):
    pass
# getitem 0
# getitem 1
# getitem 2
print("lamp" in OldCrate(["vase", "lamp"]))
# getitem 0
# getitem 1
# True

Without __iter__, iter() builds an iterator that calls __getitem__ with 0, 1, 2 and so on. When it gets an IndexError, it takes that as "the end" and stops quietly. That is why it printed up to getitem 2 and finished without an error. Without __contains__, in takes items out one by one this way and compares them with ==. It stops as soon as it finds a match.

This route exists for old-style compatibility. For new classes, fit the proper handles.

class Crate:
    def __init__(self, items):
        self.items = list(items)
    def __len__(self):
        return len(self.items)
    def __iter__(self):
        yield from self.items
    def __contains__(self, x):
        return x in self.items

c, e = Crate(["vase", "lamp"]), Crate([])
print(list(c), "lamp" in c, bool(c), bool(e))
# ['vase', 'lamp'] True True False
if not e:
    print("empty crate")
# empty crate

• __iter__ must return an iterator. return iter(self.items) works too; using yield as above makes the function a generator, which builds the iterator for you. Generators are covered in more depth in the generators and decorators course.
• With __contains__, in does not iterate; it asks that method directly.
• Without __bool__, truthiness is decided by whether __len__ is 0. Without either, an object is always true. So once Crate has just __len__, if not e: works naturally for an empty crate.

8-4. Operators: __add__, __radd__, NotImplemented (8 min)

a + b unfolds in steps.

1. Python calls type(a).__add__(a, b).
2. If that returns NotImplemented, it means "I don't know how to handle this pair". Python hands it to the other side, type(b).__radd__(b, a).
3. If both return NotImplemented, you get a TypeError.

class Crate:
    def __init__(self, items):
        self.items = list(items)
    def __repr__(self):
        return f"Crate({self.items})"
    def __add__(self, other):
        if not isinstance(other, Crate):
            return NotImplemented
        return Crate(self.items + other.items)
    def __radd__(self, other):
        if other == 0:
            return self
        return NotImplemented

c, d = Crate(["vase"]), Crate(["cup"])
print(c + d)
# Crate(['vase', 'cup'])
print(sum([c, d]))
# Crate(['vase', 'cup'])
c + 1
# TypeError: unsupported operand type(s) for +: 'Crate' and 'int'

• sum() starts at 0 and computes 0 + c first. int.__add__(0, c) knows nothing about crates and returns NotImplemented, so Crate.__radd__(c, 0) is called. Since __radd__ accepts 0, sum works.
• For a pair you don't understand, don't raise an exception: return NotImplemented. That gives the other side its chance. It is not the same as NotImplementedError (an exception).

This is where the two faces of += from Lesson 2 are explained. a += b first looks for __iadd__ (in-place add); if there is none, it turns into a = a + b and moves the tag onto a new object. list has __iadd__; tuple and str do not.

c = Crate(["vase"])
alias = c
c += Crate(["cup"])
print(c is alias, alias)
# False Crate(['vase'])

class Crate2(Crate):
    def __iadd__(self, other):
        self.items += other.items
        return self

c = Crate2(["vase"])
alias = c
c += Crate2(["cup"])
print(c is alias, alias)
# True Crate(['vase', 'cup'])

With __iadd__, the inside of the bag changes and the method returns itself. Python hangs the c tag on that return value again, but it is the same bag, so the tag stays where it was. (class Crate2(Crate) is a spec that inherits from Crate. That is the topic of Lesson 9.)

8-5. __call__, with, @dataclass (7 min)

With __call__, you can call an object like a function. It is useful when you need a function that carries settings around (the same job as a closure from Lesson 6, done with a class).

class Scale:
    def __init__(self, limit):
        self.limit = limit
    def __call__(self, kg):
        return kg <= self.limit

check = Scale(23)
print(check(20), check(30), callable(check))
# True False True

with calls __enter__ and __exit__. Even if an exception is raised inside the block, __exit__ is always called. That is why it is used for cleaning up resources, and we meet it again in Lesson 11.

class Belt:
    def __enter__(self):
        print("belt on")
        return self
    def __exit__(self, exc_type, exc, tb):
        print("belt off", exc_type)
        return False

with Belt() as b:
    print("moving")
# belt on
# moving
# belt off None

If __exit__ returns True, it swallows the exception. In most cases False (or returning nothing) is right.

Handles you always write the same way, like __init__, __repr__ and __eq__, can be generated by @dataclass.

from dataclasses import dataclass

@dataclass
class Tag:
    code: str
    kg: int = 0

t = Tag("ICN", 12)
print(t, t == Tag("ICN", 12))
# Tag(code='ICN', kg=12) True
print(Tag.__hash__)
# None

Look at the last line. When __eq__ is defined, Python sets __hash__ to None. The same is true for classes you write by hand.

class Crate:
    def __init__(self, items):
        self.items = list(items)
    def __eq__(self, other):
        return isinstance(other, Crate) and self.items == other.items

print(Crate(["vase"]) == Crate(["vase"]), Crate.__hash__)
# True None
{Crate(["vase"])}
# TypeError: cannot use 'Crate' as a set element (unhashable type: 'Crate')

If you declare "equal values mean equal objects" while the hash still goes by identity (address), two equal crates would land in different slots of a set. To avoid breaking that promise, Python switches hashing off entirely. The promise between hashing and equality is examined in detail in Lesson 12.

Case closed (10 min)

Here is Crate with every handle fitted.

class Crate:
    def __init__(self, items=()):
        self.items = list(items)
    def __repr__(self):
        return f"Crate({self.items})"
    def __len__(self):
        return len(self.items)
    def __getitem__(self, i):
        return self.items[i]
    def __iter__(self):
        return iter(self.items)
    def __contains__(self, x):
        return x in self.items
    def __add__(self, other):
        if not isinstance(other, Crate):
            return NotImplemented
        return Crate(self.items + other.items)
    def __radd__(self, other):
        return self if other == 0 else NotImplemented

a, b = Crate(["vase", "lamp"]), Crate(["frame"])
print(len(a), a[0], "lamp" in a, [x.upper() for x in a])
# 2 vase True ['VASE', 'LAMP']
print(a + b, sum([a, b]))
# Crate(['vase', 'lamp', 'frame']) Crate(['vase', 'lamp', 'frame'])
print(a[-1], a[:1])
# lamp ['vase']

For a[:1], a slice object was passed to __getitem__, and the list's slice came straight back. If you want a crate back instead, handle that case separately with isinstance(i, slice).

Nam's crate glides quietly along the conveyor. The supervisor says, "Same shape as before. All we did was add handles."

Exercises

1. Add __eq__ to the Crate from "Case closed" and check what happens to {Crate(["vase"])}. Write down why in one sentence.
2. Make a class whose __bool__ always returns False. With __len__ returning 3, predict which one if obj: follows, then check.
3. Change __getitem__ so it returns a Crate when isinstance(i, slice), making a[:1] give Crate(['vase']).