The checkpoint: how attribute access really works

To stop negative weights, Oh made Bag check the weight when a bag is created. Yet the records contain a bag weighing -10 kg. Supervisor Do Miran flips through the entry log. "We put a checkpoint at the main gate, and it came in through the side door."

Today we look at the checkpoint Python passes through the moment obj.x is read or written. It is also the full picture of the reading rule we called "instance, then class" in Lesson 7.

10-1. @property: putting a checkpoint on an attribute (9 min)

Here is the reporter's code. The check exists only in __init__.

class Bag:
    def __init__(self, kg):
        if kg < 0:
            raise ValueError("negative weight")
        self.kg = kg

b = Bag(20)
b.kg -= 30
print(b.kg)
# -10

The bag passed the check when it was created, but the later b.kg -= 30 writes straight into the instance __dict__. That is the side door. To put a check on writing the attribute itself, use @property.

class Bag:
    def __init__(self, kg):
        self.kg = kg
    @property
    def kg(self):
        return self._kg
    @kg.setter
    def kg(self, value):
        if value < 0:
            raise ValueError(f"negative weight: {value}")
        self._kg = value

b = Bag(20)
print(b.kg, vars(b))
# 20 {'_kg': 20}
b.kg -= 30
# ValueError: negative weight: -10

• Reading b.kg calls the first kg function, and writing b.kg = v calls the @kg.setter function. From the outside it is used like a plain attribute.
• self.kg = kg in __init__ goes through the setter too. The check needs to live in just one place.
• The actual value is stored in _kg. The name kg is a property object that lives on the class, not on the instance.

Without a setter, the attribute is read-only. That suits values computed from other attributes.

class Box:
    def __init__(self, w, h):
        self.w, self.h = w, h
    @property
    def area(self):
        return self.w * self.h

x = Box(3, 4)
print(x.area, type(Box.__dict__["area"]).__name__)
# 12 property
x.area = 5
# AttributeError: property 'area' of 'Box' object has no setter

10-2. Descriptors: putting the checkpoint in the spec (10 min)

The reporter's bag has not only a weight (kg) but also a length (cm). Copying a property for every attribute multiplies the checking code. Instead, you can build one checkpoint and install it on several attributes. That is a descriptor.

A descriptor is an object with at least one of __get__ and __set__ (or __delete__). Put it on a class as a class attribute, and Python calls those methods whenever that name is read or written on an instance.

class Positive:
    def __set_name__(self, owner, name):
        print(f"set_name: {owner.__name__}.{name}")
        self.name = name
    def __get__(self, obj, owner=None):
        if obj is None:
            return self
        return obj.__dict__[self.name]
    def __set__(self, obj, value):
        if value < 0:
            raise ValueError(f"{self.name} must be >= 0: {value}")
        obj.__dict__[self.name] = value

class Bag:
    kg = Positive()
    cm = Positive()
    def __init__(self, kg, cm):
        self.kg, self.cm = kg, cm
# set_name: Bag.kg
# set_name: Bag.cm

b = Bag(20, 55)
print(b.kg, b.cm, vars(b))
# 20 55 {'kg': 20, 'cm': 55}
print(type(Bag.kg).__name__)
# Positive
b.cm = -1
# ValueError: cm must be >= 0: -1

• __set_name__ (3.6) tells the descriptor the name it was attached to, the moment the class statement ends. That is why the same Positive can be installed on both kg and cm.
• There is only one checkpoint object on the class, shared by every bag. So the values are written not on the checkpoint but in each bag's __dict__.
• Even though the value went into __dict__ under the same name "kg", reading it goes through the checkpoint rather than __dict__. That ordering is the topic of 10-4.
• @property is also a descriptor, one Python ships ready-made.

10-3. How a function becomes a method (7 min)

In Lesson 7 we saw that bag.m is bag.m is False. The reason is that functions are descriptors too. A function has __get__ but no __set__, which makes it a non-data descriptor.

class Bag:
    def weigh(self):
        return 20

b = Bag()
f = Bag.__dict__["weigh"]
print(type(f).__name__, hasattr(f, "__get__"), hasattr(f, "__set__"))
# function True False
m = f.__get__(b, Bag)
print(type(m).__name__, m.__self__ is b, m())
# method True 20
print(b.weigh is b.weigh, b.weigh == b.weigh, Bag.weigh is f)
# False True True

Every time b.weigh is read, Python calls f.__get__(b, Bag) and builds a new bound method: a small object that bundles the function with b. So is is False and == is True. This bundle is also why b.weigh() can be called without passing self. Read from the class (Bag.weigh), there is no instance to bundle, so the plain function comes out.

staticmethod and classmethod are descriptors whose __get__ changes what gets bundled.

class Bag:
    @staticmethod
    def kg_to_lb(kg):
        return round(kg * 2.2046, 1)
    @classmethod
    def empty(cls):
        return cls()

class Suitcase(Bag):
    pass

print(Bag.kg_to_lb(20), Bag().kg_to_lb(20))
# 44.1 44.1
print(type(Suitcase.empty()).__name__, Suitcase.empty.__self__ is Suitcase)
# Suitcase True
print(type(Bag.__dict__["empty"]).__name__)
# classmethod

classmethod receives the class it was called on, so Suitcase.empty() builds a Suitcase. It is used for "alternative constructors" that build the right spec even under inheritance.

10-4. The order of the checkpoint (9 min)

When reading obj.x, Python (precisely, object.__getattribute__) looks in this order.

Data descriptors come before the instance __dict__, so the checkpoint is called before any value inside the bag. That is why it was fine for Positive in 10-2 to store its value in __dict__ under the same name. Functions, on the other hand, come after __dict__, so using the same name on an instance hides the method.

Pick the attributes one at a time in the flow chart below. It shows at which step the answer comes out.

This time we deliberately put the same names into the instance __dict__. The last line is a "side door" that writes straight into __dict__ to dodge the checkpoint.

• A property without a setter is still a data descriptor. A property always has __set__, and without a setter it raises AttributeError inside it. That is why __dict__ cannot hide it.
• Writing (b.x = v) has a shorter order. If the class has a descriptor with __set__, Python calls it; otherwise it writes into the instance __dict__.

10-5. __getattr__, __getattribute__, and type (5 min)

These two similarly named handles are called at different times.

• __getattribute__: called on every attribute read, always. The whole order from 10-4 lives inside object.__getattribute__. Overriding it easily leads to infinite recursion, so it is rarely done.
• __getattr__: the last rescue, called only when the order above found nothing.

class Loud:
    def __init__(self):
        self.kg = 20
    def __getattribute__(self, name):
        print("getattribute", name)
        return super().__getattribute__(name)
    def __getattr__(self, name):
        print("getattr", name)
        return None

x = Loud()
v = x.kg
# getattribute kg
v = x.color
# getattribute color
# getattr color

Finally, if classes are objects, what is the type of a class? It is type. A class statement is really the same as calling type(name, bases, attribute dict).

class Bag:
    kind = "bag"

print(type(Bag()).__name__, type(Bag), type(type))
# Bag <class 'type'> <class 'type'>
Dyn = type("Dyn", (Bag,), {"kind": "dyn"})
print(Dyn.kind, [c.__name__ for c in Dyn.__mro__])
# dyn ['Dyn', 'Bag', 'object']

Inheriting from type to change how classes themselves are created is called a metaclass. For most jobs, __init_subclass__ (Lesson 9) and descriptors are enough, so for this course just remember the name.

Case closed (10 min)

Move the check into the Positive checkpoint, and creating, changing and decreasing with -= all pass through the same checkpoint.

class Positive:
    def __set_name__(self, owner, name):
        self.name = name
    def __get__(self, obj, owner=None):
        return self if obj is None else obj.__dict__[self.name]
    def __set__(self, obj, value):
        if value < 0:
            raise ValueError(f"{self.name} must be >= 0: {value}")
        obj.__dict__[self.name] = value

class Bag:
    kg = Positive()
    cm = Positive()
    def __init__(self, kg, cm):
        self.kg, self.cm = kg, cm

b = Bag(20, 55)
for change in (lambda: setattr(b, "kg", -1), lambda: Bag(-3, 40)):
    try:
        change()
    except ValueError as e:
        print("blocked:", e)
# blocked: kg must be >= 0: -1
# blocked: kg must be >= 0: -3
b.kg -= 30
# ValueError: kg must be >= 0: -10

Setting it with setattr, creating a new bag, decreasing with -=: all are blocked. b.kg -= 30 is two steps, reading b.kg (__get__) and writing back the result (__set__), so there is no way around the checkpoint.

Oh replays the scale records. No more negative weights come through. The supervisor says, "Better than locking doors is making every route go through the checkpoint."

Exercises

1. Change Positive so it takes a lower bound (kg = AtLeast(0), cm = AtLeast(10)). Use the value received in __init__ for the check.
2. In the code for the first flow chart in 10-4, if you add b.weigh = "x", at which step is b.weigh answered? What about b.kg = "x"?
3. In a class that defines only __getattr__, use print to check whether __getattr__ is called when reading an attribute set in __init__.