Column space, null space and rank

In Lesson 8 you saw that np.linalg.solve raises an error when det = 0. But an error alone is not enough. You want to know whether there is no solution or infinitely many, and if infinitely many, what those solutions look like.

The answer comes from looking at "what happens when the plane gets squashed" from two sides. One is the arrival side. The place where vectors can end up after the transformation is called the column space. The other is the vanishing side. The directions that get pressed down to the origin and disappear are called the null space. And the number of directions left after squashing is the rank.

You meet these three often when working with data, too. For example, if a table has both a "height (cm)" and a "height (mm)" column, it may have three columns but only two columns' worth of real information. Rank tells you this "gap between how many there seem to be and how much information there really is" in a single number.

9-1. Column space: everywhere Ax can reach (9 min)

In Lesson 5, A @ x was "x·column 1 + y·column 2". Plug in every possible value for x and y, and the results are every linear combination you can make from the two columns. In Lesson 4 we called this the span of the two columns. The place spanned by the columns of a matrix A is called the column space of A.

• If the two columns point in different directions, the column space is the whole plane. You can build any b, so Ax = b can always be solved.
• If the two columns lie on one line, the column space is one line. You can never reach a b that is off that line.
• If both columns are the zero vector, the column space is just the origin.

The lab below is the same plane as in Lesson 5. Drag the tips of î (teal) and ĵ (crimson) to change the matrix, and the rank status shows up on the right. When the rank becomes 1, a thick yellow line (the column space) and a black dashed line (the null space) appear.

The 'Squash' matrix is [[1, 2], [0.5, 1]]. Column 1 is (1, 0.5), column 2 is (2, 1), and column 2 is exactly twice column 1. So the two columns lie on one line, and that line is the yellow 'column space'. Play it and every grid point on the plane gathers onto this yellow line. Whatever x you put in, Ax cannot leave the yellow line.

This is the real reason behind 'No solution' in Lesson 8. In Ax = b, if b is off the yellow line, no x can get there. If b is on the yellow line, there is an x that gets there. In short:

Try 'Squash 2' ([[2, −1], [−1, 0.5]]) as well. This time column 1 (2, −1) and column 2 (−1, 0.5) point in opposite directions, but they still lie on one line (column 2 = −0.5 × column 1), so the rank is 1. "Same direction" or "opposite direction", if they are on one line, they are linearly dependent (Lesson 4).

9-2. Null space: the directions that get squashed away (9 min)

When the plane gets squashed onto a line, two dimensions shrink to one. One direction has disappeared. Where did it go? It got pressed down to the origin.

The set of all vectors x with Ax = 0 is called the null space of A. "Null" here means "sent to zero". In the lab, pick 'Squash' and the black dashed line is the null space. The dashed line points along (−2, 1), and if you check it yourself,

• A (−2, 1) = −2·(1, 0.5) + 1·(2, 1) = (0, 0)

it really does land on the origin. Other vectors on the same line, like (−4, 2) or (1, −0.5), all go to the origin too. The null space is also a line (one direction).

The null space has a few properties.

• The origin is always in the null space. After all, A0 = 0. A matrix of rank 2 (det ≠ 0) sends nothing but the origin to the origin. That is why it can be undone (Lesson 7).
• The null space is perpendicular to the rows of A. Read Ax as dot products with the rows (Lesson 5), and Ax = 0 says "x has dot product 0 with the first row and with the second row". In Lesson 3, a dot product of 0 meant perpendicular. For 'Squash', the dot product of the first row (1, 2) with (−2, 1) is −2 + 2 = 0.
• The zero matrix sends every vector to the origin. Pick 'Zero matrix' and the rank becomes 0, and the null space is the whole plane. The lab draws no lines in this case: the whole plane is the null space, so there is no separate line to draw.

Moving along the null space is a movement A "cannot see". It is like projecting a 3-D scene onto a 2-D screen in graphics, where the information about moving "toward or away from the camera" is lost. That direction is the null space of the projection matrix.

9-3. Rank and counting dimensions (7 min)

The rank is the dimension of the column space, that is, "the number of directions left after the transformation". For a 2×2 matrix, the rank is 0, 1 or 2.

Click through the lab's ready-made matrices one by one and fill in the table, and a curious rule shows up.

The sum is always 2, the number of columns. Each of the two input directions either "survives and builds the column space" or "gets squashed away into the null space". Books call this the rank–nullity theorem. When $A$ has $n$ columns,

It holds just the same for a 3×3 matrix and for a data matrix with 100 columns.

9-4. Why there are infinitely many solutions (6 min)

Now you can explain the equations x − y = 1, 2x − 2y = 2 from 'Infinitely many' in Lesson 8. A = [[1, −1], [2, −2]] and b = (1, 2).

1. There is a solution. The two columns (1, 2) and (−1, −2) lie on one line, and b = (1, 2) is on that line (the column space) too. For example, plug in xₚ = (1, 0) and Axₚ = column 1 = (1, 2) = b.
2. There is a null space. Plug in (1, 1) and you get column 1 + column 2 = (0, 0). The direction (1, 1) is the null space.
3. Adding the two still gives a solution. Add any amount t·(1, 1) of the null space to xₚ, and the added part becomes 0 after A and disappears. So A(xₚ + t·(1, 1)) = b + 0 = b.

t can be any real number, so there are infinitely many solutions. Every solution looks like this:

As a picture, it is "the line through the point (1, 0) running in the direction (1, 1)", and that is the very line x − y = 1 where the two lines overlapped in the lab. If the null space is just the origin (rank 2), there is nothing to add, so there is only one solution. All three cases from Lesson 8 are explained by just two things, the column space and the null space.

9-5. Checking with NumPy (12 min)

Follow the output of the lab panel one line at a time, with 'Squash' selected.

>>> import numpy as np
>>> A = np.array([[1, 2], [0.5, 1]])
>>> np.linalg.det(A)
np.float64(0.0)
>>> np.linalg.matrix_rank(A)
np.int64(1)

np.linalg.matrix_rank(A) returns the rank as an integer. To check that the column space really is a line, put in several x. Every result has a second component half of the first, so it lies along the direction (1, 0.5). Also notice that putting in (−2, 1) gives the origin.

>>> for x in ([1, 0], [0, 1], [3, 1], [-1, 2], [-2, 1]):
...     print(x, A @ np.array(x))
[1, 0] [1.  0.5]
[0, 1] [2. 1.]
[3, 1] [5.  2.5]
[-1, 2] [3.  1.5]
[-2, 1] [0. 0.]

Finding the null space direction with SVD

Instead of finding the null space direction by hand, in NumPy you usually use the singular value decomposition (SVD). You will learn what SVD is, with pictures, in Lesson 14, so for now treat it only as "a tool that takes a matrix and gives back three things".

>>> U, S, Vt = np.linalg.svd(A)
>>> S
array([2.5, 0. ])

np.linalg.svd(A) returns three values at once, and U, S, Vt = … is the Python syntax that splits them into three names. Two of them matter right now.

• S: numbers called singular values, listed from largest to smallest. Each says "how much a direction gets stretched", and the number of nonzero singular values is the rank. Here there is only 2.5, so the rank is 1. matrix_rank counts the same way under the hood.
• Vt: a 2×2 array, where each row is one direction on the input side. It pairs up with the order of S, so the last row Vt[-1] is the direction that gets stretched the least. If its singular value is 0, that direction gets pressed to the origin. In other words, it is the null space direction.

Vt[-1] takes "the last row" using the negative indexing from Lesson 1.

>>> n = Vt[-1]
>>> n
array([-0.89442719,  0.4472136 ])
>>> A @ n
array([0., 0.])
>>> np.allclose(A @ n, 0)
True

n is a unit vector of length 1 (Lesson 3), so it comes out as (−0.894…, 0.447…) rather than (−2, 1). The direction is the same. Divide by the second component and you get the (−2, 1) we found by hand. The lab panel shows the same values rounded to 6 decimal places. Also, SVD does not fix the sign, so depending on your environment it may come out with every sign flipped, like (0.894…, −0.447…). Both point along the same line, so either is correct (Lesson 14).

>>> n / n[1]
array([-2.,  1.])

To check A @ n, compare it with 0, as in np.allclose(A @ n, 0). np.allclose can also compare an array with a single number, so it checks "is every component close enough to 0" in one go.

You can check the rank–nullity theorem with numbers too. A.shape[1] is the number of columns (the second value of shape).

>>> A.shape[1] - np.linalg.matrix_rank(A)
np.int64(1)

Checking for a solution with rank

In Lesson 8, solve could not tell "no solution" apart from "infinitely many". Rank can. Attach b to A as one more column and measure the rank of the result. If b is in the column space, attaching it adds no new direction and the rank stays the same; if it is outside, the rank goes up by one.

>>> A_many = np.array([[1, -1], [2, -2]])
>>> b_many = np.array([1, 2])
>>> np.column_stack([A_many, b_many])
array([[ 1, -1,  1],
       [ 2, -2,  2]])
>>> np.linalg.matrix_rank(A_many), np.linalg.matrix_rank(np.column_stack([A_many, b_many]))
(np.int64(1), np.int64(1))
>>> A_none = np.array([[1, 2], [1, 2]])
>>> b_none = np.array([2, -2])
>>> np.linalg.matrix_rank(A_none), np.linalg.matrix_rank(np.column_stack([A_none, b_none]))
(np.int64(1), np.int64(2))

np.column_stack from Lesson 4 accepts a mix of 2-D and 1-D arrays: it keeps the columns of the 2-D array as they are and attaches the 1-D array as a new column. If the rank stays the same (1, 1), there are infinitely many solutions; if it goes up (1, 2), there is no solution.

Let's also check "one solution + null space" from 9-4. Add t times (1, 1) to xₚ = (1, 0), and the result is always b = (1, 2).

>>> xp = np.array([1, 0])
>>> null_dir = np.array([1, 1])
>>> A_many @ xp, A_many @ null_dir
(array([1, 2]), array([0, 0]))
>>> for t in (-1, 0, 2.5):
...     x = xp + t * null_dir
...     print(x, A_many @ x)
[ 0 -1] [1 2]
[1 0] [1 2]
[3.5 2.5] [1. 2.]

The same in 3×3

The tools stay the same in three dimensions, where you cannot draw. The matrix filled with 1 to 9 in order is a classic example with det 0. The third column is "2 × column 2 − column 1", so the three columns lie on one plane.

>>> M = np.array([[1, 2, 3],
...               [4, 5, 6],
...               [7, 8, 9]])
>>> np.linalg.matrix_rank(M)
np.int64(2)
>>> U, S, Vt = np.linalg.svd(M)
>>> np.round(S, 6)
array([16.848103,  1.06837 ,  0.      ])
>>> n3 = Vt[-1]
>>> np.allclose(M @ n3, 0)
True
>>> n3 / n3[0]
array([ 1., -2.,  1.])
>>> M @ np.array([1, -2, 1])
array([0, 0, 0])

The rank is 2, so the column space is a plane inside 3-D space, and the null space is 3 − 2 = 1-dimensional: the line in the direction (1, −2, 1). The last singular value is actually not 0 but a tiny number like 4.4e-16. That is why it is shown rounded with np.round(S, 6). matrix_rank uses the rule "treat very small singular values as 0" to filter out such leftovers.

Let's try it on a data table too. Here is a 4×3 matrix with the height (cm), height (mm) and weight (kg) of four people. Rows are people, columns are features.

>>> X = np.array([[170, 1700, 60],
...               [160, 1600, 55],
...               [180, 1800, 75],
...               [175, 1750, 70]])
>>> X.shape
(4, 3)
>>> np.linalg.matrix_rank(X)
np.int64(2)
>>> X @ np.array([10, -1, 0])
array([0, 0, 0, 0])

There are three columns, but the rank is 2. Since "height (mm) = 10 × height (cm)", 10·column 1 − 1·column 2 = 0, and (10, −1, 0) is in the null space. It means one feature adds no new information. PCA in Lesson 15 is a method for finding this "number of directions you actually need" in data.

9-6. Common mistakes (3 min)

Judging rank with det == 0. When decimals are involved, det may come out not as exactly 0 but as a tiny number. In the matrix below, column 2 is 0.1 times column 1, so the rank is 1, yet det is not 0.

>>> B = np.array([[3, 0.3], [1, 0.1]])
>>> d = np.linalg.det(B)
>>> d
np.float64(4.1633363423443364e-17)
>>> d == 0
np.False_
>>> np.isclose(d, 0)
np.True_
>>> np.linalg.matrix_rank(B)
np.int64(1)

For matrices like this, np.linalg.solve may not raise an error at all and instead return an absurdly large answer (around 10¹⁶). If you suspect the rank, check it with matrix_rank. (In NumPy 2, comparison results show up as np.True_ rather than True, but you can use them exactly like Python's True and False.)

>>> x_bad = np.linalg.solve(B, np.array([1.0, 2.0]))
>>> np.abs(x_bad).max() > 1e15
np.True_

Trusting Vt[-1] as the null space when the rank is 2. Vt[-1] is just "the direction stretched the least". It is a null space direction only when the last singular value is 0. The lab panel, too, shows n = Vt[-1] only when the rank is less than 2. Use it on a rank-2 matrix like 'Shear' and it does not go to the origin.

>>> shear = np.array([[1, 1], [0, 1]])
>>> np.round(np.linalg.svd(shear).S, 6)
array([1.618034, 0.618034])
>>> np.allclose(shear @ np.linalg.svd(shear).Vh[-1], 0)
False

You can also pull the results of np.linalg.svd(…) out by name. .S is the singular values, and .Vh is the same as Vt above.

Seeing only one direction for the zero matrix. Pick 'Zero matrix' and the lab panel shows just one direction, (0, 1), as n = Vt[-1]. But the null space of the zero matrix is the whole plane, so (0, 1) is only one direction in it. When two singular values are 0, the last two rows of Vt are both null space directions.

Exercises (5 min)

1. In the lab, drag î to (2, 1) and ĵ to (−4, −2). What is the rank, and which directions are the column space and the null space? Check with NumPy.
2. For A = [[1, 2], [2, 4]], use rank to decide whether Ax = b has no solution or infinitely many, for b = (3, 6) and for b = (3, 5).
3. Predict the rank and the dimension of the null space of the 3×3 matrix [[1, 2, 3], [2, 4, 6], [3, 6, 9]], then check.

Solutions

>>> P = np.array([[2, -4], [1, -2]])
>>> np.linalg.matrix_rank(P)
np.int64(1)
>>> P @ np.array([2, 1])
array([0, 0])

For question 1, column 1 of the matrix is where î went, (2, 1), and column 2 is where ĵ went, (−4, −2), so the matrix is [[2, −4], [1, −2]]. Column 2 is −2 times column 1, so the rank is 1, and the column space is the line in the direction (2, 1). The null space is the direction (2, 1) too, since 2·column 1 + 1·column 2 = 0. For this matrix, the column space and the null space are the same line. Apply it once and everything gathers onto the line; apply it twice and everything goes to the origin.

>>> A2 = np.array([[1, 2], [2, 4]])
>>> np.linalg.matrix_rank(np.column_stack([A2, [3, 6]]))
np.int64(1)
>>> np.linalg.matrix_rank(np.column_stack([A2, [3, 5]]))
np.int64(2)
>>> K = np.array([[1, 2, 3], [2, 4, 6], [3, 6, 9]])
>>> np.linalg.matrix_rank(K)
np.int64(1)
>>> K.shape[1] - np.linalg.matrix_rank(K)
np.int64(2)

For question 2, A has rank 1. Attaching (3, 6) keeps the rank the same, so there are infinitely many solutions; attaching (3, 5) raises it to 2, so there is no solution. You can pass the list [3, 6] to np.column_stack as it is. For question 3, every column is a multiple of (1, 2, 3), so the rank is 1 and the null space is 3 − 1 = 2-dimensional (a plane).

In the next lesson you change your point of view and measure the same vector with a different ruler (basis). The coordinates (1, 3) become completely different numbers in a new basis, and finding those numbers is exactly the system of equations from Lesson 8.