Linear combinations and span

In Lesson 2 you learned to stretch vectors (scalar multiplication) and to add them (addition). This lesson mixes the two. Stretch each of two vectors by as much as you like and then add them: how far can you get?

Picture a robot that moves with only two buttons. Press button A once and it moves by u; press button B once and it moves by w. Pressing a button half a time, or −2 times, is allowed too. Whether this robot can reach every spot on the plane, or ends up trapped on a single line, depends on which way u and w point.

This question runs through all of linear algebra. The systems of equations in Lesson 8 ask "can I reach this point, and if so, how many presses of each button does it take?", and the rank in Lesson 9 asks "how many dimensions do the reachable places have?" This lesson teaches the vocabulary underneath, and you will build your first matrix in NumPy.

4-1. Linear combinations: a·u + b·w (8 min)

Multiply vectors u and w by numbers a and b and add the results: that is a linear combination of u and w.

a and b can be any real numbers. The name sounds grand, but it is just one scalar multiplication and one addition from Lesson 2. For example, with $\mathbf{u} = (2, 1)$, $\mathbf{w} = (-1, 1)$ and $a = 1$, $b = 1$:

Change a and b and you land on a different point. With $a = 2$, $b = -1$ you get $(4, 2) + (1, -1) = (5, 1)$.

Move the sliders a and b in the lab below. The dashed teal arrow is $a\,\mathbf{u}$, the dashed crimson arrow attached to its tip is $b\,\mathbf{w}$, and the black point is where you land, $a\,\mathbf{u} + b\,\mathbf{w}$. It is the same "tip to tail" you saw in Lesson 2.

Turn on 'Try to land on target T' and a yellow circle T = (−2, 3) appears. Use the sliders to put the black point on T. Set a near 0.3 and b near 2.7 and "You made it!" pops up.

The exact answer is actually a = 1/3, b = 8/3. The sliders move in steps of 0.1, so they cannot hit 1/3 = 0.333… exactly, and the lab counts you as arrived once you are close enough to T. You will learn to find such a and b exactly by hand in Lesson 8, with systems of equations. What matters now is the fact that "with a well-chosen a and b, you can reach T".

4-2. Span: every place you can reach (8 min)

Let a and b range over all real numbers, and collect every point that $a\,\mathbf{u} + b\,\mathbf{w}$ reaches: that collection is the span of u and w. It means "everything u and w can produce".

That is a lot of symbols at first sight, but read one at a time they are not hard. The braces $\{\ \}$ mean "the collection (set) of such things", the vertical bar $\mid$ means "where", and what follows is the condition. Read as a whole: "the collection of vectors that can be written as $a\,\mathbf{u} + b\,\mathbf{w}$, where a and b are any real numbers".

'Leave a trail' in the lab (on by default) drops an indigo dot wherever the black point has been. The longer you move the sliders, the farther the trail spreads. With the defaults u = (2, 1), w = (−1, 1), the whole plane is faintly shaded and the status reads "you can reach the whole plane". The sliders only go from −3 to 3, so the trail cannot cover the screen edge to edge, but if you could make a and b as large as you like, you could reach any point.

Now drag the tip of crimson w to (4, 2). w becomes exactly twice u = (2, 1), and the two vectors lie on one line. The faint shading of the plane disappears, a yellow line appears, and the warning "you can only move along that line" shows up. Since w = 2u,

so in the end all you can make are scalar multiples of u alone. This is where Lesson 2's "scalar multiplication cannot turn a direction" comes in. However you move the sliders, the black point cannot leave the yellow line. And T = (−2, 3) is not on that line, so no a and b can get you there.

To sum up, in two dimensions the span of two vectors is one of three things.

In every case, the span always contains the origin. Set a = b = 0 and you always get the zero vector.

4-3. Linear independence and dependence (8 min)

There are names for what separates the first case of 4-2 from the second.

• Linearly independent: none of the vectors can be made as a linear combination of the others. For two vectors: "they do not lie on one line".
• Linearly dependent: at least one of them is a linear combination of the others. For two vectors: "one is a scalar multiple of the other".

A dependent vector adds nothing to the span. If w = 2u, then w only points back to places u can already reach, and it opens up no new direction. If the second button of the two-button robot is really the same as pressing the first button twice, the robot is no different from one with a single button.

Books also write the same idea like this. u and w being linearly independent means

If they are dependent, you can also get back to the origin with a and b that are not 0. With w = 2u, $2\,\mathbf{u} - 1\,\mathbf{w} = \mathbf{0}$. In the lab, set w to (4, 2) and the sliders to a = 2, b = −1, and you will see the black point sit on the origin.

What about three vectors in two dimensions?

u = (2, 1) and w = (−1, 1) already span the whole plane. What happens if you add a third vector v = (4, 5)? v is a point on the plane too, so u and w can already reach it. In fact,

so v = 3u + 2w. The third vector opens no new direction, so the three vectors are linearly dependent. The same is true whatever v you pick. In two dimensions, three or more vectors are always linearly dependent, because the plane only has two independent directions. In three dimensions you can have up to three independent directions, and from the fourth vector on, there is always one too many.

4-4. A first taste of bases: î and ĵ (5 min)

A set of vectors that spans the whole plane and is linearly independent is called a basis. It means "you can reach the whole plane, and no vector is wastefully doubled up". In two dimensions a basis always has two vectors. The u = (2, 1), w = (−1, 1) from above is a basis too.

The most common basis is the pair of vectors one step along the x-axis and one step along the y-axis.

These two are the standard basis, read "i hat" and "j hat". Some books write them as $\mathbf{e}_1, \mathbf{e}_2$. The coordinates you have used since Lesson 1 were secretly linear combinations of the standard basis.

"3 steps right, 2 steps up" is the same as "î three times, ĵ twice". Then you could also use a different basis as your ruler and measure coordinates with it. That story continues in Lesson 10, "Changing basis". And in Lesson 5 you will see the whole plane move just by sending î and ĵ somewhere else.

Here is the new notation in one place.

4-5. NumPy: stacking columns into a matrix (12 min)

Let's follow the lab's "Check it with NumPy" panel from the top. One new function shows up.

>>> import numpy as np
>>> u = np.array([2, 1])
>>> w = np.array([-1, 1])
>>> M = np.column_stack([u, w])
>>> M
array([[ 2, -1],
       [ 1,  1]])

np.column_stack stands vectors up as vertical pillars (columns) and puts them side by side. Column 1 is u = (2, 1) and column 2 is w = (−1, 1). Numbers laid out in a table of rows and columns like this are called a matrix. The horizontal lines are rows and the vertical lines are columns.

What is printed has two layers of square brackets. Read it as two rows, [ 2, -1] and [ 1, 1], inside the outer brackets. Since it is written row by row, you have to read the columns (u, w) vertically: the first column is 2 and 1, the second is −1 and 1.

Shape and indexing of 2-D arrays

The vectors from Lesson 1 were 1-D arrays, numbers in a single row. A matrix is a 2-D array with two axes.

>>> M.shape
(2, 2)
>>> M.ndim
2

(2, 2) means "2 rows, 2 columns". It is always in the order (number of rows, number of columns). To take out a single entry, you also write M[row, column] in that order, counting from 0 as in Lesson 1.

>>> M[0, 1]
np.int64(-1)
>>> M[1, 0]
np.int64(1)

M[0, 1] is the entry in row 0 (the first row) and column 1 (the second column), written $M_{12}$ in a book. You can also take out a whole row or column. A colon : means "all of that axis".

>>> M[:, 0]
array([2, 1])
>>> M[:, 1]
array([-1,  1])
>>> M[1]
array([1, 1])

M[:, 0] means "column 0 of every row", that is, the first column, so you get u back unchanged. Write a single number, as in M[1], and you get that row. Remember: to take a column, always use M[:, j].

M @ [a, b] is a·u + b·w

The next two lines of the lab's NumPy panel compute the same result in two different ways.

>>> M @ np.array([1, 1])
array([1, 2])
>>> 1 * u + 1 * w
array([1, 2])

In Lesson 3, @ between two 1-D arrays was the dot product. With a matrix on the left and a vector on the right, @ becomes a matrix-vector product, and the result is the matrix's columns stretched by the vector's components and added up: a linear combination.

The columns of M are u and w, so the two lines are the same calculation. This combination of columns reading is the most important way to read a matrix, and in Lesson 5 it will show you a matrix moving space around. The lab's sliders a and b were, in the end, just handles for choosing the [a, b] in M @ [a, b].

Let's also check the target point T = (−2, 3) from 4-1, comparing the exact answer a = 1/3, b = 8/3 with the slider values a = 0.3, b = 2.7.

>>> M @ np.array([1/3, 8/3])
array([-2.,  3.])
>>> np.allclose(M @ np.array([1/3, 8/3]), [-2, 3])
True
>>> M @ np.array([0.3, 2.7])
array([-2.1,  3. ])

1/3 is a decimal that goes on forever, so a tiny floating-point error may be hiding in the result. That is why it was checked with np.allclose, from Lesson 3.

Rank: the dimension of the places you can reach

The last line of the NumPy panel is np.linalg.matrix_rank.

>>> np.linalg.matrix_rank(M)
np.int64(2)

This number is called the rank of the matrix. For now it is enough to read it as "how many dimensions the span of the columns has". 2 means the whole plane (linearly independent), 1 means a single line (linearly dependent), and 0 means just the origin. Build the same matrix as in the lab when you moved w to (4, 2), and the rank drops to 1.

>>> M2 = np.column_stack([u, np.array([4, 2])])
>>> M2
array([[2, 4],
       [1, 2]])
>>> np.linalg.matrix_rank(M2)
np.int64(1)
>>> np.linalg.matrix_rank(np.zeros((2, 2)))
np.int64(0)

np.zeros((2, 2)) is the 2×2 matrix whose entries are all 0. Note that the shape is passed as the tuple (2, 2). Rank comes back in detail in Lesson 9.

Three vectors: a 2×3 matrix

Stack the three vectors from 4-3 as columns and you get a matrix with 2 rows and 3 columns.

>>> v = np.array([4, 5])
>>> M3 = np.column_stack([u, w, v])
>>> M3
array([[ 2, -1,  4],
       [ 1,  1,  5]])
>>> M3.shape
(2, 3)
>>> np.linalg.matrix_rank(M3)
np.int64(2)
>>> M3 @ np.array([3, 2, -1])
array([0, 0])

Even with three columns, the rank is 2. On a 2-D plane it cannot go above 2. The last line computes 3u + 2w − v: coefficients that are not 0 produced the zero vector, which is proof that the three vectors are linearly dependent (the definition from 4-3).

A matrix from the standard basis

Stack î and ĵ as columns and you get a matrix with 1s only on the diagonal. Multiply any vector by this matrix and you get the same vector back, because "3 × î + 2 × ĵ = (3, 2)".

>>> I = np.column_stack([np.array([1, 0]), np.array([0, 1])])
>>> I
array([[1, 0],
       [0, 1]])
>>> I @ np.array([3, 2])
array([3, 2])

You will meet this matrix again in Lesson 6, under the name identity matrix.

4-6. Common mistakes (4 min)

Thinking np.array([u, w]) stacks columns. Pass a list of vectors to np.array and each vector becomes a row. The result differs from np.column_stack, which stands them up as columns.

>>> R = np.array([u, w])
>>> R
array([[ 2,  1],
       [-1,  1]])
>>> R @ np.array([1, 1])
array([3, 0])

You expected u + w = (1, 2) and got (3, 0). The columns of R are (2, −1) and (1, 1), so it computed a completely different linear combination. Once you build a matrix, get into the habit of printing it and reading the columns vertically to check.

Mixing up row and column order. M[0, 1] is "row 0, column 1". The first number is the row (horizontal), the second is the column (vertical). Write M[0] hoping for a column and you get the first row, not the first column.

>>> M[0]
array([ 2, -1])
>>> M[:, 0]
array([2, 1])

Exercises (5 min)

1. With u = (2, 1) and w = (−1, 1), find a and b so that a·u + b·w = (0, 3). Look for them with the lab's sliders first, then check with M @ np.array([a, b]).
2. Are (1, 2) and (−2, −4) linearly independent? Check with the rank.
3. From the matrix K = np.column_stack([np.array([1, 4]), np.array([2, 5]), np.array([3, 6])]), take out all of column 2 and the entry in row 1, column 3 (K₁₃ in a book).
4. Why are three 2-D vectors always linearly dependent? Explain in one sentence.

Solutions

>>> M @ np.array([1, 2])
array([0, 3])
>>> np.linalg.matrix_rank(np.column_stack([np.array([1, 2]), np.array([-2, -4])]))
np.int64(1)
>>> K = np.column_stack([np.array([1, 4]), np.array([2, 5]), np.array([3, 6])])
>>> K
array([[1, 2, 3],
       [4, 5, 6]])
>>> K[:, 1]
array([2, 5])
>>> K[0, 2]
np.int64(3)

For question 1, a = 1 and b = 2: 2·1 − 2 = 0 and 1 + 2 = 3, so you land on (0, 3). For question 2, (−2, −4) = −2 × (1, 2), so they lie on one line and are linearly dependent, and the rank is 1. For question 3, the book's K₁₃ counts from 1, so in Python it is K[0, 2]. For question 4: "on the plane, two different directions already reach every point, so a third vector can always be made as a linear combination of the first two" (and if the first two lie on one line, those two are already dependent on their own).

In the next lesson you will see a matrix not as "a table of stacked vectors" but as a rule that moves all of space. Today's column-combination reading of M @ [a, b] is the starting point.