Eigenvalues and eigenvectors

Since Lesson 5 we have looked at a matrix as "a rule that moves space". Multiply by a matrix and almost every vector changes direction, and its length changes too. But many matrices hide special directions that do not change. A vector along such a direction stays on the same line after the transformation; only its length gets multiplied.

These special directions are called eigenvectors, and the factors are called eigenvalues. The names sound intimidating, but they are used everywhere. How a search engine ranks the importance of web pages (PageRank), the natural shapes in which a bridge or building vibrates, and principal component analysis (PCA), which we meet in Lesson 15, are all "finding the eigenvectors of some matrix". Eigenvalues also hold the answer to next lesson's question: "what happens if you repeat the same transformation 100 times?"

You caught a glimpse of the "eigenvector dashed lines" in the Lesson 5 lab. In this lesson you will find out for yourself what those lines are, then learn how to compute them by hand and with NumPy.

11-1. Vectors that keep their direction (8 min)

Suppose you multiply a vector $\mathbf{v}$ by a matrix $A$, and the result is exactly some multiple of the original vector.

A (nonzero) $\mathbf{v}$ that satisfies this equation is called an eigenvector of $A$, and the number $\lambda$ is called an eigenvalue. In words, it means this.

"Eigen" is German for "own" or "characteristic". Think of it as each matrix having directions of its own.

For example, let's feed a few vectors to $A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$.

$(1, 0)$ went to $(2, 1)$, so its direction changed. But $(1, 1)$ became 3 times as long in the same direction, and $(1, -1)$ did not change at all. So the eigenvectors of this matrix are the $(1, 1)$ direction (eigenvalue 3) and the $(1, -1)$ direction (eigenvalue 1).

Find them yourself in the lab below. The teal arrow is v and the crimson arrow is Av. At first v = (1.5, 0.5), so it says "Not yet: v and Av are 17° apart". Drag the tip of the teal v until it lines up with the crimson Av on one line, and both arrows turn bold and the message tells you the factor too, like "Eigenvector! Av = 3·v".

Press 'Spin v all the way around' and v travels around a circle of radius 1.5 while the crimson Av traces an ellipse. The points on the circle have been carried onto the ellipse by the matrix. The moments during the spin when v and Av line up on one line are exactly the eigenvector directions. With 'Symmetric', the directions of the ellipse's longest and shortest diameters are the two eigenvectors. You will meet this fact again with SVD in Lesson 14.

Turn on 'Answer (2×2 formula)' and the eigenvector directions are drawn as yellow dotted lines, along with the λ values and the eigenvectors scaled to length 1. How to compute this answer is the topic of 11-3.

One point worth stressing: change v from (1, 1) to (2, 2) or (−1, −1) and it is still an eigenvector. It is more accurate to think of an eigenvector not as "one arrow" but as one line through the origin. Every vector on that line except the zero vector is an eigenvector. The zero vector is left out because A0 = λ0 holds for any λ at all, so it tells you nothing.

11-2. What the eigenvalue λ tells you (5 min)

Along an eigenvector direction, the matrix does only one thing: "multiply by $\lambda$". So the value of $\lambda$ alone tells you what happens in that direction.

The $\lambda = 0$ row is especially important. Since $A\mathbf{v} = 0\mathbf{v} = \mathbf{0}$, an eigenvector with eigenvalue 0 is a direction of the null space you saw in Lesson 9. So if 0 is an eigenvalue, then $\det A = 0$, and there is no inverse (Lesson 7).

The lab's status message says "Eigenvector!" whenever v and Av lie on one line, whether they point the same way or opposite ways. That is because a direction with a negative eigenvalue is an eigenvector too.

11-3. Finding eigenvalues with the characteristic equation (10 min)

Spinning v around in the lab gives you intuition, but to get exact values you need to calculate. The key is to tie together what you learned in earlier lessons into a single line.

Step 1: move everything to one side. In $A\mathbf{v} = \lambda\mathbf{v}$, move the right side to the left. Writing $\lambda\mathbf{v}$ as $\lambda I\mathbf{v}$ lets you factor out $\mathbf{v}$.

Step 2: read it as a picture. A nonzero $\mathbf{v}$ went through the matrix $A - \lambda I$ and vanished into the origin. That means $A - \lambda I$ squashes the plane flat (Lesson 9: the null space is more than just the origin). A matrix that squashes flat has determinant 0 (Lesson 7).

This is called the characteristic equation. The only unknown is $\lambda$, so you can solve it.

Step 3: compute it for 2×2. If $A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$, then $A - \lambda I = \begin{bmatrix} a - \lambda & b \\ c & d - \lambda \end{bmatrix}$, and the formula $ad - bc$ from Lesson 7 gives this.

$a + d$, the sum of the diagonal entries, is called the trace, and $ad - bc$ is the determinant. So the two eigenvalues of a 2×2 matrix add up to the trace and multiply to the determinant. This is very handy for checking your work.

Example: $A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$. The trace is 4 and the determinant is $4 - 1 = 3$.

Step 4: find an eigenvector for each eigenvalue. Plug in $\lambda_1 = 3$ and you get $(A - 3I)\mathbf{v} = \mathbf{0}$, a system of equations (Lesson 8).

The two rows are the same equation, so there are infinitely many solutions (the "infinitely many" case from Lesson 8). That is how it should be, because an eigenvector is a whole line. Pick $\mathbf{v}_1 = (1, 1)$ as a representative. In the same way, $\lambda_2 = 1$ gives $x + y = 0$, so $\mathbf{v}_2 = (1, -1)$. This matches exactly what you got by multiplying directly in 11-1.

Check: $3 + 1 = 4$ (the trace), $3 \times 1 = 3$ (the determinant). Correct.

Triangular matrices are even easier. For the upper triangular $\begin{bmatrix} 3 & 1 \\ 0 & 2 \end{bmatrix}$, the characteristic equation is $(3 - \lambda)(2 - \lambda) - 1 \cdot 0 = 0$, so the eigenvalues are just the diagonal entries, 3 and 2.

11-4. No eigenvectors, only one, or perpendicular ones (5 min)

The four matrices in the lab were picked on purpose to behave differently. Switch the chips one by one and compare with the table below.

Rotation. A 90° rotation turns every vector by 90°, so no vector can keep its direction. However you turn v in the lab, the message stays at "90° apart", and the path of Av is a circle, not an ellipse. The characteristic equation $\lambda^2 + 1 = 0$ has no real solutions; you need the imaginary number $i$ to solve it, giving $\lambda = \pm i$. So you can read it this way: when the eigenvalues come out complex, the matrix has a rotation in it.

Shear. A horizontal shear does not move the points on the x-axis. So the $(1, 0)$ direction is an eigenvector with eigenvalue 1. Every other direction gets tilted to the right, though, so there is only one eigenvector line. A matrix like this, where the eigenvalue 1 appears "twice" but there is only one line, cannot describe the whole plane with eigenvectors alone. It is the classic case where next lesson's diagonalization does not work.

Symmetric. A matrix that equals its own $A^\mathsf{T}$ (the transpose from Lesson 5) is called a symmetric matrix. A symmetric matrix always has real eigenvalues, and eigenvectors for different eigenvalues are always perpendicular. You can check it: $(1, 1) \cdot (1, -1) = 1 - 1 = 0$. Covariance matrices from data are symmetric, which is why PCA in Lesson 15 relies on exactly this property.

11-5. Eigenvalues and eigenvectors with NumPy (12 min)

Reading the lab's NumPy panel

The "Check it with NumPy" panel under the lab starts out with this code. Type it in line by line as you read along.

>>> import numpy as np
>>> A = np.array([[2, 1], [1, 2]])
>>> v = np.array([1.5, 0.5])
>>> A @ v
array([3.5, 2.5])

A @ v is the matrix-vector product from Lesson 5. (1.5, 0.5) went to (3.5, 2.5). Since 3.5 / 1.5 and 2.5 / 0.5 are different, it was not "stretched by one common factor", so it is not an eigenvector.

The eigenvalues and eigenvectors come out together from a single call to np.linalg.eig. There are two results, so you receive them under two names, as in vals, vecs = … (Python's tuple unpacking). The lab's NumPy panel wraps the next lines in np.round(…, 6) to make them easier to read, rounding to six decimal places.

>>> vals, vecs = np.linalg.eig(A)
>>> np.round(vals, 6)
array([3.+0.j, 1.+0.j])
>>> np.round(vecs, 6)
array([[ 0.707107+0.j, -0.707107+0.j],
       [ 0.707107+0.j,  0.707107+0.j]])

np.round(array, 6) gives a new array with every element rounded to six decimal places. The original vals and vecs are not changed. Here are the unrounded values.

>>> vals
array([3.+0.j, 1.+0.j])
>>> vecs
array([[ 0.70710678+0.j, -0.70710678+0.j],
       [ 0.70710678+0.j,  0.70710678+0.j]])

There are three new things here.

① +0.j: complex numbers. Python writes the imaginary unit as j. 3.+0.j means "real part 3, imaginary part 0", which is just 3. NumPy 2.5's eig does not assume the matrix is symmetric, so it allows for complex eigenvalues like a rotation matrix's and always returns the answer as a complex array. If you only want the real part, add .real.

>>> vals.dtype
dtype('complex128')
>>> vals.real
array([3., 1.])

② The columns of vecs are the eigenvectors. vecs is a 2×2 matrix; column 0 is the eigenvector for vals[0], and column 1 is the eigenvector for vals[1]. Take them out with M[:, 0] from Lesson 4 (slot 0 of every row = column 0). vecs[0] is row 0, which is something else entirely.

>>> vecs[:, 0]
array([0.70710678+0.j, 0.70710678+0.j])
>>> vecs[0]
array([ 0.70710678+0.j, -0.70710678+0.j])

③ They are scaled to length 1. You got (0.7071…, 0.7071…) instead of the (1, 1) you found by hand because NumPy normalizes eigenvectors into unit vectors (Lesson 3). 0.70710678 is 1/√2. Every vector on the same line is an eigenvector, so divide by the first component and you get back the shape you found by hand.

>>> 1 / np.sqrt(2)
np.float64(0.7071067811865475)
>>> vecs[:, 0] / vecs[0, 0]
array([1.+0.j, 1.+0.j])
>>> np.linalg.norm(vecs[:, 0])
np.float64(0.9999999999999999)
>>> np.isclose(np.linalg.norm(vecs[:, 0]), 1)
np.True_

The length came out as 0.9999999999999999, not exactly 1. That is the floating-point error from Lesson 3. Compare results of real-number calculations with np.isclose instead of ==.

Notice also that column 1 is (−0.707, 0.707) rather than (1, −1), the opposite sign from what you found by hand. It is on the same line, so both are correct answers. The sign of an eigenvector is not fixed. It can differ between programs and versions, so avoid writing code that expects the signs to match.

Checking that Av = λv really holds

The last two lines of the NumPy panel test the definition directly. On the left, A @ vecs[:, 0] is Av; on the right, vals[0] * vecs[:, 0] is λv. The panel wraps both in np.round(…, 6).

>>> np.round(A @ vecs[:, 0], 6)
array([2.12132+0.j, 2.12132+0.j])
>>> np.round(vals[0] * vecs[:, 0], 6)
array([2.12132+0.j, 2.12132+0.j])
>>> A @ vecs[:, 0]
array([2.12132034+0.j, 2.12132034+0.j])
>>> vals[0] * vecs[:, 0]
array([2.12132034+0.j, 2.12132034+0.j])
>>> np.allclose(A @ vecs[:, 0], vals[0] * vecs[:, 0])
True

The rounded 2.12132 (that is 2.121320 with the trailing 0 dropped) and the original 2.12132034 are the same value with a different number of digits. Rather than comparing the two sides by eye, in code you check with np.allclose, as in the last line.

You can also check both eigenvectors at once. vecs * vals uses broadcasting (Lesson 2) to multiply column 0 by vals[0] and column 1 by vals[1]. So it must equal A @ vecs.

>>> np.allclose(A @ vecs, vecs * vals)
True

For symmetric matrices, np.linalg.eigh

When you know the matrix is symmetric, use eigh. It is made only for symmetric matrices, so it is faster and more stable, and it returns a real array with the eigenvalues sorted from smallest to largest (ascending).

>>> vals2, vecs2 = np.linalg.eigh(A)
>>> vals2
array([1., 3.])
>>> vecs2
array([[-0.70710678,  0.70710678],
       [ 0.70710678,  0.70710678]])
>>> vecs2[:, 0] @ vecs2[:, 1]
np.float64(0.0)

eig made no promise about order (this time it was 3, 1), but eigh gives 1, 3. The last line confirms that the dot product of the two eigenvectors is 0, which means they are perpendicular, as 11-4 said. Remember that the largest eigenvalue is at the end (vals2[-1]). PCA in Lesson 15 uses exactly that.

Rotation, shear, upper triangular

>>> R = np.array([[0, -1], [1, 0]])
>>> np.linalg.eigvals(R)
array([0.+1.j, 0.-1.j])

When you only need the eigenvalues, use np.linalg.eigvals. The rotation matrix gives 1.j and -1.j, that is ±i, as 11-4 said. This time the imaginary part is not 0, so you must not throw it away with .real.

>>> S = np.array([[1, 1], [0, 1]])
>>> vals, vecs = np.linalg.eig(S)
>>> vals
array([1.+0.j, 1.+0.j])
>>> vecs
array([[ 1.00000000e+00+0.j, -1.00000000e+00+0.j],
       [ 0.00000000e+00+0.j,  2.22044605e-16+0.j]])

For the shear matrix NumPy returns two columns, but the second is (−1, 0.000…0002), which is practically on the same line as the first. 2.22044605e-16 is 2.22 × 10⁻¹⁶, a zero at the level of floating-point error. Getting two columns from eig does not mean there are two eigenvector directions. You can check whether the two columns are linearly independent with the rank from Lesson 4.

>>> np.linalg.matrix_rank(vecs)
np.int64(1)

The rank is 1, so the eigenvectors form only one line. If you pick 'Shear' in the lab and turn on 'Answer (2×2 formula)', only one yellow dotted line is drawn.

With the upper triangular matrix, let's check "add up to the trace, multiply to the determinant" from 11-3. np.trace is the sum of the diagonal entries.

>>> B = np.array([[3, 1], [0, 2]])
>>> vals = np.linalg.eigvals(B)
>>> vals
array([3.+0.j, 2.+0.j])
>>> np.trace(B)
np.int64(5)
>>> np.linalg.det(B)
np.float64(6.0)
>>> vals.sum()
np.complex128(5+0j)
>>> vals.prod()
np.complex128(6+0j)

np.complex128(5+0j) is also just 5. The sum matches the trace 5, and the product matches the determinant 6.

11-6. Common mistakes (4 min)

Taking an eigenvector as a row. vecs[0] is not an eigenvector. Eigenvectors are columns, so it is vecs[:, 0]. As you saw above, with a symmetric matrix vecs[0] happens to produce plausible-looking numbers, which makes the mistake hard to notice. When in doubt, check with np.allclose(A @ v, lam * v).

Using eigh on a matrix that is not symmetric. eigh trusts that the matrix is symmetric and reads only its lower triangle. Give it a non-symmetric matrix and it returns a wrong answer without any error. The true eigenvalues of the matrix below are 5 and 2, but eigh gives something unrelated.

>>> C = np.array([[4, 1], [2, 3]])
>>> np.linalg.eigvals(C).real
array([5., 2.])
>>> np.linalg.eigh(C).eigenvalues
array([1.43844719, 5.56155281])

The results of eig and eigh are named tuples, so you can also take them out with .eigenvalues and .eigenvectors. If you are not sure whether a matrix is symmetric, check first with np.allclose(C, C.T), and if it is not, use eig.

Non-square matrices. Eigenvalues ask "does a direction stay the same within the same space", so only square matrices have them.

>>> np.linalg.eig(np.array([[1, 2, 3], [4, 5, 6]]))
Traceback (most recent call last):
  ...
numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square

The tool that does a similar job for non-square matrices is SVD, in Lesson 14.

Exercises (6 min)

1. Set up the characteristic equation of $C = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}$ and find its two eigenvalues. Check them with the trace and determinant.
2. For each eigenvalue from question 1, find one eigenvector by hand, and use NumPy's @ to check that $C\mathbf{v} = \lambda\mathbf{v}$.
3. Without calculating, give the eigenvalues and eigenvectors of the diagonal matrix $D = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}$. What does this matrix do to the plane?

Solutions

Question 1: the trace is $4 + 3 = 7$ and the determinant is $12 - 2 = 10$, so $\lambda^2 - 7\lambda + 10 = 0$, that is $(\lambda - 5)(\lambda - 2) = 0$, giving $\lambda_1 = 5$, $\lambda_2 = 2$. Check: $5 + 2 = 7$, $5 \times 2 = 10$.

Question 2: for $\lambda = 5$, the first row of $(C - 5I)\mathbf{v} = \mathbf{0}$ is $-x + y = 0$, so $(1, 1)$. For $\lambda = 2$, the first row is $2x + y = 0$, so $(1, -2)$.

>>> C @ np.array([1, 1])
array([5, 5])
>>> C @ np.array([1, -2])
array([ 2, -4])
>>> D = np.array([[2, 0], [0, 0.5]])
>>> vals, vecs = np.linalg.eig(D)
>>> vals.real
array([2. , 0.5])
>>> vecs.real
array([[1., 0.],
       [0., 1.]])

(5, 5) = 5·(1, 1) and (2, −4) = 2·(1, −2), so both are eigenvectors.

Question 3: a diagonal matrix "stretches and shrinks along the axes", doubling the x direction and halving the y direction. So the eigenvalues are the diagonal entries 2 and 0.5, and the eigenvectors are the standard basis î = (1, 0) and ĵ = (0, 1). Next lesson turns this around and uses the fact that any matrix looks like a diagonal matrix like this when seen from its eigenvector directions.

In the next lesson you will use the eigenvectors as a basis (Lesson 10) and learn diagonalization, which turns a matrix into "stretching along the axes". Then multiplying by a matrix 100 times becomes raising a few numbers to the 100th power, and you will see why the golden ratio pops out of the Fibonacci sequence.