Length, angle and the dot product

In Lesson 2 you added and stretched arrows. This lesson measures them. How long is one arrow, and how much do two arrows point the same way?

These questions come up more often than you might think. Whether an enemy is in front of the player or behind, how squarely a light hits a surface, how similar two people's tastes are in a recommendation system: all of these are the same question, "how much do two vectors point the same way?" The calculation that answers it is the dot product.

The dot product itself is nothing but multiplying and adding, so the arithmetic is easy. The goal of this lesson is to see, in pictures, why that simple calculation is tied to angles. At the end you will compute lengths and angles with NumPy, and learn how to deal with the tiny errors that always come with decimal arithmetic on a computer.

3-1. Length: Pythagoras (7 min)

Draw the vector $(3, 4)$ as an arrow and it is the hypotenuse of a right triangle that goes 3 right and 4 up. The Pythagorean theorem from school gives the length of the hypotenuse:

This is the vector's length. Books call it the norm and write it with two vertical bars on each side, $\|\mathbf{v}\|$.

In three dimensions there is just one more component; the shape of the formula is the same. The length of $(1, 2, 2)$ is $\sqrt{1 + 4 + 4} = 3$. In any number of dimensions, square each component, add them all up, and take the square root, and you have the length. So even a 784-dimensional vector, which you cannot draw, has a length.

Length has two properties. Only the zero vector has length 0; every other vector has a positive length. And scaling a vector scales its length by the same amount: $\|c\,\mathbf{v}\| = |c|\,\|\mathbf{v}\|$. Multiplying by a negative number flips the direction, but the length never becomes negative, which is why $c$ gets an absolute value.

The distance between two points also comes from length. In Lesson 2, $\mathbf{u} - \mathbf{w}$ was the arrow from w to u, so the distance between the two points is $\|\mathbf{u} - \mathbf{w}\|$.

3-2. Unit vectors: making the length 1 (4 min)

A vector of length 1 is called a unit vector. A unit vector carries only a "direction". Divide any vector by its own length and you keep its direction while setting its length to 1. This is called normalizing the vector.

The hat (^) on $\hat{\mathbf{v}}$ marks "scaled to length 1", and you read it as "v hat". If you want a game character to move toward a target at 5 units per second, normalize the vector pointing at the target and multiply by 5. Forget to normalize and you get the bug where the character moves faster the farther away the target is.

The zero vector has length 0, and you cannot divide by 0, so it cannot be normalized. You cannot pull a direction out of a vector that has none.

3-3. The dot product: multiply, then add (9 min)

The dot product of two vectors multiplies the components in the same position and then adds all of those up, giving a single number. That the result is a number, not a vector, is what sets it apart from addition and scalar multiplication.

For u = (3, 0) and w = (2, 2), it is 3×2 + 0×2 = 6. Those are exactly the defaults of the lab below. On the right you see the lengths, the dot product calculation and the angle θ between the vectors.

Turn w around and a rule appears.

All you did was multiply and add, so why does it have anything to do with angles? The lab's defaults show the answer. u = (3, 0) lies on the x-axis, so the dot product is $3 \times w_1$, that is, the length of u × the x-component of w. And the x-component of w is the length of w's shadow on the x-axis, which in the triangle is $\|\mathbf{w}\|\cos\theta$. So

This formula holds even when u is not on the x-axis. Rotating both arrows together changes neither their lengths nor the angle between them, and it does not change the dot product either. In the lab, move u to (0, 3) and w to (−2, 2) to turn the default picture by 90°. The dot product is still 6, and the angle is still 45°.

Turn the formula around and you can find the angle.

With the defaults this is $6 / (3 \times 2.828) \approx 0.707$, and the angle whose cosine is 0.707 is 45°. The cosine is 1 at 0°, 0 at 90° and −1 at 180°, so the sign of the dot product tells you "smaller or larger than 90°". That is exactly what the table above says.

The dot product of a vector with itself is its length squared: $\mathbf{v}\cdot\mathbf{v} = v_1^2 + v_2^2 = \|\mathbf{v}\|^2$. Length and the dot product are part of the same family.

3-4. Shadows: projection (6 min)

With 'Show w's shadow (projection onto u)' turned on in the lab, you see a thick yellow arrow and a dashed gray line. This is the shadow w casts on u when light shines down perpendicular to u. The dashed gray line drops from the tip of w perpendicular to the line through u, and the yellow arrow is the shadow. We say w is projected onto u; the shadow is its projection.

The shadow's length already appeared in the triangle in 3-3. It is $\|\mathbf{w}\|\cos\theta$, and with the dot product formula you can get it directly, without the angle.

The lab's "Shadow length = (u · w) / |u|" is this formula. With the defaults it is $6 / 3 = 2$: the shadow of w = (2, 2) falls 2 units along the x-axis. Drag w to the side opposite u and the shadow length turns negative, with the yellow arrow pointing away from u. When they are perpendicular the shadow is 0, shrunk to a single point.

The shadow arrow itself (the yellow vector) is the unit vector in u's direction times the shadow length. Put together, that is $\dfrac{\mathbf{u}\cdot\mathbf{w}}{\mathbf{u}\cdot\mathbf{u}}\,\mathbf{u}$. This formula comes back in Lesson 13, when you look for "the line that best fits a set of points", so for now just remember "a shadow is computed with the dot product".

In a game, this shadow tells you "how far ahead of the player an enemy is". The direction the player faces is u, and the arrow from the player to the enemy is w.

3-5. Cosine similarity: taste vectors (6 min)

A recommendation system writes people's tastes as vectors. For example, a 4-dimensional vector scoring four movie genres (action, romance, comedy, horror) from 0 to 5.

Four dimensions cannot be drawn, but the angle can still be computed. The $\cos\theta$ between two taste vectors is called cosine similarity. Close to 1 means tastes pointing the same way; close to 0 means unrelated tastes. (Since every score is 0 or more, negative values rarely show up.)

There is a reason to divide by the lengths instead of using the dot product as is. Harin's taste has exactly the same proportions as yours, but she is twice as generous with every score. The dot product grows in proportion to length, so the more generous someone's scores, the more "similar" they always look. Dividing by the lengths removes the effect of length and leaves only the direction, that is, the proportions of the taste. That is why your cosine similarity with Harin is exactly 1. You will compute it with NumPy in 3-6.

Search engines and AI models use the same calculation when they turn sentences into vectors with, say, 768 dimensions and look for "sentences with similar meaning".

3-6. Lengths, dot products and angles with NumPy (10 min)

Let's follow the lines in the lab's "Check it with NumPy" panel from the top.

>>> import numpy as np
>>> u = np.array([3, 0])
>>> w = np.array([2, 2])
>>> u @ w
np.int64(6)
>>> np.dot(u, w)
np.int64(6)

@ is the operator Python added for matrix multiplication. Between two 1-D arrays it computes the dot product. np.dot(u, w) is a function that does the same thing. Modern code mostly uses @, which is shorter and easier to read. The result is a single number (np.int64(6)), not an array.

Length comes from np.linalg.norm. linalg is short for linear algebra; it is where NumPy keeps its linear algebra functions.

>>> np.linalg.norm(u)
np.float64(3.0)
>>> np.linalg.norm(w)
np.float64(2.8284271247461903)

Even for an integer vector, the length comes out as a real number (float64) because of the square root. Writing out the Pythagorean formula gives the same answer. v ** 2 is the element-wise operation that squares each component, np.sum adds up all the components, and np.sqrt is the square root.

>>> v = np.array([3, 4])
>>> v ** 2
array([ 9, 16])
>>> np.sqrt(np.sum(v ** 2))
np.float64(5.0)
>>> np.linalg.norm(v)
np.float64(5.0)

Normalizing is the broadcasting you saw in Lesson 2: divide the array by a single number.

>>> v / np.linalg.norm(v)
array([0.6, 0.8])
>>> 5 * v / np.linalg.norm(v)
array([3., 4.])

Now for the remaining three lines of the lab's NumPy panel. Compute the cosine, get the angle back with np.arccos, and convert it to degrees (°) with np.degrees.

>>> cos = u @ w / (np.linalg.norm(u) * np.linalg.norm(w))
>>> cos
np.float64(0.7071067811865475)
>>> np.arccos(cos)
np.float64(0.7853981633974484)
>>> np.degrees(np.arccos(cos))
np.float64(45.00000000000001)

np.arccos (arc cosine) returns "the angle whose cosine is this value", and it gives the result in a unit called radians. Radians measure 180° as π (about 3.14159), so 0.785… is π/4, that is, 45°. np.degrees converts to the friendlier degrees, and np.radians goes the other way, from degrees to radians.

45.00000000000001: tiny errors in decimal arithmetic

The last line came out as 45.00000000000001, not 45. The calculation is not wrong. A computer stores decimals in binary, and numbers like 0.1 or √2 go on forever in binary, so they are cut off at a fixed number of digits (about 16). The tiny bit that was cut off shows up in the last digit after a few calculations. This is called floating-point error, and you will meet it constantly in linear algebra.

The most famous example is this one.

>>> 0.1 + 0.2
0.30000000000000004
>>> 0.1 + 0.2 == 0.3
False

So you do not compare decimals with ==. Instead you compare with np.isclose, which says "if they are very close, call them equal". To compare whole arrays at once, use np.allclose. By default they count two values as equal if the difference is under 0.00000001 (one hundred-millionth) or under one hundred-thousandth of the values' size. Floating-point error is far smaller than that, so it is filtered out.

>>> np.isclose(0.1 + 0.2, 0.3)
np.True_
>>> np.isclose(np.degrees(np.arccos(cos)), 45)
np.True_
>>> np.allclose(v / np.linalg.norm(v), [0.6, 0.8])
True

np.True_ is NumPy's true value. In an if statement and elsewhere you can use it exactly like Python's True. When you want to show a value nicely on screen, round it, as in round(float(x), 3). float(x) turns a NumPy number into an ordinary Python number.

>>> round(float(np.degrees(np.arccos(cos))), 3)
45.0

This error causes one more problem in angle calculations. For two vectors pointing the same way, the cosine should be exactly 1, but the error sometimes pushes it just above 1.

>>> a = np.array([2, 3])
>>> b = np.array([4, 6])
>>> c = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
>>> c
np.float64(1.0000000000000002)

A cosine lies between −1 and 1, so if you pass np.arccos a number above 1, you get nan (Not a Number) and a warning instead of an answer. In that case, clamp the value to between −1 and 1 with np.clip before passing it in. The lab does the same thing under the hood.

>>> np.clip(c, -1, 1)
np.float64(1.0)
>>> np.degrees(np.arccos(np.clip(c, -1, 1)))
np.float64(0.0)

Computing cosine similarity

Let's compute cosine similarity for the taste vectors from 3-5. Since you will do the same calculation several times, wrap it in a function.

>>> def cos_sim(x, y):
...     return x @ y / (np.linalg.norm(x) * np.linalg.norm(y))
>>> me = np.array([5, 1, 4, 0])
>>> jimin = np.array([4, 0, 5, 1])
>>> seojun = np.array([0, 5, 1, 4])
>>> harin = np.array([10, 2, 8, 0])
>>> round(float(cos_sim(me, jimin)), 3)
0.952
>>> round(float(cos_sim(me, seojun)), 3)
0.214
>>> cos_sim(me, harin)
np.float64(1.0)
>>> me @ jimin, me @ harin
(np.int64(40), np.int64(84))

Jimin is very similar to you (0.952), and Seojun not so much (0.214). Harin only gave twice the scores with the same proportions, so she gets 1. Look at the last line: by the dot product alone, Harin (84) looks more than twice as similar as Jimin (40). That is exactly why you divide by the lengths. In 4 dimensions or 768, the code does not change by a single character.

3-7. Common mistakes (3 min)

Mistaking u * w for the dot product. * is the element-wise multiplication from Lesson 2, so it gives a vector. The dot product is the single number you get by adding that all up.

>>> u * w
array([6, 0])
>>> np.sum(u * w)
np.int64(6)
>>> u @ w
np.int64(6)

Mistaking len for length. Python's len only tells you how many components there are (the dimension), not how long the arrow is.

>>> len(v)
2
>>> np.linalg.norm(v)
np.float64(5.0)

Mixing degrees and radians. np.cos(60) is the cosine of 60 radians, not 60°. When you have an angle in degrees, convert it with np.radians first. The result being 0.5000000000000001 rather than 0.5 is the floating-point error you saw earlier.

>>> np.cos(np.radians(60))
np.float64(0.5000000000000001)

Exercises (5 min)

1. Work out the length of the vector (1, 2, 2) by hand and check it with np.linalg.norm. What do you get when you normalize it?
2. With u = (3, 0), find the dot product with w = (−2, 1) and the angle between them. You can check by moving w to (−2, 1) in the lab.
3. Find a vector perpendicular to u = (1, 2) and check that the dot product is 0.
4. In the lab with u = (3, 0) and w = (−2, 1), what is the shadow length? What does its sign mean?

Solutions

>>> p = np.array([1, 2, 2])
>>> np.linalg.norm(p)
np.float64(3.0)
>>> p / np.linalg.norm(p)
array([0.33333333, 0.66666667, 0.66666667])
>>> u = np.array([3, 0])
>>> w = np.array([-2, 1])
>>> u @ w
np.int64(-6)
>>> cos = u @ w / (np.linalg.norm(u) * np.linalg.norm(w))
>>> round(float(np.degrees(np.arccos(cos))), 1)
153.4
>>> np.array([1, 2]) @ np.array([-2, 1])
np.int64(0)
>>> u @ w / np.linalg.norm(u)
np.float64(-2.0)

For question 1 the length is $\sqrt{1 + 4 + 4} = 3$, so normalizing gives (1/3, 2/3, 2/3). For question 2 the dot product is negative, so the angle is larger than 90°: about 153.4°. For question 3, swap the two components and flip the sign of one of them to get a perpendicular vector: (1, 2) and (−2, 1). For question 4, a shadow length of −2 means the shadow falls 2 units on the opposite side of u.

In the next lesson you will stretch and add two vectors to find out how far you can get. NumPy's 2-D arrays, that is, matrices, make their first appearance.