Module 6 · Lesson 6.6

Abstract Vector Spaces & High-Dimensional Embeddings

See how vectors, dot products, and alignment extend perfectly well beyond 3D, into the high-dimensional "meaning space" behind word embeddings.

Intuition

Every arrow you've drawn so far has lived in 2D or 3D, because that's what fits on a page. But nothing about addition, scaling, or the dot product actually requires an arrow you can draw. A vector can just as well be a list of a thousand numbers — and every rule from earlier lessons still applies exactly as written, even though there's no picture left to make of it.

Core concept

A word embedding is exactly this: a long list of numbers assigned to a word, placing it at a specific point in a high-dimensional "meaning space." Words used in similar contexts end up with embeddings that point in similar directions. The dot product still measures that alignment, component by component, no matter whether there are 2 numbers or 4096 of them — it's the same arithmetic, just with more terms to add up.

cat dog car dot = 9 dot = 1
cat, dog, car embeddings(1,2,0,1), (1,3,0,2), (0,0,4,1)
dot(cat, dog)1+6+0+2 = 9
dot(cat, car)0+0+0+1 = 1
Answercat aligns far more with dog (9 vs 1)

Interactive Lab

Slide a mystery word's embedding from exactly "dog" toward exactly "car." Even though this 4-dimensional vector can't be drawn as an arrow, its similarity to "cat" — measured with the same dot product — still tracks smoothly the whole way across.

dog-like car-like similarity to "cat"
mystery = (1.00, 3.00, 0.00, 2.00) dot(cat, mystery) = 9.00 closer to: dog

Key Idea

$$ \vec{u}\cdot\vec{v} = \sum_{i=1}^{n} u_i v_i $$

This is the exact same dot product from Lesson 6.4, just written for $n$ dimensions instead of 2. Nothing new is needed: multiply matching components together, add them all up. Whether $n$ is 2 or 4096, this single sum is still what tells you how aligned two vectors are.

$n$The number of dimensions — 4096 for many LLM word embeddings
$u_i, v_i$The $i$-th component of each vector
$\sum_{i=1}^{n}$Add up the products across every dimension
$\vec{u}\cdot\vec{v}$The dot product — still measures alignment, even in high dimensions

Think Further

  1. An LLM's word embeddings often have 4096 dimensions. Even though we can't draw a 4096-dimensional arrow, why does the dot product formula still work exactly the same way?
  2. Predict: if two word embeddings have a dot product near zero, what would that suggest about how related the two words' meanings are? (Recall the perpendicular case from Lesson 6.4.)
  3. In this lesson's interactive, the mystery vector's similarity to "cat" changed in a perfectly straight line as it slid from "dog" toward "car." Why does interpolating linearly between two vectors always produce a linear change in dot product with a fixed third vector?
Show suggested answers
  1. Because the algebraic definition — multiply matching components, add them up — never depends on being able to picture the vector. It's ordinary arithmetic on however many numbers the vector has, whether that's 2 or 4096.
  2. A near-zero dot product suggests the two words are largely unrelated in this embedding space, echoing Lesson 6.4's perpendicular case, where a zero dot product signaled no alignment at all between two directions.
  3. The dot product is linear in each argument: expanding $\vec u \cdot [(1-t)\vec p + t\vec q]$ distributes to $(1-t)(\vec u\cdot\vec p) + t(\vec u\cdot\vec q)$, which is exactly a straight-line function of $t$.