Vector Embeddings & Latent Space Geometry

Represent discrete human language tokens in continuous high-dimensional vector spaces where geometric distance reflects semantic meaning.

1. The Geometry of Latent Space

Vector embedding models project discrete token IDs \(t_i \in \mathcal{V}\) into continuous dense vector representations \(\mathbf{e}_i \in \mathbb{R}^d\) (typically \(d = 1536\) or \(d = 4096\)).

2. Cosine Similarity & Dot Product Metric

Semantic relatedness between two vector embeddings \(\mathbf{u}\) and \(\mathbf{v}\) is computed via Cosine Similarity, measuring the angular divergence regardless of vector magnitude:

\[ \cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2} = \frac{\sum_{i=1}^d u_i v_i}{\sqrt{\sum_{i=1}^d u_i^2} \sqrt{\sum_{i=1}^d v_i^2}} \]

When embeddings are normalized to unit sphere length (\(\|\mathbf{u}\| = 1\)), Cosine Similarity simplifies to a single fast matrix dot product \(\mathbf{u}^T \mathbf{v}\).

🎯 Module Mastery Certification Quiz

+100 XP

Why is Cosine Similarity preferred over Euclidean Distance for high-dimensional text embeddings?

Because Cosine Similarity is sensitive only to vector orientation and invariant to document length / vector magnitude.

Because Euclidean distance requires 16-bit float rounding.

Because Cosine Similarity always returns an integer.

Because high-dimensional spaces cannot compute subtraction.