Embeddings
An embedding is a dense vector of numbers that represents the meaning of a piece of text (or an image, or audio) so that semantically similar things land close together in vector space. "a dog on the beach" and "a puppy by the sea" produce nearby vectors even though they share few words — and that property is what makes embeddings the workhorse behind semantic search, RAG, clustering, deduplication, and recommendation.
The core move is simple: turn content into vectors with an embedding model, then compare vectors by distance instead of matching keywords. Get comfortable with that and a huge range of "find similar," "search by meaning," and "group these" problems become straightforward.
TL;DR
- An embedding is a dense vector capturing meaning; similar content → nearby vectors.
- Compare with a similarity metric (usually cosine similarity).
- Powers semantic search, RAG, clustering, deduplication, recommendation.
- Store and search at scale in a vector database.
Quick Example
Embed text, then measure semantic similarity by vector distance:
Core Concepts
Vectors that capture meaning
An embedding model maps content to a fixed-length vector (often hundreds to a couple thousand dimensions). The geometry encodes semantics: direction and distance correspond to meaning, so related concepts cluster together.
Similarity metrics
- Cosine similarity — the angle between vectors; the default for text embeddings (ranges roughly −1 to 1, higher = more similar).
- Dot product / Euclidean distance — alternatives; which to use depends on whether vectors are normalized and what the model recommends.
Semantic vs keyword search
Keyword search matches exact tokens; semantic search matches meaning via embeddings, so it finds relevant results that don't share the query's words. The strongest systems combine both (hybrid search) to get meaning and exact terms (names, IDs, codes).
What Embeddings Power
- Semantic search — retrieve by meaning, not exact words.
- RAG — find the document chunks relevant to a question (RAG).
- Clustering & topic discovery — group similar items without labels.
- Deduplication — find near-duplicate content.
- Recommendation — "more like this" via nearest neighbors.
- Classification — embed + a lightweight classifier on top.
Best Practices
- Use one embedding model consistently — never compare vectors from different models; re-embed everything if you switch.
- Normalize and pick the right metric the model recommends (often cosine).
- Chunk thoughtfully for RAG — embed self-contained passages, not whole documents (see RAG).
- Add hybrid search (semantic + keyword) when exact terms matter.
- Store vectors in a vector database with metadata for filtering at scale.
Common Mistakes
Mixing embedding models
Embedding documents that are too large
FAQ
What's the difference between an embedding and what an LLM does?
An embedding model maps content to a single fixed vector that represents its meaning — it's used for comparison (search, clustering), not generation. An LLM generates text token by token. They're complementary and often used together: in RAG, embeddings retrieve the relevant context and the LLM generates the answer from it. Embedding models are typically smaller, cheaper, and faster than generative LLMs.
How do I measure similarity between embeddings?
Most text embeddings use cosine similarity — the cosine of the angle between two vectors — where higher means more similar. Some setups use dot product (equivalent to cosine when vectors are normalized) or Euclidean distance. Use whatever the embedding model's documentation recommends, and be consistent. Vector databases implement these metrics efficiently for nearest-neighbor search.
Can I compare embeddings from different models?
No. Each model produces vectors in its own space with its own geometry, so a vector from model A is not meaningfully comparable to one from model B — similarity scores between them are noise. Always embed both your stored content and your queries with the same model and version. If you upgrade or switch models, you must re-embed the entire corpus.
How many dimensions should embeddings have?
That's determined by the model you choose (commonly a few hundred to a couple thousand). Higher dimensions can capture more nuance but cost more storage and compute and can be slower to search; some models offer adjustable dimensions to trade quality for efficiency. For most applications, use the model's default and focus on chunking, retrieval, and hybrid search — those affect quality far more than dimension count.
Related Topics
- Vector Databases — Storing and searching embeddings at scale
- Retrieval-Augmented Generation (RAG) — The biggest use case
- Large Language Models — The generation counterpart
- NLP — Where embeddings fit in text processing
- Feature Engineering — Text features