Embeddings Explained

Embeddings explained in one line: an embedding turns a piece of text into a list of numbers—a vector—positioned so that things with similar meaning sit close together in space. This guide covers what that vector actually represents, how models learn to produce it, how "similarity" is measured in code, and how to pick an embedding model that fits your RAG system without overpaying. By the end you'll understand the single component that most determines whether semantic search returns the right results.

What an embedding actually is

An embedding is a fixed-length array of floating-point numbers that represents the meaning of an input. Feed the sentence "the cat sat on the mat" to an embedding model and you get back something like [0.021, -0.118, 0.34, ...]—typically 384 to 3,072 numbers long. That array is a coordinate in a high-dimensional space, and the model is trained so that texts with related meaning land near each other.

The useful mental model is meaning as geometry. "How do I reset my password?" and "I forgot my login credentials" use almost no words in common, yet a good embedding model places their vectors close together because they mean nearly the same thing. A keyword search would miss that connection; an embedding captures it. This is what powers semantic search—matching by meaning rather than exact words.

Dense vs. sparse, and the "dimensions" you'll hear about

The embeddings in this guide are dense vectors: every position holds a meaningful value, and the vector is relatively compact. They contrast with sparse vectors (like TF-IDF or BM25), which are mostly zeros and track exact term frequencies. Each position in a dense vector is a dimension. More dimensions give the model more room to encode nuance, but they cost more to store and compare—a tradeoff we return to below.

A classic illustration from older word embeddings: the vectors arrange themselves so that king − man + woman lands near queen. Modern sentence and document embeddings work on whole passages rather than single words, but the same principle holds—relationships in meaning become relationships in geometry.

How embedding models learn meaning

Embedding models are neural networks trained with contrastive learning. The model sees pairs of texts that should be close (a question and its correct answer, a sentence and its paraphrase) and pairs that should be far apart (unrelated texts). Over millions of examples, it adjusts its weights to pull related pairs together and push unrelated ones apart in vector space. The output layer produces the embedding.

Two practical consequences fall out of how these models are trained, and both trip people up.

Query and document embeddings can be asymmetric

Many retrieval models are trained to embed a short query and a longer document into the same space, but they expect you to flag which is which—sometimes with an instruction prefix like "query: ..." versus "passage: ...". Skip that and retrieval quality quietly drops. Always read your model's documentation for the expected input format.

The model defines what "similar" means

Two models can place the same two sentences at very different distances because they were trained on different data and objectives. A model trained heavily on code will cluster code differently than a general-purpose model. There's no universal embedding—only embeddings tuned for particular kinds of meaning, which is why domain fit matters more than a single benchmark score.

How similarity is measured

Once text is a vector, comparing two texts means comparing two vectors. Three metrics dominate:

• Cosine similarity — the cosine of the angle between vectors, ranging from −1 to 1. It measures direction, ignoring magnitude. This is the default for text embeddings.
• Dot product — multiplies magnitude and direction together. When vectors are normalized to unit length, dot product and cosine similarity are identical, which is why many systems normalize once and use the cheaper dot product.
• Euclidean (L2) distance — straight-line distance between the two points. Lower means more similar.

Here's the whole idea in a few lines of Python using sentence-transformers:

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, free, runs on CPU

texts = [
    "How do I reset my password?",
    "I forgot my login credentials.",   # same meaning, different words
    "What time does the store close?",  # unrelated
]
# normalize so dot product == cosine similarity
vecs = model.encode(texts, normalize_embeddings=True)

query = model.encode("can't remember my account password", normalize_embeddings=True)
scores = vecs @ query                  # cosine similarity to each text

for text, score in sorted(zip(texts, scores), key=lambda x: -x[1]):
    print(f"{score:.3f}  {text}")

The two password-related sentences score far higher than the store-hours sentence, even though the query shares few exact words with either. That gap—relevant text scoring measurably above irrelevant text—is what makes retrieval work. If your embeddings don't produce that gap on your own data, no downstream component can fix it.

How to choose an embedding model

The model you pick sets the ceiling on retrieval quality. As of early 2026, the field splits into hosted APIs and self-hostable open models, and the right choice depends on accuracy needs, language coverage, cost, latency, and whether you need multimodal (text-plus-image) support.

A few caveats keep this honest. Public MTEB scores (the Massive Text Embedding Benchmark) cluster within a few points at the top, and the 2026 MTEB v2 numbers are not directly comparable to the older v1 figures—so treat any leaderboard rank as a starting point, not a verdict. Benchmark the shortlist on your data, because generic scores often don't transfer to your domain. One more useful fact for LLM builders: Anthropic doesn't offer an embedding model, so teams using Claude for generation pair it with OpenAI, Google, Voyage, or an open model for the embedding step.

Dimensions, cost, and Matryoshka

Dimensions drive your storage and search bill. A 1,024-dimension float32 vector is about 4 KB; at 10 million documents that's roughly 40 GB of vector storage, and doubling the dimensions doubles it. Many modern models (OpenAI's and Voyage's among them) support Matryoshka Representation Learning, which lets you truncate a vector—say from 3,072 dimensions to 512 or 256—with only graceful quality loss. That's a fast lever for cutting storage and speeding up search when your accuracy budget allows. Quantization (storing values as int8 or even binary) cuts cost further.

When to fine-tune

Off-the-shelf models are excellent generalists. If your domain is highly specialized—legal, medical, internal jargon—fine-tuning an embedding model on your own labeled pairs can add meaningful retrieval gains. It's worth the effort only after you've confirmed a strong base model and good chunking still leave a measurable gap.

Where embeddings fit in a RAG pipeline

Embeddings are the connective tissue of retrieval. In a system that uses retrieval augmented generation to ground an LLM in your data, you embed every document chunk once during indexing, store those vectors, and at query time embed the user's question and search for the nearest chunks.

Two neighbors in the pipeline depend directly on embedding quality. First, what you embed matters as much as how: splitting documents well is its own discipline, covered in our guide to chunking strategies for RAG—embed chunks that are too large and the vector blurs several topics into one muddy point. Second, the vectors have to live somewhere built for fast nearest-neighbor search, which is where your choice among the best vector databases for RAG comes in; that's also where the dimension and quantization decisions above start to bite.

Embeddings give you a fast, approximate first pass. They're powerful but coarse, which is why high-quality systems add a second stage of reranking the retrieved results with a cross-encoder to sharpen the final ordering. Getting the embedding step right is also foundational to reducing hallucinations with RAG: if retrieval surfaces the correct context, the model has the facts it needs to stay grounded.

Common mistakes with embeddings

These are the errors that show up in production, not in tutorials.

Using different models for indexing and querying. Embeddings are only comparable within the same model's space. Embed your documents with one model and your queries with another—or even a different version of the same model—and similarity scores become meaningless. Re-embed everything when you change models.

Ignoring the query/passage distinction. If your model expects instruction prefixes or separate query and document modes, skipping them silently degrades recall. Read the model card.

Over-buying dimensions. Reaching for a 3,072-dimension model when a truncated 512-dimension vector would clear your accuracy bar wastes storage and slows every search. Measure first.

Trusting the leaderboard over your data. The top MTEB models differ by small margins, and rankings don't always hold on your specific documents and queries. Run a quick recall test on a representative sample before committing—the same discipline that underpins evaluating RAG quality overall.

Forgetting to re-index when content or models change. A stale vector index serves outdated meaning. Treat re-embedding as part of your content-update workflow.

Frequently asked questions

Are embeddings the same as the LLM? No. An embedding model is a separate, usually much smaller network whose only job is to turn text into vectors for comparison. The LLM generates text. In a RAG system they work together: the embedding model retrieves context, the LLM writes the answer.

How many dimensions should my embeddings have? Enough to hit your accuracy target and no more. Common sizes range from 384 to 3,072. If your model supports Matryoshka truncation, start high, then trim dimensions while watching retrieval quality to find the cheapest size that still works.

Can I use OpenAI or Google embeddings with Claude? Yes. Embedding and generation are independent steps, so it's common to pair one provider's embedding model with another's LLM. Anthropic doesn't currently offer embeddings, so Claude users routinely use OpenAI, Google, Voyage, or an open-source model for the retrieval step.

What's the difference between dense and sparse embeddings? Dense embeddings are compact vectors that capture meaning and power semantic search. Sparse vectors (like BM25) are mostly zeros and match exact terms. Hybrid retrieval combines both to catch meaning and exact keywords like product codes.

Do I need to fine-tune an embedding model? Usually not. Strong general models cover most use cases. Fine-tune only when a specialized domain leaves a measurable retrieval gap after you've already optimized your base model and chunking.

The takeaway

With embeddings explained at the level that matters for building, the practical lesson is simple: embedding quality sets the ceiling for everything downstream in retrieval, so choose the model deliberately and verify it on your own data rather than a leaderboard. Your next step is to take two or three candidate models, embed a representative sample of your real documents and queries, and measure recall before you wire up the rest of the pipeline. Get this layer right and the rest of your RAG system has a foundation it can build on.