Embeddings Explained

An embedding is a fixed-length vector of floating-point numbers that represents a piece of text in a high-dimensional space. Texts with similar meaning cluster together; dissimilar texts are far apart. This geometric property is what makes semantic search possible: you embed the query, then find document chunks whose embeddings are nearest.

How embeddings are produced

Embedding models are encoder-only transformers (like BERT) or encoder heads on top of larger architectures, trained with contrastive objectives on pairs of similar and dissimilar texts. They output a single vector per input — typically 768 to 3072 dimensions — regardless of input length up to their token limit.

Calling an embedding model

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(texts: list[str]) -> np.ndarray:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return np.array([d.embedding for d in response.data])

query_vec = embed(["How do I reset my password?"])[0]
doc_vecs = embed(["Steps to reset your account password.", "How to cancel a subscription."])

Choosing an embedding model

• Accuracy: evaluate on your domain with a retrieval benchmark like MTEB or a custom eval
• Dimensionality: higher dimensions give more expressiveness but larger storage and slower search
• Max tokens: most models cap at 512 or 8192 tokens; chunks beyond the limit are truncated
• Cost: API-based models charge per token; self-hosted models have compute and ops overhead
• Matryoshka embeddings: some models let you truncate the vector to trade accuracy for speed

Query and document asymmetry

Queries are typically short and informal; documents are long and formal. Some embedding models expose separate query and document encoding modes (e.g., prefix 'query: ' vs. 'passage: ') to handle this asymmetry — use the correct mode for each to get the best retrieval accuracy.