Vector Search Basics

Once documents are embedded into vectors, retrieval becomes a nearest-neighbor search: given a query vector, find the K document vectors with the smallest distance (or highest cosine similarity). Doing this at scale — millions of vectors in milliseconds — requires approximate nearest-neighbor (ANN) indexing.

Distance metrics

• Cosine similarity: measures angle between vectors; insensitive to magnitude. Most common for text.
• Dot product: equivalent to cosine if vectors are unit-normalized; faster to compute.
• Euclidean (L2): measures absolute distance; sensitive to magnitude. Less common for text embeddings.

ANN index algorithms

HNSW (Hierarchical Navigable Small World) is the dominant algorithm for production RAG. It builds a multi-layer graph where each node connects to nearby neighbors. Search traverses from coarse to fine layers, finding approximate neighbors in O(log n) time. It offers an accuracy/speed tradeoff controlled by the ef_construction and ef_search parameters.

# Using Qdrant as an example vector store
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

# Query
results = client.search(
    collection_name="docs",
    query_vector=query_embedding,
    limit=5,
    score_threshold=0.75,
)

Metadata filtering

Most vector stores let you attach metadata (document ID, date, category, access group) to each vector and filter on it at query time. This is essential for multi-tenant systems, time-bounded retrieval, and permission scoping. Apply metadata filters before or alongside ANN search — the implementation varies by database.

Choosing a vector store

• Qdrant, Weaviate, Milvus: purpose-built, self-hosted, strong filtering and payload support
• Pinecone, Turbopuffer: managed SaaS, minimal ops overhead, pay-per-use
• pgvector (Postgres): great for teams already running Postgres; simple, no new service to operate
• ChromaDB: lightweight, file-backed, excellent for local development and small datasets