RAG Overview and When to Use It

Retrieval-augmented generation (RAG) is the pattern of fetching relevant documents at query time and including them in the model's context before generating a response. Rather than relying solely on knowledge baked into model weights during training, RAG grounds the model's output in documents you control and can update.

Problems RAG solves

• Knowledge cutoff: the model has no awareness of events or documents after its training data
• Hallucination on specific facts: grounding answers in retrieved passages gives the model authoritative text to cite rather than generating from memory
• Private knowledge: your internal docs, codebase, or customer data are not in any foundation model's weights
• Auditability: retrieved passages give you a traceable source for every answer

The basic RAG loop

# Simplified RAG request cycle
query = user_message

# 1. Retrieve
passages = vector_store.search(query, top_k=5)
context = "\n\n".join(p.text for p in passages)

# 2. Augment
prompt = f"""Use the passages below to answer the question.

Passages:
{context}

Question: {query}"""

# 3. Generate
response = llm.complete(prompt)

When RAG is the right choice

RAG is a good fit when your knowledge base is larger than a context window, when it changes frequently, or when you need traceable citations. It is less appropriate for tasks that are entirely creative, for simple classification that does not need external facts, or when the required knowledge is already well-represented in the model's weights and hallucination risk is low.

RAG vs. fine-tuning

Fine-tuning teaches the model new behavior or style by updating weights, but it does not reliably inject factual knowledge — it is expensive to retrain every time documents change. RAG is cheaper to update and provides explicit sourcing. In practice, the two are complementary: fine-tune for tone and format, use RAG for facts.