Query Rewriting and Decomposition
Also known as: multi-query retrieval, query fan-out, HyDE, sub-question decomposition
Transform the user's question into the queries your index can actually answer — resolved, expanded, and split into parts.
The problem
The question a user asks is rarely a good search query. It carries conversational references ("what about the enterprise tier?"), it uses their vocabulary rather than the corpus's, and it frequently bundles several questions into one ("how do refunds work in the EU and how is that different from the US?"). A single embedding of that string retrieves a muddled average — chunks that are somewhat about refunds, somewhat about the EU, and decisively about nothing.
The pattern
Insert a rewriting step between the user's turn and retrieval. Resolve conversational references against the history into a standalone question, expand it with corpus vocabulary, and if it contains multiple independent information needs, split it into sub-queries that are retrieved in parallel. Fuse the result sets, deduplicate, and rerank the union.
Use when
- The interface is conversational, so follow-up turns depend on earlier context
- Users ask comparison or multi-hop questions that need facts from separate documents
- There is a vocabulary gap between how users speak and how the corpus is written
- Retrieval evaluation shows single-query recall is the bottleneck, not ranking
Avoid when
- Single-turn search where the query is already well-formed and keyword-like
- Tight latency budgets — rewriting adds a serial model call before retrieval even starts
- Very small corpora, where fan-out just returns the same chunks several times
Tradeoffs
GainHandles follow-ups and compound questions that single-shot retrieval simply cannot
CostA serial LLM call on the critical path, plus N parallel retrievals instead of one
GainDecomposition makes multi-hop answers possible without an agent loop
CostMore retrieved chunks competing for the context window; reranking becomes mandatory
GainThe rewritten query is a useful debugging artifact when an answer goes wrong
CostA bad rewrite fails silently — the pipeline retrieves confidently for the wrong question
Retrieval is only as good as the query it is given, and the raw user turn is usually the worst possible query. Query rewriting is the cheapest large win available in most RAG systems, because it fixes failures that no amount of ranking work can reach.
Three transformations, one step
The rewriting step does up to three things. Resolution turns a context-dependent turn into a standalone question by substituting what pronouns and ellipses refer to. Expansion adds synonyms and domain vocabulary that the corpus is likely to use. Decomposition splits a compound question into independent sub-questions. Do all three in a single structured call rather than chaining three calls.
REWRITE_PROMPT = """Rewrite the user's latest message into search queries.
Rules:
- Resolve pronouns and references using the conversation history.
- If the message contains multiple independent information needs, emit one query per need.
- Otherwise emit exactly one query.
- Use terminology likely to appear in the documents, not the user's phrasing.
- Never invent constraints the user did not state.
Conversation:
{history}
Latest message: {message}"""
class RewriteResult(BaseModel):
queries: list[str] = Field(min_length=1, max_length=4)
standalone_question: str
def rewrite(history: list[Message], message: str) -> RewriteResult:
return structured_call(
REWRITE_PROMPT.format(history=render(history[-6:]), message=message),
schema=RewriteResult,
)Fan out, then fuse
Each sub-query is retrieved independently and in parallel. The result sets are then fused — the same reciprocal rank fusion used for hybrid search works here — and deduplicated by chunk id before reranking. Rerank against the standalone question rather than any individual sub-query, so the final ranking reflects the user's actual need.
async def retrieve_for_turn(history: list[Message], message: str) -> list[Chunk]:
plan = rewrite(history, message)
result_lists = await asyncio.gather(*(search(q) for q in plan.queries))
fused_ids = reciprocal_rank_fusion(result_lists)[:60]
candidates = load_chunks(fused_ids)
# Rank against what the user actually asked, not against a sub-query.
return rerank_top_k(plan.standalone_question, candidates, k=8)Cap the fan-out at three or four sub-queries. Models will happily decompose a simple question into eight, and every extra query costs a retrieval round-trip while pushing genuinely relevant chunks further down the fused list.
HyDE: retrieve with a hypothetical answer
A variant worth knowing. Instead of embedding the question, ask the model to write a short hypothetical answer — a passage that would answer the question if it existed — and embed that. Answers look like documents; questions do not, so the hypothetical passage often sits closer in embedding space to the real passage. It is unreliable in domains the model knows nothing about, where the hypothetical answer is fabricated in the wrong vocabulary entirely, so treat it as an additional retrieval arm rather than a replacement.
- Resolve, expand, and decompose in one structured call — not three chained ones
- Retrieve sub-queries in parallel; the wall-clock cost is one retrieval, not N
- Rerank against the resolved standalone question, never against a sub-query
- Log the rewrite with every trace — most bad answers are visibly bad rewrites
Rewriting can silently drop constraints. If the user said "in 2024" and the rewrite omits it, you will retrieve and answer confidently from the wrong year. Assert in evaluation that named entities, dates, and numbers from the original survive into at least one query.