Worked Interview Example: AI Customer Support

The best way to internalize the design framework is to walk through a complete example from start to finish. We will design an AI-powered customer support system for a software company with a large product knowledge base. This is one of the most common AI design interview prompts, and working through it concretely illustrates every major decision category.

Problem statement and clarifications

The prompt is: 'Design an AI system that handles customer support queries for a software product.' Before designing anything, clarify: how many queries per day (scale), what is an acceptable response latency (SLA), what is the escalation path when the AI cannot answer, what sources of truth exist (docs, ticket history, knowledge base), and whether responses require source citations. Assume: 50,000 queries per day, p95 under 3 seconds, escalation to human agents for unanswerable queries, a 10,000-document knowledge base updated weekly, and citations required.

High-level architecture

• Ingestion pipeline: crawl and parse docs, chunk into 400-token segments, embed with a text embedding model, index in a vector database
• Query pipeline: receive user message, classify query type and language, retrieve top-5 relevant chunks, rerank, assemble prompt
• Generation: call LLM with assembled context, stream response tokens to the user
• Post-processing: validate response format, extract cited sources, check for policy violations
• Escalation: if classifier confidence is low or model signals inability to answer, route to human agent queue

Key design decisions and tradeoffs

Why retrieval over fine-tuning? The knowledge base changes weekly; retrieval allows updates without retraining. Why reranking? Initial embedding retrieval optimizes for semantic similarity but the cross-encoder reranker can pick up on exact entity matches and relevance nuances that embedding search misses. Why a query classifier? It lets us route off-topic queries, non-English queries, and simple FAQ queries to different handlers without consuming expensive LLM tokens.

# Simplified request handler
async def handle_query(user_message: str, session_history: list) -> AsyncIterator[str]:
    # Step 1: classify
    intent = await classify_intent(user_message)
    if intent == "off_topic":
        yield "I can only help with questions about our product."
        return

    # Step 2: retrieve and rerank
    chunks = await retrieve(user_message, top_k=20)
    top_chunks = rerank(query=user_message, candidates=chunks, top_n=5)

    # Step 3: generate with streaming
    prompt = build_prompt(user_message, top_chunks, session_history)
    async for token in stream_llm(prompt):
        yield token

    # Step 4: post-process happens outside the stream
    # (validate, log, check safety)

Evaluation strategy

Golden dataset: 200 curated queries with reference answers sourced from past resolved tickets and annotated by the support team. Metrics tracked: faithfulness (do responses contain only claims supported by retrieved docs?), resolution rate (did the user stop querying within the session?), escalation rate (how often did AI hand off to humans?), and user satisfaction rating. LLM judge evaluates response quality weekly on a random sample of production queries. Any prompt or retrieval change runs the full golden dataset before deployment.

Failure modes and mitigations

Stale knowledge: docs are re-indexed on a weekly schedule but a product change mid-week could produce incorrect answers. Mitigation: critical doc updates trigger an immediate partial re-index rather than waiting for the weekly batch. Retrieval misses: if the query is about a feature not in the knowledge base, the model may hallucinate. Mitigation: confidence threshold on retrieval similarity score; queries below the threshold escalate automatically. Rate limiting: 50,000 queries per day is roughly 2 requests per second on average, with spikes. Mitigation: Redis-based request caching for repeated queries, async LLM calls, and a fallback to a smaller model under high load.