Online Feedback Capture
Also known as: implicit signals, thumbs up/down, production labelling
Instrument the product so real usage produces labelled data, then turn that data into evaluation cases and fixes.
The problem
Offline evaluation measures the cases you thought of. Production is full of the ones you did not — and the signal is right there, unrecorded: users rephrasing because the answer missed, abandoning mid-conversation, copying one answer and ignoring another, escalating to a human. Meanwhile explicit thumbs-down clicks arrive with no context about what was wrong, and nobody has time to read them anyway.
The pattern
Capture both signal types deliberately. Explicit feedback needs a structured reason, not a bare thumb. Implicit signals — rephrasing, abandonment, copying, escalation, task completion — need instrumenting, and they are far higher volume and less biased. Join both to the trace that produced them, cluster into failure themes, and convert the recurring ones into evaluation cases.
Use when
- The product is live and has enough traffic for patterns to emerge
- Offline evaluation has stopped surfacing the failures users report
- You need real distribution data rather than the cases you imagined
- Some outcome — task completed, escalated, abandoned — is observable
Avoid when
- Pre-launch, with no real traffic to learn from
- Privacy constraints prevent retaining the content needed to act on feedback
- There is no capacity to triage what you collect — unread feedback is just cost
Tradeoffs
GainReal failure distribution instead of imagined cases; the eval set writes itself
CostInstrumentation across the product, plus storage and retention policy for traces
GainImplicit signals are high volume and unbiased by who bothers to click
CostThey are proxies — a rephrase might be a follow-up, not a failure
GainFeedback tied to traces makes root cause visible rather than guessed at
CostTraces contain user content and need access control and retention limits
Explicit feedback is a biased sample: it comes from users annoyed enough or delighted enough to click. Implicit signals cover everyone, which makes them the more useful half — and almost nobody instruments them.
Make explicit feedback structured
A thumbs-down with no reason tells you something is wrong somewhere. A thumbs-down with a category tells you which subsystem to look at. Offer a short reason list chosen so each option maps to a different fix — wrong facts point at retrieval, missing information points at coverage, wrong format points at the prompt.
class Feedback(BaseModel):
trace_id: str # joins to the full generation record
rating: Literal["up", "down"]
reason: Literal[
"incorrect", # -> retrieval or grounding
"incomplete", # -> coverage or context budget
"wrong_format", # -> prompt
"too_slow", # -> latency
"misunderstood", # -> query rewriting or routing
"other",
] | None = None
note: str | None = NoneInstrument the implicit signals
These are the ones that cover all your traffic. Each is a proxy and none is conclusive alone, but together they separate good turns from bad ones reliably enough to prioritize with. Weight them into a single quality proxy per turn, and be honest that it is a proxy.
IMPLICIT_WEIGHTS = {
"rephrased_within_60s": -0.8, # strongest negative: the answer missed
"abandoned_mid_task": -0.7,
"escalated_to_human": -0.9,
"copied_response": +0.7, # strongest positive: it was used
"followed_link": +0.4,
"task_completed": +0.9,
"thanked": +0.3,
}
def implicit_score(events: list[Event]) -> float:
signals = {e.name for e in events}
raw = sum(w for name, w in IMPLICIT_WEIGHTS.items() if name in signals)
return max(-1.0, min(1.0, raw))
def is_rephrase(prev: str, current: str, gap_s: float) -> bool:
"""A quick, semantically similar re-ask is a failure signal, not a follow-up."""
return gap_s < 60 and cosine(embed(prev), embed(current)) > 0.82Rephrase detection is the single highest-value implicit signal to build first. A user re-asking the same thing in different words within a minute is a near-unambiguous statement that the first answer failed, and it costs nothing for them to produce.
Cluster, then convert to cases
Individual feedback items are noise; clusters are a backlog. Group negative-signal traces by embedding the user input, rank clusters by volume and severity, and turn the top ones into evaluation cases. That closes the loop: production failures become the regression tests that stop them recurring.
def weekly_triage(since: datetime) -> list[FailureTheme]:
negatives = traces.query(since=since, implicit_score_below=-0.4)
clusters = cluster([embed(t.user_input) for t in negatives], min_size=8)
themes = []
for c in clusters:
members = [negatives[i] for i in c.indices]
themes.append(FailureTheme(
label=summarize_theme([m.user_input for m in members]),
volume=len(members),
example_traces=[m.id for m in members[:5]],
common_stage=mode(m.likely_failure_stage for m in members), # retrieval? generation?
))
return sorted(themes, key=lambda t: -t.volume)- Every response carries a trace id; every signal joins back to it
- Explicit feedback needs a reason category that maps to a subsystem
- Instrument implicit signals — they cover all traffic, not just the vocal minority
- Cluster negatives weekly; individual items are noise, clusters are a backlog
- Convert recurring themes into golden-set cases so they cannot come back
Set a retention policy before you start collecting. Traces contain user content, and "we keep everything forever" is a decision that gets made by default and is expensive to unwind later.