Extraction with Provenance

The shift is from extracting values to extracting claims. A claim carries its own evidence, which is what makes it checkable — by a person or by code.

A field is a value plus its evidence

Wrap every extracted field in a generic container. The critical constraint is that `source_text` must be quoted verbatim from the document — this is what makes mechanical verification possible, and it also measurably suppresses fabrication, because a model that must quote its source is less willing to invent.

T = TypeVar("T")


class Extracted(BaseModel, Generic[T]):
    value: T | None
    source_text: str | None = Field(
        description="The EXACT text from the document this came from, copied verbatim. "
                    "Null only if the field is not present in the document."
    )
    page: int | None = None
    confidence: float = Field(ge=0, le=1)


class Contract(BaseModel):
    counterparty: Extracted[str]
    effective_date: Extracted[date]
    termination_notice_days: Extracted[int]
    auto_renews: Extracted[bool]
    governing_law: Extracted[str]

Verify the span mechanically

A quoted span you never check is decoration. Verify each one against the source, allowing for normalization — whitespace, hyphenation, OCR substitutions — via fuzzy matching. A field whose span does not appear in the document is not low-confidence, it is unverified, and it should never be auto-accepted regardless of what confidence the model claimed.

def verify(field: Extracted, source: str, threshold: float = 0.90) -> Verification:
    if field.value is None:
        return Verification(status="absent")
    if not field.source_text:
        return Verification(status="unverified", note="no span quoted")

    norm_source = normalize(source)      # collapse whitespace, fix common OCR pairs
    norm_span = normalize(field.source_text)

    if norm_span in norm_source:
        return Verification(status="verified", offset=norm_source.index(norm_span))

    best, ratio = fuzzy_locate(norm_span, norm_source)
    if ratio >= threshold:
        return Verification(status="verified_fuzzy", offset=best, similarity=ratio)

    # The model quoted text that is not in the document. Treat as fabricated.
    return Verification(status="unverified", note=f"best match {ratio:.2f}")

Route by verification and confidence

Provenance pays off in the routing. Fields that are verified and confident are auto-accepted; anything unverified, low-confidence, or failing a cross-field rule goes to a human — who sees the highlighted span rather than the whole document. That combination is what turns a 20-minute manual review into a 30-second one.

AUTO_ACCEPT = 0.85


def route(extraction: Contract, source: str) -> Routing:
    needs_review = []

    for name, field in extraction:
        v = verify(field, source)
        if v.status == "unverified":
            needs_review.append((name, "span not found in document", v))
        elif field.confidence < AUTO_ACCEPT:
            needs_review.append((name, f"low confidence {field.confidence:.2f}", v))

    for rule in CROSS_FIELD_RULES:            # e.g. termination date after effective date
        if (problem := rule(extraction)):
            needs_review.append((rule.fields, problem, None))

    return Routing(
        auto_accepted=not needs_review,
        review_items=needs_review,
        highlights=[v.offset for _, _, v in needs_review if v and v.offset],
    )

• Every field carries value, verbatim span, location, and confidence
• Verify spans with fuzzy matching; treat unverified spans as fabrication
• Allow null for genuinely absent fields, or the model will invent them
• Route on verification status and confidence; show reviewers the span, not the document
• Track per-field accuracy over time — it tells you exactly what to improve