Scenario: Structured Data Extraction You are building a structured data extraction system using Claude. The system extracts information from unstructured documents, validates output using JSON schemas, and maintains high accuracy. It must handle edge cases gracefully and integrate with downstream systems. You are about to process 80,000 archived vendor contracts overnight. A new extraction prompt and schema passed a few hand-picked examples, but historical documents vary by template, scanned OCR quality, and contract addenda. Reprocessing failures delays a downstream migration. What should you do next?
Context Evaluation, Observability, and Governance
Loading lesson content...
Scenario: Structured Data Extraction You are building a structured data extraction system using Claude. The system extracts information from unstructured documents, validates output using JSON schemas, and maintains high accuracy. It must handle edge cases gracefully and integrate with downstream systems. A pilot processes 30 contracts per run. Accuracy is high on early documents but declines later: clauses copied from earlier contracts appear in later extractions, and page references from middle documents are dropped. Trace logs show raw outputs from OCR, table parsing, and validation tools dominate the request history, including coordinates, styling data, alternate OCR hypotheses, debug IDs, and validator traces. Downstream extraction only uses clause text, page number, document ID, confidence, and validation status. The operations team also needs original tool responses available for traceability. What change best addresses the reliability problem?