Prompt Engineering and Structured Output
How to make Claude produce consistent, schema-valid output on the first try through explicit criteria, few-shot patterns, constrained decoding, validation loops, and batch discipline.
Prompt engineering for production is not about clever phrasing but about removing ambiguity from every input that shapes Claude's next token. The system prompt, the few-shot examples, the JSON schema, and the validation loop each remove a different class of variance: vague criteria produce inconsistent classification, missing examples produce format drift, a missing schema produces malformed JSON, and a missing validator lets wrong values pass as correct structure. When all four are in place the model still makes mistakes, but those mistakes become detectable and correctable rather than silent and random.
This domain's six tasks map directly to the six places a production pipeline drifts. Task 4.1 makes the system prompt explicit by replacing subjective instructions with categorical criteria and the RACCE diagnostic. Task 4.2 adds few-shot examples with reasoning so the model matches a demonstrated pattern instead of interpreting an adjective. Task 4.3 moves from probabilistic JSON in text to constrained decoding through tool schemas and structured outputs. Task 4.4 closes the loop with semantic validation and retry with error feedback that distinguishes fixable from unfixable failures. Task 4.5 chooses batch versus synchronous for cost without breaking latency guarantees. Task 4.6 replaces single-pass self review with multi-pass independent review plus calibrated confidence routing.
Two principles cut across every task. First, structure removes variance more reliably than additional prose. An explicit category list beats a paragraph defining conservatism, a concrete input-output pair beats a page defining warmth, and a JSON schema with constrained decoding beats an instruction to respond only in JSON. Whenever the exam offers more instructions versus an example or a schema, the example or schema is the correct answer. Second, schema guarantees structure but never guarantees correctness. Line items can sum to the wrong total inside perfectly valid JSON, and a field can be populated with a plausible invention that passes type checks. Production therefore needs both structural enforcement and semantic validation, with retry that names the specific error rather than repeating the same prompt.
System Prompts
Replace vague guidance with explicit categorical criteria, RACCE structure, and primacy-aware placement so every response is judged by the same testable boundary.
What you need to know
The system prompt is the single most influential input you control when deploying Claude because it persists unchanged across every turn of the conversation. It is the right place for anything that should shape every response rather than just the current one: persona, behavioural constraints, output conventions, and the criteria by which quality is judged. Anything time bound or query specific belongs in a user message, while any standard that must apply without an invocation step belongs in the system prompt.
The most common production failure is a vague instruction that sounds engineering grade but gives the model no actionable decision boundary. Phrases such as be conservative, only report high-confidence findings, use your best judgement, be ethical, and be careful cannot be verified from the output and mean different things in different contexts. The exam uses exactly this language as distractors because it feels responsible while being untestable. The correct replacement is explicit categorical criteria that name what to flag and what to skip with concrete patterns per category, for example flag comments only when claimed behaviour contradicts actual code behaviour and otherwise skip minor style preferences and local patterns.
RACCE is the framework that makes a system prompt diagnosable when output quality drops. Role activates domain knowledge and perspective, without it Claude defaults to generic helpful assistant behaviour. Audience calibrates vocabulary, tone, and assumed knowledge, without it tone mismatches the consumer. Criteria define how quality is evaluated and which tradeoffs are favoured, without them Claude decides whether correctness or brevity matters. Constraints are hard boundaries not to cross, without them Claude improvises scope. Examples demonstrate the target format and quality through demonstration rather than description. When a system prompt underperforms the fault almost always maps to one missing RACCE component.
System prompts exhibit a primacy effect where Claude attends more strongly to content at the very beginning and end of context than to content buried in the middle. A critical behavioural constraint placed in the middle of an 8000 token system prompt receives reliably weaker attention than the same constraint placed at the top. The fix is structural: put the most critical behavioural constraints first, supporting detail in the middle, and examples last where they also benefit from recency. A practical budget is to keep the system prompt under 10 percent of total token budget, otherwise it competes with the task for attention.
Confidence-based filtering is not a substitute for explicit criteria. Instructions such as only report high-confidence findings fail because LLM self-reported confidence is poorly calibrated, the model is often sure about wrong findings and hesitant about right ones. Confidence scoring earns its place as a routing mechanism for uncertain outputs to human review, which is covered in the multi-pass review task, but it never replaces explicit criteria that define what counts as a valid finding in the first place. The hierarchy is explicit criteria first, confidence routing second, and skipping the first step cannot be compensated by tuning the second.
RACCE as a diagnostic and the explicit criteria replacement
Vague guidance fails the verifiability test. A constraint such as never provide a specific diagnosis or always escalate when the user describes an emergency can be checked directly in the output, while be ethical cannot. Explicit categorical criteria pass the same test: report bugs and security vulnerabilities and flag comment mismatches only when claimed behaviour contradicts code, skip style preferences and local patterns, each paired with a concrete code snippet for the critical severity and a snippet for the minor severity so severity is recognised by pattern not by prose interpretation.
RACCE makes the gap explicit during debugging. Walk each component when a prompt underperforms and ask whether role, audience, criteria, constraints, or examples is underspecified. Underperforming documentation mismatch detection almost always maps to criteria, missing the definition of what counts as a mismatch, not to a model capability limit. Adding high confidence language after the fault is still criteria missing, so it still fails.
Primacy, negative constraints, and size discipline
When a prompt grows by appending detail, critical constraints drift toward the middle where primacy is weakest. The presentation that a long plausible prompt asks which instruction is reliably followed keys to the instruction at the top, not the middle, and the fix is to reorder rather than to add emphasis words. Supporting detail and long examples belong in the middle and at the end where recency still helps.
Negative constraints are more reliable than sampling adjustments for suppressing unwanted behaviour. Lowering temperature reduces randomness but does not guarantee the absence of a preamble, while a specific negative constraint such as do not include text outside the JSON object or do not wrap output in markdown fences directly suppresses the pattern. Keep negative constraints specific and verifiable, the same rule as positive constraints.
Caching reduces per-token price but not attention cost. A 10000 token system prompt still consumes 10000 tokens of attention budget per call, the cache only discounts the price, so size discipline still applies even when caching is enabled.
Mechanism and API surface
- RACCE framework
- Role, Audience, Criteria, Constraints, Examples. Each maps to a failure mode and the fix for an underperforming prompt almost always maps to one component.
- Explicit categorical criteria with examples per severity
- Define what to flag and what to skip per category with concrete code patterns, for example critical as a SQL injection snippet and minor as inconsistent variable naming, not prose descriptions.
- Primacy and recency placement
- Critical behavioural constraints at the very top, supporting detail in the middle, examples last. Content in the middle of a long prompt receives weaker attention than the same content at either end.
- Negative constraints over temperature
- Specific suppressions such as Output ONLY the JSON object, no preamble, no explanation, no markdown fences suppress a pattern directly. Temperature controls randomness, not pattern absence.
- Size budget and update discipline
- Keep system prompt under 10 percent of token budget, version in source control, shadow test a sample of real inputs on old versus new prompt, change one RACCE component at a time, and monitor satisfaction, tool call patterns, and refusal rate.
Distinctions that decide answers
| This | Not this | How to tell them apart |
|---|---|---|
| System prompt | User message | System persists across every turn and frames the assistant, user is per-turn input that can request a temporary shift. Universal standards belong in system, query-specific instructions belong in user. |
| Explicit categorical criteria | Confidence-based filtering | Criteria define what to flag and what to skip with code examples per category. Confidence tries to filter by self-reported certainty, which is poorly calibrated. Hierarchy is criteria first, confidence routing second. |
| Negative constraints | Lower temperature | Negative constraints directly suppress a pattern such as a preamble. Lowering temperature reduces randomness but does not guarantee pattern absence. |
| Primacy at top | Recency near end | Both ends receive strong attention, primacy shapes core behaviour, recency shapes the specific output format. Put behavioural constraints first and examples last. |
| RACCE framework | Free-form prose prompts | RACCE decomposes the prompt into five testable components. Free-form prose reads like a specification but cannot be diagnosed systematically when quality drops. |
Traps
The tempting answer. Add the phrases be conservative or only report high-confidence findings because they sound engineering grade.
Why it fails. Both phrases give the model no actionable decision boundary. Conservative varies by context and high-confidence is an uncalibrated subjective threshold. The exam uses them as distractors for exactly their plausible tone.
What is correct. Replace them with explicit categorical criteria and concrete code examples per category that define what counts as a valid finding.
The tempting answer. Add only report high-confidence documentation issues to curb a category running at 40 percent false positives.
Why it fails. LLM self-reported confidence is poorly calibrated and does not address the missing definition of what counts as a valid finding in that category, so the same false positives continue at a new cutoff.
What is correct. Define what counts as a valid finding with criteria and examples, then use confidence only as a routing signal to human review.
The tempting answer. Keep the high false positive category live during repair because disabling it feels like losing functionality.
Why it fails. Trust bleeds across categories. A 40 percent false positive rate on one category destroys trust in security findings that remain highly accurate.
What is correct. Temporarily disable the unreliable category, rework its criteria and examples, and re-enable only once precision improves, with shadow testing before deployment.
The tempting answer. Grow the prompt by appending detail and leave the most important constraints where they were originally written.
Why it fails. Content at the very top receives stronger attention than content buried in the middle due to the primacy effect, so middle-placed constraints are reliably followed less often.
What is correct. Move critical behavioural constraints to the very top, supporting detail to the middle, and examples to the end.
The tempting answer. Add aspirational constraints such as be ethical or be careful to cover safety.
Why it fails. Aspirational constraints are not verifiable from the output, there is no way to check whether Claude was ethical from the text alone.
What is correct. Use testable constraints such as never provide a specific diagnosis or always escalate when the user describes an emergency.
The tempting answer. Treat a 10000 token system prompt as free once prompt caching is enabled.
Why it fails. Caching reduces repeat-call price but not per-call attention footprint. The 10000 tokens still occupy attention budget on every call.
What is correct. Keep the system prompt under 10 percent of working context even when caching, and treat caching as a price discount not an attention discount.
Few-Shot Prompting
Show the pattern with 2 to 4 reasoning-annotated examples so Claude matches a demonstrated judgement instead of interpreting an adjective.
What you need to know
Few-shot examples are the most effective technique for achieving consistent, well formatted output from Claude, more effective than more instructions, confidence thresholds, or temperature adjustments. When output is inconsistent across invocations despite detailed instructions, the first tool to reach for is a small set of examples showing the exact format and judgement you want. The exam presents exactly that scenario and tests whether you choose more instructions or few-shot examples, with the latter as the correct answer.
Few-shot works through in-context learning: Claude generalises patterns from in-context examples more reliably than it follows verbal descriptions of those same patterns. Telling the model to respond in a warm, concise, professional tone with a brief empathy statement leaves warm and concise open to interpretation, while three actual responses exhibiting that tone remove the interpretation step entirely. The model matches the demonstrated pattern rather than reconstructing it from an adjective.
Three triggers tell you few-shot is needed. Detailed instructions produce inconsistent formatting, sometimes a bulleted list and sometimes a table for the same request, more prose will not converge it. Judgement calls are inconsistent on ambiguous cases, for code review the model flags variable shadowing as critical in one file and minor in another, or for routing it sends check my order to different tools depending on phrasing. Extraction produces empty or null fields for information that exists but appears in an unexpected format, inline citations versus bibliographies or narrative text versus structured tables. All three resolve faster with examples than with additional description.
The rules for constructing effective examples are tight. Use 2 to 4 targeted examples, fewer than 2 does not establish a pattern and more than 4 wastes tokens without proportional benefit. Point each example at a specific ambiguous scenario causing problems, not at generic happy paths. Each example must show reasoning, not just input and output, explaining why one action was chosen over plausible alternatives so the model learns the decision principle and generalises to novel inputs. Without reasoning the model learns literals such as queries mentioning order numbers go to lookup_order, with reasoning it learns the principle that specific identifiers route to specific lookup tools.
Examples also reduce hallucination in extraction. Varying document structures such as inline citations, bibliographies, narrative descriptions, structured tables, headers, and embedded text taught through examples help the model handle structural variety without inventing data. A report that lists expenses in a table on one page and buries them in a paragraph on the next is the canonical case where, without examples, the model extracts the table correctly but leaves narrative fields empty or fabricated, and with examples for both structures extraction quality climbs.
Where examples live and how many to use
Few-shot examples appear either embedded in the system prompt as part of the Examples component of RACCE for patterns that should shape every response, or inline in the messages array as a sequence of user and assistant pairs before the actual query for patterns specific to that query. Proximity matters, examples immediately before the user message influence the response more than examples earlier in context due to recency, the same effect that shapes system prompt placement.
Task complexity determines count, not input volume. Simple three-category sentiment classification needs 2 to 3 examples, a complex extraction with 20 fields, nested objects, and conditional logic may need 8 to 10. Few-shot with 2 to 5 offers the best reliability to cost ratio for most applications, zero-shot is cheapest but least reliable for format-sensitive tasks, and many-shot with 5 or more helps nuanced tasks at the cost of context and potential dilution of system instructions.
Ordering, balance, and format hygiene
Example ordering affects output because later examples carry more influence near the query. For balanced classification mix categories rather than grouping all positives then all negatives, grouping biases the model toward the last group seen. If chain-of-thought examples are used the last example's reasoning pattern is most likely to be emulated, and a 5 to 10 percent accuracy swing between orderings is common.
Label balance creates a prior that biases predictions. Four positives and one negative in a five-example sentiment set teaches the model a positive prior comparable to adding hundreds of majority-class training examples. For balanced classification use equal class counts, for genuinely skewed production distributions a proportionally skewed set may be intentional but trades minority accuracy for majority accuracy. Format consistency is non-negotiable, even a missing colon or inconsistent capitalisation is modelled as intentional and may be reproduced, so every example must use identical formatting.
Dynamic few-shot selection is the production pattern when the example bank is too large for a static prompt. Embedding similarity retrieval finds semantically similar examples, keyword overlap such as TF-IDF or BM25 finds term-sharing examples, hybrid combines both, and cluster-based selection serves very large banks of 1000 or more where nearest neighbour is noisy.
Examples with reasoning and the email classification case
Examples without reasoning teach literal pattern matching that fails on novel phrasing. Examples with reasoning that explains why a classification was chosen teach the underlying principle that transfers. For email triage the principle becomes the primary intent is what the customer asks the agent to do, not which keywords appear most often, and a billing question that mentions a refund in passing stays a billing question when the ask is about the bill, while an explicit money back request becomes a refund request.
The same principle resolves code review inconsistency for variable shadowing. Shadowing within an arrow function scope is limited and does not create a bug, so the example classifies it as minor with reasoning about scope. Shadowing across a function boundary that hides a parameter creates real risk and the example classifies it as critical with reasoning about the hidden parameter. With those two examples the model generalises that severity depends on scope and whether something important is hidden, not on whether the word shadow appears.
Mechanism and API surface
- In-context learning from 2 to 4 examples
- Targeted examples pointed at the ambiguous scenarios causing problems, fewer than 2 does not establish a pattern, more than 4 wastes tokens without proportional benefit for most tasks.
- Examples with reasoning over bare pairs
- Each example shows input, reasoning for why one choice beats alternatives, then output, so the model learns the principle that generalises rather than the literal mapping that does not.
- Placement and proximity
- Examples in system prompt shape every response, examples in messages as user assistant pairs before the query shape that query. Closer to the query means stronger influence due to recency.
- Ordering and label balance
- Later examples influence output more, so mix classes for classification rather than grouping positives then negatives. Equal class counts produce neutral prior, skewed counts produce a prior for the majority class.
- Format consistency down to punctuation
- Every example uses identical formatting, a missing colon or capitalisation difference is modelled as intentional and may be reproduced in the response.
- Dynamic few-shot selection
- Embedding similarity, keyword overlap, or hybrid retrieval selects the most relevant examples at request time for large banks, cluster-based selection scales to 1000 or more where nearest neighbour is noisy.
Distinctions that decide answers
| This | Not this | How to tell them apart |
|---|---|---|
| Zero-shot | Few-shot | Zero-shot relies on instructions alone, few-shot shows examples. Few-shot is more reliable for format sensitive tasks, zero-shot is cheaper for simple transformations. |
| Few-shot examples | More detailed instructions | Examples demonstrate the pattern via input-output pairs, more instructions describe the pattern in prose. Examples eliminate interpretation variance, prose reduces it but does not eliminate it. |
| Examples with reasoning | Examples without reasoning | With reasoning the model learns the decision principle and generalises to novel patterns, without reasoning it learns surface pattern matching that may not transfer. |
| Static few-shot | Dynamic few-shot selection | Static uses a fixed set baked into the prompt, dynamic retrieves the most similar examples at request time using embedding or keyword search. Dynamic scales to large example banks, static is simpler to maintain. |
| Few-shot for classification | Schema changes for fabrication | Few-shot teaches consistent extraction from varied structures, schema changes such as making fields optional or nullable prevent fabrication of absent values. The two address different problems and often stack. |
| Few-shot examples | Confidence thresholds | Few-shot teaches the correct judgement for ambiguous cases, confidence thresholds filter by self-reported certainty which is poorly calibrated. Few-shot fixes judgement, confidence fixes routing. |
Traps
The tempting answer. Add more detailed prose when detailed instructions already produce inconsistent output structures across invocations.
Why it fails. Prose descriptions still rely on interpretation and more precise prose does not eliminate interpretation ambiguity.
What is correct. Add few-shot examples demonstrating the exact desired format, with reasoning when the format depends on a judgement.
The tempting answer. Treat input-output pairs as templates the model copies rather than principles it generalises.
Why it fails. Bare pairs teach surface matching, so novel phrasing fails when the literal token does not appear.
What is correct. Include reasoning for why each decision was made alongside the pair, so the model learns the principle behind the mapping that transfers to new inputs.
The tempting answer. Filter inconsistent judgement calls with a confidence cutoff instead of teaching the judgement.
Why it fails. Thresholds are poorly calibrated and do not address the root cause of inconsistency.
What is correct. Show few-shot examples demonstrating the correct judgement for ambiguous cases, with reasoning that names the decision boundary.
The tempting answer. Present all positives first then all negatives because grouping feels organised.
Why it fails. Later examples have more influence than earlier ones, grouping biases the model toward the last group seen and a 5 to 10 percent accuracy swing is common.
What is correct. Mix classes so no class dominates the recency window.
The tempting answer. Add many more examples to feel thorough rather than selecting a small well-chosen set.
Why it fails. Two to four well-chosen examples covering the standard case and key edge cases are enough, beyond that token cost rises without proportional benefit and system instructions may be diluted.
What is correct. Keep 2 to 4 targeted examples covering standard plus edge, then verify generalisation on a novel case.
The tempting answer. Treat small formatting differences such as a missing colon or inconsistent capitalisation as harmless.
Why it fails. The model treats inconsistency as intentional pattern and may reproduce it, so a colon drop in one example becomes a random colon drop in output.
What is correct. Use identical formatting down to punctuation across every example.
Structured Output
Guarantee schema-valid JSON through tool schemas and constrained decoding, then design nullable enums and redundancy fields that handle real document variation.
What you need to know
When you need guaranteed schema-compliant structured output from Claude there is a clear reliability hierarchy, and committing it to memory pays directly on the exam. At the top is tool_use with JSON schemas or the API-level structured outputs mode, which eliminates JSON syntax errors entirely through constrained decoding. Below that is prompt-based JSON where the model is asked to output JSON in a text response, which is probabilistic and will periodically produce malformed output with missing brackets, trailing commas, unquoted keys, or markdown code fences that break parsers.
With tool_use the tool's JSON schema constrains the shape of what Claude returns, and the separate tool_choice parameter controls whether the model must call a tool at all. Prompt-based extraction gives no structural guarantees, so a production parser that expects pure JSON will eventually receive a preamble such as Here is the JSON, a forgotten required field, or commentary before the object, and will fail. The exam builds directly on this hierarchy by asking which approach guarantees schema-valid JSON and which still requires a fallback parser.
The tool_choice parameter has three modes that the exam tests directly. Auto is the default and lets the model decide whether to call a tool or return text, so it may answer conversationally without ever producing structured output. Any forces the model to call a tool but lets it choose which one from the tools array, which is correct when you have multiple extraction schemas and document type is unknown. Forced tool selection with type tool and name guarantees the model calls exactly that tool regardless of input, and the name must character-for-character match a tool in the array or the request returns a 400 error. Use forced selection when a step must run before other steps, such as metadata extraction that must precede enrichment.
Schema design prevents entire classes of errors at the structural level before any model judgement is involved. Optional and nullable fields are the primary defence against fabrication: if a field is required the model is pressured to produce a value even when the source has none, if the field is nullable the model can honestly return null. An unclear enum value handles genuinely ambiguous cases, an other value paired with a freeform detail string handles long-tail categories the predefined enum does not cover, and format normalisation rules in the prompt such as all dates in ISO 8601 or all currency as decimal numbers without symbols enforce consistency alongside the schema.
The structured outputs mode at the API level works through constrained decoding at the sampler. The sampler maintains the set of valid next tokens given the partial output and the target schema, and tokens that would violate the schema such as a string where a number is required or a closing brace before required fields are filled are assigned zero probability at generation time. The model cannot generate them. This is a logit-level intervention, not post-processing. Strict tool use with strict true on a tool definition applies the same mechanism to tool call arguments, guaranteeing the arguments exactly match the provided JSON schema with no missing required fields, no extra fields, and correct types throughout, provided additionalProperties false is set.
Hierarchy and the tool_choice contract
Prompt-based JSON depends on model compliance, which is best effort, while API-level structured outputs and strict tool_use depend on constrained decoding, which is enforced. That difference is why a pipeline that must produce machine-parseable output for a downstream system should never rely on prompt instructions alone; the exam marks the schema-backed path as the only guarantee.
Choosing among auto, any, and forced selection is a design decision. Use auto only when a conversational response is a legitimate alternative to a tool call. Use any when a document could match several schemas and the model should choose the best fit. Use forced selection when a mandatory step must run and the model's preference must not override the workflow order.
Schema patterns that prevent fabrication and brittleness
Make fields nullable with a type union such as string or null or with nullable true depending on the schema dialect, so absence is an honest signal rather than an invented value. Add unclear to enums for genuinely indeterminate cases rather than forcing the model to pick a precise label from an incomplete set.
Pair an other enum value with a freeform detail string so unseen categories do not fail validation and are still captured verbatim. Add a description to every property so a field named tier with description Customer tier one of bronze, silver, gold is unambiguous where tier alone could be a string, integer, or label. Add format normalisation rules in the prompt because the schema enforces structure while the prompt enforces formatting consistency.
Build self-correction into the schema with redundancy fields such as calculated_total versus stated_total plus total_discrepancy, or conflict_detected for contradictory source text, so discrepancy detection surfaces inline without external recomputation for every document.
What tool_use guarantees and what it does not
Tool_use with JSON schemas eliminates JSON syntax errors, the class that includes malformed JSON, missing required fields, wrong data types for structurally enforced fields, and extra fields when strict. It does not prevent semantic errors: values that do not sum correctly, data placed in the wrong field, or fabricated values for missing information inside structurally valid JSON. Those require validation logic in a separate layer, which is the subject of the next task.
This separation is why production needs both layers. Schema catches structure, semantic validation catches correctness, and the two together determine whether a document is accepted, retried, or routed to human review.
Mechanism and API surface
- Reliability hierarchy tool_use and structured outputs over prompt JSON
- tool_use with JSON schemas and tool use with a JSON schema with json_schema eliminate syntax errors via constrained decoding, prompt-based JSON in text is probabilistic and will produce unparseable output in production.
- tool_choice auto, any, and forced
- auto lets the model return text, any forces a tool call but lets the model choose which, forced with type tool and name guarantees exactly that tool and 400s on a name mismatch.
- Nullable, unclear, and other patterns
- Nullable type unions allow honest null when information is absent, unclear handles ambiguous cases, other paired with a freeform detail string captures long-tail categories without validation failure.
- Self-correction redundancy fields
- calculated_total versus stated_total plus total_discrepancy boolean, conflict_detected for contradictions, and detected_pattern per finding enable inline discrepancy detection and dismissal-driven prompt improvement.
- Format normalisation alongside schema
- Schema enforces structure, prompt rules such as all dates in ISO 8601 and all currency as decimal numbers without symbols enforce formatting consistency.
Distinctions that decide answers
| This | Not this | How to tell them apart |
|---|---|---|
| Prompt-based JSON | API-level structured outputs | Prompt-based JSON is probabilistic and may produce malformed output. API-level structured outputs use constrained decoding to enforce the schema at generation time. |
| tool_choice auto | tool_choice any | auto lets the model respond with text instead of calling a tool, any requires a tool call but lets the model choose which. For guaranteed structured output with unknown document type use any. |
| tool_choice any | Forced tool selection | any lets the model pick which tool, forced with type tool and name requires exactly the named tool. Use forced when a step must run before others such as mandatory metadata extraction. |
| Optional or nullable fields | Required fields | Required pressures the model to fabricate when information is absent, nullable allows honest null. Use nullable when the source may lack the field. |
| Schema validation for structure | Semantic validation for correctness | Schema catches type errors and missing required fields, semantic validation catches arithmetic errors, misplaced values, and fabricated content. Both are required. |
| strict true on a tool | Non-strict tool use | Strict applies constrained decoding guaranteeing schema conformance, non-strict relies on best-effort compliance. Strict requires additionalProperties false and correct optional handling. |
Traps
The tempting answer. Treat tool_use with JSON schemas as a complete guarantee against any extraction mistake.
Why it fails. Structural enforcement feels comprehensive but only covers syntax. Semantic errors such as wrong sums, values in wrong fields, or fabricated values for absent data remain inside valid JSON.
What is correct. Add semantic validation and retry with error feedback for correctness, the schema guarantees structure, validation guarantees correctness.
The tempting answer. Use auto when you need guaranteed structured output because both involve tools.
Why it fails. auto allows the model to return text instead of calling any tool, so structured output is not guaranteed. Any guarantees a tool call.
What is correct. Use any for guaranteed structured output with unknown document type, use forced selection when a specific tool must run.
The tempting answer. Mark every field required to ensure nothing is missed.
Why it fails. Required fields pressure the model to invent plausible values when the source lacks information, which is worse than an honest null.
What is correct. Make fields where the source may lack information nullable or optional so the model can return null honestly.
The tempting answer. Enable strict true without additionalProperties false because it looks like a minor detail.
Why it fails. Strict requires additionalProperties false to explicitly reject extra fields, without it the schema cannot be enforced through constrained decoding.
What is correct. Always set additionalProperties false and include all optional properties in properties even when not required when using strict.
The tempting answer. Keep prompt-based JSON because adding a JSON schema feels like overhead.
Why it fails. Prompt-based JSON periodically produces malformed output and downstream parsers break on that failure class.
What is correct. Use structured outputs or strict tool_use at the API level, which cost nothing extra and eliminate the syntax failure class.
The tempting answer. Omit description on schema properties because the field name seems self-explanatory.
Why it fails. Descriptions guide the model's interpretation of ambiguous names, a field named tier without description could be a string, integer, or label, with description it is unambiguous.
What is correct. Add a description to every property, especially any name that admits multiple interpretations.
Validation and Retry Loops
Add semantic validation and retry with document plus failed output plus specific error so fixable mistakes self-correct and unfixable ones route to human review.
What you need to know
Production extraction systems fail for reasons no schema can prevent. Documents arrive in unexpected formats, line items do not sum to the stated total, values end up in the wrong fields, and plausible looking values appear for data that was never in the source. The question is not whether failures happen but how the system responds, and the validation retry pattern turns those failures into self-correcting workflows.
The correct retry pattern sends three pieces of information back to the model: the original document so the model has the source to re-examine, the failed extraction so it can see what it produced, and the specific validation error so it knows exactly what went wrong. This beats naive retry by a wide margin, because without the specific error the model has no guidance for what to fix and usually reproduces the same mistake, while with it the model can target its correction such as re-examining the document for missed line items, checking field placement, or recalculating a total.
Retries have a clear effectiveness boundary and the exam tests this boundary most aggressively in this task. Retries are effective for format mismatches such as wrong date format or inconsistent currency notation, structural output errors such as values in wrong fields or incorrect nesting, misplaced values where data exists but was extracted into the wrong field, and mathematical errors such as a missed line item affecting a total. Retries are not effective for information genuinely absent from the source document, data that exists only in an external document not provided to the model, or fields requiring knowledge the model does not have. If a document genuinely does not contain a department name, no amount of retrying will produce a correct value; the correct action is to flag for human review or return null when the schema allows it.
The exam distinguishes two error categories that must be handled separately. Schema syntax errors are malformed JSON, missing required fields, and wrong data types, and are eliminated entirely by tool_use with JSON schemas from the previous task. Semantic validation errors are correct JSON structure but incorrect values: line items that do not sum, dates with invalid ordering, values in the wrong fields. Schema validation cannot catch semantic errors because the JSON is structurally valid, only business-rule validation can. The overlap between the two tasks is intentional and the exam tests whether you understand what tool_use solves and what it does not.
Pydantic is the canonical Python validation layer alongside JSON Schema. A Pydantic model does two jobs at once: parsing enforces structure such as types, required fields, and enums, while validators enforce semantics such as cross-field arithmetic or date ordering. Both failure kinds surface through one ValidationError with machine-readable errors naming the field and the broken rule, and the except branch of that validation is the retry-with-error-feedback pattern where Pydantic supplies the specific error in a form that can be formatted straight into the retry prompt. Self-correction schema design reinforces this by building error detection into the extraction shape itself, for example calculated_total versus stated_total with total_discrepancy, conflict_detected for contradictory source text, or detected_pattern per finding for systematic prompt improvement.
The three-part retry message and the fixable boundary
A retry message that contains only the original prompt gives the model the same ambiguous input and produces the same mistake. A retry message that contains the original document, the failed JSON, and the specific validation error in the form what was expected versus what was found gives the model a target, such as Validation errors line_items sum to 450.00 but stated_total is 500.00, please re-extract ensuring all line items are captured.
The boundary decision precedes the retry. Format mismatches, structural errors, misplaced values, and arithmetic errors are fixable and should be retried. Absent information, external data not in source, and knowledge the model lacks are not fixable and should route to human review. The exam presents both shapes and expects the boundary to be applied before choosing retry, not as an afterthought.
Pydantic parsing versus validation and the SDK integration
Parsing enforces structure at parse time through model_validate_json and related entry points, while validation runs after parsing via model_validator with mode after and can access all fields to enforce cross-field rules. Raising ValueError inside the validator adds to the ValidationError returned by model_validate, with loc for the path to the failing field, msg for the error message, and type for the category, all directly formattable into the retry prompt without manual translation.
External validation logic recomputes checks after extraction, while self-correction fields in the schema surface discrepancies without external logic. The two are complementary: external validators are the enforcement, self-correction fields are the surface that makes discrepancy visible inline.
Schema redundancy and dismissal-driven improvement
Schema redundancy builds error detection into the extraction output. calculated_total is populated by summing line_items amount, stated_total is read verbatim from the document, total_discrepancy is true when they differ, conflict_detected flags contradictory text in the source such as payment due 30 days in one section and net 60 in another, and detected_pattern on each finding names the specific construct that triggered the finding.
The detected_pattern field creates a systematic improvement loop. When developers dismiss findings triggered by a specific pattern, that pattern likely needs prompt refinement, not a generic quality complaint. Tracking frequency and dismissal rate per pattern prioritises which prompts to refine next, closing the loop from extract to validate to collect dismissal data to refine prompts to repeat.
Mechanism and API surface
- Three-part retry with error feedback
- Original document plus failed extraction plus specific validation error in the retry message. Without the specific error the model reproduces the same mistake, with it the model targets correction such as missed line items or misplaced fields.
- Fixable versus unfixable boundary
- Fixable: format mismatches, structural errors, misplaced values, arithmetic errors. Not fixable: genuinely absent information, data in an external document not provided, knowledge the model lacks. Retries only on the first category.
- Pydantic structure versus semantics
- Parsing enforces types, required fields, and enums. Validators with model_validator mode after enforce cross-field arithmetic, date ordering, and conditional rules. Both surface via ValidationError with loc, msg, and type.
- Self-correction fields in the schema
- calculated_total versus stated_total plus total_discrepancy, conflict_detected for contradictions, and detected_pattern per finding build detection into the output shape rather than only in external logic.
- Retry budget and routing
- Maximum typically three retries, beyond that route to human review. Routing for absent fields uses nullable schema and null return, while low-confidence routing uses calibrated thresholds from labelled validation sets.
Distinctions that decide answers
| This | Not this | How to tell them apart |
|---|---|---|
| Schema syntax errors | Semantic validation errors | Syntax is malformed JSON, missing required fields, wrong types, eliminated by tool_use with schemas. Semantic is correct JSON with incorrect values such as wrong sums or misplaced fields, requiring validation logic. |
| Retry with error feedback | Naive retry | Naive resends the same prompt, retry with error feedback sends document, failed extraction, and specific error. The latter is dramatically more effective at producing corrected output. |
| Fixable errors | Unfixable errors | Fixable is format mismatches, structural errors, misplaced values, and math errors. Unfixable is absent information, external data not in source, or unavailable knowledge. Retry only the first category. |
| Schema validation | Pydantic validation | Schema enforces structure via JSON Schema, Pydantic enforces both structure and semantics via validators. Schema catches syntax, Pydantic catches arithmetic and business-rule violations. |
| Single total field | calculated_total plus stated_total | A single total hides discrepancies, the dual field plus total_discrepancy surfaces them inline, the schema redundancy pattern for math consistency. |
| External validation logic | Self-correction fields in schema | External recomputes checks after extraction, self-correction builds the check into the schema surfacing discrepancies without external recomputation for each document. |
Traps
The tempting answer. Retry any extraction failure because retry is the default reflex when something fails.
Why it fails. Retries fix format mismatches, structural errors, misplaced values, and math errors, but they cannot produce information genuinely absent from the source, such as a department name the document never contains.
What is correct. Distinguish fixable from unfixable before retrying and route absent-data cases to human review or null when the schema allows it.
The tempting answer. Retry by resending the same prompt or only the original document because the retry message is easier to build that way.
Why it fails. Without the specific error the model has no guidance and typically reproduces the same mistake.
What is correct. Include the original document, the failed extraction, and the specific validation error stating what was expected versus what was found.
The tempting answer. Rely on schema validation because tool_use enforces the schema.
Why it fails. Schema validation catches syntax errors only, semantic errors such as wrong sums, misplaced values, and fabricated data remain inside valid JSON.
What is correct. Add semantic checks with Pydantic validators and route detected discrepancies through the retry loop.
The tempting answer. Drop Pydantic because the schema already enforces structure.
Why it fails. Schemas cannot express cross-field semantic rules such as sums that must match or dates that must be ordered, and validators produce the specific per-field messages the retry loop feeds back.
What is correct. Keep Pydantic validators for business rules and use their ValidationError loc, msg, and type directly in the retry prompt.
The tempting answer. Keep retrying the same document because more attempts feel more thorough.
Why it fails. Beyond a budget of three the cost exceeds value and unfixable cases will never succeed regardless of attempts.
What is correct. Cap retries at three and route to human review on exhaustion.
The tempting answer. Route any failure through the retry path because the retry infrastructure already exists.
Why it fails. Retries on absent information never succeed and waste API cost while the correct action would have been a null return or a human queue.
What is correct. Apply the fixable versus unfixable boundary before invoking retry, and treat absent data as a routing decision not a correction problem.
Batch Processing
Use the Message Batches API for latency-tolerant work at 50 percent savings and keep blocking workflows synchronous, correlating every batch result through custom_id.
What you need to know
The Message Batches API is a cost optimisation tool with hard constraints that the exam tests directly, and you design around those constraints rather than working around them. The fixed properties are 50 percent cost savings compared to synchronous API calls, results that become available when every request finishes or at the 24 hour ceiling whichever comes first, no per-item latency promise even though most batches complete well inside an hour, and custom_id fields for correlating request-response pairs. These are not tuning knobs, they are the contract.
The matching rule is the single most tested concept from this task. The synchronous Messages API is for blocking workflows where someone or something is waiting for the result: pre-merge checks in CI, real-time code review feedback, or any workflow where developers are blocked pending completion. The Batch API is for latency-tolerant workflows where results are consumed later: overnight technical debt reports, weekly code audit summaries, nightly test generation runs, or batch document extraction. The exam presents a manager who proposes switching everything to batch for the cost savings, and the correct answer keeps blocking workflows synchronous while moving only latency-tolerant workflows to batch, because a 50 percent saving that breaks a merge gate is not a saving.
When designing batch schedules you must plan for the worst case, not the typical case. Results often arrive in well under an hour, but the platform guarantee is within 24 hours, so latency-tolerant designs must budget for the full window. If an organisation requires a 30 hour SLA for a report, the math is 30 hours minus 24 hours of processing equals 6 hours of buffer for collecting requests, validating inputs, and absorbing operational delay. The final batch must therefore be submitted at least 24 hours before the deadline, with submissions every 4 to 6 hours inside the buffer so a fresh batch is always in flight. Working backwards from the SLA is the pattern the exam tests, not polling for status.
A batch request is a single asynchronous unit with no interactive loop between your code and the model in the middle of it. Documentation confirms that tool use including server tools, multi-turn conversation history, system messages, vision, and extended thinking can all appear inside a batched request, so the naive claim that tools are unavailable is too strong. What is genuinely unavailable is a control flow where your own code inspects a tool result and then decides the next request, because nothing runs on your side until the batch ends. Any workflow whose next step depends on reading an intermediate result therefore needs separate requests, and if that decision must happen within seconds it needs the synchronous path. The unsupported request parameters are the ones that assume interactivity or per-request scheduling: streaming, fast-mode speed selection, stateful thread fields, cache and context routing hints, and a zero token ceiling.
Two production disciplines determine batch economics more than the per-token discount. First, sample set refinement before full submission: take 5 to 10 representative documents covering the range of formats, edge cases, and types, iterate on the sample refining prompts, examples, and schema until accuracy is high, then submit the full batch. A 90 percent first-pass success rate on 1000 documents means 100 retries, a 60 percent rate means 400 retries, four times the resubmission cost, so refinement is the highest leverage activity. Second, failure handling resubmits only the failures with targeted modifications, not the entire batch, with modifications such as chunking oversized documents, simplifying prompts for unusual structures, or adding format-specific few-shot examples, each retry batch reusing custom_id with a suffix so correlation stays unambiguous.
Request shape, limits, and correlation
The Message Batches API accepts up to 100000 requests per batch or 256 MB total request size, whichever limit is reached first. Each request carries a unique custom_id that is the correlation key to its eventual response, and the array of requests wraps parameters identical to a synchronous Messages call such as model, max_tokens, and messages. The batch creation response returns a batch identifier whose status can be queried and whose results are retrieved once processing completes.
Results iterate over the array and are checked per item. A successful item has type succeeded and carries the same shape as a synchronous response, a failed item has type errored with an error field. Failure handling filters by type, looks up the original document by custom_id, and constructs a retry batch containing only the failed items with targeted modifications. The per-request model and parameters can vary, so a single batch may contain heterogeneous requests as long as each has its own custom_id.
The matching rule and the 24 hour schedule math
Blocking workflows have someone waiting, latency-tolerant workflows consume results later, and the label determines the API. The exam probes this by naming overnight reports, weekly audits, and nightly runs as batch-eligible and pre-merge checks and real-time screening as synchronous-only, with a manager proposing to batch everything as the distractor. The correct answer preserves synchronous for anything blocking and reserves batch for anything tolerant, because batch has no SLA.
The 24 hour maximum is the planning number, not the observed typical latency. The schedule works backwards from the consumer deadline: consumer deadline minus 24 hours is the latest submission time for the final batch, with a buffer for collection and validation. Submitting every 4 to 6 hours inside that buffer keeps throughput high while respecting the guarantee, and a fallback that tries batch then switches to synchronous on timeout defeats the batch contract because latency cannot be predicted per item.
Cost leverage through sample refinement and targeted resubmission
Submitting a large unrefined batch feels fast but is the most expensive path. Refining on a small representative sample first raises first-pass success from the 60 percent range into the 90 percent range, and the difference on 1000 documents is 400 failures versus 100 failures that each require a retry batch. The cost of a few sample iterations is repaid in avoided resubmission.
Resubmission must be targeted, not wholesale. Oversized documents need chunking into line-item requests, documents with unusual structures need simplified prompts, and structurally varied documents need format-specific few-shot examples. Each pattern gets its own modification, and the retry batch carries the same custom_id convention with a retry suffix so a second failure can be traced to the original document without ambiguity.
Mechanism and API surface
- Fixed batch contract
- 50 percent savings versus synchronous, results available when every request finishes or at the 24 hour ceiling whichever comes first with no per-item latency promise, and custom_id correlation for request-response pairs.
- Matching rule blocking versus latency-tolerant
- Synchronous for blocking workflows where someone waits such as pre-merge checks, batch for latency-tolerant workflows such as overnight reports, weekly audits, and nightly generation where results are consumed later.
- Batch limits and shape
- Up to 100000 requests per batch or 256 MB total size whichever is first, requests array with custom_id plus params per item matching a synchronous call, models and parameters can vary per item in the same batch.
- Result handling with custom_id
- Iterate results, filter by type succeeded versus errored, use custom_id to look up original inputs, retry batch contains only failures with modifications and a suffixed custom_id for correlation.
- SLA schedule working backwards
- Consumer deadline minus 24 hours is latest submission for the final batch, with 4 to 6 hour submission cadence inside the buffer. Use worst-case latency for planning, not observed best case.
- Sample refinement as cost leverage
- Refine prompts on 5 to 10 representative documents before the full batch, covering formats, edge cases, and types. Higher first-pass success directly reduces retry count and total batch cost.
Distinctions that decide answers
| This | Not this | How to tell them apart |
|---|---|---|
| Synchronous Messages API | Batch API | Synchronous returns immediately at full price, batch returns within 24 hours at 50 percent lower cost. Use synchronous for blocking, batch for latency-tolerant. |
| Blocking workflow | Latency-tolerant workflow | Blocking has someone waiting for the result such as pre-merge checks, latency-tolerant consumes results later such as overnight reports. The matching rule is the most tested concept in this task. |
| Best-case latency | Worst-case latency | Best case is often much less than 24 hours, worst case is the guaranteed 24 hours. Design around worst case, not best case, because there is no SLA. |
| custom_id correlation | Anonymous requests | custom_id is the unique identifier per request used to match responses to requests. Without it batch results cannot be correlated and failure handling is impossible. |
| Resubmitting only failures | Resubmitting the entire batch | Resubmitting only the failed custom_id values with targeted modifications saves cost on already successful documents, resubmitting the whole batch wastes cost. |
| Queue with bounded latency | Batch with fixed collection | A queue processes items as they arrive with bounded per-item latency, batch processes a fixed collection together with no per-item guarantee. Queues suit continuously arriving tolerant workloads, pure batch suits one-shot large jobs. |
Traps
The tempting answer. Move every workflow to batch because 50 percent savings applies uniformly.
Why it fails. Blocking workflows where developers wait for results must remain synchronous. Batch has up to 24 hours with no SLA and will block merges unpredictably regardless of how often it finishes early in practice.
What is correct. Keep blocking workflows on synchronous and move only latency-tolerant workflows to batch, with the saving applied to the tolerant subset alone.
The tempting answer. Design a blocking workflow around typical batch speed such as minutes rather than the guaranteed maximum.
Why it fails. Typical latency is much less than 24 hours but there is no SLA, so designing for best case is unsafe for anything blocking.
What is correct. Design around the 24 hour maximum and keep any workflow that cannot tolerate that window on synchronous.
The tempting answer. Read the absence of an interactive loop as tools being unsupported inside a batch.
Why it fails. Documentation lists tool use including server tools, multi-turn history, and extended thinking among what can be batched. What is missing is your own code reading an intermediate result and choosing the next request, because nothing runs on your side until the batch ends.
What is correct. Batch anything that completes as one asynchronous unit, and use the synchronous path only when your control flow must inspect an intermediate result before deciding what to send next.
The tempting answer. Resubmit the full batch because tracking which items failed requires extra bookkeeping.
Why it fails. Already successful documents are billed again for no reason, which directly wastes the cost savings the batch was meant to capture.
What is correct. Identify failures by custom_id, look up the original inputs, and resubmit only the failed items with the modification each pattern needs.
The tempting answer. Submit the full 5000-document batch first and refine after seeing results because refinement feels like extra work.
Why it fails. A 60 percent first-pass rate means four times the retry count of a 90 percent rate, so the cost of unrefined submission exceeds the cost of a few sample iterations.
What is correct. Refine on 5 to 10 representative documents first, then submit the full batch with the improved prompts.
The tempting answer. Monitor batch status with polling to decide when to submit the next batch.
Why it fails. Polling does not solve the scheduling problem, the platform may still hold a batch for the full window regardless of polling frequency.
What is correct. Work backwards from the consumer deadline to determine the latest submission time for the final batch, then submit at fixed intervals of 4 to 6 hours inside the buffer.
Multi-Pass Review
Split large audits into per-file passes plus an integration pass from an independent instance, with calibrated confidence routing to focus human review where it matters.
What you need to know
When Claude reviews its own output in the same session it starts at a disadvantage because it still carries the reasoning it used to generate that output. The model remembers why it chose each approach, classified each finding at a particular severity, or selected certain values, and is less likely to question those decisions. This is not a defect to argue with but a property to design around, and the design that works is an independent review instance: a separate Claude invocation without the prior reasoning context that approaches the output fresh and judges it on what it actually says.
The exam tests this decision directly. When presented with options for improving review quality, an answer that adds please review carefully to the same session is never the correct improvement over an answer that uses a separate model instance. The separate instance is significantly more effective at catching subtle issues because it carries no prior commitment to the decisions it is about to evaluate, while the self-review instance filters evidence through the lens of why it already chose that path.
Large reviews such as multi-file pull requests, complex extraction pipelines, and broad code audits suffer from attention dilution when processed in a single pass. The symptoms are recognisable: detailed feedback on some files and superficial comments on others, obvious bugs missed in the middle of the review, and contradictory findings where a pattern is flagged as problematic in one file while identical code is approved in another. A larger context window does not fix this. The problem is not capacity but attention quality, the model can hold more text but still spreads its attention unevenly across files.
The fix is to split the review into focused passes. Pass one is per-file local analysis, where each file is examined individually with a focused review prompt so every file receives full attention regardless of its position in the batch. This eliminates the inconsistent depth and the middle-file misses, because each invocation sees only one file. Per-file analysis also parallelises naturally since files are independent inputs, so all per-file reviews can run concurrently. Pass two is cross-file integration, where a separate pass receives all per-file findings and checks for cross-file issues that no single-file review could identify: data flow between modules, consistent API usage across services, dependency conflicts, and contradictions among the per-file findings themselves.
Confidence-based routing complements multi-pass review. The review prompt asks the model to report a confidence value typically from 0.0 to 1.0 alongside each finding plus the reasoning for the score. High confidence findings report directly to developers, low confidence findings route to a human review queue. Raw self-reported confidence is poorly calibrated and must not be used for automated decisions without calibration, which uses labelled validation sets where correct outputs are already known. Running the system against the labelled set and measuring how reported confidence tracks actual accuracy identifies the threshold that best separates usually correct from often wrong, and the set should cover the full range of finding types and severity levels rather than an aggregate accuracy number.
Independent instances and why self-review confirms
Independent review uses a fresh API call whose messages array starts without the generation system prompt or conversation history, containing only the output to review plus a focused review prompt. The fresh invocation approaches the output as if seeing it for the first time, which is why its findings are measurably more thorough on subtle issues even though the model and prompt may be the same otherwise.
The cost-quality tradeoff is explicit. Independent multi-pass review with confidence routing costs more than single-pass self-review, and the tradeoff is worth it when review quality directly affects production reliability such as CI pipelines, financial extraction, or compliance analysis. It is not worth it for low-stakes outputs where occasional misses are tolerable, which is why the exam frames the choice around downstream impact rather than abstract thoroughness.
Per-file local passes plus integration pass
Per-file passes ensure consistent depth by giving each file a dedicated context window with the same focused prompt such as Review this file for bugs, security issues, and logic errors. Each file is an independent input that can run concurrently, so total wall-clock time drops even though more API calls are made, and middle files receive the same scrutiny as the first and last.
The integration pass runs sequentially after all per-file reviews complete because it depends on their findings. Its prompt names the cross-file checks explicitly: data flow inconsistencies between modules, contradictory patterns flagged in different files, and API contract violations across service boundaries. Skipping the integration pass saves one call but removes the only mechanism that catches these systemic issues.
Calibrated confidence routing and dismissal-driven refinement
Confidence scores are reported in the structured output alongside each finding with reasoning for the score, and the routing decision of direct report versus human review is based on the score relative to a calibrated threshold. Calibration validates the threshold against labelled sets, so the score becomes a routing signal with a known accuracy boundary rather than an uncalibrated claim.
The detected_pattern field per finding creates a systematic improvement loop beyond calibration. When developers dismiss findings triggered by a specific pattern, that pattern's dismissal rate indicates a prompt refinement priority. Patterns with high dismissal rates need refined prompts, patterns with low dismissal rates are working correctly, and tracking frequency plus dismissal rate per pattern produces a prioritised refinement backlog that continuously improves review quality from production data.
Mechanism and API surface
- Independent review instance
- Separate API call with no prior generation system prompt or conversation history, only the output to review plus a focused review prompt, so the model judges the code on its content alone without bias from I chose this because.
- Per-file local analysis
- One focused review invocation per file ensuring consistent depth for every file including middle files, parallelisable because files are independent, carries the same prompt and criteria for each file.
- Cross-file integration pass
- Sequential pass after all per-file reviews that receives all per-file findings and checks data flow, API contract violations, dependency conflicts, and contradictions among per-file findings.
- Confidence reporting per finding
- Review prompt requests a 0.0 to 1.0 confidence value plus reasoning per finding, enabling per-finding routing rather than a single document-level threshold.
- Calibration on labelled validation sets
- Run the system against a labelled set where correct outputs are known, compare reported confidence to actual accuracy, and set the routing threshold that best separates usually correct from often wrong across the full range of finding types and severities.
- detected_pattern for systematic refinement
- Each finding carries the specific pattern that triggered it, so dismissal tracking by pattern identifies which prompts need refinement and creates a feedback loop from production data.
Distinctions that decide answers
| This | Not this | How to tell them apart |
|---|---|---|
| Self-review in same session | Independent review instance | Self-review retains reasoning context and tends to confirm prior decisions, independent has no prior reasoning and approaches output fresh, with significantly higher effectiveness on subtle issues. |
| Single-pass review | Multi-pass review with per-file plus integration | Single pass produces inconsistent depth, missed bugs, and contradictions from attention dilution. Multi-pass splits into per-file local passes plus a cross-file integration pass for consistent depth and systemic coverage. |
| Larger context window | Focused per-file passes | Larger capacity does not improve attention quality, the model still gives uneven attention across files. Focused per-file passes ensure consistent depth regardless of window size. |
| Raw confidence scores | Calibrated confidence thresholds | Raw confidence is poorly calibrated self-reported certainty, calibrated thresholds are validated against labelled sets and suitable for routing decisions. Using raw confidence for automation is an anti-pattern. |
| Per-file analysis | Cross-file integration | Per-file catches local issues within a single file such as bugs, security, and logic errors. Integration catches systemic issues across files such as data flow, contract violations, and contradictions. |
| Routing to human review | Auto-approving findings | High confidence findings auto-report to developers, low confidence findings route to human review. The threshold determines treatment and must be calibrated. |
Traps
The tempting answer. Review the generated output in the same session to avoid extra API calls because the model already has full context.
Why it fails. The model retains its generation reasoning and is less likely to question decisions it already justified, so subtle issues survive that an independent instance would catch.
What is correct. Use a separate Claude instance with a fresh session and no prior generation history so the reviewer approaches the output without the bias of prior reasoning.
The tempting answer. Run a single review pass over fourteen files because single-pass is simpler to implement.
Why it fails. Single-pass multi-file reviews produce inconsistent depth, miss bugs in the middle, and generate contradictory findings due to attention dilution.
What is correct. Split into per-file local passes for consistent depth plus a cross-file integration pass for systemic issues.
The tempting answer. Switch to a higher-tier model with a larger context window so fourteen files fit more easily.
Why it fails. The problem is attention quality, not capacity. A larger window holds more text but the model still spreads attention unevenly across files.
What is correct. Keep focused per-file passes, which ensure consistent depth regardless of window size.
The tempting answer. Route findings automatically by raw confidence scores such as treating 0.85 and above as high confidence.
Why it fails. Raw self-reported confidence is poorly calibrated and the score does not reliably track actual accuracy.
What is correct. Calibrate thresholds against labelled validation sets that cover the range of finding types and severities, then route by the calibrated boundary.
The tempting answer. Run per-file reviews sequentially because sequential execution is simpler to code.
Why it fails. Per-file reviews are independent inputs that can run concurrently, so sequential execution wastes wall-clock time for no quality gain.
What is correct. Parallelise per-file reviews and run the integration pass sequentially only after all per-file results are available.
The tempting answer. Skip the integration pass to save an API call when per-file reviews appear self-consistent.
Why it fails. The integration pass is the only mechanism that catches cross-file issues such as data flow between modules, API contract violations, and contradictions among per-file findings, which per-file passes cannot evaluate.
What is correct. Always run the integration pass after per-file analysis, regardless of whether per-file findings appear independent.