Prompt Engineering and Structured Output

Mechanism reference: The system parameter: what it is and what it is not

The Messages API exposes system as a top-level parameter alongside model, max_tokens, and messages. It is not a message role. The valid role values inside messages are user and assistant and those messages must strictly alternate, while system sits outside that array and persists across every turn of the conversation. This structural placement matters. Content placed in system is present before the first user message and is reapplied unchanged on every subsequent turn, so it frames how every user message is interpreted. Content placed inside a user message competes for attention as one instruction among many and is more vulnerable to being overridden by later user turns.

system can be supplied as a single string or as an array of content blocks. The array form is the form that enables prompt caching and structured organisation of complex system prompts. When system is an array, each entry has type: "text", a text string, and optionally cache_control: { type: "ephemeral" } on stable sections. A typical construction keeps the role, task framing, and guidelines in a cached block and leaves per-request context outside it. Systematic prompt caching of the system prompt plus tool definitions typically represents a large share of token usage, and caching that block can reduce cost at scale, but the mechanism itself is API-level persistence and caching, not model-level authority.

The distinction between steering and enforcement is central. The system prompt is higher-authority guidance that the model is trained to weigh more heavily than an individual user turn, but it is not a hard enforcement boundary. Documentation and lessons explicitly state that a determined or adversarial user message can still degrade adherence to a system instruction, especially for nuanced or unusual requests. The lesson draw is that system is the correct and primary place for standing behavioural constraints, and that repeating critical constraints in every user message is not the recommended repair. Better prompt design and, for hard requirements, application-layer validation are the documented repairs. Where the reference page emphasises putting role and constraints in the system prompt, it is describing this steering-without-guarantee property accurately.

Ownership in this mechanism sits with the developer at authoring time and with the API at delivery time. The SDK constructs the request, the API delivers system with higher-authority framing, and application code owns enforcement where guarantees are needed, such as schema validation after the response or a structured outputs configuration that constrains the sampling grammar. The model owns interpretation, but interpretation is bounded by what the system prompt actually defines as a decidable predicate.

Mechanism reference: RACCE decomposition: where role, audience, criteria, constraints, and examples belong

System prompt design lessons decompose an effective system prompt into five components: Role, Audience, Criteria, Constraints, and Examples, abbreviated RACCE. Each component maps to a failure mode when omitted.

Role activates domain knowledge and perspective. A concrete role such as You are a senior security engineer specializing in web application security. Review code for security vulnerabilities with attention to OWASP classifications. reliably surfaces relevant decision patterns, while You are a helpful assistant does not change behaviour in a measurable way. Lesson guidance on role specificity describes an additive narrowing effect: each qualifier such as seniority, specialization, and context reduces the chance of generic output. Role also interacts with placement. A role established in system is persistent across turns, while a role introduced in a user message provides temporary or supplementary framing for a single turn, such as asking the architect to review from a QA perspective for one exchange. When multiple roles appear, they compete, and specificity, recency, and system-level persistence determine which influence dominates. Conflicting roles produce averaging rather than selection.

Audience calibrates vocabulary, technical depth, and tone. The reference task for system prompts treats audience as a first-class decision rather than a stylistic afterthought. Without an audience, the model defaults to a generic register that may be too technical or too shallow. An audience definition such as explaining to a junior developer who knows Python but not distributed systems removes ambiguity about how much background can be assumed. Audience matters for code review prompts because the consumer is often a developer reading bot comments under time pressure, which strengthens the need for crisp severity labels and concise remediation rather than expansive explanation. The audience component is therefore not decorative. It determines how findings are framed for the developer who must act on them.

Criteria define how quality is evaluated and which tradeoffs to favour when objectives compete. Prioritized criteria outperform unordered lists of desirable properties. A line such as Prioritize correctness over brevity. If you must choose between a complete answer and an accurate one, choose accuracy and note what was omitted. gives the model a clear tiebreaker, while Be helpful, accurate, and concise forces the model to resolve the tradeoff silently and inconsistently. For code review, criteria are the REPORT and SKIP sets, the conjunctive conditions that must all hold before a finding is emitted, and the per-severity examples that anchor classification. Without criteria, the model makes its own quality assumptions, and those assumptions are the precise source of false positives that the reference page targets.

Constraints are hard boundaries the model must not cross, regardless of what the user asks. Good constraints are specific, testable, and few enough to be followable. Lesson material notes research on instruction-following degrading significantly beyond roughly ten distinct behavioural rules, and recommends keeping constraints to five to ten prioritized items. The test for a constraint is whether compliance can be verified from output alone. Never provide a specific diagnosis, you can describe symptoms and suggest consulting a doctor passes the test, while Be ethical does not. For code review pipelines, constraints include negative constraints such as Output ONLY the JSON object. Do not generate any text before or after it which are more reliable than sampling adjustments such as lowering temperature for suppressing preambles. Negative constraints matter most where format drift would break parsing.

Examples embedded in the system prompt demonstrate the exact interaction pattern, format, and edge-case handling that the application requires. Placement at the end of the system prompt maximises influence because examples closest to the user message carry the most weight. Two to four input/output pairs are the documented range for the examples component, and the lesson example inventory shows patterns such as clear request, escalation, and out-of-scope handling to establish both format and judgement boundaries. In the context of task 4.1, examples serve a calibration function for severity and a boundary function for category membership. Concrete code examples per severity tier create stable, checkable anchors that produce consistent classification across invocations, while contrastive pairs teach where the boundary sits rather than merely reinforcing a surface feature.

The recommended ordering of these components matches how the model processes the prompt: role first to establish context for everything that follows, then audience, then criteria, then constraints, then examples last and closest to the user content. This ordering also supports caching, because role, audience, criteria, and constraints are stable across requests while examples and the user task vary more. Structured with this ordering and tagged with delimiters, the system prompt becomes auditable: a reviewer can see which component handles which instruction and can change one component at a time during iteration, which is the documented safe update practice of shadowing new output against old before broad rollout.

Mechanism reference: Why instructions must be verifiable predicates, not adverbs

An instruction is only useful if the model can decide whether a candidate finding satisfies it without appeal to an ungrounded judgement. Lesson material formalises this as the testability requirement: a constraint passes if two reasonable people would agree on whether output followed it, and fails if that judgement remains debatable. Vague instructions fail the test because they are adverbs without operands. Be conservative does not name which construct to flag or which to skip. Only report high-confidence findings names a threshold but attaches it to confidence, a self-reported score that tracks reported certainty rather than membership in a defined category. Use your best judgement and prioritize precision over recall behave identically: they gesture at a tradeoff without defining the categories whose precision should improve. Each of these phrases can be appended to any prompt without changing the set of observable properties the model checks, so the classification function does not gain a new predicate and the false positive distribution does not move.

Verifiable predicates have a different logical form. They compare observables: flag a comment only when its claimed behaviour contradicts the code's actual behaviour, report only when a query uses string concatenation, the concatenated value is unsanitized, and the value originates from an untrusted entry point, report only when an amount exceeds the threshold and the merchant is first-seen and the shipping country mismatches the billing country. Each clause names a property that can be checked in the artifact alone, so a reviewer can determine which clause failed for a dismissed finding. Forensics records that replacing a vague line with a conjunctive predicate dominates precision more than any surrounding prose, because membership becomes decidable and auditable. The predicate also explains why repeating or intensifying the vague line never repairs precision: extremely conservative and be careful and avoid false positives map to the same ambiguous prior under harsher pressure, trimming volume without changing which families misfire. The dial that moves precision is not pressure but the presence of a new checkable clause.

A practical consequence is that every vague verb in a system prompt is an opportunity for inconsistency. Review this code without criteria leaves the dimension open, so code review outputs vary across dimensions such as style, bugs, security, and performance depending on whatever prior was most salient at sampling time. Replacing review with review for exploitable injection and missing null guards, flagging only reachable untrusted input that reaches a sink without sanitization eliminates the open dimension. Similarly, summarize without audience and length is vague: summarize for a business executive in three bullets of one sentence each is a predicate that removes interpretation. The exam tests this by presenting vague prompts that sound reasonable and asking which improvement removes ambiguity, and the correct answer consistently introduces an observable criterion or a concrete dimension rather than a stricter adverb.

Mechanism reference: How examples and delimiters interact with the system layer

XML structured prompting lessons describe tags such as <instructions>, <context>, <contract>, <example>, <thinking>, and <answer> as labels that give the model a reliable way to identify which part is which when a prompt contains multiple sections. Claude's training included large amounts of structured XML content, so XML labels function as reliable content separators rather than decorative formatting. Tags help in four situations relevant to system prompts: when a prompt has three or more distinct sections, when multiple documents or sources are provided, when few-shot examples must be separated from the real task, and when user input could otherwise appear to be instructions.

The system layer and the delimiter layer are complementary, not interchangeable. The system layer establishes persistence and authority, while delimiters establish boundaries within the content that the system layer frames. Putting instructions in system without delimiters leaves internal sections unlabeled, so a long contract excerpt or a code diff can bleed into the instruction region. Wrapping the same content with tags while leaving it inside a user message leaves it without persistence, so a later user turn can still degrade adherence. The documented pattern places the standing frame in system and uses tags to separate its internal sections and to isolate volatile inputs. For example, placing the review criteria under <criteria> and the code under <code> inside the system prompt, or wrapping user-supplied input under <user_message> and instructing the system that content inside those tags must be treated as data only, combines authority with boundary clarity.

Guidance on nesting matters here: lessons recommend a maximum of three to four levels of nesting, favour flattening beyond that, and warn that empty tag pairs such as <reference_documents></reference_documents> are semantically ambiguous because the model can interpret an empty section as implying that content should exist and may invent documents to fill the gap. Two safer patterns are omitting the tag entirely when its content is absent, or including explicit placeholder text such as None provided so absence is stated rather than implied. This directly affects system prompts that conditionally include large context blocks. A conditional <reference_documents> tag that is sometimes empty should not be rendered as an empty pair in the system prompt; it should be omitted or given an explicit absence marker. The broader point is that delimiters are not about aesthetics. They change how the model assigns spans to roles such as instruction versus data, and they make that assignment robust when inputs contain characters that resemble instructions, such as </instructions> appearing inside user content.

Examples interact with this structure in a specific way. Lessons advise that examples embedded in the system prompt should sit at the end, closest to the user message, so that proximity increases their influence. XML makes the example boundary explicit: <examples> containing <example> elements with <input> and <output> children cleanly separates training signal from the live task input. In system prompts for code review, severity calibration examples and contrastive boundary pairs belong inside such an examples block rather than scattered through prose. The model then classifies by nearest-example matching rather than by interpreting an abstract adjective such as dangerous.

Mechanism reference: Placing critical instructions in long inputs

Long-context guidance warns that critical instructions buried in the middle of long content are disproportionately overlooked. When a prompt exceeds many thousands of tokens, later clauses receive diluted processing relative to early tokens when the task demands uniform scrutiny. Forensics shows that this manifests as position-correlated miss rates, such as reviewing long SOW documents or 150 endpoint references or large file sets where coverage of later sections measurably degrades, and where an instruction to weight every schedule equally or allocate equal scrutiny reduces misses only partially without structural change.

The documented repairs are structural rather than exhortative. First, keep critical instructions at the boundaries of the prompt where attention is strongest rather than in the middle of a long data block. Second, shard inputs that demand uniform coverage into per-unit independent calls, such as per-clause calls for a long contract where each call receives one clause plus the relevant playbook rule, followed by a separate cross-clause integration pass to catch interactions that span units. Third, use delimiters so that instructions and data occupy distinct tagged sections rather than running together as an indistinguishable stream. Adding more adverbs inside the same long prompt does not change the underlying attention distribution. The exam tests this by contrasting add an instruction to weight every section equally while keeping the entire document in one prompt with split the document into per-clause calls plus an integration pass, and only the latter is treated as structural.

For system prompts specifically, this means the system prompt itself should remain compact where possible, rely on caching for large stable guidelines, and avoid stuffing all per-request data into the system content. Large reference documents belong in the user message under labeled tags or in a cached block with explicit structure, not interleaved with the core instructions where they bury them. Where an instruction must survive in a long input, the combination of early placement, shallow XML structure, and per-unit sharding is the documented path, not intensifying the same instruction.

Mechanism reference: Guidance in a prompt versus enforcement in code or schema

Two distinct reliability strategies appear in this domain. Prompt-level guidance steers the model toward the right judgement by defining what to check, what to report, and what to skip. Enforcement at the API or application layer guarantees a structural property regardless of whether the model followed the guidance. The reference page covers the first strategy thoroughly but the full domain distinguishes them explicitly, and exam scenarios require knowing which layer owns which guarantee.

Prompt guidance includes the categorical criteria, the severity anchors, the negative constraints, and the placement of role and audience. It is the right tool for judgements that require interpretation of code and intent, such as deciding whether a comment contradicts actual behaviour or whether a pattern described on a page matches a listed vulnerability shape. Its failure mode is non-determinism and sensitivity to phrasing. Documented mitigation is careful design, examples, and verification, not stronger sampling settings.

Enforcement includes schema-constrained decoding and application-layer validation. Documentation describes output_config.format with {"type": "json_schema", "schema": ...} as a mechanism that enforces schema compliance by grammar-constrained sampling, and strict: true on a tool definition as the separate feature that enforces compliance on tool names and inputs. The older top-level output_format field and the associated beta header are accepted only for a transition period, while output_config.format is the current form. Structured outputs guarantee valid JSON matching a schema but do not guarantee semantic correctness: a field such as quantity: "30 minutes" can be syntactically valid while semantically wrong, and checks for semantic correctness still require validation logic outside the model. Two other enforcement realities matter: a refusal returns stop_reason: "refusal" with a 200 status and billed tokens, with refusal text taking precedence over the schema, and hitting the token limit truncates before the schema can be satisfied, so schema compliance is not absolute.

The split between guidance and enforcement determines how pipeline code is written. For a code review pipeline that must produce machine-readable findings, guidance defines what counts as a finding and what severity to assign, while enforcement validates that the response parses as valid JSON with the expected fields such as file, line, detected_pattern, and severity before findings are surfaced. Lesson material warns that asking for XML output without validation is fragile because the model may produce malformed XML such as missing closing tags or extra text outside tags, and recommends regex-based fallback extraction when XML is used as output. The same logic applies to severity labels: prompting Classify each finding as CRITICAL, HIGH, MEDIUM, or LOW is guidance that benefits from code examples per tier, but the downstream validator should still reject an unknown label or a missing field rather than assume the model always obeyed. Reserve prompts for steering judgement and schemas for guaranteeing shape, and add application checks for semantic plausibility where schema alone cannot catch the error.

Mechanism reference: Severity calibration and why prose fails

Prose severity definitions such as Critical: Issues that could cause system failures or data loss or Minor: Issues that affect code readability but not functionality share a structural defect: they map an abstract label to context-dependent estimates such as could cause or dangerous or slightly suboptimal without naming an observable property of the artifact. The model must infer what counts as dangerous from surrounding tokens, so identical null-pointer shapes in two files receive different labels without any artifact change. Widening the prose into longer paragraphs adds more interpretable words rather than new tests, so variance persists. Forensics records that even seemingly precise variants such as poses an immediate danger still produced substantial misclassification, and that a review of many critical flags found a large share belonging in a lower tier, while identical patterns were labeled critical in one file and low in another.

Concrete code examples per severity tier repair this by replacing interpretation with nearest-example matching. A labeled fragment such as a query that concatenates unsanitized external input directly into a sink is observable in the code before any judgement about danger. A naming inconsistency such as userName versus user_name within the same module is observable as a textual mismatch without functional risk. Once each tier is bound to such examples, classification asks which example the candidate resembles most closely, which is stable across runs. Evidence favours one worked example per level as the minimum viable anchor, with two to three for the most contested boundary or five to seven spanning the full range drawn from consensus or borderline cases where disagreement was historically highest. Providing many examples for one tier while omitting anchors for neighbouring tiers leaves novel findings without comparative context, so distribution across tiers matters as much as existence.

Mechanism reference: Confidence: why self-reported scores are not calibrated probabilities and how routing uses them correctly

Self-reported confidence is a score the model emits alongside a finding, such as confidence: 0.94 or confidence: 8 on a 1 to 10 scale, expressing how sure the model reports itself to be. That report comes from the same reasoning that produced the finding. It is not an independent measurement of correctness and it is not a calibrated probability that tracks empirical accuracy. Lesson and forensics material is explicit on this point: the model is often sure about wrong findings and hesitant about right ones, so raising the bar from 0.85 to 0.90 or 0.99 still lets confident false positives through while hiding correct but cautiously scored findings. The reference page phrases this compactly as LLM self-reported confidence is poorly calibrated and states the hierarchy that explicit criteria come first and confidence-based routing comes second. The documented path to reliability is criteria plus verification, with routing on top.

Three confusions make confidence attractive as a filter and explain why each fails. First, a quantitative form such as only report above 0.85 appears more precise than be conservative but preserves the same defect: the threshold is applied to an uncalibrated signal, so the gate filters by reported certainty, not correctness. Second, lowering temperature to zero is sometimes proposed as an alternative to a confidence gate, but determinism without a decidable criterion reproduces the same vague judgement deterministically rather than correcting it. Third, confidence can be scoped per category rather than globally, such as separate thresholds for input handling versus authentication versus data exposure, but per-category thresholds tuned to historical dismissal rates still tune around the symptom while leaving the misclassification logic intact. Each bar gates an underspecified category, so surface-feature errors such as parameterized queries flagged as injection or env-var secret loads flagged as hardcoded credentials continue to clear the gate, because the reported score for those findings was already high for the wrong reason.

Confidence becomes useful after criteria exist and after it has been calibrated against labeled truth. Calibration is a post hoc mapping built from a validation set per field and per segment, such as invoice_total versus indemnification clause or comparison table versus appendix, where the set is labeled by human adjudication or a known-correct reference and is held out from prompt iteration. The calibration step records what a reported 0.95 actually meant empirically for each segment, for example 0.94 measured on party name at reported 0.95 versus 0.71 measured on indemnification at the same reported score. Once fitted, the calibrated score can drive thresholds that reflect measured precision: a threshold where the fitted curve crosses the target precision for auto-release, with a lower band routed to review and contradictory material routed regardless of score. Without this per-segment fit there is no principled way to choose a threshold, because the same reported number means different things in different segments.

Two additional practices make a confidence router trustworthy. First, stratified random sampling of a fixed fraction of high-confidence outputs, weekly and across document type, field type, or confidence band, surfaces confident-but-wrong errors that no filter would show and yields a pipeline-wide error estimate that can be tracked over time. Lowering the threshold or adding heuristic rules for known error sources such as comparison tables covers only previously seen shapes and cannot estimate the overall high-confidence error rate, while re-extracting and comparing variance fails for structural errors that have zero variance and return the same wrong value on every run. Stratified sampling is the mechanism that measures whether explicit criteria actually reduce the high-confidence error rate from cycle to cycle.

Second, granularity matters. Field-level or category-level scores enable targeted routing while a single document-level score masks the weak segment by averaging. A document whose unit field is extracted at 0.94 correctness while quantity is at 0.31 can look healthy under a document score, so reviewers miss the concentrated weakness. Lessons on confidence scoring describe this as achieving efficiency by routing the 20 percent that contains 60 percent of errors once granularity and calibration are in place. The aggregate score is adequate only when disposition is whole-document and per-field accuracy is genuinely uniform.

The production distinction is therefore between filtering and routing. Filtering discards low-confidence material silently, which loses information needed for safety and hides the right population in the wrong bucket. Routing preserves every finding but sends low-confidence or contradictory or not-yet-validated material to a human or an independent fresh-context verifier, while high-confidence material in validated segments may proceed. Nothing is suppressed for scoring low; low simply means more scrutiny. That is why the reference hierarchy exists and why the exam penalises any answer that proposes confidence as the primary gate before explicit criteria define what counts.

Mechanism reference: Trust, false positive spillover, and the role of criteria

A pipeline rarely emits one undifferentiated stream. It tags findings by category such as security, correctness, performance, style, naming, or documentation, or by family such as injection versus hardcoded-secret, or by rule such as color contrast versus alt text versus focus order. Per-category precision often diverges, with accurate categories operating at low false positive rates while one or two noisy tags operate at very high rates. Developers consume the stream as a single trust object. When one tag repeatedly delivers dismissed items, consumers develop a global heuristic such as ignore the bot or skim every high ticket rather than a per-tag reliability judgement. The noisy tag then bleeds credibility from the accurate tags, and the accurate findings lose effective recall even though their precision never changed.

Spillover explains why aggregate statistics mislead. A pipeline whose overall rate looks moderate can still suffer cross-category distrust when most false positives concentrate in one tag. Lowering the total count by trimming about a fifth through be conservative does not restore engagement when the remaining mix still contains frequent bad tags, because the prior that the bot is noisy was built from frequency of bad experiences rather than from the exact ratio. Showing confidence per finding or carrying a uniform strictness reduction across all categories fails for the same reason: the consumer applies the global ignore before evaluating any score, and the accurate categories are suppressed alongside the noisy ones. The harm model here is category-weighted, not aggregate, so per-category measurement is required to reason about trust.

Criteria are the lever that breaks the spillover, because they let the noisy category be identified and repaired without touching the accurate categories. Report-versus-skip sets that enumerate both sides, conjunctive conditions that require every clause such as reachable AND impactful before a finding is emitted, and per-severity code examples that anchor classification each reduce the contribution of a specific tag without changing the accurate stream. Trust restoration is therefore not an aggregate problem. It is a per-category precision problem solved by explicit definitions that make each noisy tag decidable.

Mechanism reference: The detected_pattern diagnostic and per-category analysis

When trust has already collapsed or when a team is deciding which repair to prioritize, coarse signals do not help a diagnosis. A finding stream that records only file, line, and description, or only a broad category such as security flagged 500 times, hides which trigger shape drove dismissals. Grouping by a self-rated confidence or a circular is_false_positive boolean asks the same classifier to judge itself. The forensics record identifies detected_pattern as the finest-grained diagnostic signal: a field that records the construct that triggered the flag, such as string concatenation in database query, user input passed to eval(), parameterized query as the safe counterpart, or list comprehension versus generator expression.

With detected_pattern present, dismissals can be aggregated per pattern, such as 73 percent dismissed on concatenation versus 8 percent on eval, and the dominant pattern becomes the target for explicit criteria and for examples. Without it, a team cannot distinguish two situations that demand different repairs: one where a single pattern accounts for most dismissals and needs a focused predicate, and one where dismissals are diffuse and point to a broader criteria gap. The field is diagnostic rather than curative: adding detected_pattern while leaving a noisy tag in the delivered stream does not restore trust, but adding it alongside a quarantine gives the team the signal needed to rewrite the right clause and to measure whether that rewrite moved the per-pattern dismissal rate. The rule for diagnostics is therefore to pair measurement with isolation: log the finest-grained available signal, act on delivery immediately, and verify the repair off-line.

Mechanism reference: Why contrastive few-shot pairs and single-polarity examples behave differently

Two ways of adding examples inside a system prompt produce opposite outcomes on the exam, and the distinction maps directly to whether the task is format imitation or boundary learning.

Format imitation is served by positive-only examples. A prompt that demonstrates the interaction pattern User: I need to return my order to Assistant: Could you share your order number teaches the structure to imitate. For code review, this would be examples that show how a finding should be formatted as JSON, which severity label to attach, and which fields to populate. Where the problem is format variance rather than judgement variance, a few positive demonstrations are the correct fix, consistent with the lesson note that examples calibrate tone and format when placed closest to the user input.

Boundary learning requires contrastive pairs that hold the surface feature constant across two sides that differ only on the precise property that flips the label. For example, two findings that both mention a string concatenation in a query but differ on whether the concatenated value traces to an untrusted entry point, or two messages that both contain a sensitive term but differ on whether the term is a directed attack versus self-reference or quotation for condemnation versus endorsement. Each pair carries a short because line naming the decision feature. The model given only the bad side learns feature implies flag as a rule and applies it to the benign side. Pairs that hold the surface token constant force the model to locate the actual decision feature such as directed versus self-referential or reachable versus isolated via parameterization. Once exposed, that feature transfers to phrasings never shown. Lesson framing that combines role with few-shot examples makes the same point: roles prime the reasoning approach while examples demonstrate the expected quality level and boundary, and together they outperform either alone.

Single-polarity positive-only sets such as eight few-shot, two per high-dismissal category, each pairing a flagged snippet with an instruction to down-weight fail because they increase the prior for feature implies flag without conditioning on the surrounding intent that actually determines correctness. The exam consistently treats four contrastive pairs as correct while treating six to eight genuine-violation-only examples or twelve weighted toward the positive side as incorrect even though the latter set is larger, because size does not substitute for the missing negative side. The negative side is not about balance for aesthetics. It is the only way to teach the model that the same surface feature can be benign and that a neighbouring predicate such as sanitization or parameterization or quoting determines the label.

The boundary for choosing which example strategy to use is therefore functional: if novel phrasings will continue to appear and the current rules are already written but the model still treats a surface token as sufficient evidence, contrastive pairs are the repair. If the prompt never defined its positive and negative sets in the first place, pairs without defined sets are premature: writing explicit report-versus-skip criteria comes first, then pairs that refine the boundary those criteria established. The reference page omits this nuance but the adjacent lesson on anti-patterns anticipates it: specificity is a dial, and review for O(n squared) time complexity and memory usage is better than review this code, which is better than no dimension at all, and refinement toward enumerated criteria precedes refinement of the boundary between its members.

Mechanism reference: System prompt role placement and the system versus user distinction revisited

Placement is worth a deeper treatment because the system prompt concept is frequently tested as a multiple-choice distinction: where to put role definition, hard constraints, output format, and the immediate task. Lesson structure treats the answer as a table rather than a rule of thumb: role, hard constraints, and output format that should hold for every turn belong in system, while the actual task for this turn and any one-off format change belong in the user message. The user versus system choice is not about emphasis. It is about persistence and overridability.

A role that belongs in system but is placed in a user message loses persistence. It applies for a turn and then decays, so a later user turn can drift the persona without any explicit override. A role that belongs in a user message, such as a temporary Now act as a QA engineer and review my test plan shift on top of a senior software architect system baseline, gains temporary framing without losing the primary persona. Conversely, a constraint that belongs in system but is left as a user instruction is not framed as higher-authority guidance and is more easily degraded by a subsequent user turn that pushes against it. The documented repair for a constraint that drifted mid-conversation is not to repeat the constraint in every user message, but to strengthen the system wording where it was underspecified or to add application-layer validation for the hard requirement.

Nested roles illustrate why persistence, recency, and specificity together determine outcome. A system prompt that says You are concise competing with a user message that says provide exhaustive detail does not deterministically choose one side. It averages them. A specific user role can then override a vague system role, but a specific system role dominates a vague user role. These hierarchy effects are predictable: system over user when both are comparably specified, recency adding weight to the most recent turn even when that instruction is less authoritative, specificity beating generality, and conflicts producing averaging. Choosing where a role lives is therefore a control over which hierarchy effect should dominate for the task.

Three implementation properties follow. First, the system prompt should take the most critical instruction first, because ordering inside the system prompt itself carries a primacy effect, and the first instruction is weighted more. Second, stable sections of the system prompt should be stored and delivered as cacheable blocks, because the system prompt plus tool definitions is the share of tokens most amenable to caching across requests. Third, changes to the system prompt should touch one component at a time, with shadow testing that compares old and new behaviour on a sample of real inputs and monitoring of regression signals such as tool call patterns, escalation rate, and refusal rate. Updating role and constraints in the same deployment is a characteristic anti-pattern because attribution of any behavioural shift becomes ambiguous.

Mechanism reference: Positive versus negative instruction framing

A separate framing choice within the system prompt is whether to describe what not to do or what to do instead. The reference page does not foreground this distinction, but forensics shows it is consistently tested: negative prohibitions such as NEVER include the original content in your explanation or You MUST NEVER, under ANY CIRCUMSTANCES, include original content risk reinforcing the undesired pattern, while positive instructions such as Describe flagged content by violation category, pattern type, and severity level only give the model an explicit allowed set to produce instead. Negative definitions leave the model to choose a replacement behaviour that often preserves the undesired pattern under different wording, while positive definitions constrain the output space to a specific shape. The same contrast appears in severity filtering. Only report HIGH and MEDIUM. Omit LOW unless asked with a positive classification ladder such as Classify each finding as CRITICAL, HIGH, MEDIUM, or LOW outperforms a vague negative such as Do not report minor findings, because the positive ladder names the exact alternative.

Lesson guidance on negative constraints adds precision to this picture: negative constraints are most valuable for output formatting where a positive rule alone still leaves a gap, such as respond only in JSON still allowing a Here is the JSON preamble that breaks parsing, so the reliable fix is an explicit negative such as Do not generate any text before or after the JSON object, no preamble, no explanation, no markdown code fences. That is a narrow exception to a general preference for positive framing. Where the task is to avoid a verbatim repetition, such as reproducing flagged content, positive framing dominates because negatives such as add examples showing what not to include demonstrate the bad shape the model is being asked to avoid and post-processing filters that remove quoted content are brittle on edge cases. The documented pattern is to prefer positive instruction that names the desired action, using negative constraints only for the specific gap an affirmative alone cannot close, and always pairing any negative with the positive replacement so the model has a well-defined alternative to emit.

Mechanism reference: Long inputs, attention dilution, and the verification gap

Two additional mechanisms explain why verification strategies succeed or fail, and both interact directly with system prompt design. The first is attention dilution across long inputs. The second is anchoring in self-review.

Long-contract and large-batch examples in forensics show that a single prompt ingesting a long artifact, such as an entire redlined SOW plus the playbook, a reference list of many endpoints, or a batch of many files, causes coverage of later sections to degrade even after the prompt explicitly asks the model to weight every schedule or section equally. Miss rates remain elevated because the instruction does not change the underlying attention distribution. The structural repair is to replace uniform weighting in a single call with per-unit independent calls. In its SOW form, that is per-clause calls where each call receives one clause plus the relevant playbook rule and aggregate findings, followed by a separate cross-clause integration pass that catches exposure emerging from interactions between schedules. In code review form, that is per-file processing where each file is evaluated against the same checkable rules, avoiding the burial of later files in a single session. The integration pass is not optional where interactions matter, because per-unit sharding alone would miss cross-unit effects.

Anchoring explains why same-thread verification fails. A pipeline that feeds the first pass finding and its reasoning back into the same conversation and asks the model to re-examine it inherits the prior framing, so reaffirmation remains high even though a material share of findings are non-issues. Even a second instance that is seeded with the original reasoning is instructed to uphold or overturn that reasoning, which frames the task as defence rather than fresh judgement. The repair is a second instance in a fresh conversation that receives only the clause or diff, the relevant playbook rule, and the asserted severity, without the first pass reasoning, and that independently judges whether the item is genuine. Any shared reasoning channel reintroduces the anchoring, so verification is trustworthy only when it is independent and grounded in the same explicit criteria that the first pass should have used.

Mechanism reference: Structured outputs and the boundary between prompting and enforcement

The DOC-URLS companion for this domain clarifies a terminology shift that matters for grounding. Structured outputs are now requested with output_config.format containing {"type": "json_schema", "schema": ...}. The older output_format top-level field and its associated beta header remain accepted for a transition period only. Strict tool use is the separate feature with strict: true on a tool, enforcing schema compliance on tool names and inputs by grammar-constrained sampling. The two features are independent and composable in one request. Schema compliance is therefore not a property of prompt wording at all. It is a sampling constraint enforced by the API.

That enforcement has limits that pipeline code must account for. A refusal returns stop_reason: "refusal" with a 200 status and billed tokens, and the refusal text takes precedence over the schema. Reaching the token limit also truncates output before the schema can be satisfied, so a large or unconstrained prompt that exhausts the budget will still produce incomplete output even though a schema was requested. For system prompt design, the consequence is that no amount of system wording replaces the structured outputs configuration when the consumer parses results programmatically. The system prompt defines what counts as a finding, structured outputs and application validation ensure that definition is delivered in a parseable shape, and confidence routing under calibration handles the residual uncertainty that neither enforcement can resolve. Each layer owns a different guarantee, and the pipeline fails when it asks one layer to do another layer's job, such as asking confidence to enforce a schema or asking a system sentence to guarantee format.

Ownership map

Version and terminology currency

Three currency notes affect how this material should be read by a candidate who may encounter older exam guide phrasing.

First, structured outputs terminology has shifted. Older material references output_format as the top-level mechanism for schema-constrained output. The current mechanism is output_config.format with {"type": "json_schema", "schema": ...}. Strict tool use with strict: true on a tool definition is the separate feature that constrains tool names and inputs. Community material that presents prompt-requested JSON versus tool use as the complete hierarchy is therefore incomplete. The full hierarchy is prompt-requested JSON, structured outputs through output_config.format, and strict tool use, with the last two composable and independent. A candidate should answer with output_config.format when asked which field enforces schema, and distinguish it from strict tool use.

Second, lesson variants that call system a message role use older phrasing. Messages carry role: "user" and role: "assistant", while system is a top-level parameter outside the alternating messages array. Behaviour is unchanged: system is the intended channel for standing, persistent instructions, carried as a string or as an array of text blocks with optional cache controls. Forensics distinguishes a role in system versus a role in a user message as persistence versus per-turn override.

Third, prompt caching is now a stable API feature rather than preview. The cache_control: { type: "ephemeral" } marker on stable system blocks is documented, and the note that a large system prompt plus tool definitions is worth caching because it is identical across requests is current. This affects what counts as a production-ready construction when shown a cached guidelines block, not which severity answer is correct.

No other version shifts affect task 4.1. Cost, throughput, and batch-window numbers belong to adjacent task statements and are treated there, not here. Where the reference page uses the legacy name for the trust bleed, the mitigation naming of temporary disable in policies is the stable one, and the additional variants quarantine, suppress from posted output, downgrade to a non-blocking channel, and stop surfacing while those rules are validated describe the same delivery quarantine rather than distinct mechanisms.

Official versus community divergence

Community material that appears in research feeds diverges from documentation on two points that directly affect this task. A candidate should answer with the documented position in each case and be prepared to recognise the community phrasing as a distractor.

Divergence 1: Whether confidence is a precise filter that tightens as the numeric gate rises. Community write-ups frequently present confidence as though raising 0.85 to 0.90 or 9 of 10 or 0.99 mechanically raises precision, and recommend per-category self-rated bars tuned to dismissal rates as though the score were measuring correctness. Forensics records that even 0.94 to 0.97 fields are later corrected while 0.80 items are correct, and that tightening the gate preserves the confident-but-wrong tail while hiding correct findings that were reported with lower numbers. The documented position is that self-reported confidence is not a calibrated probability and that raising the threshold before criteria exist and before calibration on labeled data does not change which families misfire. Community material that recommends global or per-category self-rated filtering as the first fix therefore contradicts documentation. A candidate should prefer explicit criteria first, calibrated routing second and should treat any answer that filters by self-rated confidence before defining what counts as incorrect.

Divergence 2: Whether vague replacement phrasing such as extremely conservative, be careful, use your best judgement, and avoid false positives constitutes an incremental improvement. Community posts often suggest that intensifying a vague line or scattering careful throughout the prompt will gradually improve precision, sometimes paired with show confidence or lower temperature so that the change appears technical. Lesson material formalises the opposite view as the testability criterion: if two reasonable people would disagree about whether a constraint was followed, the wording is not specific enough to be a boundary, no matter how strongly it is stated. Forensics records that these intensified variants produce the same single-category failure with only volume trimmed, and that they compete with explicit rules when both are present and can suppress genuine findings that feel uncertain. The documented repair is to replace the adverb with a predicate over observables or to replace a prose severity paragraph with a code example per tier, rather than to restate the adverb more forcefully. A candidate should treat any answer that strengthens a vague instruction without adding a new checkable clause as incorrect.

A narrower divergence appears in how severity consistency is presented. Community suggestions sometimes propose a mapping table of issue types to default severities or a requirement to explain reasoning for manual recalibration as the cure for inconsistency. Both are documented as insufficient. Type alone does not determine risk magnitude without a concrete code shape, and tracing does not fix the underlying prose mapping. Only per-tier code examples move consistency. The exam uses this incomplete community suggestion as a distractor precisely because it looks systematic without addressing the missing observable anchor.

Beyond the task statement

The reference page for 4.1 covers four of the mechanisms it assigns, but the adjacent lesson material contains topics that enrich a system prompt design answer and that the reference omits entirely.

Meta-prompting as a system prompt authoring aid. Meta-prompting treats prompt authoring as a generative task where a model helps write or improve a prompt, then the improved prompt is tested and iterated. For task 4.1, this matters because improving noisy review categories is an iterative rewrite where a meta prompt can propose explicit criteria and code examples, but only a held-out measurement can validate that precision moved. A candidate should therefore distinguish use a meta prompt to draft the rewrite as a workflow aid from re-enabling because the explicit rules themselves demonstrate improved precision as a validation claim. The first is legitimate, the second is not.

Extended thinking and its interaction with prompt wording. Extended thinking shifts reasoning into a dedicated thinking content block at the API level rather than a tag the application must parse from text. It helps when step-by-step reasoning genuinely benefits the task and can hurt where it does not, and the reference build exercise notes that reasoning does not fix vague criteria. For system prompts, enabling thinking or requesting output inside <thinking> does not correct an undefined boundary. Longer reasoning against an unclear standard yields longer justifications for the same ambiguous decisions. Thinking is complementary to explicit criteria and examples but never a substitute for them.

Role hierarchy and persona drift in long conversations. Lessons on role prompting detail how persona consistency can decay across many turns and recommend reinforcement patterns such as reminders to stay in role, explicit behavioural constraints such as never breaking character, and behavioural anchors such as always providing production examples. This is adjacent material that the reference does not connect to trust. A review bot that drifts from its code-review persona mid-conversation will amplify spillover, because a drifted persona is more likely to emit a finding that crosses a category boundary. The system prompt's role definition is therefore part of trust design: the persona frames which categories the assistant should treat as in-scope in the first place, and examples at the end reinforce that frame each turn.

Prompt caching and its placement implication. Lessons on system prompt design and on context management both note that system prompts are ideal candidates for prompt caching because they are identical across requests, and that large stable guidelines should be placed in a cached block where subsequent calls pay a fraction of the cost. Beyond cost, caching affects reliability. A long system prompt rewritten as two cached blocks, one for role and constraints and one for the large guidelines, is easier to audit than a monolithic string, and the cached boundary discourages the antipattern of interleaving volatile per-request data inside the system content. Candidates who recognise that a code example shows system: [{ type: "text", text: LARGE_GUIDELINES, cache_control: { type: "ephemeral" } }] are seeing a production-ready construction, not a debugging choice.

Negative constraints and the positive-over-negative preference. Updated lesson content separates negative constraints that are layered on top of positive instructions from standalone prohibitions, and notes that negatives are most reliable for format rather than judgement. That distinction refines the exam's positive-over-negative guidance: where the goal is to suppress verbatim reproduction of flagged content or to enforce a severity ladder, the correct answer names the desired action rather than restating the prohibition. Where the goal is to guarantee machine-parseable JSON, a precise negative added to the positive request, such as Do not generate any text before or after the JSON object, is the expected pattern.

Few-shot taxonomy beyond formatting. The few-shot lesson presents examples as the strongest lever for showing output shape and for handling edge cases, placed closest to the task. Lesson and forensics clarify the taxonomy: positive-only examples calibrate format, while contrastive pairs that hold a surface feature constant across flag versus skip calibrate the boundary. A system prompt for code review therefore needs both, in exam order: explicit criteria first, then the right example shape for the residual failure. Adding two per category positive-only demonstrations to a prompt whose criteria remain underspecified is therefore marked incorrect. Examples do not rescue criteria, they refine what criteria already define.

Instruction hierarchy and recoverability. Lesson material treats hierarchy explicitly: system instructions are the standing frame, user turns are per-request input, later instructions can override earlier ones when they compete, and system is intended to dominate without achieving a hard guarantee. For a review pipeline, hierarchy determines recoverability: a noisy category in the system prompt is recoverable by editing the persistent frame, while a per-turn user override is recoverable by removing the override and relying on the cleaner baseline. This is why trust-recovery designs disable delivery from the system configuration rather than adding a user turn that says ignore the noisy category this time. The former changes standing behaviour, the latter creates a turn-local exception that must be renewed.

The build exercise link. The reference page includes a multi-step build exercise that starts from a vague prompt, measures false positive behaviour on a small test set, rewrites to explicit criteria and code examples per severity tier, and then applies the trust strategy of disabling problematic categories. That exercise is the bridge between the system prompt task and production practice. It makes the transition from consistent classification as an impression to stable across repeated runs on the same snippets as an observable behaviour. Adjacent lessons treat build exercises as the way to prove that a prompt change survived shadowing and that the system layer still dominates after the change.

Worked production examples: Example A: Replacing a vague review prompt with explicit categorical criteria

A team runs a CI bot on every pull request. The original system prompt is a single sentence appended during a complaint-driven hotfix:

> Review this code. Be conservative. Only report high-confidence findings.

Measured behaviour matches the anti-pattern described in the reference page and in lessons on prompt anti-patterns: classification is inconsistent across runs, style nits are flagged as critical in one file and skipped in another, and genuine injection patterns are missed or marked minor depending on surrounding context. Intensifying the line to extremely conservative or use your best judgement or avoid false positives keeps the same prior under harsher pressure and preserves the single-category failure.

The rewrite replaces the adverb with checkable predicates, enumerated report and skip sets, and a conjunctive condition for vulnerability findings. It lives in system as a cached block, separates criteria from code with delimiters, and puts format constraints at the end with their negative companion. The observable change is that membership becomes decidable: a reviewer can verify whether userName versus user_name belongs in SKIP for locally consistent style, or whether a query with string concatenation that reaches a sink from an untrusted entry point without sanitization belongs in REPORT.

// Example 1: Vague prompt rewritten into explicit categorical criteria
// System prompt construction with RACCE ordering and XML delimiters

const SYSTEM_PROMPT_EXPLICIT_CRITERIA = [
  "You are a senior application security reviewer specializing in TypeScript.",
  "Audience: developers reading CI comments under time pressure. Be concise and actionable.",
  "Criteria for findings:",
  "<criteria>",
  "  REPORT: bugs where claimed comment behaviour contradicts actual code behaviour,",
  "          security findings where untrusted input reaches a sink without sanitization",
  "          with both reachability from user input AND impact of data exposure or unauthorized action.",
  "  SKIP: style, naming, formatting, linter-enforced checks, intentionally broad",
  "        exception handlers in batch jobs, and house logging style.",
  "  SEVERITY: defined by code examples per tier in <examples> below, not by prose labels.",
  "</criteria>",
  "<constraints>",
  "  1. Output valid JSON matching the response schema. Output ONLY the JSON object, no preamble, no markdown fences.",
  "  2. Never include the original flagged code verbatim. Describe category, pattern type, and severity only.",
  "  3. If evidence is insufficient to decide a clause, mark the finding as insufficient evidence and skip.",
  "</constraints>",
  "<examples>",
  "  <!-- severity and boundary examples live here, closest to the user input -->",
  "</examples>",
].join("\n");

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();

async function reviewDiff(diffText: string) {
  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 2048,
    system: [
      { type: "text", text: SYSTEM_PROMPT_EXPLICIT_CRITERIA, cache_control: { type: "ephemeral" } as const },
    ],
    messages: [
      { role: "user", content: `<code_diff>${diffText}</code_diff>\n<task>Apply <criteria> to the diff. Produce JSON findings.</task>` },
    ],
  });
  // Application-layer enforcement: parsing, field validation, severity enum check
  return JSON.parse(response.content[0].type === "text" ? response.content[0].text : "{}");
}

What this block proves: explicit criteria replace interpretation with a membership test, conjunctive conditions prevent single-property over-flagging such as flagging string concatenation that is config-internal, and delimiter separation prevents the diff from appearing to be instructions. The failure boundary is that explicit criteria do not help when a trust boundary is scoped too narrowly, such as restricting reachability to external HTTP requests while message queues are legitimate entry points. Widen that conjunct to the team's actual boundary rather than dropping the conjunction.

Worked production examples: Example B: Severity calibrated by concrete code, not prose

The second failure is severity instability. A system prompt that defines Critical: system failure or data loss. Minor: readability only. still maps an abstract adjective to a continuous estimate of could cause, so the same missing null guard is labelled critical in one file and low in another without artifact change. Expanding the paragraph adds more adjectives without adding a checkable property. Even adding numeric risk 0 to 100 preserves the defect because the number inherits the same prose calibration problem.

The repair binds each tier to labeled code fragments with a one-line rationale, so classification becomes nearest-example matching rather than estimation. The documented minimum is one code snippet per severity value, with stronger coverage at two to three for the contested boundary or five to seven spanning the full range drawn from consensus or borderline cases. Strings of the form userName versus user_name within the same module sit at the minor anchor. Unsanitized external input concatenated into a query sits at the critical anchor. Missing null guards on a may-be-null service sit at the major anchor. The start of the ladder must not be assumed to be format imitation: without anchors, numbers and prose together remain insufficient.

// Example 2: Severity anchored with concrete code per tier
// Each tier carries a snippet observable in the artifact, not a prose definition

const SEVERITY_ANCHORS_XML = `
<examples>
  <example tier="CRITICAL" rationale="external input concatenated without sanitization, reachable sink">
    <code>query = f"SELECT * FROM users WHERE id = {user_input}"</code>
    <fields><severity>CRITICAL</severity><category>security</category><detected_pattern>string concatenation in database query</detected_pattern></fields>
  </example>
  <example tier="CRITICAL" rationale="unsanitized value passed to eval-like sink from external origin">
    <code>eval(user_input)</code>
    <fields><severity>CRITICAL</severity><category>security</category><detected_pattern>user input passed to eval()</detected_pattern></fields>
  </example>
  <example tier="MAJOR" rationale="crash path, observable missing guard on may-be-null">
    <code>const user = cache.get(id); return user.name;</code>
    <fields><severity>MAJOR</severity><category>bug</category><detected_pattern>missing null check on cache lookup</detected_pattern></fields>
  </example>
  <example tier="MINOR" rationale="readability only, no functional risk">
    <code>let userName = "alice"; let user_name = "bob";</code>
    <fields><severity>MINOR</severity><category>style</category><detected_pattern>naming inconsistency within module</detected_pattern></fields>
  </example>
  <example tier="MINOR" rationale="style idiom already enforced by tooling, no novel risk">
    <code>if (x == null) throw new Error("missing"); // defensive guard per team lint</code>
    <fields><severity>LOW</severity><category>style</category><detected_pattern>defensive null check per linter</detected_pattern></fields>
  </example>
</examples>
`.trim();

const SYSTEM_WITH_SEVERITY_ANCHORS = [
  "You are a senior TypeScript reviewer reporting severity by example match.",
  "<criteria>",
  "  Classify severity by which anchored example the candidate most closely resembles.",
  "  Do not estimate danger from surrounding tokens. Compare to the nearest snippet and its rationale.",
  "</criteria>",
  SEVERITY_ANCHORS_XML,
  "<constraints>Output valid JSON. Only use severity values that appear in <examples>.</constraints>",
].join("\n");

What this block proves: severity labels become stable across invocations and files because the standard is observable. The failure boundary is per-tier coverage: anchoring only CRITICAL with multiple examples while omitting MAJOR or MINOR counterparts leaves novel findings without a comparative context, and a single idiosyncratic example per tier overfits that tier to a narrow shape. Choosing anchors from a consensus set such as consensus cases or borderline disagreement cases where human reviewers previously diverged strengthens transfer beyond hand-crafted snippets.

Worked production examples: Example C: system parameter used correctly with structured sections and delimiters

The third repair concerns where things live. A common construction puts everything in the user message, so standing instructions compete as one paragraph among many. The correct construction places the standing frame in system with stable blocks cached and volatile inputs in the user message wrapped with tags that prevent injection.

// Example 3: Correct system parameter construction with delimiters and caching

import Anthropic from "@anthropic-ai/sdk";

const SYSTEM_ROLE = "You are a senior code review assistant specializing in TypeScript security and correctness.";
const SYSTEM_GUIDELINES = `
<criteria>
  <report_when>
    <rule>claimed comment behaviour contradicts actual code behaviour</rule>
    <rule>query uses string concatenation AND value is unsanitized AND value originates from untrusted entry point</rule>
  </report_when>
  <skip_when>style preferences, local naming patterns, linter-enforced checks, sanctioned exception handlers</skip_when>
</criteria>
<response_format>
  JSON array of objects with fields: file, line, category, severity, detected_pattern, remediation.
</response_format>
`.trim();

const SYSTEM_NEGATIVE = "Output ONLY the JSON array. Do not generate any text before or after it. No preamble, no code fences.";

const client3 = new Anthropic();

async function reviewWithSystemAndDelimiters(code: string, fileName: string) {
  const response = await client3.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 4096,
    system: [
      { type: "text", text: SYSTEM_ROLE },
      { type: "text", text: SYSTEM_GUIDELINES, cache_control: { type: "ephemeral" } as const },
      { type: "text", text: SYSTEM_NEGATIVE },
    ],
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "Review the file below against <criteria>. Respect <response_format>." },
          { type: "text", text: `<file name="${fileName}"><code>${code}</code></file>` },
        ],
      },
    ],
  });
  const text = response.content.find((b) => b.type === "text")?.text ?? "[]";
  const parsed = JSON.parse(text);
  // Enforcement owns shape: reject unknown severity, missing file, or verbatim code echo before surfacing
  return Array.isArray(parsed) ? parsed.filter((f) => ["CRITICAL","HIGH","MEDIUM","LOW","MAJOR","MINOR"].includes(f.severity)) : [];
}

What this block proves: system carries persistence and authority, the cached block SYSTEM_GUIDELINES amortizes token cost, tags <criteria> and <file> keep instruction and data in distinct spans, and the negative constraint handles the preamble gap that temperature adjustment cannot close. The failure boundary is empty-tag ambiguity and over-nesting. Conditional sections such as <reference_documents> should be omitted or given None provided rather than rendered as <reference_documents></reference_documents> where the model may invent missing content, and nesting should be flattened beyond three to four levels. The observable output is parseable JSON with no surrounding prose, ready for downstream enforcement such as severity enum rejection.

Worked production examples: Example D: Category-disable and re-enable for trust recovery

Noisy categories that dominate dismissals destroy system-wide trust because developers apply a global ignore before evaluating any per-finding signal. The correct first bleed stop is to remove the noisy categories from delivered output. They continue offline against a validation set, are iterated with explicit criteria and code examples, and are re-enabled only once per-category precision is validated on held-out data. Lowering a threshold or showing confidence does not restore engagement when the remaining mix still contains frequent bad tags, and keeping all categories active while improving each gradually leaves the attention cost in place.

Delivery quarantine has many surface names that describe the same intervention: temporarily disable the documentation mismatch category, suppress the style category from posted output for now, downgrade to a non-blocking review channel, and stop surfacing the high false-positive categories while those rules are validated. The essential property is isolation of the noisy stream from the primary attention channel.

// Example 4: Category quarantine with held-out validation before re-enablement

type ReviewCategory = "bug" | "security" | "performance" | "style" | "naming" | "documentation";

interface ReviewConfig {
  enabledCategories: ReviewCategory[];
  quarantined: ReviewCategory[];
  thresholds: { perCategoryFalsePositiveMax: number; heldOutSetSize: number };
}

const CANDIDATE_CONFIG: ReviewConfig = {
  enabledCategories: ["bug", "security"],
  quarantined: ["style", "naming", "documentation"],
  thresholds: { perCategoryFalsePositiveMax: 0.15, heldOutSetSize: 80 },
};

// Offline iteration: rewrite quarantined prompts with explicit criteria + code examples
// then validate before broad rollout; reporter-only pilot or single-reviewer check is not sufficient

type ValidationResult = { category: ReviewCategory; falsePositiveRate: number; sample: string };

function shouldReEnable(result: ValidationResult, cfg: ReviewConfig): boolean {
  // Per-category gate on a held-out set of past pull requests with known findings
  // such as an 80-case consensus set, not the complaint sample
  if (result.falsePositiveRate > cfg.thresholds.perCategoryFalsePositiveMax) return false;
  // Also require per-pattern inspection via detected_pattern before re-enabling
  // e.g. concatenation shape versus eval shape must both clear the gate
  return true;
}

// Delivery path posts only enabledCategories. Quarantined categories run offline
// against the held-out set and are considered for re-enablement one at a time.

function filterFindingsForDelivery(findings: Array<{ category: ReviewCategory }>, cfg: ReviewConfig) {
  return findings.filter((f) => cfg.enabledCategories.includes(f.category));
}

function promoteAfterValidation(cfg: ReviewConfig, result: ValidationResult): ReviewConfig {
  if (!shouldReEnable(result, cfg)) return cfg;
  return {
    enabledCategories: [...cfg.enabledCategories, result.category],
    quarantined: cfg.quarantined.filter((c) => c !== result.category),
    thresholds: cfg.thresholds,
  };
}

What this block proves: quarantine attacks the consumer prior directly, so trust in accurate categories returns without change to those categories and without suppressing genuine findings in the accurate set. Offline iteration preserves coverage for later refinement, and per-category validation on a held-out set supplies evidence that the rewritten criteria improved measured precision on the failure shapes that drove dismissals. The failure boundaries are uniform disablement and premature re-enablement. Disabling is incorrect when noise is uniformly distributed and no clean high-precision remainder exists to preserve, where global criteria work scoped to blast radius is the expected path instead. Re-enabling immediately after new criteria are added, or after a reporter-only pilot or a single senior reviewer's small manual check, is consistently marked incorrect because the check does not cover the production distribution.

Worked production examples: Example E: Confidence used for routing, not for suppressing findings

Confidence belongs after criteria and after calibration. Before those steps, any gate over confidence is a filter over self-assessment from the same reasoning that produced the finding, which is why raising 0.85 to 0.90 or switching 1 to 10 versus 0 to 100 preserves the defect, and why confidence exceeds 0.7 suppresses correct uncertain material alongside confident errors. After explicit criteria and calibration, the same signal becomes a router that preserves every finding while directing low-certainty, contradictory, or not-yet-validated material to a human or an independent fresh-context verifier. High-confidence material in validated segments may proceed, and nothing is silently dropped for scoring low.

Granularity strengthens the router. Field-level scores such as quantity: 0.31 versus unit: 0.94 enable targeting of the weak segment, while a single document-level score hides it. Calibration method strengthens it further. Reported confidence must be mapped per field and per segment against a labeled set where human adjudication is truth, with stratified random sampling of a fixed fraction of high-confidence outputs each week across document type, field type, or confidence band to surface confident-but-wrong errors and to measure cycle-to-cycle improvement. The following production skeleton implements routing rather than suppression.

// Example 5: Confidence as a calibrated router, not a silent filter

type ExtractionField = "quantity" | "unit" | "party_name" | "indemnification_clause";

interface FieldConfidence {
  field: ExtractionField;
  reported: number;   // self-reported score from the model
  calibrated?: number; // mapped from a labeled validation set per field and segment
  contradictory?: boolean;
}

interface Calibration {
  field: ExtractionField;
  segment: string; // e.g. comparison_table, appendix, scanned_handwritten
  mapping: Record<string, number>; // reported bucket to measured accuracy, e.g. "0.95": 0.71
  threshold: { autoRelease: number; review: number };
  source: string; // label for the validation set that produced the mapping
}

const CALIBRATIONS: Calibration[] = [
  {
    field: "party_name",
    segment: "typed_contract",
    mapping: { "0.95": 0.99, "0.85": 0.93, "0.75": 0.82 },
    threshold: { autoRelease: 0.95, review: 0.80 },
    source: "labeled_validation_set_1000_docs_typed_contract",
  },
  {
    field: "indemnification_clause",
    segment: "appendix_heavy",
    mapping: { "0.95": 0.71, "0.85": 0.64, "0.75": 0.51 },
    threshold: { autoRelease: 0.97, review: 0.85 },
    source: "labeled_validation_set_600_docs_appendix_segment",
  },
];

type Route = "auto_release" | "human_review" | "fresh_context_verifier";

function routeField(f: FieldConfidence, calib: Calibration | undefined): Route {
  if (f.contradictory) return "human_review";
  if (!calib) return "human_review"; // uncalibrated segments do not auto-release
  const measured = calib.mapping[String(f.reported)] ?? 0;
  if (measured >= calib.threshold.autoRelease) return "auto_release";
  return "human_review";
}

// Never filter silently. Low or uncalibrated simply means more scrutiny.
function emitWithRouting(fields: FieldConfidence[]): Array<FieldConfidence & { route: Route }> {
  return fields.map((f) => {
    const calib = CALIBRATIONS.find((c) => c.field === f.field);
    return { ...f, route: routeField(f, calib) };
  });
}

// Stratified sampling of high-confidence material, weekly, to catch confident-but-wrong
// and to measure whether explicit criteria moved the high-confidence error rate

interface SamplingPlan {
  fraction: number; // fixed fraction of high-confidence extractions
  cadence: "weekly";
  strata: Array<{ dimension: "document_type" | "field_type" | "confidence_band"; values: string[] }>;
}

const SAMPLING_PLAN: SamplingPlan = {
  fraction: 0.05,
  cadence: "weekly",
  strata: [
    { dimension: "document_type", values: ["product_pages", "comparison_tables", "appendix_heavy"] },
    { dimension: "field_type", values: ["quantity", "indemnification_clause"] },
    { dimension: "confidence_band", values: ["85-90", "90-95", "95-100"] },
  ],
};

What this block proves: routing preserves information under scrutiny where filtering would discard it, per-field granularity prevents a strong field from masking a weak one, and per-segment calibration gives a principled basis for choosing thresholds that reflect measured precision rather than intuition. The sampling plan closes the loop: a fixed weekly fraction stratified across sensitive dimensions guarantees coverage of rare but error-prone types that pure random would miss and allows the team to track whether fixes such as few-shot examples for comparison tables actually moved the pipeline-wide estimate. The failure boundary is global thresholds, document-level scores, and lowering a global bar to match review capacity. A threshold is a quality signal, not a volume dial. The global document score is adequate only when disposition is whole-document and per-field accuracy is uniform, which the 99 versus 71 pattern shows is frequently not the case.

Build exercise material

The reference page prescribes a hands-on exercise that makes the abstract claim concrete by showing that vague instructions produce inconsistent classification while explicit criteria with code examples produce stable output. The following steps translate that prescription into verifiable build steps with observable outcomes. Each step states what to do, why it tests a task claim, and what success looks like. Steps assume access to the Messages API. All keywords use inline backticks as required for identifiers such as system, messages, content, cache_control, and detected_pattern.

Step 1: Establish a vague baseline. Create a system prompt with a vague line such as Review this code. Be conservative. Only report high-confidence findings. and a generic role such as You are a helpful assistant. Drive it with five code snippets that span known categories: one reachable injection with user_input reaching a concatenation sink, one parameterized query that is safe, one missing null guard, one naming inconsistency userName versus user_name, and one intentionally broad exception handler that the team lint permits. Send each snippet as <code> inside the user message with temperature at its default. Record the findings. Observable outcome that proves the step worked: classification is inconsistent. The same file run twice produces different severity labels, a style nit is labeled CRITICAL on one run, and a genuine injection is missed or marked MINOR on another. Lesson mapping predicts this variance because no dimension or criterion was defined, so the model guesses the dimension each time.

Step 2: Rewrite with explicit categorical criteria and positive framing. Replace the vague line with a RACCE-structured system prompt in system with three concrete components: a REPORT set bugs, security vulnerabilities, claimed versus actual comment contradictions and a SKIP set style preferences, local patterns, linter-enforced checks, house logging style, a conjunctive security predicate uses string concatenation AND concatenated value is unsanitized AND value originates from untrusted entry point, and a positive output directive Describe flagged content by violation category, pattern type, and severity level only with its negative companion Output ONLY the JSON object, no preamble. Wrap the code under <code> and the task under <task>. Observable outcome: the same five snippets now produce a stable binary decision per snippet. The injection that satisfies all three conjuncts is reported, the parameterized safe counterpart is skipped, and a reviewer can point to which clause failed for any skipped item. The change is attributable to the new predicate because no other parameter changed.

Step 3: Anchor severity with code examples per tier and place them last. Add an <examples> block at the end of the system prompt with one anchored example per severity value: CRITICAL with query = f"SELECT * FROM users WHERE id = {user_input}", MAJOR with const user = cache.get(id); return user.name; where the guard is missing, and MINOR with userName versus user_name within the same module, each with a one-line rationale. Keep tier prose minimal because prose alone preserves inconsistency even when it reads as precise. Observable outcome: severity labels converge across runs. Identical null-pointer shapes in two files receive the same label, and the model justifies labels by nearest-example resemblance rather than by estimating danger. The failure case to watch for is anchoring only one tier with multiple examples while omitting anchors for neighbouring tiers, which leaves novel findings without comparative context.

Step 4: Measure false positive rate on a held-out set and apply the trust strategy. Assemble a held-out set of past pull requests whose ground truth is known, such as an 80-case consensus set not used during rewrite. Compute per-category false positive rate before and after the rewrite. If any category remains above the team's gate, for example 0.15, apply delivery quarantine: remove that category from posted output, keep accurate categories at existing precision, and iterate offline with explicit criteria, code examples, and contrastive pairs that hold the surface feature constant across flag versus skip. Observable outcome: the delivered stream reflects the high-precision remainder and the quarantined category can be remeasured each cycle without polluting trust. Re-enablement requires the held-out rate to drop below the gate, not a reading that prose looks improved or a single-reviewer check on seen examples.

Step 5: Replace confidence filtering with calibrated routing. Add a confidence field alongside each finding, but do not gate delivery on it. Build a per-field calibration mapping from a labeled validation set per field and per segment, such as invoice_total versus indemnification clause or comparison table versus appendix, where reported 0.95 maps to different measured accuracy in different segments. Route fields whose calibrated score clears the segment-specific auto-release threshold, route low-confidence or contradictory material to human review, and route not-yet-validated material to a fresh-context verifier that sees only the artifact and rule. Supplement the router with stratified sampling of a fixed weekly fraction of high-confidence outputs across document type, field type, or confidence band. Observable outcome: low-confidence material is no longer silently suppressed and high-confidence blind spots become measurable. The step fails if the threshold is chosen globally or at the document level, where a strong field masks the weak one.

Step 6: Verify placement and enforcement layers end to end. Confirm that system carries role, criteria, constraints, and severity examples with caching on stable blocks, and that the user message carries only the labeled artifact such as <code> and <task>. Confirm that structured outputs with output_config.format are configured where JSON is parsed and that application code validates enums such as severity values and detected_pattern before surfacing. Confirm that long inputs use per-file or per-clause sharding plus integration where interactions span units, and that CRITICAL verification uses a fresh-context verifier rather than same-thread review. Observable outcome: one-component changes can be shadow-tested and parse failures or refusals with stop_reason: "refusal" do not surface as malformed findings because enforcement sits outside the prompt.

Mechanism reference: Count guidance and the documented versus exam divergence

The effective example count is the most tested surface of this task. The exam material centers on 2 to 4 examples and treats 2 as the floor below which no stable pattern forms and 4 as the practical ceiling beyond which token cost grows while accuracy does not. The live documentation states "include 3 to 5 examples for best results" and adds that you can ask Claude to evaluate your examples for relevance and diversity or to generate additional ones.

These two ranges are not in conflict once you read them as different lenses on the same underlying behavior. The documentation's 3 to 5 is a best-results recommendation for general use. The exam's 2 to 4 is the decision rule for the scenario where you are diagnosing an inconsistency and choosing the smallest demonstration that resolves it. Our lesson splits the difference by describing few-shot as 2 to 5 and then giving an exam tip that 2 to 4 well-chosen examples are optimal.

The shared, defensible principle is diminishing returns past four. Below two, the model cannot establish a pattern and may overfit to a single case. Above four to six, each added example buys little accuracy while linearly increasing token cost and latency, and a large homogeneous set can even dilute attention. The forensic analysis notes an open question where items range from "2-4" to "3-6"; the resolution is to present 2 to 4 as the recommended start and 3 to 6 as the nuanced ceiling for harder tasks, while always treating four as the point of diminishing returns. The number is a starting point to measure against, not a magical constant; the lesson explicitly says to benchmark your specific task rather than assume a fixed number.

The token-cost argument reinforces this: each example beyond the point of diminishing returns adds linear cost and latency while buying little accuracy, so the disciplined practice is to start at three, measure the failure modes that remain, and add a targeted example only for each uncovered edge case rather than scaling the whole set. A set of fifteen near-identical examples collapses to five diverse ones with equal quality once the pattern is saturated, which is why diversity and targeting outrank raw volume. The exam's recurring distractor of "ten to twenty examples for maximum coverage" fails for exactly this reason: near-identical examples add no information the model has not already generalized, and they crowd the context window that could hold diverse coverage instead.

One nuance the documentation adds that the exam material does not emphasize: you can use Claude itself to critique and extend your example set. Asking the model to evaluate your examples for relevance and diversity is a documented way to close the gap between a too-small set and a well-balanced one.

Mechanism reference: Relevance and diversity requirements

The live documentation names three properties every example set should have: relevant, diverse, and structured. These map directly onto exam principles.

Relevant means the examples mirror the actual use case closely. An example taken from a different domain teaches a pattern that does not transfer and can dilute the target signal. A representative example mirrors production length and content, demonstrating the pattern without irrelevant detail. Over-long or out-of-domain examples are explicitly discouraged.

Diverse means the examples vary enough that the model does not pick up unintended patterns, and that they cover edge cases. This is the formal grounding for two exam rules: target the failing scenario rather than the easy one, and cover the failing structure rather than the working one. If the model already extracts correctly from tables but returns empty fields on narrative text, adding more table examples controlled for nothing; the diversity requirement says to add the narrative-structure example. Diversity beats raw count: a handful of representative examples that include edge cases outperforms a large set of near-identical happy-path examples.

Structured means wrapping examples in <example> tags, with multiple examples inside <examples> tags, so Claude can distinguish them from instructions. This is the documented delimiter pattern and it is the safest way to satisfy the exam's requirement that examples have clean input/output demarcation. The XML lesson confirms that <example> inside <examples> makes it unambiguous which content is an example and which is the real task.

Mechanism reference: Why reasoning inside an example generalizes where a bare pair does not

This is the subtle mechanism the exam tests hardest, and the documentation supports it through the thinking note. A bare input-output pair teaches a literal mapping: for these surface features, emit this label. A pair that also shows the reasoning for the decision teaches the underlying principle, so the model can apply the same logic to novel inputs it has never seen.

The live documentation makes the same point from the other direction: multishot examples work with thinking, and placing <thinking> tags inside a few-shot example shows Claude the reasoning pattern, which it then generalizes to its own extended thinking blocks. In other words, the reasoning content inside an example is not discarded as decoration; it is absorbed as the decision logic and reused on new inputs.

The boundary matters. For purely mechanical format demonstration, such as "always emit three bullets," bare pairs suffice because there is no judgement to teach. For ambiguous or edge-case judgement, such as a borderline severity rating or a near-miss tool selection, the reasoning is what lets the model generalize instead of overfitting to the literal case shown. The common false claim that reasoning in examples forces the model to emit a chain-of-thought in its output is wrong: the reasoning teaches decision logic, it does not mandate that the output reproduce the trace.

The documented note also clarifies placement. When thinking is enabled, the model generalizes the <thinking> style from the example to its own internal reasoning. When thinking is off, you can still encourage step-by-step reasoning with structured tags such as <thinking> and <answer> to separate reasoning from the final output.

Mechanism reference: Ordering, label balance, and the nearest-example principle

Two secondary mechanisms are documented or lesson-supported. First, ordering: the model exhibits recency bias, so placing the example most similar to the target input last leverages that bias productively, provided every example uses consistent formatting. Our lesson states that later examples have more influence because they sit closer to the query in the context window. Second, label balance: the distribution of labels in the examples creates a prior that biases predictions, so unbalanced sets skew the model toward the over-represented class. The documentation does not quantify this prior, so any specific percentage about its strength should be treated as lesson-supported rather than documented.

Mechanism reference: How examples interact with structured output

The technique-selection boundary is where candidates most often lose marks, so it deserves full treatment. The live structured-outputs documentation defines two complementary, composable features: JSON outputs via output_config.format with {"type": "json_schema", "schema": ...}, and strict tool use via strict: true on a tool, which enforces schema compliance on tool names and inputs by grammar-constrained sampling. The older output_format top-level field still works for a transition period but is legacy; the current shape is output_config.format.

These features are the structural-enforcement layer. Few-shot examples are a different layer: they teach field semantics and cross-structure extraction. The two are not in competition; they are stacked. A schema guarantees that the response is valid JSON matching the shape, but it does not, by itself, teach the model where in a messy document a field lives, or how to classify a borderline value. That semantic teaching is what examples provide. The documentation states that structured outputs guarantee schema-compliant responses through constrained decoding, eliminating JSON.parse() errors, missing required fields, and inconsistent types. It also states, critically, that schema compliance is not absolute: a refusal returns stop_reason: "refusal" with a 200 status and billed tokens, and the refusal text takes precedence over the schema; hitting the token limit also truncates before the schema can be satisfied. This is why a validation layer still belongs in the pipeline even when a schema is present.

The exam's right answer for "malformed JSON output" is therefore tool use with JSON schemas, not few-shot, because the schema enforces structure on every call while examples only demonstrate it. The exam's right answer for "empty fields for data that exists in narrative text" is few-shot showing cross-structure extraction, because the schema is already satisfied (valid JSON) but the population is wrong. The two failures live in different layers, and the fix lives in the matching layer.

Mechanism reference: How examples interact with a validation layer

A validation layer checks the output against computable invariants that neither a schema nor a set of examples can guarantee. The clearest case is reconciliation: when extracted line items must sum to a stated total, a schema can require that both a line_items array and a total number exist, but it cannot require that they agree. That arithmetic closure is a validation concern, not a demonstration concern. The forensic analysis frames this as the distinction between a demonstration problem and a verification problem: few-shot examples show input-output pairs, not arithmetic closure, so a validation step that recomputes the discrepancy and feeds it back as specific error feedback is the correct intervention.

The documentation supports the need for a separate verification step through its guidance on self-correction and self-checking. The prompting best practices page recommends appending an instruction such as "before you finish, verify your answer against the test criteria," and it describes the common chaining pattern of generating a draft, reviewing it against criteria, and refining. That is a validation-retry loop at the application layer. A blind retry without specific feedback reproduces the same error; the loop must carry the specific discrepancy.

The interaction is therefore: schema enforces shape, examples teach semantics and cross-structure location, and the validation layer enforces computable invariants the other two cannot. Examples do not remove the need for validation, and validation does not remove the need for examples. They are stacked, and the technique-selection matrix tells you which to reach for first when a specific symptom appears.

The validation layer has its own boundaries. It can catch computable mismatches, such as a sum that fails to reconcile or a cross-field type conflict, because those are deterministic checks the application code performs after the response is parsed. It cannot catch semantic hallucinations that are internally consistent, such as a plausible but wrong funder name, because no invariant distinguishes a wrong name from a right one without an external source of truth. That is precisely where few-shot examples earn their keep: they teach the model to locate the correct funder in varied structures, reducing the hallucination at the source rather than detecting it after the fact. A blind retry ("regenerate valid output") is explicitly the wrong move for a reconciliation mismatch, because the same prompt reproduces the same error; the retry must carry the specific discrepancy, such as "sum is 520 but line items total 500," so the model has new information to act on. The documentation's self-check guidance supports this by recommending verification against explicit test criteria before finishing.

A subtle point: the validation layer and the schema are both "enforcement," but at different stages. The schema acts during generation via constrained decoding, before the response is accepted. The validation layer acts after generation, on the accepted response, and may trigger a chained retry. Few-shot sits outside both, acting during generation as demonstration. Recognizing which stage owns which guarantee is the core of the technique-selection boundary and the most reliable way to defeat the exam's distractors.

Mechanism reference: The technique-selection boundary, stated precisely

A demonstration problem is one where the model already understands the goal but applies it inconsistently across invocations: drifting format, inconsistent judgement on ambiguous cases, or empty fields for present data in an unexpected container. The fix is few-shot demonstration. A schema problem is one where the output must satisfy a hard structural contract, such as valid JSON with a fixed schema or a guaranteed tool-input shape; the fix is output_config.format or strict: true. A verification problem is one where the output must satisfy a computable invariant the model cannot enforce by demonstration, such as a sum matching a total or a cross-field consistency check; the fix is a validation-retry loop. A description problem (for tool selection) precedes demonstration: sharpen the tool descriptions first, then add few-shot only for the residual ambiguous requests.

The live documentation adds a composability point the older reference material omits: JSON outputs and strict tool use are independent and can be used together in one request, and few-shot examples sit on top of both to teach semantics. The hierarchy is not "prompt JSON versus tool use" as the older material implied; it is "enforce shape with a schema, teach meaning with examples, verify invariants with a loop."

Mechanism reference: Negative examples and contrastive pairs

Two specialized example shapes address false positives and near-neighbor confusion. A negative example shows a rejected or benign case and explains why it is out, paired with positive examples so the model sees both sides of the boundary. This is the "show what to ignore as well as what to flag" principle. A contrastive pair places an acceptable borderline next to a genuine violation that looks almost the same, with reasoning naming the separating feature, so the model generalizes the boundary to novel phrasings it never saw. The documentation's diversity requirement supports both: varying enough that the model does not pick up unintended patterns includes varying the sign of the example (accept versus reject). Negative examples must be clearly marked (for example Correct:/Incorrect:) to avoid confusing the model; a poorly labelled negative is worse than none.

Mechanism reference: Representative length and the out-of-domain trap

Examples should mirror production length and content. An example far longer than production inputs introduces irrelevant structure the model may imitate and wastes tokens; an example from a different domain teaches a pattern that does not transfer. The goal is a representative input with the correct output that clearly demonstrates the pattern, not the longest or most exotic pair available. Edge cases matter, but the set should still cover typical cases, not only the hardest, because the model sees typical inputs most often.

Mechanism reference: Dynamic and per-type example selection

When one prompt must serve multiple document types, a single static set mixing all types forces the model to reconcile conflicting extraction patterns and raises token cost on every call. The lesson and forensic analysis recommend classifying the document type first, then selecting the relevant examples for that type, keeping the count low and the patterns coherent. The loop engineering lesson describes dynamic few-shot selection that retrieves the most relevant examples per input using embedding similarity, keyword overlap, or hybrid retrieval, and notes that a diverse static set is the cache-friendly baseline. A within-type axis also matters: the same field can appear as a table, a bulleted list, or embedded prose, and few-shot coverage must address that presentation-format variation explicitly, not just document-type variation.

Mechanism reference: Caching and the static-prefix discipline

When the few-shot examples are fixed and sit in the static prefix of the prompt, the prefix stays byte-identical across calls, which enables prompt caching and a real cost saving. Rotating fresh random examples into the prefix on every call mutates it and defeats caching, trading a certain saving for an uncertain coverage gain. The disciplined design keeps curated examples static and appends only the item to classify. Dynamic per-type selection intentionally varies the prefix per call, which is a deliberate trade for relevance; there, caching is per-type rather than global. Examples in the system prompt cost the same tokens as examples in messages, so the saving comes from prefix stability, not from location.

Mechanism reference: When few-shot is the wrong tool

Our lesson enumerates situations where few-shot backfires: tasks requiring precise numerical output (use tool use with function calling), outputs that must follow a very long specification (break into calls or use structured outputs), examples that would leak private data (use synthetic examples or strict format), tasks already handled zero-shot (test zero-shot first), examples that require frequent updating (use dynamic retrieval), and tasks needing creativity or diversity (zero-shot or a single broad example). These are the inverse of the three triggers and they reinforce the technique-selection boundary: demonstration is for consistency, judgement, and present-but-misplaced data, not for arithmetic, structural contracts, or genuine absence.

Mechanism reference: Ordering details and the nearest-example rule restated

The recency bias means the example placed last has the strongest influence on the next output. The productive use is to place the example most similar to the target input last, while keeping every example in consistent format. A malformed example given primacy (placed first) worsens drift rather than helping, so the twin rule is never to give a bad example primacy. If examples are diverse with no single nearest match, consistent formatting matters more than ordering. Random ordering is suboptimal because it wastes the exploitable recency bias.

Mechanism reference: Schema enforcement limits and how they interact with examples

The structured-outputs documentation specifies concrete limits that matter when you stack a schema under few-shot examples. Strict tools are capped at 20 per request; optional parameters across all strict schemas and JSON output schemas are capped at 24; parameters using union types (such as {"type": ["string", "null"]}) are capped at 16 because they create exponential compilation cost. These limits are the reason the nullable-schema technique in Example 5 must be applied judiciously: making many fields nullable pushes toward the optional-parameter and union-type caps, and deeply nested objects with optional fields compound grammar complexity. The documented mitigation order is to mark only critical tools as strict, reduce optional parameters by making them required where a default exists, flatten nested structures, and split complex schemas across separate requests.

The interaction point for few-shot is that examples teach the model which fields to populate and how to locate them, while the schema enforces the shape those populated values must take. When the schema is large, examples become more valuable, not less, because they guide the model through the field space the grammar alone does not explain. But examples cannot rescue a schema that violates the documented limits; a 400 error for "schema too complex for compilation" is an API ownership issue, not a demonstration issue.

A further documented caveat: property ordering in structured output places required properties first, then optional ones, so the output order may differ from the schema order unless all properties are required. Few-shot examples that show a specific field order do not override this API behavior; the application must account for reordering in its parsing logic. This is a concrete place where the demonstration layer and the enforcement layer have independent rules, and the candidate must know which owns the behavior.

Mechanism reference: Label balance and recency bias in depth

The label-balance prior deserves more than a one-line mention because it is a frequent distractor. If all examples belong to one class, the model develops a prior that over-predicts that class, including on ambiguous inputs that should land elsewhere. The documentation frames this through the diversity requirement (vary enough that the model does not pick up unintended patterns), and our lesson quantifies the effect as a strong prior comparable to many training examples of the majority class, though that specific strength is lesson phrasing rather than a documented measurement. The practical rule: for binary tasks use one example of each class; for multi-class use one to two per class; keep the distribution roughly proportional to the real task distribution, or equal representation when the real distribution is unknown. A set of four positive and zero negative examples biases the model toward positive, which is the exact failure the exam tests.

Recency bias is the mirror image at the sequence level. The last examples weigh most, so the nearest match to the target should sit last, and a malformed example must never sit first (primacy makes drift worse). Consistency of format across all examples is the precondition: if one example uses a different delimiter or casing, the model may latch onto that variant regardless of position. The documentation's structured-example guidance (wrap in <example> tags) is what makes consistent demarcation achievable.

Ownership map

Which layer owns which guarantee matters for answering "who is responsible" questions on the exam.

The model owns pattern matching and generalization from the examples it is shown. It decides, from the demonstrated input-output mappings, how to transform the real query. It does not own structural enforcement; it can deviate from a demonstrated shape.

The application code (your prompt assembly) owns example selection, demarcation, ordering, and the static prefix that enables caching. In a loop harness, the harness owns which examples are retrieved per call and whether the set is static or dynamic.

The SDK and API own schema enforcement when you use output_config.format or strict: true. Constrained decoding is performed by the runtime, so a valid schema means the response matches the shape regardless of what the model "intended".

The validation layer (your code, or a chained follow-up call) owns computable invariants. Neither the model nor the schema enforces reconciliation; your code does, by recomputing and feeding back specific errors.

The configuration layer (prompt caching) owns the cost saving from a stable example prefix. A fixed curated set in the static prefix is cached and reused; rotating fresh examples each call defeats that cache.

The boundary between model-owned and API-owned is the single most important ownership distinction for this task: format consistency is model-owned and fixed by demonstration, while JSON structural validity is API-owned and fixed by schema.

Version and terminology currency

The documented API surface has shifted. The current structured-outputs entry point is output_config.format carrying {"type": "json_schema", "schema": ...}. The older output_format top-level field and the beta header structured-outputs-2025-11-13 are accepted only for a transition period; the Python SDK (v1.0 and later) no longer accepts output_format on client.beta.messages.create() or count_tokens() and raises a TypeError, so output_config must be used.

Strict tool use is a separate, independent feature set by strict: true on an individual tool. It enforces schema compliance on tool names and inputs through grammar-constrained sampling. It is not the same as JSON outputs, and the two compose in one request.

The historical name "multishot prompting" now redirects into the prompting best practices page under the "use examples effectively" section. The terms few-shot and multishot are used interchangeably in the live documentation. Candidates should answer with either term; the exam material uses "few-shot."

Thinking configuration has also moved. On Claude 4.6 and later models, extended thinking uses adaptive thinking (thinking: {type: "adaptive"}) and the effort parameter, rather than a manually set budget_tokens. The prompting best practices page notes that manual chain-of-thought prompting is a fallback when thinking is off. This matters because the reasoning-in-examples mechanism described earlier interacts with thinking: <thinking> tags inside few-shot examples generalize to the model's own extended-thinking blocks.

Prompt prefilling for format control is deprecated on Claude 4.6 and later; the documentation's migration path is structured outputs, tool calling, or direct instructions instead of a prefilled assistant message.

Official versus community divergence

The clearest divergence is the example count. The exam material and our lesson exam tip say 2 to 4; the live documentation says 3 to 5. A candidate should answer with the exam's 2 to 4 in an exam scenario (because the scenario is phrased around the smallest fix), while understanding that the documentation's 3 to 5 is the general recommendation and that both converge on "diminishing returns past four.".

A second divergence is hierarchy framing. Older reference material presents "prompt-requested JSON versus tool use" as the whole structure. The live documentation adds JSON outputs through output_config.format and strict tool use as separate, composable guarantees, with few-shot examples as a third, orthogonal layer for semantics. Documentation wins; the defensible answer is that shape enforcement, meaning demonstration, and invariant verification are three stacked concerns.

A third area is confidence thresholds. Community and some training material imply a confidence gate can fix inconsistent judgement. The lesson and forensic analysis state that self-reported confidence is poorly calibrated and cannot create the missing standard; the fix is explicit criteria plus examples. The live documentation does not address confidence thresholds in this context, so this position is lesson-supported and should be cited as such rather than as documented fact.

A fourth item the forensic analysis flags is the reasoning requirement. The exam insists every example carry reasoning. The documentation does not mandate it, but it does confirm that reasoning patterns inside examples generalize (via the thinking note). So the exam rule is REFINED against documentation: reasoning is required for ambiguous judgement examples and is documented as beneficial through thinking generalization, but it is not mandated for pure format pairs.

Beyond the task statement

The task statement covers the core demonstration technique, but our lesson set extends well past it. These adjacent topics are part of the same domain and a candidate should recognize how each connects to few-shot prompting.

chain-of-thought (slug: chain-of-thought) is the reasoning technique that interacts most directly with few-shot. The lesson distinguishes zero-shot CoT, few-shot CoT (providing two to three worked reasoning examples), and structured CoT (XML-tagged reasoning). The key boundary: few-shot CoT is the right tool when the task is ambiguous or conditional reasoning, while plain few-shot example pairs are right for format and pattern recognition. The forensic analysis frames chain-of-thought as the targeted exception that outranks examples for genuinely conditional logic and for genuinely ambiguous source text. The lesson also warns that CoT cannot fix a bad base prompt and that explicit CoT beats implicit "think step by step" for accuracy.

xml-structured-prompting (slug: xml-structured-prompting) supplies the delimiter pattern few-shot depends on. The lesson shows that <examples> containing <example> blocks make it unambiguous which content is a demonstration and which is the real task, and that <thinking>/<answer> tags separate reasoning from the final output. It also warns about empty optional tags: an empty <reference_documents></reference_documents> is semantically ambiguous and can cause the model to invent content, so either omit the tag or state "None provided". This is the structural companion to clean example demarcation.

prompt-anti-patterns (slug: prompt-anti-patterns) explains why more prose fails. Its "vague instructions" and "conflicting instructions" sections are the root cause of the instruction-plateau trigger: when instructions are already detailed and output is inconsistent, the problem is not missing prose but missing demonstration. The lesson's over-flagging and aggressive-safety sections also connect to the false-positive half of few-shot, where negative and contrastive examples teach the model what to ignore.

loop-eng-prompt-engineering (slug: loop-eng-prompt-engineering) reframes few-shot inside autonomous loops. In a loop, examples become training data the agent sees on every run, and dynamic few-shot selection retrieves the most relevant examples per input using similarity search. The lesson warns about example bias amplification: a biased static set can be amplified across iterations, so diversify and include counter-examples. It also describes example accumulation, where a successful output is promoted to a few-shot example for later iterations, and meta-prompting, where the loop rewrites its own example library.

extended-thinking (slug: extended-thinking) is the API capability that the thinking note in the best-practices page references. When thinking is enabled, the model reasons in a separate thinking content block, and few-shot <thinking> tags generalize to that block.

system-prompt-design (slug: system-prompt-design) is where the example prefix typically lives for caching, and role-prompting (slug: role-prompting) focuses tone before examples refine format. meta-prompting (slug: meta-prompting) can generate or critique example sets, which parallels the documentation's suggestion to ask Claude to evaluate examples for relevance and diversity.

The adjacent reliability lessons matter for the validation layer. validation-strategies (slug: validation-strategies) covers the retry-with-feedback loop that handles reconciliation mismatches, and guardrails (slug: guardrails) covers the trust-erosion problem where high false-positive categories poison confidence in accurate ones, which few-shot alone cannot repair.

system-prompt-design (slug: system-prompt-design) is where the example prefix typically lives, and the RACCE framing (Role, Audience, Criteria, Constraints, Examples) places examples as one of five components rather than the whole prompt. A common mistake is to pour all effort into examples while leaving criteria vague; the anti-patterns lesson shows that vague criteria are the root cause of inconsistency that examples then have to rescue.

role-prompting (slug: role-prompting) sets tone and scope before examples refine format, and it pairs naturally with few-shot when the demonstration must match a persona's voice. meta-prompting (slug: meta-prompting) can generate or critique example sets, which directly parallels the documentation's suggestion to ask Claude to evaluate examples for relevance and diversity.

sampling-parameters (slug: sampling-parameters) is where the temperature myth is settled. Lowering temperature reduces sampling randomness but does not define a missing standard, so it cannot fix criteria ambiguity; the forensic analysis is explicit that temperature zero makes the model deterministically wrong in the same way. This is why the exam treats "set temperature to zero" as a distractor for inconsistent judgement calls.

constrained-decoding (slug: constrained-decoding) and schema-definition (slug: schema-definition) are the lessons that explain how output_config.format is enforced at the token level, complementing the structured-outputs documentation. json-mode (slug: json-mode) covers the legacy approach that the current output_config.format replaces, useful for understanding version currency.

tool-definition-schemas (slug: tool-definition-schemas) and tool-use-blocks (slug: tool-use-blocks) are the lessons behind the "wrong tool selection" branch of the technique-selection matrix: sharpen tool descriptions first, then add few-shot only for residual ambiguity.

Worked production examples

The following examples are realistic end-to-end demonstrations. Each shows the reasoning chain and the failure mode it avoids. Every example block uses a language tag and inline backticks for identifiers, per the citation discipline.

Worked production examples: Example 1: a wrapped example set with clear delimiters

The first requirement is a wrapped set where the model can tell examples from instructions and from the real query. The documented pattern is <example> tags inside <examples>, and each example uses consistent Input:/Output: markers. This resolves the format-drift trigger: the demonstration removes interpretation.

<task>
Classify each support ticket into exactly one category: BILLING, TECHNICAL, ACCOUNT.
Return only the uppercase category label and nothing else.
</task>

<examples>
  <example>
    <input>My card was charged twice for the March subscription and I want a refund.</input>
    <output>BILLING</output>
  </example>
  <example>
    <input>The desktop app crashes every time I export a PDF larger than 10 MB.</input>
    <output>TECHNICAL</output>
  </example>
  <example>
    <input>I cannot log in after the password reset email, it says the link expired.</input>
    <output>ACCOUNT</output>
  </example>
</examples>

<input>I was billed for a plan I cancelled last month, please reverse the charge.</input>
<output>

What this proves: the <examples> wrapper separates demonstrations from the live <input>, and the uniform Input:/Output: shape teaches the model to emit a single uppercase label with no surrounding text. The failure boundary is format drift: if one example said sentiment: negative while others said NEUTRAL, the model would imitate the inconsistency. Observable output is a bare BILLING token for the final input, with no preamble.

Worked production examples: Example 2: an example that carries its own reasoning for a borderline judgement

The second requirement is a borderline example whose reasoning shows why one action was chosen over a plausible alternative. This is the mechanism that lets the model generalize instead of matching surface form.

Select the correct tool for each request. Show the reasoning, then the tool.

Input: "check my order #12345"
Selected tool: lookup_order
Reasoning: The user provides a specific order number (#12345), which indicates they
want order-specific information. Even though this could be read as a general customer
question, the concrete order identifier routes it to lookup_order rather than the
broader get_customer tool.

Input: "I think my account details are wrong after the merger"
Selected tool: get_customer
Reasoning: There is no order identifier and the concern is about the customer profile
itself, not a specific purchase, so get_customer is the right scope.

Input: "help with my recent purchase, not sure which one"
Selected tool: lookup_order
Reasoning: "recent purchase" implies a specific transaction even without a number. The
phrase "not sure which one" signals the user expects us to find it, which is exactly
what lookup_order does when given the customer context. This is the borderline case:
absent an explicit number, a bare get_customer would lose the purchase intent.

What this proves: the Reasoning: line names the decisive feature (a specific identifier, or the presence of purchase intent), so the model learns the principle that specific identifiers route to specific lookup tools. Without the reasoning, the model would learn only "mentions order numbers go to lookup_order" and would misroute the borderline "recent purchase" case. The failure boundary is overfitting to a literal cue; the reasoning protects against it. Observable output for a new borderline request is a tool name chosen by the same principle, not by keyword matching.

Worked production examples: Example 3: examples covering the failing structure, not the working one

The third requirement is to demonstrate the structure that currently fails. Here the model extracts correctly from tables but returns empty fields on narrative text, so the example set leads with the narrative (failing) structure.

Extract the funder name and funding amount from each grant paragraph.

Example (narrative, the failing structure):
Text: "The city council awarded Riverside School a grant, and the total committed
comes to four hundred thousand dollars spread across three years."
funder: city council
recipient: Riverside School
amount: 400000
currency: USD
note: amount written as words, normalized to digits

Example (narrative, a second variant):
Text: "According to the memo, the Brown Foundation will support the clinic with
roughly half a million dollars."
funder: Brown Foundation
recipient: the clinic
amount: 500000
currency: USD
note: "roughly" preserved, not converted to an exact figure

Now extract:
Text: "The county board approved aid for the harbor project; the contribution is
around two hundred and fifty thousand dollars."

What this proves: both examples show extraction from prose, the structure that was failing, so the model learns to locate the funder and amount regardless of presentation. The table structure is deliberately not demonstrated because it already works. The failure boundary is structural recognition: if the example set had shown only tables, the narrative empty-field rate would not improve. Observable output is a populated amount of 250000 for the final narrative input rather than null.

Worked production examples: Example 4: contrastive pairs showing what to flag and what to ignore

The fourth requirement is a contrastive pair that teaches the boundary between a genuine issue and a benign look-alike, reducing false positives while preserving real detection.

Review the following code for variable shadowing. Flag severity only if the shadow
causes a real risk.

Example A (flag as minor):
Code:
  function process(items) {
    const result = items.map(item => {
      const result = transform(item); // shadows outer 'result'
      return result;
    });
    return result;
  }
Severity: minor
Reasoning: The inner 'result' shadows the outer variable but only within the arrow
function scope. The code stays readable and the shadow does not cause a bug. This is a
style preference, not a defect. Flag as minor only if style consistency is in scope.

Example B (do NOT flag):
Code:
  function render(user) {
    const config = loadConfig();
    const config = sanitize(config); // assignment to a new const in same scope
    return paint(user, config);
  }
Severity: none
Reasoning: This is the benign look-alike. The second 'config' is a re-declaration that
the JavaScript engine rejects at parse time, so it is caught by the build, not a subtle
runtime shadow. Treating it as a shadowing defect would be a false positive. Ignore it.

What this proves: the pair sits the acceptable pattern (Example B) next to the genuine but minor issue (Example A), with the separating feature named in each Reasoning: line. The model learns to distinguish a real shadow from a parse-time re-declaration, cutting false positives without losing the ability to flag true shadows. Observable output is none for a new benign re-declaration and minor for a true scope-limited shadow.

Worked production examples: Example 5: an example set paired with a nullable schema so absence is demonstrated

The fifth requirement is a nullable schema with few-shot examples that return null for absent fields, so the model demonstrates absence rather than inventing a value.

{
  "type": "object",
  "properties": {
    "company_name": {"type": "string"},
    "founded_year": {"type": ["integer", "null"]},
    "headquarters": {"type": ["string", "null"]},
    "source_excerpt": {"type": "string"}
  },
  "required": ["company_name", "founded_year", "headquarters", "source_excerpt"],
  "additionalProperties": false
}

Extract company facts. Return null for a field when the source is silent about it.
Always populate source_excerpt with the supporting text or "not stated".

Example (field present):
Text: "Acme Logistics was founded in 2009 and is based in Rotterdam."
company_name: Acme Logistics
founded_year: 2009
headquarters: Rotterdam
source_excerpt: "founded in 2009 and is based in Rotterdam"

Example (field absent, demonstrate null):
Text: "Globex Health provides telemedicine services across the region."
company_name: Globex Health
founded_year: null
headquarters: null
source_excerpt: "not stated"

What this proves: making founded_year and headquarters nullable removes the schema pressure to populate them, and the second example demonstrates returning null plus a source_excerpt of "not stated" when the source is silent. Together they eliminate fabrication at the source rather than detecting it after the fact. The failure boundary is a required field with genuinely absent data: a required field alone still pressures fabrication, and examples alone cannot make null legal, so both halves are required. Observable output is null for absent fields instead of an invented year.

Worked production examples: Example 6: reasoning tags that generalize to extended thinking

The documented note that <thinking> tags inside few-shot examples generalize to the model's own extended-thinking blocks is worth a concrete example.

Decide whether a request is COMPLIANT or REQUIRES_REVIEW.

Example:
Request: "Our vendor agreement lets us store EU customer data in a US region."
<thinking>
The request mentions EU customer data, which triggers a geographic constraint. The
agreement explicitly permits US-region storage, so the stated practice matches the
contract. There is no obvious conflict with the documented policy.
</thinking>
Decision: COMPLIANT

Now decide:
Request: "We will keep audit logs for 30 days then delete them."

What this proves: the <thinking> block inside the example teaches the model the reasoning shape, and when thinking is enabled the model reuses that shape in its own thinking content block. This grounds the claim that reasoning inside an example generalizes, not merely as output formatting but as internal reasoning structure. Observable output is a COMPLIANT or REQUIRES_REVIEW decision preceded by a private reasoning trace.

Worked production examples: End-to-end scenario: the technique-selection decision in practice

A realistic pipeline makes the boundary concrete. Suppose an invoice-extraction system receives mixed documents: some have line items in a table, some bury totals in narrative prose, and the stated total must equal the sum of line items. The symptoms arrive separately.

Symptom A: the response is not valid JSON, or required keys are missing. This is a schema problem. The fix is output_config.format with a json_schema defining the invoice object, or strict: true on an extraction tool. Few-shot alone cannot enforce this; examples only demonstrate.

Symptom B: the JSON is valid, but total is null for invoices where the total sits in narrative prose while tables extract fine. This is the cross-structure empty-field problem. The fix is few-shot examples showing extraction from narrative prose, leading with the failing structure as in Example 3.

Symptom C: the valid JSON populates every field, but total is 520 while the line items sum to 500. This is a verification problem. Neither the schema nor the examples can guarantee arithmetic closure. The fix is a validation step that recomputes the sum, detects the 20 discrepancy, and feeds it back as specific error text before a retry.

Symptom D: a required field such as tax_id is absent from the source, and the model invents a plausible value. This is a fabrication problem. The fix is a nullable schema plus few-shot examples demonstrating null, exactly as in Example 5.

The decision matrix a candidate should internalize is: malformed or missing structure goes to schema; empty fields for present data go to few-shot; sum or reconciliation mismatches go to validation; fabricated absent values go to nullable schema plus demonstration; ambiguous tool routing goes to sharper descriptions then few-shot. Few-shot is the right first move only when the failure is pattern or judgement inconsistency, which is exactly the scenario the exam presents with "detailed instructions already exist but output is still inconsistent."

Observable outcome of applying this matrix: after adding narrative-structure few-shot examples, the empty-field rate on prose invoices drops while table invoices stay correct; after adding a nullable tax_id with null demonstrations, fabrication stops; after adding a validation-retry loop, sum mismatches are caught and corrected rather than passed downstream. Each intervention is measurable against the baseline from Step 1 of the build exercise.

Worked production examples: How the reasoning-in-example mechanism connects to extended thinking

The documentation note that <thinking> tags inside few-shot examples generalize to the model's own extended-thinking blocks is the formal basis for why reasoning inside an example outperforms a bare pair on novel inputs. The example's reasoning is not surface text the model copies; it is absorbed as the decision logic and replayed in the private thinking block on new inputs. This is distinct from forcing the model to emit a chain-of-thought in its visible output. The lesson on chain-of-thought makes the same distinction: structured CoT with <thinking>/<answer> tags separates reasoning from the answer so the application can parse and discard the trace, and the model still gains the accuracy benefit of writing intermediate steps. Few-shot reasoning examples are the demonstration-time version of that separation.

Build exercise material

The reference page's build exercise asks you to construct a few-shot enhanced extraction prompt and measure its impact. The steps below give verifiable outcomes so you can prove each step worked.

Step 1: Build a base extraction prompt with detailed instructions but zero examples. Test it against ten documents with varied structures (tables, narrative paragraphs, mixed). Observable outcome: inconsistent extraction across the ten documents, with fields filled correctly from tables but empty or wrong from narrative paragraphs, and different output shapes across runs. This establishes the baseline that proves the consistency problem.

Step 2: Record which fields are consistently empty or inconsistent per document type. Observable outcome: a table or log naming the failing field-document combinations, such as dates extracted from tables but missed in narrative, or amounts inconsistent when written as words.

Step 3: Create three few-shot examples targeting the failing patterns, each with reasoning explaining the extraction decision. Observable outcome: three examples, each showing a different document structure (table, narrative, mixed) with both the correct extraction and a Reasoning: section.

Step 4: Re-run the same ten documents with the few-shot enhanced prompt and compare empty-field rate, format consistency, and accuracy. Observable outcome: a measurable reduction in empty fields on narrative documents and improved format consistency, with the largest gain on the document types that previously failed.

Step 5: Document which structural patterns benefited from few-shot and which require a different technique. Observable outcome: a decision matrix. Narrative extraction and format consistency improve with few-shot. Fabrication of missing data does not improve and needs a nullable schema instead. Malformed JSON needs a schema. Sum discrepancies need a validation loop.

Build exercise material: Worked trace of the build exercise

A concrete trace makes the measurable improvement explicit. Baseline run on ten documents: table invoices extract with total populated in 10 of 10 cases, but narrative invoices populate total in only 2 of 4 cases and funder in 1 of 4, with output shape drifting between a JSON object and a prose sentence. After adding the three narrative-structure few-shot examples from Example 3 and the nullable tax_id demonstrations from Example 5, the second run populates total on all four narrative invoices and returns null for tax_id instead of an invented value, with a stable JSON shape across all ten. The decision matrix then routes any remaining sum mismatch to the validation loop, which catches a 20 discrepancy on one document and returns the corrected total on retry. The observable, quantified delta is the proof the exam expects: few-shot moved the narrative empty-field rate from 50 percent toward zero, and the schema plus nullable demonstration removed fabrication, while the validation loop handled the arithmetic case few-shot cannot.

Resolved flagged uncertainties

The forensic analysis flagged several open questions and tensions. Each is resolved here against the live documentation.

Uncertainty 1: the example-count range (2 to 4 versus 3 to 6). Resolution: the live documentation recommends 3 to 5, our lesson exam tip says 2 to 4, and both converge on diminishing returns past four. Present 2 to 4 as the recommended start and 3 to 6 as the nuanced ceiling for harder tasks; treat four as the practical bound.

Uncertainty 2: examples versus chain-of-thought for ambiguous source or conditional logic. Resolution: this is a scope boundary, not a contradiction. Few-shot is the default for pattern and judgement inconsistency; chain-of-thought outranks examples for genuinely conditional logic and for genuinely ambiguous source text where the model must locate evidence rather than pattern-match.

Uncertainty 3: examples versus explicit criteria. Resolution: they are complementary, not opposed. Explicit criteria name the boundary; examples generalize it; the strongest fixes combine both, especially for false-positive reduction.

Uncertainty 4: the older "prompt JSON versus tool use" hierarchy. Resolution: the live documentation adds JSON outputs via output_config.format and strict tool use as separate, composable guarantees, with few-shot as a third orthogonal layer for semantics.

Uncertainty 5: whether reasoning is required in every example. Resolution: REFINED. The documentation does not mandate it but confirms reasoning patterns inside examples generalize via the thinking note; reasoning is required for ambiguous judgement examples and optional for pure format pairs.

Documentation gaps and unverified claims

This section records where the documentation is less prescriptive than the exam material, and where a claim cannot be sourced.

The live documentation does not state a hard upper bound of four examples; it says 3 to 5 for best results and invites you to ask the model to critique diversity. The exam's 2 to 4 is therefore more prescriptive than the documentation, and the candidate should treat four as the diminishing-returns point rather than a documented ceiling.

The documentation does not mandate a Reasoning: field inside every example. It supports reasoning generalization through the thinking note but stops short of requiring it. The exam's "each example must show reasoning" is therefore lesson- and forensic-supported, not documentation-mandated.

The documentation does not discuss confidence thresholds or temperature as fixes or non-fixes for inconsistent judgement. The claim that they do not repair ambiguous criteria is sound per our lessons and the forensic analysis but is not stated in the live prompting page, so it is marked lesson-supported.

Specific percentages that appear in the forensic analysis, such as a 16 to 38 percent fabrication rate on required fields when the source omits data, or a 35 to 40 percent edge-case misclassification rate with label-only examples, are derived from the forensic study of assessment material, not from official Anthropic documentation. They are cited as and must not be presented as official measured figures.

The label-balance prior strength ("comparable to adding hundreds of training examples") is phrased in our lesson and is lesson-supported rather than documented with a measured value.

Everything else in this grounding is either directly quoted from or strictly inferred from the live documentation at the URLs cited, or from the named lessons.

A further note on prescriptiveness: the live documentation does not enumerate the three exam triggers (inconsistent formatting, ambiguous judgement, empty fields for present data) as a numbered list. It states the general principle that examples improve accuracy and consistency and should be relevant, diverse, and structured, which subsumes those triggers without naming them. The exam material is more prescriptive about triggers and counts; a candidate should treat the documentation's principle as the foundation and the exam's triggers as the operational checklists built on top of it. Likewise, the documentation does not use the phrase "few-shot beats more instructions"; it expresses the same idea by ranking examples as one of the most reliable steering mechanisms and by the migration guidance that newer models follow complex schemas when told to, reducing the need for format prompting altogether.

The documentation is also silent on the precise token-cost curve of adding examples beyond four; it confirms that examples cost tokens and that a static prefix is cached, but it does not quantify the diminishing-returns point. The "diminishing returns past four" framing is therefore lesson- and forensic-supported rather than documented with a measured threshold. Candidates should answer with the principle (measure, don't assume a fixed number) and avoid presenting any specific count as an official measured figure.

Consolidated checklist for the exam

The following checklist distills the verified material into the decision rules the exam tests. Each rule is grounded in the sections above.

When detailed instructions already exist but output is still inconsistent, add two to four few-shot examples that demonstrate the exact desired format rather than more prose. This is the instruction-plateau trigger and the single most reliable fix.

When the model makes inconsistent judgement calls on ambiguous cases, require reasoning inside the examples so the model learns the principle, not the literal pair. Temperature and confidence thresholds do not repair this because they do not define the missing standard.

When extraction returns empty fields for data that exists in an unexpected container, add cross-structure examples showing extraction from the failing structure, leading with that structure. Increasing the context window or pre-processing into a canonical format are the wrong moves.

When JSON is malformed or a hard structural contract is required, use output_config.format or strict: true rather than few-shot, because the schema enforces shape on every call.

When a field is required but the source omits it, make the field nullable and add few-shot examples demonstrating null; doing only one half leaves fabrication in place.

When a sum or reconciliation fails, add a validation-retry loop that carries the specific discrepancy; a blind retry reproduces the error.

When tool selection is wrong, sharpen tool descriptions first, then add few-shot only for residual ambiguity. For genuinely conditional logic or genuinely ambiguous source text, chain-of-thought outranks examples.

Wrap examples in <example> tags inside <examples>, keep formats consistent, place the nearest match last, balance the label distribution, and keep the set static to enable caching.

Mechanism reference

This section documents every mechanism in Task 4.3 at full depth. The organizing principle is that Task 4.3 now spans two independent structured-output features that share a compilation pipeline, plus a control parameter that governs invocation, plus a schema design layer that governs truthfulness under the shape guarantee.

Mechanism reference: Two independent features that share a pipeline

Structured outputs on the platform consist of two complementary features that can be used independently or together in the same request.

1. JSON outputs (output_config.format). The request carries output_config: {format: {type: "json_schema", schema: {...}}}. The model response is constrained to return valid JSON matching that schema in the text content block. This is the direct successor to prompt-based JSON and replaces the need to parse free text.

1. Strict tool use (strict: true on a tool definition). The request carries tools: [{name, description, strict: true, input_schema: {...}}]. Each tool call input is constrained to match its input_schema and each tool name is constrained to be one of the provided tools. This is the successor to non-strict tool use where shape was advisory.

Both features compile their schemas into grammar artifacts that constrain token sampling. The documentation states that grammar compilation is cached for 24 hours from last use and invalidated when the schema structure or the tool set changes; changing only name or description does not invalidate.

Feature compatibility reported by the structured-outputs page includes batch processing, token counting, and streaming as compatible, and citations plus message prefilling as incompatible with JSON outputs. The page also notes that grammars apply only to direct model output, not to tool_result blocks or to thinking blocks, so the model can think freely while still producing structured final output.

Mechanism reference: Current canonical form

The generally available (GA) form requires no beta header and is enabled per request:

{
  "model": "MODEL_ID",
  "max_tokens": 1024,
  "messages": [{"role": "user", "content": "Extract the key information from this email: ..."}],
  "output_config": {
    "format": {
      "type": "json_schema",
      "schema": {
        "type": "object",
        "properties": {
          "name": {"type": "string", "description": "The person's full name"},
          "age": {"type": "number", "description": "The person's age in years"}
        },
        "required": ["name", "age"],
        "additionalProperties": false
      }
    }
  }
}

Requirements for this form include: type of the format object is json_schema, the schema root is type: "object", and additionalProperties must be false on objects when structured outputs are in use. The page also documents that output_config.format can be set to {"type": "text"} to explicitly request plain text when that is desired for a particular request.

SDK helpers surface the same capability through typed interfaces: Python client.messages.parse() with a Pydantic model, TypeScript zodOutputFormat() or typed JSON Schema literals via jsonSchemaOutputFormat(), Java plain classes through outputConfig(Class), Ruby BaseModel classes, PHP classes implementing the structured-output interface, C# generic Create<T>(), and Go structs reflected into schemas. In the TypeScript and Python SDKs, unsupported schema constraints are automatically folded into field description text and validated client side after the response; hand-constructed requests must manage that fallback manually.

Property ordering for JSON outputs follows a documented caveat: required properties are emitted first in schema order, followed by optional properties in schema order, regardless of the order in which they were declared in properties. Applications that depend on a specific key order should either mark all properties as required or handle reordering after parsing.

Mechanism reference: Legacy field and header accepted only transitionally

Two older surfaces are documented as accepted for a transition period only: the top-level output_format field (the beta-era location for the same schema) and the anthropic-beta: structured-outputs-2025-11-13 header that formerly gated the feature. New integrations must use output_config.format, and the Python SDK v1.0 and later raises TypeError if output_format is passed to client.beta.messages.create() or count_tokens() and requires output_config instead. The legacy surfaces remain backward compatible only to avoid breaking existing callers and are not to be used in new code.

Mechanism reference: What strictness means

Strict mode guarantees that every tool_use block input strictly follows its input_schema and that every tool name is one of the tools provided in the request. The platform page phrases this as guarantees on both input and name. The retrieval for tool-definition-schemas confirms the same guarantee and adds that the toolset entries computer_toolset_20260801 and browser_toolset_20260801 do not accept strict: true and are rejected if set.

Mechanism reference: Exact requirements that make strictness enforceable

A strict tool must satisfy all of the following, each of which is enforced at request validation time:

• strict is set to true as a top-level property on the individual tool definition, alongside name, description, and input_schema. Strictness is per tool, so a single request can mix strict and non-strict tools.
• input_schema is a JSON Schema object whose root is type: "object" with properties, required, and additionalProperties: false. The additionalProperties: false clause is required for strict mode and is also recommended generally.
• All properties that the tool may return, including optional ones, are present in properties. Optional means not listed in required and typed to allow absence, not omitted from properties altogether.
• input_schema uses only the supported JSON Schema subset documented on the structured-outputs page. Supported features include basic types (object, array, string, integer, number, boolean, null), enum over primitives (no complex types), const, anyOf and allOf with limitations, $ref/$defs/definitions (external $ref not supported), default, required, additionalProperties: false, string format values from a fixed set (date-time, time, date, duration, email, hostname, uri, ipv4, ipv6, uuid), and array minItems limited to 0 or 1. Unsupported features produce 400.

The constrained-decoding lesson notes that SDKs strip unsupported numeric and string constraints from the wire schema, fold them into description, and validate client side after the response; hand-built requests must treat those constraints as guidance and validate post-response.

Mechanism reference: Strictness share of the contract

Strict mode is the correct remedy when a pipeline sees type drift such as "2" instead of 2, missing required fields, or extra hallucinated properties. It is not, however, a remedy for values that satisfy the type but are factually wrong. The lesson states this explicitly as a design principle: strict mode guarantees shape, not business validity, and does not replace server-side validation that checks authorization, existence, or cross-field truth.

Mechanism reference: How the two features compose in one request

The structured-outputs page shows that output_config.format and strict: true can appear together. In a composed request the JSON format constrains any final text block and each strict input_schema constrains any tool_use input for that name, giving two disjoint grammar targets in one turn. The page combines a summary/next_steps JSON schema with a strict search_flights tool in one messages.create call to illustrate the branching. Cache identity includes both the JSON schema and the set of strict tools; changing only description text does not invalidate. Batch processing remains compatible with the composed form, with the incompatible batch parameters being stream: true, fast-mode speed, thread fields, cache hints, and max_tokens: 0.

Mechanism reference: Constrained decoding under the hood

Both features share the same implementation mechanism: at each generation step the sampler holds a constraint automaton compiled from the JSON Schema, identifies the valid next tokens given the partial output, sets logits for invalid tokens to negative infinity, and samples from the filtered distribution. The compilation cost is incurred once per distinct schema (or composition) and cached for 24 hours as noted above. When the schema and the model prior disagree about which token is most likely, the schema wins unconditionally by logit masking, which is also why constrained decoding can enforce shape but cannot force the model to pick the correct schema-valid value among the remaining options.

Mechanism reference: Documented cases where output can still fail to match a schema

The guarantee of schema compliance is strong but not absolute. The platform page enumerates specific failure modes with distinct observables.

Refusal (stop_reason: "refusal"). Claude retains safety behavior even under structured outputs. On a refusal the response carries stop_reason: "refusal", is returned with HTTP status 200, is billed for the tokens that were generated, and the refusal text takes precedence over the schema so the output may not match the JSON shape. The response also carries stop_details with a policy category when present.

Token limit truncation (stop_reason: "max_tokens"). When max_tokens is reached before the JSON object can be closed, the output is truncated and therefore incomplete and not schema-valid. The observable is stop_reason: "max_tokens". The fix is to retry with a larger max_tokens value. This is distinct from the cache-warming case where max_tokens: 0 intentionally produces no generation.

Enum capitalization drift. The page documents a narrow caveat that enum and const string values are not guaranteed on capitalization: the model may return a value that differs only in capitalization, typically the first letter after a space, for example Conversation Topic 3 versus Conversation topic 3. The response completes normally with no error and no special stop_reason. Applications must compare such values case-insensitively and avoid enum sets that differ only in casing.

Schema complexity and compilation limits. Strict tool count, optional parameter count, and union-type parameter count each have explicit limits (roughly 20 strict tools per request, 24 optional parameters, 16 union-type parameters), plus internal grammar-size limits and a 180 second compilation timeout. Exceeding them returns 400 with a complexity or schema error rather than a partial enforcement.