Context Management and Reliability

What you need to know

Context window management is the daily practice of deciding what occupies the finite token budget that every Claude request draws from. Within the window, the system prompt, tool definitions, MCP server payloads, conversation history, retrieved documents, and the response itself including extended thinking all compete for the same allocation. Accuracy and recall degrade as the token count grows, an effect documented under the name context rot, so passive filling reliably degrades output quality long before any limit is hit.

The single most important pattern is the persistent case facts block, sometimes called the immutable facts block. Its purpose is to defeat the progressive summarisation trap. When a long-running conversation is compressed between turns, abstractive summarisation systematically destroys the most critical information in transactional systems. A refund of 247.83 for order 8891 placed on March 3rd, after two summarisation passes, becomes the customer wants a refund for a recent order. The three facts the agent needs to process the refund are gone, and the model proceeds confidently with the wrong data. The fix is structural rather than prompt-based. Extract transactional facts into a structured block that is included in every prompt and is never summarised, never FIFO truncated, and never compressed. This block sits immediately after the system prompt, before any conversation history, and is updated in place as new critical facts emerge. The protected facts survive every compression cycle because the compression pipeline explicitly excludes them.

A related degradation mode is the lost-in-the-middle effect. Findings buried in the middle of a long context receive less attention than findings at the beginning or end, and this is a positional phenomenon rather than an instruction-following problem. Telling the model to pay attention to everything is unreliable because the attention pattern is shaped by position, not by intent. The structural fix is to place key findings summaries at the top of any aggregated input, organise detailed results with explicit section headers throughout, and use recency at the end of the input for the current user message so it gets strong attention. For a multi-source synthesis the format is always key findings summary first, detailed findings under explicit headers, and the synthesised answer last.

The Claude API is stateless. Every request must include the complete conversation history. Omitting earlier messages does not free context on the server, it just causes the model to lose conversational coherence. This creates the central tension of context management: the history must be present for coherence, but it grows with every turn. The persistent case facts block resolves this tension by separating critical facts from summarisable narrative, letting you compress the conversation flow while preserving every transactional detail verbatim.

The persistent case facts block and the summarisation trap

Abstractive summarisation collapses exact values into vague generalities that look reasonable but cannot drive correct downstream action. Numbers, dates, percentages, and explicit expectations are the most vulnerable fields because they are precise by nature, and a summariser that turns 247.83 on March 3rd into recent order has destroyed the transactional payload while preserving narrative gist. The case facts block is the only structural protection because it is excluded from the summarisation pipeline by construction.

The block is a runtime discipline, not an API feature. The application extracts facts from tool results and conversation turns, stores them in a structured object with fields such as customerId, orderId, refundAmount, orderDate, and status, and prepends the serialised object to every prompt construction. The block is updated in place as new facts emerge and for multi-issue sessions each issue gets its own entry with order IDs, amounts, and statuses so compressing one issue narrative does not bleed into another issue facts.

Lost in the middle, context rot, and token budget shape

Quality does not hold flat across the window. Anthropic documents the degradation directly: as token count grows, accuracy and recall fall, an effect named context rot, which is why curating what is in context matters as much as how much room is left. No official threshold is published, and community study material commonly quotes an effective ceiling somewhere in the high 140,000s, so treat any specific number as an estimate and reason about the direction of the effect rather than a cliff edge. A scenario describing a 180K context where the model misses a policy buried in the centre is testing positional degradation compounded with context rot, not a token limit being exceeded. The fix is to restructure so critical content sits at the start or end rather than buried in the middle, with key findings at the top, headers throughout, and the current question at the end for recency.

A practical allocation for a customer support agent with five tools keeps roughly 10 to 15 percent for system prompt, 10 to 15 percent for tool definitions at about 3,500 tokens each so five tools consume roughly 9 percent of a 200K budget before any conversation, 30 to 40 percent for conversation history, 10 to 20 percent for retrieved documents, and a 5 to 10 percent safety buffer for unexpected payloads. Without the buffer a single long user document can push the session deep into the degraded region mid-turn.

Mechanism and API surface

Persistent case facts block after system prompt

Structured JSON with customerId, orderId, refundAmount, orderDate, and status, prepended to every prompt, never summarised or FIFO truncated, updated in place as new facts emerge, with per-issue entries for multi-issue sessions.

Lost-in-the-middle positional fix

Key findings summary at the very top, detailed findings under explicit section headers throughout, and current user message at the end for recency. Positional effect is not fixable by instruction.

Context rot as the documented degradation

Accuracy and recall fall as token count grows, with no published cliff. Community material quotes an effective ceiling in the high 140,000s; treat that as an estimate. The hard window limit is a separate hard stop where the request errors.

Practical token allocation with safety buffer

System prompt 10 to 15 percent, tool definitions 10 to 15 percent at about 3,500 tokens each, conversation history 30 to 40 percent, retrieved documents 10 to 20 percent, plus a 5 to 10 percent safety buffer that is non-negotiable.

Tool result trimming at the boundary

Strip verbose fields before the result enters messages via a hook or wrapper. Once in history verbose data persists every turn, so trimming after the fact does not recover the cost.

Distinctions that decide answers

Traps

What you need to know

Escalation calibration determines whether a customer support agent achieves its target first-contact resolution rate or quietly degrades into a system that punts every difficult case to a human while attempting every easy case autonomously. The exam tests the precise boundaries of when to escalate, when to resolve, and which commonly proposed triggers are unreliable in practice. There are exactly three valid escalation triggers, two unreliable triggers that look rigorous but fail, and one absolute rule about explicit human requests.

The first valid trigger is an explicit human request. The moment a customer says I want to speak to a person or transfer me to a human agent, the escalation happens immediately. No investigation, no let me see if I can help you with that first, no diagnostic questions. The customer has made a clear unambiguous request and the agent must honour it without delay. This is an absolute rule with no exceptions. The temptation to try once first wastes the customer's time, frustrates an already frustrated user, and signals that the system does not respect explicit requests.

The second valid trigger is a policy gap, distinct from a policy violation. A violation is a request that documented policy explicitly addresses and the answer is usually no, such as refund outside the return window or discount beyond the authorised limit. A gap is a situation the documented policy is silent on, such as a customer asking for competitor price matching when the policy only covers own-site price adjustments. The policy does not say no and does not say yes, it is silent. The agent cannot make policy on the fly, so the gap requires human judgement about whether to make an exception.

The third valid trigger is inability to make meaningful progress. The agent has attempted resolution and cannot advance because tools returned errors that local retry logic cannot resolve, the situation requires system access the agent does not have, the issue is a technical bug that needs engineering, or the order genuinely does not exist in any system the agent can query. The catch-all is genuine but only after a real attempt. I might not be able to handle this is not sufficient. The agent has to show it tried and failed.

The two unreliable triggers deserve direct attention because they sound rigorous but consistently fail in production. Sentiment-based escalation uses frustration detection or negative sentiment scores to route to a human. Frustration does not correlate with case complexity. A customer furious about a simple late delivery is easy to resolve with an apology, compensation, and reship. A calm polite customer asking about a competitor price match needs human judgement on a policy gap. Sentiment measures emotional state not case difficulty, so routing by sentiment inverts the correct priority. The same applies to politeness and tone classification. Self-reported confidence scoring is the second unreliable trigger. Having the model output a 1 to 10 score and escalating below a threshold sounds engineering grade but LLM self-reported confidence is poorly calibrated. The model is often incorrectly confident on hard cases and unnecessarily uncertain on straightforward cases. This produces the exact failure the exam tests, an agent that escalates simple cases while attempting complex ones. Confidence earns its place only as a routing signal in a calibrated system with external ground truth, not as a primary trigger. Objective signals like self-consistency checks that run the same prompt three times and escalate on disagreement and hedging detection that flags uncertain language are more reliable than self-reported scores.

The three frustration cases and ambiguous matching

If the issue is straightforward and the customer is frustrated, acknowledge the frustration, offer the resolution, and do not escalate. I understand this is frustrating, I can process your replacement right now is the correct pattern. If the customer reiterates a preference for a human after you have offered help, escalate because they have been given an opportunity to accept agent resolution and declined. If the customer explicitly says I want a human from the start, escalate immediately with no investigation step. The distinction is between a frustrated customer with a resolvable issue where you resolve and a customer who explicitly wants a human where you escalate.

Ambiguous customer matching is its own failure mode. When a tool returns multiple matches for a name search such as three John Smith records, the agent must ask for additional identifiers like email address, phone number, or order number. The agent must not select the most recent, most active, or any heuristic-based record. Selecting the wrong customer risks privacy violations by exposing one customer's data to another and incorrect actions such as processing a refund on the wrong account. The only safe response is to ask for clarification.

Calibration in the system prompt and structured handoff

The most effective first fix is to add explicit escalation criteria with few-shot examples to the system prompt. Examples demonstrate when to escalate for explicit request, policy gap, and inability to progress, when to resolve autonomously for straightforward frustrated cases, and the exact format of the escalation. Prompt optimisation should always precede architectural changes such as classifier models. A classifier is an addition only when prompt engineering has plateaued.

Escalation is implemented at the application layer. The system prompt carries the criteria and few-shot examples, the runtime detects triggers and either invokes an escalation tool or routes to a human queue. A common tool definition is escalate_to_human with required fields for customerId, issueSummary, attemptedActions, recommendedAction, and urgency, which makes the handoff machine readable for downstream routing. When the application detects an explicit request or a flagged policy gap it can force the structured handoff with tool_choice type tool and name escalate_to_human rather than relying on free-form text.

Logs, tiebreakers, and what inability really means

Every escalation event should log the trigger that fired, the customer identifier, the issue summary, the attempted actions and their outcomes, the recommended next action, and a timestamp. This structure is what makes the escalation useful to the human reviewer who can pick up without re-asking the customer for context, which reduces average handle time per case.

When two automated components disagree about whether to escalate, one flags a gap while another classifies the same case as a violation, the escalation policy needs a tiebreaker such as priority rules, timestamp ordering, or default to escalate. Without a tiebreaker the disagreement produces inconsistent handling across similar cases.

Inability to progress means concrete failed attempts, such as tool errors that retry cannot resolve, missing system access, or a confirmed technical bug. Subjective difficulty is the agent guessing without trying. Only the first justifies the inability trigger, which is why the exam tests whether escalation is grounded in failed attempts rather than in a feeling of difficulty.

Mechanism and API surface

Explicit human request as an absolute trigger

Phrases like I want to speak to a person or transfer me to a human require immediate escalation with no investigation step and no let me try first.

Policy gap versus policy violation

A gap is a situation the documented policy is silent on and requires human judgement about whether to make an exception. A violation is explicitly addressed with a documented answer and the agent communicates that answer.

Inability to make meaningful progress

Grounded in concrete failed attempts where tools, retries, and accessible systems did not advance the case. Subjective guessing without trying does not qualify.

Sentiment and confidence as unreliable signals

Frustration and negative sentiment measure emotional state not case complexity. Self-reported confidence is poorly calibrated and often confident on hard cases and hesitant on easy ones. Use calibrated external signals if confidence is used at all.

Ambiguous matching via clarification request

On multiple matches ask for an additional identifier such as email, phone, or order number. Never select by recency, activity, or other heuristic.

Structured handoff and forced escalation tool

Tool escalate_to_human with customerId, issueSummary, attemptedActions, recommendedAction, urgency, and timestamp, invocable with tool_choice type tool and name escalate_to_human for a complete machine readable handoff.

Distinctions that decide answers

Traps

What you need to know

Error propagation in multi-agent systems determines whether a failure in one component brings down the whole workflow or whether the system recovers gracefully with partial results. The exam tests the structural shape of error context, the two catastrophic anti-patterns that sit on opposite ends of handling, and a critical distinction that most developers get wrong on first reading: access failures versus valid empty results. Getting this pattern right means every failure carries enough context for the coordinator to decide intelligently, and getting it wrong means either silent data loss or disproportionate waste of completed work.

The correct pattern is structured error context. When a subagent fails it returns a structured object that enables the coordinator to make intelligent recovery decisions. The context must include four elements. First, failure type as a categorical label that drives recovery logic: transient failures such as timeouts, rate limits, or network blips that may succeed on retry, validation failures that need the query fixed, business failures such as rule violations that need escalation or alternative action, and permission failures that cannot be retried without authorisation changes. Second, what was attempted: the specific query, parameters, target system, and any identifiers, so searched academic database for renewable energy policy with date range 2022 to 2024 is actionable while search failed is not. Third, partial results gathered before failure: if a subagent retrieved three of five sources before timing out, those three are valuable and discarding them because the overall operation failed is wasteful. Fourth, potential alternative approaches: the subagent knows its domain and can suggest a different database, broader search terms, or cached results that may help the coordinator decide on recovery.

The two anti-patterns are catastrophic in different ways and sit on opposite ends of handling. Silent suppression is the worst: a subagent catches a timeout and returns results empty with status success. The coordinator believes the search ran and found nothing, will not retry, will not try alternatives, and produces a synthesis that silently omits an entire research area. The final output looks complete but is missing critical content. Silent suppression is especially dangerous because it is invisible: the output looks correct and the gap cannot be detected from the output alone. In a customer support context this might mean reporting no orders found when the order lookup system was actually down, leading the agent to tell the customer they have no account. Workflow termination is the other anti-pattern: killing the entire pipeline on a single failure. One subagent times out and the entire research pipeline crashes, throwing away results from four subagents that completed successfully. This is a disproportionate response with no recovery path. The correct middle ground is structured error propagation: the failing subagent reports what happened with partial results preserved, the coordinator assesses the damage, and the system continues with targeted recovery.

The access-failure versus valid-empty-result distinction is critical and the exam tests it directly. An access failure is when the tool could not reach the data source, such as a timeout, connection error, or permission denial. The search did not execute, so retry with the same or modified parameters should be considered. A valid empty result is when the tool reached the source and executed the query but found no matches. This is the answer, no retry is needed because the system worked correctly and there simply are no results for this query. Conflating them produces two opposite failure modes: treating access failures as valid empties means never retrying when you should, treating valid empties as access failures means wasting time retrying a query that will always return nothing. The fix is a structured response with explicit status, failureType, and shouldRetry fields so the coordinator can distinguish the two programmatically.

Coverage annotations are the structural mechanism for handling partial failures in synthesis. When a synthesis agent combines findings from multiple subagents the output should note which topic areas are well supported and which have gaps. If one subagent failed to retrieve sources on geothermal energy the synthesis should explicitly state section on geothermal energy is limited due to unavailable journal access during research. This is far better than silently omitting the topic, because a gap otherwise looks like the topic was not relevant rather than the source being unavailable. Local recovery for transient failures happens before propagation. Subagents implement local retry logic with exponential backoff and jitter, fallback sources, and degraded responses before propagating. Only persistent errors that survive local recovery should propagate, always with attempted action and partial results included, which reduces coordinator complexity because the coordinator does not manage retry for every transient failure across every subagent.

Structured error shape and the shouldRetry signal

The minimum fields for a structured error are status with success or error, failureType from transient, validation, business, and permission, attemptedAction as an object with tool, query, and parameters, partialResults as an array of any retrieved data, alternativeApproaches as an array of suggested recovery strategies, and shouldRetry as a boolean. The failureType enum drives coordinator logic: transient triggers retry with same or modified parameters, validation triggers a query fix attempt, business triggers escalation or alternative, and permission triggers immediate escalation without retry.

In the tool layer this surfaces with isError true, errorCategory from transient, permanent, auth, not-found, and validation, and isRetryable as the actionable boolean. A transient category with isRetryable true means the coordinator should retry, an auth category with isRetryable false means retrying will not help. The shouldRetry and isRetryable fields make the retry decision explicit rather than relying on the model to infer it from an error message, and the exam tests whether you route by those fields rather than by prose interpretation.

Local retry before propagation and exponential backoff

Local retry logic uses exponential backoff with jitter. A typical implementation attempts once immediately, then after 1 second plus jitter, then after 2 seconds plus jitter, then after 4 seconds plus jitter, giving up after three retries. The jitter prevents synchronised retry storms where multiple clients retry the same failing service at the same instant. When a Retry-After header is present the retry logic honours it instead of computing its own backoff.

The discipline is to handle transient failures locally and propagate only persistent ones. Most transient failures resolve on the first or second retry, so propagating immediately floods the coordinator with recoverable errors. Each subagent handles its own transient failures, preserves partial results across retries, and escalates only the persistent error with the full structured context.

Coverage annotations that make gaps visible by construction

Coverage annotations are appended to the synthesis output as a structured section listing each topic area with its data quality status: well supported, limited with reason, or unavailable with reason. Failed subagent topics are explicitly noted with the failure type and any alternative approaches that were attempted. The annotation is what makes a gap visible: without it a missing geothermal section looks like the topic was not relevant, with it the same gap reads as source unavailable and the downstream consumer knows not to trust absence as relevance.

External validation pipelines mirror the same pattern for Claude outputs. When a Claude output fails schema or semantic validation the pipeline returns a structured error with the specific validation message, the failed output, and a recommended fix. Model downgrade from Opus to Sonnet to Haiku and circuit breaker patterns provide the fallback and protection layer that complements propagation: the downgrade chain is invoked when a primary model returns repeated transient failures, and the circuit breaker opens after a threshold of failures, allows test requests after a cooldown, and closes on recovery to protect downstream services from cascading load.

Mechanism and API surface

Four-element structured error context

Failure type, attemptedAction with tool and query and parameters, partialResults array, and alternativeApproaches array, all required alongside status and shouldRetry so the coordinator has actionable context.

Access failure versus valid empty result

Access failure means the tool could not reach the source and the query did not execute, valid empty means the tool executed correctly and found no matches. The first has isError true and shouldRetry true, the second has status success with empty results and shouldRetry false.

isError and isRetryable signals

errorCategory from transient, permanent, auth, not-found, and validation plus isRetryable boolean. Transient with isRetryable true routes to retry, auth with isRetryable false routes to escalation without retry.

Exponential backoff with jitter for local retry

Delay computed as base times two to the power of attempt plus random jitter, typically 1 second base with 1 second, 2 seconds, 4 seconds plus jitter, and Retry-After header honoured when present to avoid synchronised storms.

Partial results preservation across failure boundaries

The partialResults field specifically preserves data retrieved before failure so the coordinator can use it. Discarding treats the operation as all or nothing and wastes completed work.

Coverage annotation on synthesis output

Structured section listing each topic area as well supported, limited with reason, or unavailable with reason. Failed topics are explicitly noted with failure type and attempted alternatives, never silently omitted.

Distinctions that decide answers

Traps

What you need to know

Extended codebase exploration is one of the most context-intensive tasks a Claude-based agent performs, and it produces a specific failure mode that has nothing to do with running out of tokens. Context degradation is the model losing its grip on earlier findings as the conversation fills with verbose discovery output. It manifests as a concrete observable behaviour: the model starts referencing typical patterns instead of the specific classes, methods, and dependency chains it discovered earlier. After investigating several modules an agent might say this follows the typical repository pattern instead of the OrderRepository class at src slash repos slash order dot ts implements the Repository interface with custom caching in the findById method.

Context degradation happens because each exploration step generates verbose output such as file contents, search results, and directory listings that accumulates in the conversation context. Earlier precise discoveries are pushed further into the context while more recent verbose output dominates. The model's attention shifts to recent output and it loses specific references to earlier findings. The critical insight is that increasing the context window does not fix degradation. The model is not running out of space, it is losing track of specific details as they get buried under newer more verbose output. The fix is structural, not capacity based.

The primary mitigation is scratchpad files. The agent writes key findings to a file and references it for subsequent questions. This persists knowledge outside the conversation context and makes it immune to degradation. A scratchpad for codebase exploration typically captures three categories: key classes and their locations with class name, file path, and brief description, dependency chains stating which classes depend on which, and critical findings such as bugs, missing tests, and architectural concerns. The scratchpad is a deliberate strategy from the start of any extended exploration session, not a rescue move once things degrade.

Subagent delegation is the second major mitigation. Instead of the main agent doing all exploration directly and filling its context with verbose output from every file read and search, delegate specific investigation tasks to subagents such as find all test files for the order service and report their coverage status, trace the refund flow from API endpoint to database and list all intermediate services, and identify all external API integrations and their error handling patterns. Each subagent operates with its own isolated context, can explore verbosely without polluting the main agent's context, and returns a structured summary to the coordinator. The obvious reading is parallelisation for speed, the real value is context isolation. The main agent's context stays clean for high-level coordination while subagents handle the verbose exploration.

Summary injection between phases prevents the cold start problem when exploration happens in phases. Phase 1 might be understanding the architecture, phase 2 might be investigating specific components. Summarise key findings from Phase 1 before spawning Phase 2 subagents and inject those summaries into the initial context of Phase 2 subagents. Without injection Phase 2 duplicates Phase 1 exploration because it was not given the previous findings. The injection ensures Phase 2 agents have the architectural understanding needed to ask the right questions without rediscovering the system structure. The compact command in Claude Code provides API-level summarisation during extended sessions, use it proactively during exploration not only when hitting limits, because it protects context quality not only quantity. Crash recovery via structured state manifests is the resilience layer: each agent exports its current state to a known file location with what has been explored, key findings, current phase, next steps, and pending questions, so on resume the coordinator loads the manifest and the agent picks up where it left off.

Scratchpad discipline and what counts as a finding

The scratchpad file is a runtime discipline using ordinary file operations such as create, view, str_replace, and delete exposed through the file system API. A common shape is Markdown with explicit sections for key classes, dependency chains, and critical findings. The agent writes after each discovery and reads at the start of each subsequent step. The discipline fails if the scratchpad tries to hold the entire history verbatim, because that re-introduces the verbose output problem in a different file. Capture only specific findings the agent needs to reference across turns, not every intermediate step.

A scratchpad that captures everything is a history dump, a scratchpad that captures specific findings is a reference. The three categories keep the file lean: class name plus file path plus one-line description, dependency edges such as OrderService orchestrates OrderRepository and PaymentGateway, and critical findings such as RefundProcessor has no retry logic for Stripe failures or RefundProcessor test coverage 12 percent. When the agent at turn 16 reads the scratchpad, the specific identifiers are restored regardless of how much verbose output has accumulated since they were discovered.

Subagent delegation for isolation and summary injection between phases

Subagent delegation in Claude Code uses the Task tool with a focused role description and task prompt. The role should name the investigation target and the expected return format as a structured summary with key findings, file paths, and class names. The coordinator receives the summary without the verbose intermediate file contents and search results that would otherwise pollute its context. The primary benefit for codebase exploration is isolation, parallel execution is secondary and only available when investigations are independent.

Summary injection is the application-side mechanism that places Phase 1 findings into Phase 2 prompts. After Phase 1 the coordinator extracts key findings from the Phase 1 scratchpad and prepends them as a Phase 1 Summary section. The Phase 2 subagent receives both the architectural understanding and the specific investigation target so it does not need to rediscover system structure. Without injection the same files are read again, the same searches are run again, and API cost and time are wasted on duplication that also pollutes the new subagent's context.

Proactive compaction and crash recovery manifests

The compact command is invoked via slash command inside Claude Code and triggers conversation-level summarisation that preserves decisions, findings, and tool calls while discarding verbose intermediate output. Proactive use early and mid-session keeps attention on specific findings throughout, while reactive use only when context is nearly full is a last resort that may already have lost findings because verbose output pushed them into the middle where positional degradation applies.

The crash recovery manifest is a JSON file at a known location with sessionId, phase, exploredPaths as an array of file paths, keyFindings with architecture, criticalIssue, and testCoverage, and nextSteps as an array of pending targets. On resume the coordinator reads the manifest and injects the structured state into the agent's initial prompt. Upstream agents that return verbose reasoning chains rather than structured summaries defeat the same principle downstream: downstream agents with limited budgets cannot use verbose prose, so upstream agents should return key facts, file paths, and relevance scores that downstream agents can process without re-parsing.

Mechanism and API surface

Scratchpad file outside conversation context

Markdown file with sections for key classes, dependency chains, and critical findings, written after each discovery and read before each subsequent step, capturing specific class names and file paths not summaries or full history.

Subagent delegation for context isolation

Spawn focused subagents with a specific investigation task and expected structured summary format. Each runs in its own isolated context and returns findings without verbose intermediate output. Primary benefit is isolation, speed is secondary.

Summary injection between phases

Extract Phase 1 findings from the scratchpad and prepend them as a Phase 1 Summary section in Phase 2 subagent prompts so Phase 2 starts with architectural context and does not re-explore.

Proactive compact command

Slash command that summarises the conversation preserving decisions and findings while discarding verbose output. Use proactively during extended sessions to maintain quality, not only reactively at the limit.

State manifest for crash recovery

JSON manifest at a known location with sessionId, phase, exploredPaths, keyFindings with architecture, criticalIssue, and testCoverage, and nextSteps, loaded on resume and injected into the agent's initial prompt.

Upstream structured summary over verbose reasoning

Upstream agents return key facts, file paths, and relevance scores rather than reasoning chains, because downstream agents with limited budgets cannot apply verbose prose and waste tokens re-parsing it.

Distinctions that decide answers

Traps

What you need to know

Human review is the safety net for automated extraction and classification systems. The exam tests the allocation problem, not whether to use human review but how to allocate limited reviewer capacity to maximise accuracy while minimising cost. This requires understanding confidence calibration, the trap of aggregate metrics, and stratified sampling strategies. The most dangerous misconception is the aggregate metrics trap: a system reports 97 percent overall accuracy, the team celebrates, and management approves full automation for all high-confidence extractions. The problem is that the aggregate hides catastrophic failure rates on specific segments where errors have the highest business impact.

A system that extracts dates from standard invoices at 99.5 percent accuracy might extract dates from handwritten receipts at 60 percent, from scanned PDFs with poor OCR at 72 percent, and from international documents with non-standard formatting at 45 percent. The aggregate looks excellent because standard invoices dominate volume, but three document types have unacceptable accuracy hidden by the volume-weighted average. Automating high-confidence extractions on handwritten receipts at 95 percent confidence would let through errors at 40 to 55 percent rates. The rule is to always validate accuracy by document type and field segment before automating and never make automation decisions based on aggregate metrics alone. International invoices from new suppliers, handwritten receipts from field staff, and scanned historical documents for compliance are the canonical segments where aggregate approval fails.

Stratified random sampling is the ongoing verification mechanism. After validating by document type and field you need continuous verification because accuracy can drift as documents age, schemas change, or new edge cases emerge. Stratified random sampling means selecting a representative sample from each stratum defined by document type, confidence band, and field type and having humans verify it. The critical insight is that you must sample high-confidence extractions, not just low-confidence ones. Low-confidence items are already routed to human review while high-confidence items are automated. If the model develops a novel error pattern that affects high-confidence extractions, only stratified sampling that includes the high-confidence band will catch it. Sampling serves two purposes: ongoing accuracy measurement to confirm each segment maintains its validated rate over time, and novel error pattern detection to discover new failure modes that did not exist in the original validation set.

Field-level confidence calibration is the mechanism for converting raw model confidence into actionable routing decisions. The model can output confidence per field, for example vendorName with value Acme Corp at 0.98 and totalAmount at 0.72, but raw scores are not calibrated. A model that reports 0.95 confidence might actually be correct 88 percent of the time on certain field types or 99 percent on others. The score is relative not absolute. Calibration requires labelled validation sets with ground truth. Take documents with known correct extractions, run the model, compare its confidence scores to actual accuracy, and build a calibration curve by binning predictions by reported confidence and computing actual accuracy in each bin. The curve might show when the model reports 0.90 on date fields it is actually correct 94 percent of the time while 0.90 on amount fields means only 82 percent. Calibrated thresholds then drive routing where fields above the calibrated threshold route to automation with stratified sampling, fields below route to human review, and fields in the ambiguous zone route to prioritised review.

Reviewer capacity prioritisation is where system design matters most. Human reviewers are expensive and limited, so route the highest-uncertainty items first: low model confidence fields, extractions from ambiguous or contradictory sources, document types with historically poor accuracy, and fields where the model expresses multiple possible interpretations. Do not spread reviewer capacity evenly across all extractions, because even distribution wastes time re-verifying high-confidence items that the model handles well while leaving insufficient capacity for the uncertain items that actually need judgement. The queue should be a dynamic priority queue ordered by uncertainty where the next item served is always the highest-uncertainty item remaining, not the next in chronological order, and when new extractions arrive the queue reorders dynamically. The validation sequence before automation is fixed in order: measure accuracy by document type and field segment not aggregate, calibrate confidence scores using labelled sets, set calibrated thresholds for automation versus review, implement stratified sampling for ongoing verification of automated extractions, and only then reduce human review on segments that demonstrate consistent validated accuracy. Skipping to the last step based on aggregate metrics is the trap and every step exists to prevent a specific failure mode.

Per-stratum accuracy, calibration curves, and routing thresholds

Aggregate accuracy is a single number across all documents, per-type and per-field accuracy is a matrix showing accuracy for each document type and each field type separately. The matrix exposes segment failures that the aggregate hides, and validation must use the matrix before any automation decision. Standard invoices at 99.5 percent on dates and handwritten receipts at 60.1 percent on dates can coexist inside a 97 percent aggregate, and the decision to automate must be made per cell not per aggregate.

The calibration module takes a labelled validation set and produces calibrated thresholds per field type per document type as a curve mapping confidence ranges to actual accuracy. For example 0.90 to 0.95 on date fields on invoices at actual accuracy 94 percent versus 0.90 to 0.95 on amount fields on invoices at 82 percent. Raw model confidence is not comparable across field types until it has been calibrated, and a single static threshold across all extractions lets through errors on segments where the same score means lower accuracy. Objective signals that complement calibration include self-consistency that runs the same prompt three times and escalates on disagreement, hedging detection that flags uncertain language such as I think or possibly, log-probability analysis that inspects token distributions, and meta-cognitive prompting that asks the model what it would need to verify before answering.

Stratified sampling that must include the high-confidence band

The stratified sampling function selects a representative sample from each stratum proportionally to its volume in the population, not uniformly. A stratum with 1,000 invoices at high confidence and 50 handwritten receipts at high confidence contributes samples at a 20 to 1 ratio, not 1 to 1. High-confidence sampling includes automated extractions in the verification sample, low-confidence-only sampling verifies only items already routed to human review. High-confidence sampling is what catches novel error patterns in automated extractions, low-confidence-only sampling misses them entirely.

Stratified sampling is a permanent feature, not a temporary verification step that ends after validation. As prompts, schemas, or routing logic change the golden dataset pattern provides regression testing for accuracy on canonical examples, while red-teaming deliberately constructs adversarial inputs that target known failure modes. Validation pipelines with schema, semantic, and business-rule stages produce structured errors that feed into the calibration data, and content filter pipelines handle appropriateness while confidence calibration handles accuracy, the two running in parallel with a structured routing decision.

Dynamic priority queue over even distribution

The review router implements a priority queue ordered by uncertainty. Items enter with their confidence scores and the queue orders them so the next item served is always the highest-uncertainty item remaining. When new extractions arrive the queue dynamically reorders and reviewers consume from the top, never in chronological order. Even distribution spreads reviewer time across all extractions, priority queue concentrates effort where judgement matters, and average handle time drops because reviewers spend time on items that need it rather than re-verifying easy high-confidence items.

The calibration loop is continuous, not a one-time setup activity. Accuracy drifts as documents age, schemas change, or new edge cases emerge, and calibration must be maintained via ongoing stratified sampling that captures novel patterns. Production-traffic calibration uses reviewer decisions on real extractions to adjust thresholds initially set from validation-set calibration. The detected_pattern field on each finding turns dismissal data into systematic improvement: patterns with high dismissal rates become candidates for prompt refinement, closing the extract to validate to refine loop.

Mechanism and API surface

Per-stratum accuracy matrix over aggregate

Accuracy measured by document type and field type as a matrix, not a single aggregate number. The matrix exposes segments such as international documents at 45 percent that a 97 percent aggregate hides, and automation decisions are made per cell.

Field-level confidence with calibration curves

Raw confidence per field from 0.0 to 1.0 is mapped to actual accuracy via binning on a labelled validation set, producing a per-stratum curve where 0.95 on a date field can mean 99 percent while the same score on an amount field means 56 percent.

Stratified sampling proportional to volume including high confidence

Sampling selects from each stratum defined by document type and confidence band proportionally to its volume, and must include the high-confidence automated band to catch novel error patterns there.

Dynamic priority queue ordered by uncertainty

Review backlog ordered so the next item served is always the highest-uncertainty item remaining. The queue dynamically reorders as new extractions arrive and reviewers never consume in chronological order.

Three-tier threshold plus validation sequence

Low confidence escalates immediately, medium routes to prioritised review, and high proceeds with stratified sampling. Sequence gates automation: measure by type and field, calibrate, set per-stratum thresholds, implement sampling, then reduce review only on validated segments.

Calibration maintenance and detected_pattern loop

Calibration is continuous via ongoing stratified sampling and production-traffic adjustment, and the detected_pattern field on findings with high dismissal rates identifies which prompts to refine next.

Distinctions that decide answers

Traps

What you need to know

Information provenance, knowing where every claim comes from and how confident you should be in it, distinguishes a research system that produces trustworthy outputs from one that produces plausible-sounding fiction. The exam tests how attribution survives or dies through multi-agent synthesis pipelines, how to handle conflicting sources, and how temporal context prevents false contradictions. Without provenance the final output cannot be verified, cited, or audited, and without deliberate preservation the pipeline destroys it on first paraphrase.

Every finding in a multi-agent research system must carry its provenance as a structural guarantee, not optional metadata. Each finding must include five elements: the claim as the specific assertion, the source URL where the information was found, the document name as the title of the source document, the relevant excerpt as the specific passage that supports the claim, and the publication date when the source was published or the data was collected. Without all five the finding cannot be verified. The application should reject findings with missing fields at the source so discipline holds.

The critical challenge is that attribution dies during summarisation. When a synthesis agent combines findings from multiple subagents it naturally compresses and paraphrases. Without explicit instructions to preserve claim-source mappings the synthesis produces statements such as investment in renewable energy has grown significantly with no amount, no source, and no date. The information was correct but the attribution was destroyed. Downstream agents must explicitly preserve and merge claim-source mappings through synthesis: subagents output findings in the structured claim-source format, the synthesis agent is instructed to maintain those mappings when combining findings, and the final output includes inline citations or a structured reference section that traces each claim to its source.

Conflict handling is the second critical pattern. When two credible sources report different statistics for the same measure the wrong approach is to arbitrarily select one value. Selecting the more recent source, averaging the values, or picking from the more authoritative publisher all destroy information and present false certainty. Source A reporting 12 percent growth and source B reporting 8 percent are both credible and the difference may reflect different reporting periods or methodological approaches. The correct approach is to annotate with both values and full source attribution letting the consumer decide, such as market growth estimates vary by source with 12 percent from the IEA World Energy Report published June 2024 using 2023 calendar year data and 8 percent from the Bloomberg NEF Annual Review published March 2024 using July 2022 to June 2023 data, with a note on the possible explanation.

Temporal awareness explains many apparent conflicts and is the third pattern. Different publication dates explain different numbers without contradiction. Source A published 2023 reporting 8 percent growth and source B published 2024 reporting 12 percent are not contradictory but a trend where growth accelerated. Require publication and data collection dates in all structured outputs, because without temporal context valid trends get misread as data quality issues and the synthesis agent may incorrectly flag or suppress findings that are actually consistent. Content-appropriate rendering completes the discipline: financial data is best as tables where numbers, comparisons, and trends are scannable, news and current events read naturally as prose for narrative and cause and effect, and technical findings are clearest as structured lists for hierarchy. Synthesis should not flatten everything into a uniform format, and attribution preservation through every step from research to analysis to synthesis to report generation must be explicitly required in the synthesis agent's prompt with coverage annotations that mark well supported, limited, conflicting, and unavailable areas.

The five-field mapping and the synthesis merge that preserves it

The structured claim-source mapping is a JSON object with five required fields: claim, sourceUrl, documentName, relevantExcerpt, and publicationDate preserved as an ISO 8601 string. The schema should be enforced at the boundary where subagent findings enter the pipeline, rejecting any finding with a missing field so provenance discipline holds at the source rather than being recovered later.

The synthesis merge takes two arrays of claim-source mappings and returns a unified array where every original mapping is preserved. Related findings are grouped by topic but source attribution is not collapsed, so each merged topic carries the union of its source mappings and every claim remains traceable to its origin. The prompt must contain an explicit instruction that every claim in the output must be traceable to a specific source and that mappings are preserved through the merge process, because summarisation actively destroys attribution unless instructed otherwise.

Conflict preservation with full attribution and temporal grouping

The conflict handling schema extends the mapping with detection and resolution fields. A typical conflict object carries field name, conflictDetected true, a values array with each entry holding value, source, and context such as audited financial statements for fiscal year ending December 2023 versus preliminary unaudited figures for calendar year 2023, and a possibleExplanation field noting likely reasons such as audited versus preliminary or different reporting periods or methodology. Completing analysis with conflicts intact is the rule: the analysis agent returns the structured conflict without resolving it, because the resolution decision belongs to the coordinator or consumer with full information.

Publication dates make correct interpretation possible by distinguishing a trend from a contradiction and by enabling a changelog-aware synthesis pattern where a newer changelog supersedes older documentation. The synthesis merge groups findings by date when temporal context matters, surfacing trends across publication periods rather than treating different dates as conflicting. Coverage annotations close the output with each topic area marked as well supported where multiple sources agree, limited with single source or partial coverage, conflicting where both values are preserved with attribution, or unavailable where no source was retrieved, with failed topics explicitly noting the failure type and attempted alternatives.

Content-appropriate rendering and per-source attribution

Financial data renders as tables with columns for year, value, source, and publication date so comparisons are immediate. News findings render as prose paragraphs for narrative context and chronological development. Technical findings such as architectural patterns, API specifications, and configuration options render as bulleted or numbered lists with clear hierarchy. A rendering function can detect content type by topic label or schema marker and apply the appropriate format, which the exam tests by checking that synthesis does not flatten everything into one format that degrades comprehension.

Per-source attribution extends beyond RAG findings to the six context sources: external knowledge carries source URLs and publication dates, tools carry execution logs and tool identifiers, memory carries storage location and retrieval timestamp, and state carries session identifier and update history. Evaluation adds a provenance check that periodically re-fetches a sample of cited sources and confirms they actually contain the claimed excerpts, catching citation drift where the synthesis agent has paraphrased away from the source, and a streaming protocol that includes citation markers inline so attribution is visible as synthesis arrives rather than only at the end.

Mechanism and API surface

Five-field claim-source mapping

Required fields of claim, sourceUrl, documentName, relevantExcerpt, and publicationDate as ISO 8601 string, enforced at ingestion so any finding with a missing field is rejected.

Synthesis merge that preserves every mapping

Merge function over two arrays of mappings that returns a unified array where the union of source mappings is preserved per topic and no mapping is discarded without explicit handling. Grouping by topic never collapses attribution.

Conflict preservation not resolution by the synthesis agent

When values overlap and disagree return a conflict object with conflictDetected true, values array carrying both entries with full attribution and context, and possibleExplanation, and let the coordinator or consumer decide. Averaging or picking one value destroys information.

Temporal context and changelog-aware grouping

Publication and data collection dates distinguish a trend from a contradiction and allow grouping by date to surface trends. A newer changelog supersedes older documentation and both are presented with dates rather than treating the older as authoritative.

Content-appropriate rendering

Financial data as tables, news as prose, and technical findings as structured lists, selected by content type so comparison, narrative, and hierarchy each use the format that maximises comprehension.

Coverage annotation and inline citation during streaming

Synthesis closes with well supported, limited, conflicting, or unavailable per topic, and streaming includes citation markers inline so attribution is visible as synthesis arrives, not only at the end.

Distinctions that decide answers

Traps