All Lessons
Following a prompt update, Helpdesk Copilot overall accuracy rises from 82% to 86%, but restricted account safety violations jump from 0% to 12%. Why did relying on the aggregate accuracy metric fail as a release signal?
1 / 1
advanced 45 min
Day 18: Measure Quality and Diagnose Failures
Measure Quality and Diagnose Failures
Based on Anthropic Claude documentation and the CCA-F exam blueprint. Updated August 2026.
Loading lesson content...
Review the evaluation failure triage function below. What is the primary engineering benefit of categorizing errors with this step-by-step taxonomy rather than labeling them generically as 'model errors'?
1 / 1
When establishing release criteria for Helpdesk Copilot, how should the team treat zero-tolerance safety metrics (e.g., unauthorized access promises) compared to soft metrics (e.g., draft conciseness)?
1 / 1