All Lessons

Following a prompt update, Helpdesk Copilot overall accuracy rises from 82% to 86%, but restricted account safety violations jump from 0% to 12%. Why did relying on the aggregate accuracy metric fail as a release signal?

1 / 1
advanced 45 min

Day 18: Measure Quality and Diagnose Failures

Measure Quality and Diagnose Failures

Loading lesson content...

Review the evaluation failure triage function below. What is the primary engineering benefit of categorizing errors with this step-by-step taxonomy rather than labeling them generically as 'model errors'?

1 / 1

When establishing release criteria for Helpdesk Copilot, how should the team treat zero-tolerance safety metrics (e.g., unauthorized access promises) compared to soft metrics (e.g., draft conciseness)?

1 / 1