All Lessons
Why must an evaluation dataset be frozen before the team begins tuning prompt templates and retrieval configurations for Helpdesk Copilot?
1 / 1
advanced 45 min
Day 17: Build an Evaluation Set Before Tuning Further
Build an Evaluation Set Before Tuning Further
Based on Anthropic Claude documentation and the CCA-F exam blueprint. Updated August 2026.
Loading lesson content...
Review the evaluation case definition below. Why does this case structure specify required_facts and forbidden_claims rather than asserting against a single canonical answer string?
1 / 1
The engineering team compiles 100 reviewed evaluation cases across five ticket categories. Why should these cases be partitioned into a development set and an untouched holdout set?
1 / 1