An engineering team evaluates a candidate embedding model and observes that aggregate Recall@5 across 500 test questions improved by 2.5 percent. However, post-deployment safety incidents increase. What statistical pitfall caused this failure?
Day 29: Run Controlled Experiments and Release Changes Safely
Run Controlled Experiments and Release Changes Safely
Loading lesson content...
In the PolicyDesk CI/CD release gate configuration above, why is the allowable regression on safety slices and security ACL breaches set to exactly 0.00 (zero tolerance), while general metrics allow statistical variance?
The FieldDesk engineering team prepares to deploy a new neural reranker. They need to validate real-world latency, token costs, and output divergence under actual production query distribution without exposing field technicians to unverified responses. Which deployment topology satisfies this requirement?