All Lessons

Why is evaluating an enterprise RAG system solely by inspecting the final generated text (the 'vibe check') fundamentally untrustworthy?

1 / 1
beginner 45 min

Day 3: Create a Gold Set with Evidence Labels

Create a Gold Set with Evidence Labels

Loading lesson content...

In the FieldDesk gold set item above, why does the ground_truth_evidence field explicitly specify chunk_id instead of merely citing MAN-X840-D.pdf as the target document?

1 / 1

The evaluation harness runs on PolicyDesk across 500 benchmark queries. Retrieval metrics show Recall@5 is 0.97 and MRR is 0.92, but end-to-end Faithfulness is only 0.69 due to the LLM introducing ungrounded assertions during synthesis. Based on this decoupled evaluation, what is the correct engineering priority?

1 / 1