All Lessons

In FieldDesk, why is Mean Reciprocal Rank (MRR @ K) considered the primary ranking metric for evaluating factoid and diagnostic lookup queries (such as alarm code meanings or pinout diagrams)?

1 / 1
advanced 45 min

Day 19: Measure Retrieval with Ranking Metrics

Measure Retrieval with Ranking Metrics

Loading lesson content...

For a given test query in FieldDesk, gold_ids contains {'manual_chunk_42'}. The retrieval engine returns candidate list ['manual_chunk_10', 'manual_chunk_20', 'manual_chunk_42', 'manual_chunk_88', 'manual_chunk_99']. What are the resulting hit_rate and mrr values calculated by the function above?

1 / 1

In PolicyDesk, an evaluation query compares international parental leave addenda across three countries, where the gold standard dataset contains four relevant passages graded on an ordinal relevance scale from 3 (binding statutory top-up) down to 1 (general HR overview). Why is Normalized Discounted Cumulative Gain (nDCG @ K) preferred over Recall @ K or MRR @ K for this evaluation?

1 / 1