In FieldDesk, why is Mean Reciprocal Rank (MRR @ K) considered the primary ranking metric for evaluating factoid and diagnostic lookup queries (such as alarm code meanings or pinout diagrams)?
Day 19: Measure Retrieval with Ranking Metrics
Measure Retrieval with Ranking Metrics
Loading lesson content...
For a given test query in FieldDesk, gold_ids contains {'manual_chunk_42'}. The retrieval engine returns candidate list ['manual_chunk_10', 'manual_chunk_20', 'manual_chunk_42', 'manual_chunk_88', 'manual_chunk_99']. What are the resulting hit_rate and mrr values calculated by the function above?
In PolicyDesk, an evaluation query compares international parental leave addenda across three countries, where the gold standard dataset contains four relevant passages graded on an ordinal relevance scale from 3 (binding statutory top-up) down to 1 (general HR overview). Why is Normalized Discounted Cumulative Gain (nDCG @ K) preferred over Recall @ K or MRR @ K for this evaluation?