RelevanceEvaluatorPlus

class trove.evaluation.relevance_evaluator.RelevanceEvaluatorPlus(query_relevance, measures, relevance_level=1)

__init__(query_relevance, measures, relevance_level=1)

Expands pytrec_eval.RelevanceEvaluator with new metrics.

User facing API is the same as pytrec_eval.RelevanceEvaluator and should be a drop-in replacement. See pytrec_eval.RelevanceEvaluator for more details.

Parameters:

• query_relevance (Dict[str, Dict[str, Dict[str, int]]]) – see pytrec_eval.RelevanceEvaluator.__init__()

• measures (Union[Set[str], List[str]]) – see pytrec_eval.RelevanceEvaluator.__init__()

• relevance_level (int) – see pytrec_eval.RelevanceEvaluator.__init__()

evaluate(scores)

Calculate IR metrics for the given similarity scores.

User facing API is the same as pytrec_eval.RelevanceEvaluator.evaluate() and should be a drop-in replacement. See pytrec_eval.RelevanceEvaluator.evaluate() for more details.

Parameters:

scores (Dict[str, Dict[str, Dict[str, float]]]) – predicted similarity scores. see pytrec_eval.RelevanceEvaluator.evaluate()

Return type:

Dict[str, Dict[str, Dict[str, float]]]

Returns:

a mapping where output[qid][metric_name] is the value of “metric_name” for query “qid”. see pytrec_eval.RelevanceEvaluator.evaluate()