INitiative for the Evaluation of XML Retrieval XML Entity Ranking (XER) 2008

21.Jan.2009 UPDATE: The evaluation results have been updated because of a bug found in the evaluation script. A new version of the script is available as well.

23.Jun.2009 UPDATE: Test collection released: final set of topics and relevance judgements for the XER and LC tasks available.

Testing Data

The testing data consists of 35 topics (inex08-xer-topics-final.xml) and their assessments in trec_eval format.

Genuine XER Topics

Topics 101-149 are genuine XER topics, in that the participants created these topics specifically for the track, and (almost all) topics have been assessed by the original topic authors.

From the originally proposed topics, we have dropped topics with less than 7 relevant entities (that is, 149, 111, and 120) and topics with more than 74 relevant entities (that is, 103, 101, 102, 146, 137, 148, 105, and 142).

Topic 145 has been excluded on request of the topic assessor.

Topics 107 and 131 have been dropped because their assessments were never finished.

The final set consists of 35 genuine XER topics (inex08-xer-topics-final.xml) with assessments.

Entity Relationship Topics

34 Entity Relationship topics have been developed based on the 49 XER Topics.

After the selection described in the previous section, 23 Entity Relationship topics are part of the final set of genuine XER topics considered in the evaluation.

Relevance assessments have not yet been performed for Entity Relationship topics.

Assessments Data Format (Qrels)

Assessments for the entity ranking topics are provided in qrels format, with the following structure:

• topicID
• Q0
• EntityID
• Stratum
• Relevance

The identifier for article ###.xml in the collection is given as WP### .

Use inex08-xer-testing-qrels-entity-ranking.txt to evaluate your results on the entity ranking tasks, and inex08-xer-testing-qrels-list-completion.txt to evaluate results for the list completion task. The difference between the two files is whether the entity examples are included as relevant answers or left out. Notice that your system should not include the given example entities in the answer set when evaluating the list completion task!

In the case of the entity ranking task, the organizers checked and fixed all the cases where the examples of relevant entities provided at topic creation time were inserted in the pool and judged as non-relevant at assessment time.

The official evaluation measure is xinfAP as defined in [1] which makes use of the stratified sampling for estimating Average Precision. For computing such measure the script sample_eval.pl (gently provided by Emine Yilmaz) can be used together with the qrels and the run files.

Guidelines and results

The evaluation results measured with xinfAP for the entity ranking task are:

The evaluation results measured with xinfAP for the list completion task are:

Guidelines are archived in the original INEX 2008 Entity Ranking guidelines document.

Judging INEX Entity Ranking

See the judging pages.

References

[1] A simple and efficient sampling method for estimating AP and NDCG. Emine Yilmaz, Evangelos Kanoulas, and Javed A. Aslam. SIGIR'08.