INitiative for the Evaluation of XML Retrieval XML Entity Ranking (XER) 2009

25 Feb 2010. Official results posted. Qrel files will be posted here when INEX 2009 proceedings will be published.
12 Jul 2010. Test Collection (i.e., qrel files) posted.

Testing Data

The testing data consists of 55 topics (inex09-xer-topics-final.xml) and their assessments in trec_eval format.

XER Topics

60 genuine XER topics have originally been selected from the last 2 editions to be run on the new INEX Wikipedia collection. Topics have been re-assessed by INEX-XER 2009 participants on the new collection.

As last year, from the originally proposed ones, topics with less than 7 relevant entities (that is, 104, and 90) and topics with more than 74 relevant entities (that is, 78, 112, and 85) have been excluded.

The final set consists of 55 genuine XER topics with assessments.

LC Topics

Out of the 55 XER topics, 3 topics have been excluded for the LC task (i.e., 143, 126, and 132).

The reason is that example entities for these topics were not relevant as the underlying Wikipedia collection has changed.

After this selection, 52 List Completion topics are part of the final set and are considered in the evaluation.

Assessments Data Format (Qrels)

Assessments for the INEX XER topics are provided in qrels format, with the following structure:

• topicID
• Q0
• EntityID
• Strata
• Relevance

The identifier for article ###.xml in the collection is given as WP### .

The possible relevance values are:

• 0 meaning not relevant
• 1 meaning relevant
• 2 meaning not-an-entity (to be considered as not relevant for xinfap calculation)

Guidelines and results

Use inex09-xer-testing-qrels-entity-ranking.txt to evaluate your results on the entity ranking tasks, and inex09-xer-testing-qrels-list-completion.txt to evaluate results for the list completion task. The difference between the two files is whether the entity examples are included as relevant answers or left out and that 3 topics have been removed for LC. Notice that your system should not include the given example entities in the answer set when evaluating the list completion task!

In the case of the entity ranking task, the organizers checked and fixed all the cases where the examples of relevant entities provided at topic creation time were inserted in the pool and judged as non-relevant at assessment time.

The official evaluation measure is xinfAP as defined in [1] which makes use of the stratified sampling for estimating Average Precision. For computing such measure the script sample_eval_xer09.pl (gently provided by Emine Yilmaz) can be used together with the qrels and the run files. The new version of the script takes into account not-an-entity judgements as non-relevant results.

The evaluation results measured with xinfAP for the entity ranking task are:

The evaluation results measured with xinfAP for the list completion task are:

Guidelines are archived in the original INEX 2009 Entity Ranking guidelines document.

Judging INEX Entity Ranking

See the judging pages.