Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/763
Title: | A universal method of information retrieval evaluation: the "missing" link M and the universal IR surface | Authors: | EGGHE, Leo | Issue Date: | 2004 | Source: | Information Processing & Management, 40(1). p. 21-30 | Abstract: | The paper shows that the present evaluation methods in information retrieval (basically recall R and precision P and in some cases fallout F) lack universal comparability in the sense that their values depend on the generality of the IR problem. A solution is given by using all "parts" of the database, including the non-relevant documents and also the notretrieved documents. It turns out that the solution is given by introducing the measure M being the fraction of the not-retrieved documents that are relevant (hence the "miss" measure). We prove that - independent of the IR problem or of the IR action - the quadruple (P,R,F,M) belongs to a universal IR surface, being the same for &l IR-activities. This universality is then exploited by defining a new measure for evaluation in IR allowing for unbiased comparisons of all IR results. We also show that only using 1, 2 or even 3 measures from the set {P,R,F,M} necessary leads to evaluation measures that are nonuniversal and hence not capable of comparing different IR situations. | Keywords: | Universal IR surface; miss measure; precision; recall; fallout; silence; evaluation | Document URI: | http://hdl.handle.net/1942/763 | ISSN: | 0306-4573 | e-ISSN: | 1873-5371 | DOI: | 10.1016/S0306-4573(02)00094-8 | ISI #: | 000187738300002 | Category: | A1 | Type: | Journal Contribution | Validations: | ecoom 2005 |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
universal.pdf | Non Peer-reviewed author version | 357.29 kB | Adobe PDF | View/Open |
universal 1.pdf Restricted Access | Published version | 256.97 kB | Adobe PDF | View/Open Request a copy |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.