Archive (2016–2006)

Learning the Preferences of Physicians for the Organization of Result Lists of Medical Evidence Articles

Journal: Methods of Information in Medicine
Subtitle: A journal stressing, for more than 50 years, the methodology and scientific fundamentals of organizing, representing and analyzing data, information and knowledge in biomedicine and health care
ISSN: 0026-1270

Focus Theme
Pervasive Intelligent Technologies for Health
Guest Editors: O. Mayora, P. Lukowicz, M. Marschollek

Issue: 2014 (Vol. 53): Issue 5 2014
Pages: 344-356

Learning the Preferences of Physicians for the Organization of Result Lists of Medical Evidence Articles

Original Article

D. O´Sullivan (1, 2), S. Wilk (2, 3), W. Michalowski (2), R. Slowinski (3, 4), R. Thomas (5), M. Kadzinski (3), K. Farion (6)

(1) School of Informatics, City University London, London, United Kingdom; (2) Telfer School of Management, University of Ottawa, Ottawa, Canada; (3) Institute of Computing Science, Poznan University of Technology, Poznan, Poland; (4) Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland; (5) Sprott School of Business, Carleton University, Ottawa, Canada; (6) Departments of Pediatrics and Emergency Medicine, University of Ottawa, Ottawa, Canada


Information Retrieval, Decision Support Systems, Evidence-based medicine, Organization of medical evidence, physician preferences, rank-ordered lists, clinical


Background: Online medical knowledge repositories such as MEDLINE and The Cochrane Library are increasingly used by physicians to retrieve articles to aid with clinical decision making. The prevailing approach for organizing retrieved articles is in the form of a rank-ordered list, with the assumption that the higher an article is presented on a list, the more relevant it is.

Objectives: Despite this common list-based organization, it is seldom studied how physicians perceive the association between the relevance of articles and the order in which articles are presented. In this paper we describe a case study that captured physician preferences for 3-element lists of medical articles in order to learn how to organize medical knowledge for decision-making.

Methods: Comprehensive relevance evaluations were developed to represent 3-element lists of hypothetical articles that may be retrieved from an online medical knowledge source such as MEDLINE or The Cochrane Library. Comprehensive relevance evaluations asses not only an article’s relevance for a query, but also whether it has been placed on the correct list position. In other words an article may be relevant and correctly placed on a result list (e.g. the most relevant article appears first in the result list), an article may be relevant for a query but placed on an incorrect list position (e.g. the most relevant article appears second in a result list), or an article may be irrelevant for a query yet still appear in the result list. The relevance evaluations were presented to six senior physicians who were asked to express their preferences for an article’s relevance and its position on a list by pairwise comparisons representing different combinations of 3-element lists. The elicited preferences were assessed using a novel GRIP (Generalized Regression with Intensities of Preference) method and represented as an additive value function. Value functions were derived for individual physicians as well as the group of physicians.

Results: The results show that physicians assign significant value to the 1st position on a list and they expect that the most relevant article is presented first. Whilst physicians still prefer obtaining a correctly placed article on position 2, they are also quite satisfied with misplaced relevant article. Low consideration of the 3rd position was uniformly confirmed.

Conclusions: Our findings confirm the importance of placing the most relevant article on the 1st position on a list and the importance paid to position on a list significantly diminishes after the 2nd position. The derived value functions may be used by developers of clinical decision support applications to decide how best to organize medical knowledge for decision making and to create personalized evaluation measures that can augment typical measures used to evaluate information retrieval systems.

You may also be interested in...


Original Article

E. M. Zini (1), G. Lanzola (1), P. Bossi (2), S. Quaglini (1)

Methods Inf Med 2017 56 4: 283-293


State-of-the-Art Article

J. P. Agrawal (1), B. J. Erickson (1), C. E. Kahn, Jr. (2)

Yearb Med Inform 2016 Suppl. 1: S23-S31


Section 5: Decision Support


S. Quaglini, L. Sacchi, G. Lanzola, N. Viani

Yearb Med Inform 2015 : 106-118