Archive (2016–2006)

Formative Evaluation of Ontology Learning Methods for Entity Discovery by Using Existing Ontologies as Reference Standards

Journal: Methods of Information in Medicine
Subtitle: A journal stressing, for more than 50 years, the methodology and scientific fundamentals of organizing, representing and analyzing data, information and knowledge in biomedicine and health care
ISSN: 0026-1270

Focus Theme
GMDS 2012 – Medical Informatics, Medicine and Neighboring Disciplines
Guest Editor: J. Stausberg

Issue: 2013 (Vol. 52): Issue 4 2013
Pages: 308-316

Formative Evaluation of Ontology Learning Methods for Entity Discovery by Using Existing Ontologies as Reference Standards

Original Article

K. Liu (1), K. J. Mitchell (1), W. W. Chapman (2), G. K. Savova (3), N. Sioutos (4), D. L. Rubin (5), R. S. Crowley (1, 6)

(1) Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA; (2) Division of Biomedical Informatics, University of california San Diego, San Diego, CA, USA; (3) Childrens´ Hospital Boston and Harvard Medical School, Boston, MA, USA; (4) Lockheed Martin Corporation, Fairfax, VA, USA; (5) Department of Radiology, Stanford University, Stanford, CA, USA; (6) Department of Pathology, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA


natural language processing, Ontology learning from text, ontology enrichment, statistical ontology learning method, statistical ontology, learning algorithm, ontology evaluation


Objective: Developing a two-step method for formative evaluation of statistical Ontology Learning (OL) algorithms that leverages existing biomedical ontologies as reference standards.

Methods: In the first step optimum parameters are established. A ‘gap list’ of entities is generated by finding the set of entities present in a later version of the ontology that are not present in an earlier version of the ontology. A named entity recognition system is used to identify entities in a corpus of biomedical documents that are present in the ‘gap list’, generating a reference standard. The output of the algorithm (new entity candidates), produced by statistical methods, is subsequently compared against this reference standard. An OL method that performs perfectly will be able to learn all of the terms in this reference standard. Using evaluation metrics and precision-recall curves for different thresholds and parameters, we compute the optimum parameters for each method. In the second step, human judges with expertise in ontology development evaluate each candidate suggested by the algorithm configured with the optimum parameters previously established. These judgments are used to compute two performance metrics developed from our previous work: Entity Suggestion Rate (ESR) and Entity Acceptance Rate (EAR).

Results: Using this method, we evaluated two statistical OL methods for OL in two medical domains. For the pathology domain, we obtained 49% ESR, 28% EAR with the Lin method and 52% ESR, 39% EAR with the Church method. For the radiology domain, we obtain 87% ESA, 9% EAR using Lin method and 96% ESR, 16% EAR using Church method.

Conclusion: This method is sufficiently general and flexible enough to permit comparison of any OL method for a specific corpus and ontology of interest.

You may also be interested in...


Original Article

K. Liu (1), W. W. Chapman (1, 2), G. Savova (3), C. G. Chute (3), N. Sioutos (4), R. S. Crowley (1, 2, 5)

Methods Inf Med 2011 50 5: 397-407



S. M. Meystre (1), G. K. Savova (2), K. C. Kipper-Schuler (2), J. F. Hurdle (1)

Yearb Med Inform 2008 : 128-144


Section 5: Decision Support, Knowledge Representation and Management


A.-M. Rassinoux, Managing Editor for the IMIA Yearbook Section on Decision Support, Knowledge Representation and Management

Yearb Med Inform 2008 : 80-82