A Framework for Benchmarking Entity-Annotation Systems
Venue
Proceedings of the International World Wide Web Conference (WWW) (Practice & Experience Track), ACM (2013)
Publication Year
2013
Authors
Marco Cornolti, Paolo Ferragina, Massimiliano Ciaramita
BibTeX
Abstract
In this paper we design and implement a benchmarking framework for fair and
exhaustive comparison of entity-annotation systems. The framework is based upon the
definition of a set of problems related to the entity-annotation task, a set of
measures to evaluate systems performance, and a systematic comparative evaluation
involving all publicly available datasets, containing texts of various types such
as news, tweets and Web pages. Our framework is easily-extensible with novel entity
annotators, datasets and evaluation measures for comparing systems, and it has been
released to the public as open source. We use this framework to perform the first
extensive comparison among all available entity annotators over all available
datasets, and draw many interesting conclusions upon their efficiency and
effectiveness. We also draw conclusions between academic versus commercial
annotators.
