Source: This example was kindly contributed by André Lamúrias, LaSIGE, Faculdade de Ciências, Universidade de Lisboa, Portugal.

This project consisted in comparing crowd annotations with automatic annotation approaches. We organized two annotation sessions using WebAnno as the annotation platform, which we installed in our server. On the first annotation session, we used the Human Phenotype corpus, annotation with concepts from the human phenotype ontology, and evaluated a crowd approach with the gold standard. On the second occasion, we annotated radiology terms on clinical reports, and compared with pattern-based and machine learning approaches.

We choose WebAnno since it was versatile enough for our needs, easy to install and set-up in our server, and free of charge.

Publications

• Luis F. Campos, Andre Lamurias, Francisco M. Couto. Can the Wisdom of the Crowd Be Used to Improve the Creation of Gold-standard for Text Mining applications?. INForum - Simpósio de Informática, Aveiro, Portugal. [PDF]

Data

• HPO Crowd Annotation Experiment