An experimental study measuring human annotator categorization agreement on commonsense sentences. (18th June 2021)

• Record Type: Journal Article
• Authors: Ramírez Rivera, Adín; Santos, Henrique; Kejriwal, Mayank; Mulvehill, Alice M.; Forbush, Gretchen; McGuinness, Deborah L.
• Is Part Of: Experimental results. Volume 2(2021)
• Publication Date: 2021-06-18
• Subjects: Commonsense reasoning -- benchmarking -- annotation -- annotator agreement -- commonsense theories; Science -- Experiments -- Periodicals; Science -- Methodology -- Periodicals; 507.24
• DOI: 10.1017/exp.2021.9
• Languages: English
• ISSNs: 2516-712X

Abstract

Abstract: Developing agents capable of commonsense reasoning is an important goal in Artificial Intelligence (AI) research. Because commonsense is broadly defined, a computational theory that can formally categorize the various kinds of commonsense knowledge is critical for enabling fundamental research in this area. In a recent book, Gordon and Hobbs described such a categorization, argued to be reasonably complete. However, the theory's reliability has not been independently evaluated through human annotator judgments. This paper describes such an experimental study, whereby annotations were elicited across a subset of eight foundational categories proposed in the original Gordon-Hobbs theory. We avoid bias by eliciting annotations on 200 sentences from a commonsense benchmark dataset independently developed by an external organization. The results show that, while humans agree on relatively concrete categories like time and space, they disagree on more abstract concepts. The implications of these findings are briefly discussed.