Can you fool AI by doing a 180? — A case study on authorship analysis of texts by Arata Osada. Issue 5 (September 2021)

• Record Type: Journal Article
• Authors: Nieuwazny, Jagna; Nowakowski, Karol; Ptaszynski, Michal; Masui, Fumito
• Is Part Of: Information processing & management. Volume 58:Issue 5(2021)
• Publication Date: 2021-09
• Subjects: Authorship analysis -- Single authorship identification -- Authorship verification -- Similarity detection -- Binary text classification -- Transformers -- Personal ethics; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2021.102644
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: This paper is our attempt at answering a twofold question covering the areas of ethics and authorship analysis solutions. Firstly, since the methods used for performing authorship analysis imply that an author can be recognized by the content he or she creates, we were interested in finding out whether it would be possible for an author identification system to correctly attribute works to authors if in the course of years they have undergone a major psychological transition. Secondly – and from the point of view of the evolution of an author's ethical values – we checked what it would mean if the authorship attribution system encounters difficulties in detecting single authorship. We set out to answer those questions through performing a binary authorship analysis task using a text classifier based on a pre-trained transformer model and a baseline method relying on conventional similarity metrics. For the test set, we chose several works of Arata Osada, a Japanese educator and specialist in the history of education, with half of them being books written before the Second World War and another half in the 1950s, in between which the author underwent a transformation in terms of political opinions. As a result, we were able to confirm that in the case of texts authored by Arata Osada in a time span of more than 10 years, while the classification accuracy drops by a large margin and is substantially lower than for texts by other non-fiction writers, confidence scores of the predictions remain at a similar level as in the case of a shorter time span, indicating that the classifier was in many instances tricked into deciding that texts written by Arata Osada over a time span of multiple years were actually written by two different people, which in turn leads us to believe that such a change can affect authorship analysis, and that historical events have great impact on a person's ethical outlook as expressed in their writings. Highlights: A shift in writer's opinions poses a challenge to author analysis solutions. Time-related changes have to be considered in an authorship analysis task. Drop in performance of solution highest for fiction books due to topical diversity.