Algorithmic failure as a humanities methodology: Machine learning's mispredictions identify rich cases for qualitative analysis. Issue 2 (July 2022)

• Record Type: Journal Article
• Authors: Rettberg, Jill Walker
• Is Part Of: Big data & society. Volume 9:Issue 2(2022)
• Publication Date: 2022-07
• Subjects: Machine vision -- machine learning -- qualitative methodology -- machine anthropology -- digital humanities -- algorithmic failure; Big data -- Social aspects -- Periodicals; Social sciences -- Research -- Data processing -- Periodicals; Social sciences -- Research -- Methodology -- Periodicals; Data mining -- Periodicals; 300.28557
• DOI: 10.1177/20539517221131290
• Languages: English
• ISSNs: 2053-9517

Abstract

This commentary tests a methodology proposed byMunk et al. (2022) for using failed predictions in machine learning as a method to identify ambiguous and rich cases for qualitative analysis. Using a dataset describing actions performed by fictional characters interacting with machine vision technologies in 500 artworks, movies, novels and videogames, I trained a simple machine learning algorithm (using the kNN algorithm in R) to predict whether or not an action was active or passive using only information about the fictional characters. Predictable actions were generally unemotional and unambiguous activities where machine vision technologies were treated as simple tools. Unpredictable actions, that is, actions that the algorithm could not correctly predict, were more ambivalent and emotionally loaded, with more complex power relationships between characters and technologies. The results thus support Munk et al.'s theory that failed predictions can be productively used to identify rich cases for qualitative analysis. This test goes beyond simply replicating Munk et al.'s results by demonstrating that the method can be applied to a broader humanities domain, and that it does not require complex neural networks but can also work with a simpler machine learning algorithm. Further research is needed to develop an understanding of what kinds of data the method is useful for and which kinds of machine learning are most generative. To support this, the R code required to produce the results is included so the test can be replicated. The code can also be reused or adapted to test the method on other datasets.