Assessing reproducibility in faunal analysis using blind tests: A case study from northwestern North America. (February 2017)

• Record Type: Journal Article
• Authors: Nims, Reno; Butler, Virginia L.
• Is Part Of: Journal of archaeological science. Volume 11(2017)
• Page Start: 750
• Page End: 761
• Publication Date: 2017-02
• Subjects: Blind reanalysis -- Data quality control -- Reproducibility -- Zooarchaeology -- Agreement index -- Protocol drift; Archaeology -- Periodicals; Archaeology -- Research -- Periodicals; 930.1
• DOI: 10.1016/j.jasrep.2017.01.012
• Languages: English
• ISSNs: 2352-409X

Abstract

Abstract: Zooarchaeologists have long recognized that assigning taxonomic identifications to animal remains is a subjective process, and recent studies have highlighted the need for data quality assurance standards in archaeofaunal research. Our study contributes to this growing interest in quality assurance by presenting simple quantitative methods for assessing the reproducibility of analytic results through blind reanalysis of animal remains that we developed during analysis of fishbone from Čḯx w icən, a large Native American village on the coast of Washington State, U.S.A. Given the large scale of the Čḯx w icən project – over 112, 000 fish remains were documented by five different analysts over three years – there was a real possibility that inconsistencies in laboratory practices affected analytic results (e.g., number of identified specimens, taxa present and relative abundance, elements identified, burning frequencies). To evaluate the reproducibility of the Čḯx w icən fishbone data, and the possibility of "protocol drift" – changes in how specimens and bone modifications were identified over the course of analysis – we reanalyzed samples of fish remains that were previously documented during three discrete stages (beginning, middle, and end) of the Čḯx w icən project. The original data and the reanalysis results show close agreement in each stage, with only minor differences in the numbers of recorded specimens, taxonomic representation at family- and finer taxonomic levels, and body part representation assessed for a single taxonomic order. Identifying burning on bone was not very reproducible. Reproducibility studies such as this are useful for highlighting ambiguous identification criteria (e.g., in taxonomic assignment, bone modification), and could stimulate dialog among researchers about ways to address such issues in future studies. Increasing the implementation of these, and other, widely applicable methods should improve zooarchaeological data quality and stimulate further research on quality assurance in archaeology overall. Highlights: Blind reanalysis is effective for assessing the reproducibility of zooarchaeological data. Simple statistical methods for diagnostic agreement are useful tools for quantifying intra- and inter-analyst disagreement. Fishbone data for the Čḯx w icən fishbone assemblage is demonstrably reproducible. Quality assurance should be included in zooarchaeological research designs.