Reviewing the Test Reviews: Quality Judgments and Reviewer Agreements in the Mental Measurements Yearbook. Issue 2 (3rd April 2021)

• Record Type: Journal Article
• Authors: Hogan, Thomas; DeStefano, Marissa; Gilby, Caitlin; Kosman, Dana; Peri, Joshua
• Is Part Of: Applied measurement in education. Volume 34:Issue 2(2021)
• Page Start: 75
• Page End: 84
• Publication Date: 2021-04-03
• Subjects: Educational tests and measurements -- United States -- Periodicals; Educational tests and measurements -- Periodicals; Education -- Standards -- Periodicals; 371.26
• DOI: 10.1080/08957347.2021.1890742
• Languages: English
• ISSNs: 0895-7347

Abstract

ABSTRACT: Buros' Mental Measurements Yearbook ( MMY ) has provided professional reviews of commercially published psychological and educational tests for over 80 years. It serves as a kind of conscience for the testing industry. For a random sample of 50 entries in the 19th MMY (a total of 100 separate reviews) this study determined the level of qualitative judgment rendered by reviewers and the consistency of those independent reviewers in rendering judgments. Judgments of quality distributed themselves almost uniformly from very good to very bad across the 100 reviews. Agreement among reviewers for a given test was positive but relatively weak. We explore implications of the results and suggest follow-up investigations.