Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research. (March 2014)

• Record Type: Journal Article
• Authors: Stevens, Mark R.; Lyles, Ward; Berke, Philip R.
• Is Part Of: Journal of planning education and research. Volume 34:Number 1(2014:Spring)
• Page Start: 77
• Page End: 93
• Publication Date: 2014-03
• Subjects: plan quality -- plan evaluation -- content analysis -- intercoder reliability -- percent agreement -- Krippendorff's alpha; Regional planning -- Study and teaching -- United States -- Periodicals; City planning -- Study and teaching -- United States -- Periodicals; 307.12
• DOI: 10.1177/0739456X13513614
• Languages: English
• ISSNs: 0739-456X

Abstract

Plan quality evaluation researchers typically evaluate plans in relation to whether they contain certain desirable features. Best practice dictates that plans be evaluated by at least two readers and that researchers report a measure of the extent to which the readers agree on whether the plans contain the desirable features. Established practice for assessing this agreement has been subject to criticism. We summarize this criticism, discuss an alternative approach to assessing agreement, and provide recommendations for plan quality evaluation researchers to follow to improve the quality of their data and the manner in which they assess and report that quality.