The whole is more than the sum of its parts – assessing writing using the consensual assessment technique. Issue 9 (2nd September 2021)

• Record Type: Journal Article
• Authors: Zahn, Daniela; Canton, Ursula; Boyd, Victoria; Hamilton, Laura; Mamo, Josianne; McKay, Jane; Proudfoot, Linda; Telfer, Dickson; Williams, Kim; Wilson, Colin
• Is Part Of: Studies in higher education. Volume 46:Issue 9(2021)
• Page Start: 1907
• Page End: 1918
• Publication Date: 2021-09-02
• Subjects: Consensus -- writing -- 'Academic Literacies' -- assessment -- 'ICC'; Education, Higher -- Periodicals; 378.005
• DOI: 10.1080/03075079.2019.1711044
• Languages: English
• ISSNs: 0307-5079

Abstract

ABSTRACT: Evaluating the impact of Academic Literacies teaching (Lea and Street [1998. "Student Writing in Higher Education: An Academic Literacies Approach." Studies in Higher Education 23 (2): 157–72. doi:10.1080/03075079812331380364]) is difficult, as it involves gauging whether writers (1) gain better understanding of what influences written social interactions, and (2) improve their ability to manipulate language to address readers. Self-report can assess the first; the second can only be evaluated by examining texts and their effect on readers. Texts are commonly assessed with rubrics-based tools focussing on textual features, but their insensitivity to communicative context and readers' perception makes these inappropriate for an Academic Literacies framework (Canton [2018. "'It's Hard to Define Good Writing, but I Recognise it When I See it': Can Consensus-Based Assessment Evaluate the Teaching of Writing?" Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). Consensual assessment used by Amabile ([1996. Creativity in Context . Boulder, CL: Westview Press]) for creativity, offers a potential solution (Canton [2018. "'It's Hard to Define Good Writing, but I Recognise it When I See it': Can Consensus-Based Assessment Evaluate the Teaching of Writing?" Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). This paper presents a new instrument based on consensual assessment and empirically tests it. Intra-class Correlation Coefficients (ICCs) found moderate (Koo and Li [2016. "A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research." Journal of Chiropractic Medicine 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012]) to good (Cicchetti [1994. "Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology." Psychological Assessment 6 (4): 284–90. https://pdfs.semanticscholar.org/50d7/f68422d0c0424674f6b235ac23be8300da38.pdf]) agreement among raters, which offers proof of concept for capturing the readers' perception of the complex interactions in writing.