Measuring the amorphous: Substantive and methodological outcomes from concept maps. Issue 1 (19th January 2021)

• Record Type: Journal Article
• Authors: Foley, Rider W.; Ferguson, Sean M.; Pollack, Catherine C.
• Is Part Of: Journal of engineering education. Volume 110:Issue 1(2021)
• Page Start: 161
• Page End: 183
• Publication Date: 2021-01-19
• Subjects: accreditation criteria -- assessment tools -- concept inventory -- knowledge gain -- knowledge retention; Engineering -- Study and teaching (Higher) -- Periodicals; 620.00711
• DOI: 10.1002/jee.20373
• Languages: English
• ISSNs: 1069-4730

Abstract

Abstract: Background: Accreditation organizations broadened program assessment criteria in ways that present challenges for the evaluation of learning outcomes. This is especially the case in courses where there are not narrowly defined questions or definitive solutions, such as engineering ethics. While protocols for learning assessment exist, there is limited research exploring sociotechnical learning outcomes in a manner that combines theoretical, empirical, and procedural aspects of assessment. Purpose/Hypothesis: This paper shares 3 years of research into the effectiveness of concept maps as an assessment tool for engineering students in courses that prioritize professional skills development and sociotechnical thinking. We show that concept maps can offer evidence of knowledge formation and learning outcomes associated with courses that introduce complex problems with multiple possible interpretations or viable solutions. Design/Method: A concept mapping activity was completed by 614 undergraduate engineering students at the start and end of three different courses to evaluate sociotechnical thinking. Student‐level longitudinal changes were evaluated using paired t tests and simple linear regression. The concept mapping activity was improved iteratively in response to pilot tests and focus groups. Results: Undergraduate engineers demonstrated greater complexity in the representation of sociotechnical relationships as observed in the structure and content of the concept maps. The methodological results offer lessons about the administration and analysis of concept maps often excluded from conversations on course and curriculum level assessment. Conclusions: Concept maps offer insights into student learning outcomes, and they can be deployed and analyzed with minimal resources. However, assessments must be carefully designed to account for administrative and analytical challenges.