Conference Paper

From Manual Coding to Machine Understanding: Students' Feedback Analysis

A. Alsharif, A. Katz

Proceedings of the 131st Annual Conference of the American Society for Engineering Education2024

Contribution Summary

The paper demonstrates a privacy-conscious, locally deployed generative-AI workflow for semi-automating qualitative coding of student feedback while retaining human validation.

Draft enrichment generated from extracted publication text; pending human review.

Plain-Language Summary

This conference paper evaluates a locally run generative-AI workflow for developing qualitative themes from open-ended engineering student feedback. The workflow is applied to student responses about the rapid transition to online learning during COVID-19 and compared with prior human-generated qualitative themes.

Research Question

How can a locally run generative AI system be leveraged to effectively develop qualitative themes from students' feedback?

Methods

  • Analyzed open-ended survey responses from a large first-year engineering course after the Spring 2020 transition to online learning.
  • Used a four-step workflow: text extraction with a locally deployed Dolphin-Mistral model, embedding with UAE-Angle, clustering similar ideas, and generating codes from clusters.
  • Compared generated topics with human qualitative coding and used human review to consolidate overlapping sub-codes.

Key Findings

  • The generative-AI workflow produced themes that substantially overlapped with human-generated coding, including feedback, communication, instructor support, engagement, flexibility, resources, and teaching methods.
  • The workflow helped organize large volumes of student feedback into interpretable topic structures, including treemap visualizations.
  • Human review remained necessary to consolidate overlapping sub-topics, refine terminology, and preserve context-specific meaning.

Implications

Local deployment can reduce privacy concerns when analyzing sensitive educational feedback.

Generative AI can accelerate first-pass theme development, but expert review is still needed for naming, organization, and interpretation.

Visual topic structures can help researchers and instructors inspect patterns in open-ended student feedback.

Research Artifacts

protocolFour-step local GAI workflowText extraction, embedding, clustering, and code generation workflow for student feedback analysis.
figureTopic treemapsVisual summaries of generated topics and sub-topics from student feedback prompts.

Abstract

Publication on From Manual Coding to Machine Understanding: Students' Feedback Analysis

All publications