Expanding possibilities for generative AI in qualitative analysis: Fostering student feedback literacy through the application of a feedback quality rubric
Summary
This study investigates the reliability and utility of generative AI to apply a feedback quality rubric to peer feedback comments from first-year engineering students, exploring how LLMs can assist in qualitative analysis.
Findings from this paper
-
A language model scoring student peer feedback struggled on the same rubric criteria that human raters found hardest
Applying a four-part feedback-quality rubric to 295 peer comments, an open-source language model agreed well with researchers on whether a comment named a problem or suggested a fix, and poorly on the two criteria where the researchers also agreed least.
Cite this paper
Drinkwater Gregg, K., Ryan, O., Katz, A., Huerta, M., & Sajadi, S. (2025). Expanding possibilities for generative AI in qualitative analysis: Fostering student feedback literacy through the application of a feedback quality rubric. Journal of Engineering Education. https://doi.org/10.1002/jee.70024
BibTeX
@article{drinkwatergregg2025expanding,
title = {Expanding possibilities for generative AI in qualitative analysis: Fostering student feedback literacy through the application of a feedback quality rubric},
author = {Drinkwater Gregg, Katherine and Ryan, Olivia and Katz, Andrew and Huerta, Mark and Sajadi, Susan},
journal = {Journal of Engineering Education},
year = {2025},
doi = {10.1002/jee.70024}
}