Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching

Plain-Language Summary

This paper presents an NLP and generative text workflow for creating qualitative codebooks from student evaluations of teaching. The workflow extracts ideas from thousands of comments, embeds and clusters them, and uses a generative model to summarize candidate codes for researcher review.

Research Question

How can NLP and generative text models support qualitative codebook generation for large collections of student evaluations of teaching?

• Analyzed a corpus of 5,000 student evaluations of teaching from undergraduate science and engineering courses.
• Used an extract, embed, cluster, and summarize workflow to identify semantically related ideas and generate candidate code labels.
• Compared the generated codebook with prior human-generated categories and educational frameworks.

• The workflow produced a detailed codebook with substantial overlap with human-generated categories.
• The approach surfaced granular themes across a larger dataset than would typically be feasible through first-pass manual coding alone.
• The paper emphasizes that researchers still need to judge relevance, redundancy, abstraction level, and saturation.