How can researchers combine qualitative judgment with open-source generative AI to scale thematic analysis without hiding methodological choices?
Using Large Language Models and Generative AI to Scale Qualitative Data Analysis
Leveraging open-source large language models and generative AI to create workflows to conduct large-scale qualitative data analysis
Findings
-
An open-model workflow for thematic analysis found nearly all the themes planted in synthetic test data
When the lab planted known themes in three synthetic datasets, the GATOS workflow produced a good match for 166 of 187 planted sub-themes, a partial match for 18 more, and missed 3.
Approach
- Design reproducible workflows for inductive qualitative codebook development
- Compare model-assisted coding outputs with researcher interpretation and validation practices
- Document where AI systems can support analysis and where human qualitative judgment remains essential
Objectives
- Investigate how to combine modern machine learning methods with open-source large language models and generative AI to identify themes in qualitative data
- Combine language models with traditional qualitative methods to enhance analysis speed and scalability
Evidence and materials
- Thematic analysis with open-source generative AI and machine learning
- Featured paper introducing an open-source generative AI and machine learning method for inductive qualitative codebook development.
- Using generative AI for large-scale qualitative analysis of social media posts
- Journal of Engineering Education study using generative AI to analyze more than 10,000 Reddit posts about why people leave computer science.
Tools from this project
Team
- PI