IDEEAS Lab

An open-model workflow for thematic analysis found nearly all the themes planted in synthetic test data

When the lab planted known themes in three synthetic datasets, the GATOS workflow produced a good match for 166 of 187 planted sub-themes, a partial match for 18 more, and missed 3.

GATOS matched 166 of 187 planted sub-themes well Unit chart with one square for each of the 187 sub-themes planted in three synthetic datasets. Teammate feedback: 60 sub-themes, 54 good matches, 6 partial, 0 with no match. Organizational culture of ethics: 64 sub-themes, 52 good matches, 9 partial, 3 with no match. Returning to the workplace: 63 sub-themes, 60 good matches, 3 partial, 0 with no match. In total, 166 good, 18 partial, and 3 with no match, all three in the ethics dataset. Good match 166 Partial match 18 No match 3 Each square is one planted sub-theme (187 in all), judged by the research team Teammate feedback 60 sub-themes: 54 good, 6 partial, 0 none Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Good match Teammate feedback: Partial match Teammate feedback: Partial match Teammate feedback: Partial match Teammate feedback: Partial match Teammate feedback: Partial match Teammate feedback: Partial match Organizational culture of ethics 64 sub-themes: 52 good, 9 partial, 3 none Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Good match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: Partial match Organizational culture of ethics: No match Organizational culture of ethics: No match Organizational culture of ethics: No match Returning to the workplace 63 sub-themes: 60 good, 3 partial, 0 none Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Good match Returning to the workplace: Partial match Returning to the workplace: Partial match Returning to the workplace: Partial match

Each square is one sub-theme the team planted in the synthetic data, judged against the themes GATOS produced. Almost every square is a good match; the only three misses came from the ethics dataset.

Drawn from the match counts reported in Katz, Coloyan Fleming & Main (2026), pp. 11-12. CC BY 4.0.

GATOS, the lab’s workflow for building qualitative codebooks with open-source language models, found nearly all of the themes it was supposed to find. Across three synthetic datasets with known themes built in, the research team judged that the workflow’s output was a good match for 166 of 187 planted sub-themes and a partial match for 18 more. Three had no match, all in one dataset. Every one of the 24 broader planted themes (eight per dataset) had a good match.

Some matches were word for word: “resistance to traditional office hours” came back as exactly that. The misses were subtler. The planted sub-theme “avoiding blame” came back only as “personal bias management” and “negative behaviors and their impact”, which circle the idea without naming it.

Why it matters

Thematic analysis is how many social scientists turn open-ended text into findings, and it is slow. A small team can work through tens or hundreds of responses, but thousands strain both time and consistency. Prior studies of language models for qualitative analysis mostly worked with small datasets in a single context. GATOS runs on open-weight models on local hardware, so data need not be sent to a commercial provider, and researchers control which model version they use.

How the lab did it

The test borrows a habit from quantitative methods research: simulate data where you control the answer, then check whether the method recovers it. The team used language models to write 823 to 1,110 realistic responses per dataset on three topics (teammate feedback, organizational cultures of ethics, and returning to the office after the pandemic). Each dataset was built from eight themes with eight sub-themes apiece, mixed with varied personas, contexts, and writing styles. A different model then ran GATOS on those responses. The workflow summarizes each response into short points, clusters similar points, and for each cluster asks the model whether an existing code already covers it or a new one is needed, showing it only the closest existing codes. A final pass merges near-duplicate codes into themes.

Andrew Katz wrote the prompts and ran the analysis. Gabriella Coloyan Fleming, a research scientist in the lab, wrote the paper’s review of prior work, and Joyce B. Main of Purdue University helped conceptualize and edit it.

Synthetic data are a controlled test, not the real thing. Text written by language models may be easier for other language models to summarize than human writing is; the team used different models for writing and analysis to reduce that risk. The test asked whether planted ideas were found, not whether every generated theme was useful: the workflow produced 75 to 110 themes per dataset against eight planted themes, and near-duplicate codes sometimes survive. The responses were short answers to a single question rather than interview transcripts, and the authors list scaling to tens of thousands of responses as an open problem.

The next question is whether GATOS reproduces what human analysts found in real data. The authors report that comparison is under way on three datasets from educational research.

The paper

Katz, A., Fleming, G. C., & Main, J. B. (2026). Thematic analysis with open-source generative AI and machine learning: A new method for inductive qualitative codebook development. Humanities and Social Sciences Communications. https://doi.org/10.1057/s41599-026-06508-5

Related project

How can researchers combine qualitative judgment with open-source generative AI to scale thematic analysis without hiding methodological choices?

All findings