Abstract
This paper extends surprisal-based minimal-pairs evaluation of language models from binary grammaticality judgments to ordinal-scaled classification and scoring tasks. Instead of asking models to generate answers, it measures the surprisal a model assigns to each position on a rating scale, yielding surprisal curves that reveal both the model's preferred response and its uncertainty via entropy, demonstrated across four applied domains including social-ecological-technological systems classification and deductive qualitative coding.