Date: Tuesday, October 27, 2026
Time: 1:00pm-2:30pm ET
Format: Virtual/Zoom
Description: Organizations increasingly use human ratings to evaluate AI outputs and then train automated evaluators to reproduce those ratings at scale. Yet agreement with human ratings shows that a model can reproduce an observed score, not that the score captures the intended construct. Ratings of qualities such as helpfulness, accuracy, and task success depend on how those constructs are defined and how the rating instrument presents them.
This talk uses evidence from human annotation experiments, large-language-model studies, and human review of AI-generated suggestions to show how reasonable changes in task structure, prompts, response scales, batching, and review interfaces can alter ratings and downstream conclusions. LLMs can produce highly consistent answers within a fixed design while remaining sensitive to defensible changes in that design. As a result, conventional sampling uncertainty can substantially understate total measurement uncertainty.
AI has made measurement cheap and made knowing what to measure more important than ever. Survey scientists and other measurement experts should be involved before human or LLM ratings are collected.
Speaker: Stephanie Eckman
Stephanie Eckman is a Researcher at the University of Maryland's Social Data Science Center, with a Ph.D. in Statistics & Methodology. Her research bridges survey methodology and natural language processing, focusing on how data collection and annotation practices shape the training data behind AI systems. She has published at venues including EMNLP, ICML, and Harvard Data Science Review, and speaks regularly on bringing survey science to AI. Before Amazon, she worked at RTI International and held research and teaching roles at the Institute for Employment Research and the University of Mannheim in Germany.