Shannon L. Coleman, PhD

Coleman Advisory

Research consulting and fractional research leadership for product teams, especially teams building with AI.

I take on a small number of projects. Most start with one of these questions.

How should research work here?

Centralized or embedded, built in house or bought, and how to show leadership what the function is worth.

How do we study an AI product?

Standard UX methods miss a lot when a product gives different answers to the same question. I help teams design evaluative research and find out how much users trust the system and how well that trust is calibrated to its accuracy.

Can we trust these numbers?

Survey design, sampling, weighting, and knowing when a question needs a regression model and when it needs an hour of listening.

I'm a mixed methods researcher and strategist with a PhD in psychology and quantitative methods. I write Ground Truth, a newsletter on research rigor and the gap between how confident a finding sounds and how accurate it is.

I also built The Capability Question, an AI research design tool, and an open source tool that checks whether an AI system's confidence matches how often it's right.

Setting up research? Building an AI product? Not sure your numbers hold up?

These are some of the problems I like working through the most with teams. Email me with a few lines about what you're working on.

shannon@shannonlcoleman.com