Resume

Shannon L. Coleman, PhD

Principal Mixed Methods Researcher and Strategist
Qualitative and Quantitative Research Methods  •  Product Strategy  •  10+ Years
Ground Truth Substack • Writing on AI research rigor, measurement, and the gap between confident and correct
The Capability Question • An AI-powered research design tool for measuring what AI enablement does to human judgment
AI Calibration Audit Tool • An open-source Python tool operationalizing a published four-layer AI evaluation framework

Principal Mixed Methods Researcher and Strategist with 10+ years shaping high-stakes product roadmaps, go-to-market strategies, and AI platform implementations. Leverages advanced quantitative methods (predictive modeling, regression, survival analysis, A/B testing) alongside deep qualitative inquiry (interviewing, contextual inquiry, usability testing) with an AI-forward approach to research and synthesis. Proven track record of presenting to C-suite and executive leadership, mentoring, standardizing research operations, and leading cross-functional teams. Formerly with Fisher-Price, DoD (Secret Clearance, inactive), and Crowley Webb/Praxis (clients include AAA, AbbVie, and M&T Bank).

Coleman Advisory 2024 to present
Principal Researcher and Strategist, Consultant

Consulting practice setting research and product strategy, working directly with founders and executives.

  • 16,000+ Users, First Six-Figure Agreement: Uncovered a trust gap through onboarding interviews and product data analysis; recommended a go-to-market strategy that borrowed credibility from self-funded employers, unions, and advocacy groups, growing an AI-powered health platform from fewer than 20 users to over 16,000, which led directly to its first six-figure service agreement
  • $50,000–$65,000 in Engineering Costs Prevented: Ran evaluative prototype testing with 6 users, which identified an AI resume feedback tool’s score-first design undermined trust in its recommendations; redirected the design to lead with actionable feedback, preventing unnecessary rework and raising task completion confidence in 5 of 6 participants
  • 22% Retention Lift, ~50% Fewer Support Tickets: Diagnosed 17 friction points in a UX and regulatory compliance audit of a Web3 tokenized asset platform, where users mistook normal transaction delays for lost funds; delivered a prioritized UX and compliance roadmap to lift 90-day retention 22% and cut support tickets ~50%
  • 35–50% Faster Compliance Approval: Cut approval cycles from 6–8 weeks to 3–5 weeks by merging UX and compliance into one workstream, aligning cross-functional stakeholders on onboarding and transaction flows, eliminating late-stage rework
  • UX Research for Early-Stage AI Startup: Synthesized recorded customer demos into a thematic analysis and recommended next steps and fundamental platform fixes and a follow-up round of testing, applying responsible AI practices to catch transcription errors
DoD / US Air Force 2022 to 2024
Lead Researcher and Product Strategist

Led teams across two programs with a combined 900,000+ users, setting strategy and earning leadership buy-in without formal authority.

Advana  |  DoD CDAO Enterprise AI & Analytics Platform
  • Foundational Mixed Methods Program, 1,400 Research Participants: Set the agenda for and led a mixed methods research program (segmentation and SUS surveys, in-depth interviews, card sorting, tree testing, and two rounds of usability testing) across Advana’s enterprise users, establishing the evidence base for AI platform strategy and roadmap priorities
  • Statistically Validated User Archetypes: Designed and executed a quantitative segmentation survey using exploratory factor analysis, two-step clustering, and chi-square to establish distinct, actionable user archetypes that product teams used for design
  • Six Failure Modes Reframed as One Structural Challenge: Synthesized findings into six simultaneous failure modes and reframed them for senior leadership as a single structural challenge, driving a DoD-level decision to pause development for infrastructure improvements on a platform that had scaled faster than its architecture, UX, and organizational support
  • Human-AI Trust Gap, CDAO Strategy Pivot: Used field observation on how users relied on or routinely overrode AI recommendations, identifying a systemic trust gap and adoption barriers in human-AI interaction that pivoted CDAO’s platform strategy toward persona-specific tools, infrastructure stability, and guided UX
  • 36% of Users Affected by a Content-Findability Crisis: Diagnosed an information architecture failure affecting 36% of users by uncovering persona-specific platform mental models; designed the sitemap, navigation redesign, and content model to resolve it
  • $15B Multi-Vendor Recompete, Two Product Roadmaps: Shaped strategic priorities for the $15B multi-vendor recompete and two product roadmaps, translating findings into actionable insights and recommendations, used communication tactics and presentation skills during recurring executive readouts to the Chief Experience Architect and Chief Product Officer
Air Force EITaaS  |  Enterprise IT Modernization ($5.7B program)
  • 84% Faster Startup, 59% Faster Data Transfer, 92% Less Security Workload: Built a jobs-to-be-done persona and workflow research framework from in-depth interviews, journey mapping, and moderated usability testing across nine pilot sites, setting user requirements for the infrastructure migration behind the gains and establishing the segmentation foundation for XLA standards
  • 800,000+ Personnel, XLA Standards in $5.7B Contracts: Facilitated stakeholder workshops to define success metrics and KPIs, translating research across five user personas into measurable XLA performance thresholds embedded in $5.7B in vendor contracts
  • Air Force Blueprint for Evaluating Commercial IT: Scaled a continuous VOC measurement framework and dashboard tracking user sentiment, IT satisfaction, and mission impact through experience monitoring; became the blueprint for commercial IT services
  • 74/100 Satisfaction vs. 43–46 Baseline: Validated the pilot model before the Air Force committed to enterprise-wide scaling by benchmarking IT satisfaction against comparable pre-transition bases, giving leadership the evidence to scale
  • U.S. Air Force Chief Experience Officer’s Challenge Coin: Awarded for shifting IT modernization into a user-centric service model, setting foundational standards for Air Force digital transformation
Crowley Webb / Praxis 2020 to 2022
Researcher and Strategist
  • $0 External Recruitment Spend, ~1,000-Participant Panel: Cut study recruitment timelines by 2+ weeks by building and scaling a proprietary research panel from zero aligned with GDPR, ISO 27001, and HIPAA practices, plus FDIC requirements for banking clients; democratized research
  • Phase II/III Trial Recruitment, F100 Pharma (Pfizer, Eli Lilly, Merck, J&J, AbbVie): Improved recruitment conversion and signups by capturing patient voice of customer through social listening, copy testing, and segmentation, combined with real-time performance analysis that drove site selection adjustments, messaging, and budget decisions
  • 28% Lift in Screener Conversions: Owned a continuous discovery program of full-funnel tracking across weekly sprints and quarterly longitudinal waves, synthesizing media spend, CTR, landing page, questionnaire drop-off data, and A/B tests of ad and landing page copy into consumer decision journey maps that optimized ad spend
  • 2–4 Days Faster per Study: Managed and set research operations standards for a 7–10 person team, standardizing recruitment workflows across concurrent client programs to cut project timelines and eliminate redundant overhead
University at Buffalo 2014 to 2019
Doctoral Researcher, Psychology & Quantitative Methods
  • Multi-Year Longitudinal Study, Dropout Risk Modeling: Led mixed methods study integrating qualitative methods with survival analysis and logistic regression on high-dimensional behavioral data to identify the strongest predictors of dropout
  • 60%+ Interrater Reliability Gain, 25-Person Team: Managed a 25-person research team, standardizing coding criteria, running bias-awareness training, and monitoring consistency, driving a 60%+ gain in interrater reliability and reducing costly re-coding cycles
Early Career
  • Researcher | Fisher-Price | $23M Category Entry, Four-Product Launch: Drove an estimated $23M, leading concept and usability testing, focus groups, diary studies, observation; partnering with Product, Engineering, and Design; consumer insights informed product development and market strategy for its first potty training line
  • Marketing Intelligence Analyst | Segmedica, Inc. | 35,000+ Survey Responses: Led survey design and analysis for a national segmentation study, packaged into a strategic research product that informed messaging to distinct consumer segments
University at Buffalo, State University of New York
PhD & MA, Psychology & Quantitative Methods
Fredonia, State University of New York
BS, Education
Honesty and IntegrityAccountabilityCommitment and PassionDecision Making CapabilitiesResilienceInspire OthersEmpathy
Human-Centered DesignAI-Forward Research and SynthesisAI EvaluationAgentic WorkflowsGood CommunicatorCross-Functional PartnershipInfluencing Senior LeadersGenerative ResearchStorytellingZero-to-One (0→1) DiscoveryA/B TestingAgile Research IntegrationVendor NegotiationBudget ManagementDesign ThinkingNavigating Ambiguous Problem SpacesContextual InquiryModel CalibrationAI Capability MeasurementBuilding and Scaling Research OperationsParticipant PrivacyTrust DynamicsCompliance-Driven ResearchWorkshop FacilitationResearch Planning and PrioritizationService Design
MuralAmplitudeQualtricsdscoutFigmaUserTestingOptimal WorkshopDovetailRSPSSSQLPythonTableauPower BIGA4JiraConfluenceServiceNowClaudeChatGPTGeminiPerplexityDovetail AINotion AIOtter.aiMiro AssistExcel and PowerPoint