TL;DR
- What was at stake
- The client was betting that AI feedback would make the tool worth using for job seekers.
- What changed
- The team rebuilt the flow around specific recommendations with AI-generated before-and-after examples, and moved the score to a secondary, explained role.
- My role
- Solo researcher and strategist, from study design through recommendations
- Timeline
- About 2 weeks to recommendations, then a readout and follow-up test
Engineering cost avoided. A four-person team for two sprints is about 640 hours, at fully loaded salary rates.
Participants more confident they could revise their resume in the follow-up test, measured on the same pre and post scale.
Recommendations first, with AI before-and-after examples, and the score moved below them with an explanation.
An unexplained score made users skeptical of the advice that came with it.
- 1The score came first and got all the attention, with nothing explaining how it was calculated
- 2Suggestions too general to act on, with no tie to a specific line
- 1Each recommendation tied to a specific line, with an AI-generated before-and-after example
- 2The score moved down and explained, so it supports the advice
Read the full case study
Three changes to the flow
Tie each suggestion to a line in the person’s resume and say what to change, so the next step is obvious.
Have the AI rewrite the person’s own line as an example, so they can see what “better” looks like.
Move it below the recommendations and show what it measures, so it reads as context.
The team wanted to keep the score because it gave people something concrete to watch go up, or down, as they made changes. I’d have tested dropping it. It stayed, explained, and moved below the advice.
Two views of the score
A score up front would make people want to raise it, so they’d engage with the feedback.
People fixated on the number. When it felt arbitrary, they did not trust the suggestions under it.
The tool might not need a score at all.
What would have changed my mindIf people had used the score as a target and revised more, I’d have dropped the idea of cutting it.
The score stayed, but only as context below the recommendations.
Two sprints from build
Two engineers were already choosing parsing libraries and testing prompts for a design built around the score.
The tool analyzed a resume and suggested improvements. The prototype was a high-fidelity Figma flow with simulated AI output: upload a resume, read the feedback, see a score. I came in when the team was fairly sure what they wanted to build, but before engineering had committed to building it.
A small test aimed at one decision
Six target users uploaded their own resumes into the clickable prototype. I measured whether each person could revise their resume using the feedback, and how confident they felt doing it, before and after.
Each person uploaded a real resume and tried to revise it using the feedback.
Pointed the redesign at the feedback instead of the score.
After the sprint.
Six people was enough to see the pattern without spending extra time on more sessions.
The score got the attention the feedback needed
All 6 participants looked at the number first, and the recommendations got less of their attention.
Four of the six said the score felt arbitrary. With no explanation of how it was calculated, the score made people doubt the rest of the feedback too.
People didn’t know what to change when a suggestion wasn’t tied to a specific line.
Impact
This project changed how I think about scores in AI products. The number looked authoritative, but there was no explanation for how it was calculated, and people stopped trusting the advice underneath it. For the team, this meant building the recommendation-first version from the start, instead of reworking a score-first one after launch.
Hindsight
Confidence was self-reported. Having someone rate the before and after resumes would show whether people’s revisions actually got better.I could say the redesign produced better resumes, not only more confident users.
Participants judged whether the score felt fair. Running it against recruiter ratings with my calibration audit would show whether a high score actually meant a strong resume.The team would have known whether to fix the score or drop it, instead of only moving it down the page.
With two more weeks
See whether showing how the score is calculated builds trust, or whether the score should go entirely.
Track whether people come back for a second round of revisions, which a one-session test can’t show.