TL;DR
- What was at stake
- DoD had invested millions in Advana, people weren’t using it as much as expected, and a major contract decision was coming.
- What changed
- Product teams designed against the research’s statistically validated user archetypes, and CDAO’s platform strategy shifted toward tools built for each persona, a more stable infrastructure, and more guidance in the interface.
- My role
- Lead researcher and platform strategist, directing a cross-functional team of 7, within a larger Isobar team of about 13, from study design through the prototype and roadmap
- Timeline
- About 14 months
Many people at CDAO and across DoD shaped the pause and the recompete. My team’s research set the user priorities those decisions drew on.
Development, in June 2024, to fix the infrastructure before adding features, as I recommended.
Recompete that followed, centered on a modular, multi-vendor architecture built around distinct groups of users.
A redesigned navigation, content model, and clickable prototype, plus two product roadmaps.
Baseline: of about 220 scored feedback responses before the engagement, 38% said people fully accomplished what they came to do.
Leadership had been treating each problem on its own. The research showed six failures happening at once, all because the platform had grown faster than its architecture, design, and support.
- 1Same layout for financial analysts, logistics officers, intelligence staff, and command leadership
- 2Content spread across community spaces; 36% of users had trouble finding what they needed
- 3Feedback capped at about 100 characters, so it flagged frustration without explaining it
- 1Entry points for each of the four main roles
- 2Navigation built from a card sort and checked with a tree test
- 3A clear starting point for people with less data experience
Read the full case study
Three recommendations, in order
I delivered this recommendation in writing and in a formal briefing to the Chief Experience Architect and Chief Product Officer. DoD paused development in June 2024 to make infrastructure improvements.
Make it clearer where to go and what to do for every role, and lower the effort for less technical users without limiting experienced ones. The card sort and tree test produced the new sitemap, navigation, and content model behind the prototype.
The research supported modular components built around each group’s data and workflows, which matches the modular, multi-vendor approach CDAO pursued in the recompete.
Pausing meant holding back changes that were already lined up to ship, and there was pushback. I recommended it anyway, because adding them to an unstable platform would have made the existing problems worse.
What I expected to find
People needed the platform. Low engagement, even with a mandate, came from not being able to find things.
What would have changed my mindIf people who could find content still weren’t using the platform, the problem would have been value, not navigation.
People described finding workarounds and using outdated methods instead of the platform.
The experience problems sat downstream of the infrastructure, like slow performance and broken pipelines, so interface fixes alone wouldn’t move satisfaction.
What would have changed my mindIf satisfaction had stayed low even for people with fast, stable sessions, I’d have put the interface first.
Infrastructure strain was one of six root causes, and fixing it came first in the recommendations.
One interface couldn’t serve the range of data literacy across the four main roles.
What would have changed my mindIf the segmentation hadn’t produced distinct groups, one redesigned interface would have been enough.
Usage split by role, which led to the role-based entry points in the prototype.
A platform that had outgrown its own design
Leadership needed to know who used the platform, how, and why it was getting harder to use as it grew.
Advana started as a tool to help the comptroller’s office manage financial data. By 2023 it pulled from more than 400 business systems and served more than 100,000 people across every branch, from analysts to senior leadership. CDAO leadership publicly called it a victim of its own success.
The platform’s own feedback showed the strain before we arrived. Of about 220 scored responses from August 2020 to April 2023, only 38% said they fully accomplished what they came to do. The feedback form capped answers at about 100 characters, so it could flag frustration but not explain it.
“It has taken me two months attempting to get access. Account status is still inactive.”
“I can’t find what I want and can’t figure out how to use what I’ve been given access to.”
“Navigating the site is painful. I would say more but you have restricted the input to 101 characters.”
An earlier contractor had already built a set of personas. Our job was to test them against real users and refine them. I presented findings and recommendations directly to the CDAO’s Chief Experience Architect and Chief Product Officer throughout the engagement.
A sample big enough to compare roles
The four main roles used Advana in different ways, so a small study would only have caught the loudest problems. I started with a segmentation survey of 1,382 people. Its results defined the archetypes, so every finding could be traced back to a specific kind of user. From there:
- System Usability Scale and CX surveys to measure the current experience
- Interviews with at least 8 people per group, recruited by both archetype and role
- Two rounds of usability testing, on the current site and on the redesign
- A card sort to build the new navigation, then a tree test to check it
The work produced 20 deliverables, including two product roadmaps.
OMB approval, the participant burden estimate, CAC cards, and FedRAMP checks on every tool, while the survey and documents came together. Nothing could go into the field before these cleared.
Set who we would interview and test with.
Ran side by side, with rolling debriefs as interviews came in.
Put infrastructure fixes ahead of experience changes and set which archetypes came first.
Where the platform was breaking down
No single root cause explained the drop in usability.
I observed people in the field routinely override the platform’s AI recommendations, from flagged ledger balances to suggested stock reorders. Many didn’t believe a system could know more than their own experience did. A sizable number said that once a recommendation had been wrong for them, they would not use it again. That trust gap was one of the barriers holding back adoption.
Impact
Advana reinforced my habit of looking for the pattern across problems before proposing a fix for any one of them. For the team and CDAO leadership, the research gave them the evidence they needed to pause and fix the architecture before adding more features.
Hindsight
Some findings challenged existing assumptions about the personas and the platform’s readiness to scale. Agreeing up front on who could act on which kinds of findings would have shortened the time between insight and action.Findings that sat in review for weeks could have reached product teams within a sprint.
A platform this size, with a contract decision coming, needed continuous listening: short pulse surveys and feedback built into the platform, so changes could be tracked over time.Leadership would have gone into the recompete with a trend line, not one snapshot.
The card sort turned out to be central to the new navigation. Scheduling it earlier would have given the navigation and content model a stronger base than interview synthesis alone.The second usability round could have tested role-based entry points instead of basic findability.
With two more weeks
Run a focused round with the people who have the least data experience, since the redesign was largely built around them.
Go back to watching how people used or overrode the platform’s AI recommendations, work that was cut short.