When no one owns the definition of “good,” an engineer's gut feel becomes the quality bar. I built an evaluation framework that moved ownership of agent quality to UX. It turned subjective judgments about behavior into measurable signals the team could use to improve the product.
We were improving an AI agent without a shared definition of what “good” looked like.
Issue selection depended on manually flagged conversation threads and an engineer's judgment about what seemed wrong. The team spent roughly four hours a week reading threads by hand, then had little confidence that a change actually improved the experience.
The result was a slow feedback loop:
find a problem → investigate manually → make a change → hope it helped
I introduced an evaluation framework that made experience quality explicit, measurable, and scalable.
Instead of engineering deciding which conversations looked problematic, UX and Product defined the behaviors that mattered. We translated those judgments into structured evaluators, including LLM-as-a-judge systems that could score conversations at scale.
The goal wasn't simply to automate evaluation. It was to make UX's definition of quality executable.
The time from investigating an issue to shipping a code change dropped from four weeks to one. And the team moved from manually reading a small set of flagged threads to having behavioral signals across the experience.
Evals are structured measures of whether an agent is doing what it's supposed to do. It's common to use LLM judges to score outputs consistently and at scale.
But every eval contains a judgment.
What counts as accurate?
What counts as helpful?
When has an agent actually moved someone forward?
What does an aligned response sound like?
Those aren't engineering questions alone. They're experience decisions.
When engineering owns evaluation without UX involvement, UX inherits a quality bar it never defined. When the evaluation is based on whatever happened to get flagged, the team optimizes for isolated problems instead of the experience as a whole.
So I reframed the role of UX:
If you don't own the evaluation, you don't own the experience.
The framework became a way for UX to define the behavioral contract between the user and the agent — and for the rest of the team to measure against it.
One of the most important distinctions in the framework was that accuracy alone isn't enough.
Let's look at an example of an AI agent that is built to handle customer support questions.
Customer: Can I set up different levels of access for my kid and for my partner?
AI Agent: I don't have the answer to that question. I'm escalating this to another representative. You can also submit your question by contacting us at [email] or calling [phone number].
While the agent was accurate – it's supposed to hand off the question to a human if it doesn't know the answer – it wasn't aligned with customer expectations. The customer expected that their question would be submitted by writing it in the chat.
These accurate but not aligned interactions eroded trust that the agent actually understood the job to be done. That distinction became something we could define, teach, and measure.
The framework had to work for whoever picked it up — including designers with no data access at all. So I wrote the rollout as a matrix: the same three measures, executed with whatever tools you actually have.
Momentum = helpfulness, clarity, and forward progress. The agent should either complete the task or package the next step so any human can continue without friction. Accuracy is a prerequisite; this layer evaluates motion.
You don't need access to start. You need agreement on what good looks like.
The hardest part was getting a team to say out loud, in examples, what an aligned response is. That artifact is what made the evaluators possible, and it's the part designers can start building today.
This became a UXDX talk →