Case study 01 AI agent product · Enterprise SaaS

Shaping AI agent behavior

When no one owns the definition of “good,” an engineer's gut feel becomes the quality bar. I built an evaluation framework that moved ownership of agent quality to UX. It turned subjective judgments about behavior into measurable signals the team could use to improve the product.

Role
Director of UX
Partners
Product, Engineering, ML
Contribution
Evaluation framework, LLM-as-judge evaluators, rollout strategy by access level
Also became

At a glance

Problem

We were improving an AI agent without a shared definition of what “good” looked like.

Issue selection depended on manually flagged conversation threads and an engineer's judgment about what seemed wrong. The team spent roughly four hours a week reading threads by hand, then had little confidence that a change actually improved the experience.

The result was a slow feedback loop:

find a problem → investigate manually → make a change → hope it helped

Solution

I introduced an evaluation framework that made experience quality explicit, measurable, and scalable.

Instead of engineering deciding which conversations looked problematic, UX and Product defined the behaviors that mattered. We translated those judgments into structured evaluators, including LLM-as-a-judge systems that could score conversations at scale.

The goal wasn't simply to automate evaluation. It was to make UX's definition of quality executable.

Impact

The time from investigating an issue to shipping a code change dropped from four weeks to one. And the team moved from manually reading a small set of flagged threads to having behavioral signals across the experience.

4 wks → 1 wk

Someone else was defining what “good” looks like

Evals are structured measures of whether an agent is doing what it's supposed to do. It's common to use LLM judges to score outputs consistently and at scale.

But every eval contains a judgment.

What counts as accurate?

What counts as helpful?

When has an agent actually moved someone forward?

What does an aligned response sound like?

Those aren't engineering questions alone. They're experience decisions.

When engineering owns evaluation without UX involvement, UX inherits a quality bar it never defined. When the evaluation is based on whatever happened to get flagged, the team optimizes for isolated problems instead of the experience as a whole.

So I reframed the role of UX:

If you don't own the evaluation, you don't own the experience.

The framework became a way for UX to define the behavioral contract between the user and the agent — and for the rest of the team to measure against it.

The framework

01 · Trust

Can the user rely on the agent as a collaborator?

AccuracyDid the agent do the right thing?
Behavioral alignmentDid it behave the way the user expected?
02 · Momentum

Is the agent moving the user toward resolution?

Conversational agentsClarify intent, narrow options, guide toward a decision.
Task agentsExecute autonomously, reduce human effort, hand off cleanly with context.
03 · Sentiment

How does the user actually feel?

Baseline healthLow emotional volatility, a stable or improving trajectory, frustrations that get resolved.
As a signalSentiment rarely stands alone — it points at a break in trust or momentum.

Accurate ≠ aligned

One of the most important distinctions in the framework was that accuracy alone isn't enough.

Let's look at an example of an AI agent that is built to handle customer support questions.

Customer: Can I set up different levels of access for my kid and for my partner?

AI Agent: I don't have the answer to that question. I'm escalating this to another representative. You can also submit your question by contacting us at [email] or calling [phone number].

While the agent was accurate – it's supposed to hand off the question to a human if it doesn't know the answer – it wasn't aligned with customer expectations. The customer expected that their question would be submitted by writing it in the chat.

These accurate but not aligned interactions eroded trust that the agent actually understood the job to be done. That distinction became something we could define, teach, and measure.

Implementation

The framework had to work for whoever picked it up — including designers with no data access at all. So I wrote the rollout as a matrix: the same three measures, executed with whatever tools you actually have.

No accessFigma, Miro, pen & paper
Limited accessAnalytics, customer feedback
Full accessAgent observability, dev platform
Trust
Ground the team in the customer's mental model, then build examples that show the gap between accurate and aligned.
Use sentiment as a proxy: which tools get called when feedback turns negative, where repetition and drop-off cluster.
Train an evaluator on the customer's mental model to surface where the agent breaks it.
Momentum
Map the agentic workflow visually, storyboard what flow feels like, and mark the likely “brick wall” moments.
Find those walls in the data: time on task, drop-off points, time spikes.
Run a thread-level LLM-as-a-judge evaluator that scores forward progress across the whole conversation.
Sentiment
Agree as a team on what degradation sounds like before you try to measure it.
Track where sentiment degrades and test whether it correlates with a trust or momentum break.
Use it as the tripwire: sentiment flags the thread, trust and momentum evals explain it.
fig 1.1 — trust evaluator prompt
Momentum evaluator prompt

Momentum = helpfulness, clarity, and forward progress. The agent should either complete the task or package the next step so any human can continue without friction. Accuracy is a prerequisite; this layer evaluates motion.

True when
Stays oriented to the goal · gives an actionable next step or finishes the task · packages context for the next actor.
False when
Loops without resolving · loses context or re-asks · leaves the user with no path forward.
fig 1.2 — agentic workflow map

What changed

4 → 1
Weeks from investigating an issue to shipping the code change.
4 hrs/wk
Of manual thread reading replaced by automated scoring at scale.
UX
Now owns the definition of quality and the behavioral data behind it.
Before
After
Manually flagged threads
Evaluators scoring every thread
An engineer's gut feel
Experience expertise, written down
“Did that change help?” — unknown
Rich behavioral data per release
Quality as an engineering concern
Quality as a shared, UX-led practice
What I'd carry forward

You don't need access to start. You need agreement on what good looks like.

The hardest part was getting a team to say out loud, in examples, what an aligned response is. That artifact is what made the evaluators possible, and it's the part designers can start building today.

This became a UXDX talk →