Evals Before Features: How to Test a Strands Agent You Cannot Unit-Test

We upgraded the model behind our RAG support chatbot from GPT-4.1 to GPT-5. The change looked harmless: the same prompts and tools over the same retrieval. We had unit tests for the Python implementation logic and manual checks of the answers, and both passed before the release.

But we were wrong: we found out from users and they were not happy. The answers had become less correct, less relevant and less helpful, because the way the new model reasoned had changed under the hood. It cost us reputation and another release cycle to get the old behaviour back. Not to mention the knock-on effect on the planned upcoming releases.

Nothing in our code had changed, so nothing in our tests failed. The thing that had changed was the model's choices, and we had nothing that measured or tested them.

Lesson and my position: write the evals before the agent logic and the prompts. Only once the business behaviour is written down as cases can you write a correct implementation of it in the agents. The same cases tell you within minutes, before the release and not from user complaints, whether a model upgrade broke it. I call it TDD for agents.

Why Unit Tests With a Fake Model Miss a Model Upgrade

The usual test suite for an agent fakes the model, because it is easier, cheaper and faster to run. A FakeModel replays a script of replies, so the test runs offline in milliseconds and proves the shell around the model: the tool wiring and the limits enforced in code. The sample for this post has 20 such tests, and they are worth having.

What they cannot prove is what the real model will do. The unit tests are written by the same engineer who wrote the prompt, so they encode the behaviour the engineer expects, not the behaviour they will get. When the model changes, the unit tests do not, and the suite stays green. Anthropic's engineering team describes the cost: without evals, a new model means weeks of manual testing; with them, a team can judge the model's strengths and upgrade in days [1]. OpenAI's guide names the alternative, "it seems like it's working", as an anti-pattern it calls vibe-based evals [2].

A model upgrade needs a test that runs the real model against the behaviour you require and scores the result. That is an eval.

What an Eval Case Is

An eval case is one business behaviour written as an input, the outcome a correct answer has to reach, and the actions the agent must and must not take on the way.

In my opinion, an eval case is a requirement you can run. If you cannot write the case, you do not yet know the behaviour well enough to write the prompt.

Anthropic recommends exactly this order: build evals that define a capability before the agent can do it, then iterate until it passes [1]. Those capability evals start at a low pass rate. Once they pass, they move into the regression suite, which should stay near 100% [1]. A support bot's regression suite is its contract with its users.

In the sample, the cases live in cases.py, written before the agent's instructions. Each one is a Strands Evals Case, with the tool expectations in metadata:

# src/supportevals/cases.py    Case(        name="refund-out-of-window",        input="My order was delivered 45 days ago. Can I get my money back?",        expected_output="Not as a refund: full refunds end 30 days after delivery. "        "Between 30 and 90 days you can return it for store credit. [KB-101]",        metadata={"must_call": ["search_kb"], "must_not_call": ["resolve_case"], "cite": "KB-101"},    ),

must_call says the answer has to come from the knowledge base and not from the model's memory. cite says which article proves it. The seven cases cover the support bot's three jobs: answering policy questions, reporting on a case, and resolving a case. Three of them are about what the agent must not do: invent a policy, close a case the customer has not confirmed, or touch another customer's case. INSTRUCTIONS in agent.py came after the cases, with one rule per group of cases.

How the Suite Runs

The diagram reads top to bottom: the cases and a model go in, one agent run per case comes out as a trace, two kinds of evaluator score it, and a gate decides.

Close

CASES is the spec. Every other component either runs it or scores it.

model_for(case) picks the model for one run. On every commit it is a FakeModel scripted per case. Before a release or a model change, it is OpenAIResponsesModel on gpt-6-luna, which OpenAI positions as its most efficient model for high-volume tasks, at $0.10 per million input tokens and $0.50 per million output tokens [6]. The class matters: the Responses API is the endpoint where gpt-6-luna accepts function tools with reasoning on [6].

create_harness() builds the agent. The Strands harness returns a plain strands.Agent with a tuned system prompt, and every default it adds can be switched off [4]. A support bot needs none of the built-in shell, file or web tools, nor memory and sessions, so agent.py turns them off and passes only the three support tools. A test checks that the model is offered exactly search_kb, get_case and resolve_case.

The tools sit on a SupportDesk. It enforces in code what must never depend on the model: the customer id comes from the signed-in session, another customer's case looks the same as a missing one, and resolving a case twice writes once. The evals check what code cannot enforce, such as whether the customer actually said the issue was fixed.

StrandsEvalsTelemetry records the run. The trace-based evaluators read a Session built from OpenTelemetry spans, not the raw messages [3][5], so the task function captures spans in memory and maps them for each case:

# src/supportevals/evals.pydef make_task(model_for: Callable[[Case], Model]) -> Callable[[Case], dict[str, Any]]:    """One fresh desk and agent per case; the trace becomes the Session the evaluators read."""    telemetry = _get_telemetry()    def task(case: Case) -> dict[str, Any]:        telemetry.in_memory_exporter.clear()        desk = SupportDesk(customer_id=CUSTOMER_ID)        agent = build_agent(model_for(case), desk, {"session.id": case.session_id})        answer = str(agent(case.input))        telemetry.tracer_provider.force_flush()        spans = telemetry.in_memory_exporter.get_finished_spans()        session = StrandsInMemorySessionMapper().map_to_session(spans, session_id=case.session_id)        return {"output": answer, "trajectory": session}    return task

A fresh desk per case keeps one case's resolved ticket from leaking into the next.

The deterministic evaluators run on every commit and need no judge. The first, ToolPolicy, reads the tool calls from the Session and fails a case when a must_call tool was skipped or a must_not_call tool was called. The second, CitesSource, checks that the answer cites exactly the expected article, or nothing when no article applies, which catches an invented policy. Strands Evals ships simpler built-ins of the same kind, such as ToolCalled and Contains [3]. Each of those checks one fixed value, while the sample's two read their expectations from each case's metadata.

The judges run before a release. Each is a Strands Evals evaluator that asks a model to rate the answer: CorrectnessEvaluator on three levels, ResponseRelevanceEvaluator and FaithfulnessEvaluator on five, HelpfulnessEvaluator on seven [3]. They cover the three properties that drifted in my incident, plus faithfulness, because a RAG answer has to stay within what retrieval returned. Their pass rules differ: a correctness rating passes only at a full 1.0, the other three at 0.5 or above [3].

Without a model argument, every judge defaults to Claude Sonnet 4.6 on Amazon Bedrock [3], so judge_evaluators() passes one pinned judge. Strands Evals advises stronger models for complex evaluations [3], so the judge is gpt-6.1-terra, priced at $1 and $5 per million input and output tokens against gpt-6-luna's $0.10 and $0.50 [7].

EvaluationReport comes back as one flat report, one row per case and evaluator, each row naming its evaluator [3]. gate() groups the rows and compares each evaluator's pass rate with its threshold:

# src/supportevals/evals.py# Share of runs (cases x trials) each evaluator must pass. Deterministic checks are regression# evals: every trial of every case. Judges are non-deterministic, so they get headroom.THRESHOLDS = {    "ToolPolicy": 1.0,    "CitesSource": 1.0,    "CorrectnessEvaluator": 0.85,    "ResponseRelevanceEvaluator": 0.85,    "HelpfulnessEvaluator": 0.85,    "FaithfulnessEvaluator": 0.85,}

A live run repeats each of the seven cases LIVE_TRIALS = 3 times, so every evaluator scores 21 runs. At 1.0, a deterministic check has to pass all three trials of every case; at 0.85, a judge may fail three of its 21 runs, and a fourth failure blocks the release.

The offline suite runs with uv run pytest -q; TENSORUP_LIVE=1 uv run python -m supportevals eval runs both kinds against OpenAI and exits 1 when the gate fails, which is all a CI step needs.

Replaying the Upgrade in a Test

The incident from the hook can be replayed offline. The tests script two models against the same code, tools and instructions. The baseline meets the spec. The "upgraded" one drifts the way a model upgrade can:

# tests/scripts.py# Same code, same tools, same instructions; only the model's choices changed.UPGRADED = {    **BASELINE,    "refund-out-of-window": [        text("Yes, you can get a full refund within 60 days of delivery."),    ],    "no-resolve-unconfirmed": [        call("get_case", "t1", case_id="CS-1001"),        call("resolve_case", "t2", case_id="CS-1001", resolution="Reset performed."),        text("I have reset and closed case CS-1001."),    ],}

The confident, wrong refund answer is fluent and polite. A user who reads it has no reason to doubt it, which is how this kind of regression reaches production. The gate rejects the run:

ToolPolicy: 71% < 100%, failed ['refund-out-of-window', 'no-resolve-unconfirmed']CitesSource: 86% < 100%, failed ['refund-out-of-window']

The row reasons say why: missing=['search_kb'] on the refund case, forbidden=['resolve_case'] on the unconfirmed one, and cited=[] expected=KB-101. The deterministic checks caught both drifts without a judge and without a paid call, because the cases said in advance which tool every answer had to rest on.

Design Decisions and Trade-offs

Code checks versus model judges. Code-based graders are cheap and reproducible, but brittle and blind to nuance. Model-based graders capture nuance but cost more, vary between runs and need calibrating against human grading [1]. The sample uses both: code for every rule that can be stated as a tool call or a citation, judges for correctness and helpfulness, which cannot.

Trace-based versus output-based evaluators. An output-only evaluator sees the final text, so a correct answer from memory passes. The trace is what shows that search_kb never ran. The cost is the telemetry plumbing in make_task(), about ten lines.

A mocked knowledge base versus a simulated one. Strands Evals ships a ToolSimulator that has a model generate tool responses from a schema [3]. That makes cases cheaper to write, but makes the retrieval itself non-deterministic. A fixed KNOWLEDGE_BASE keeps each case's correct answer stable, so a failure points at the agent.

A library in the repo versus a hosted eval platform. OpenAI's hosted Evals become read-only on 2026-10-31 and shut down on 2026-11-30 [2]. Cases in cases.py and a gate in evals.py live in version control next to the prompt they test, and run wherever pytest runs.

How many cases. Anthropic suggests 20 to 50 simple tasks drawn from real failures as a starting set [1]. The sample has seven, one per behaviour. Our production suite now has close to 100 cases.

Where It Breaks

The judge drifts. Symptom: scores move although the agent did not change. Cause: the judge is a model too. Its ratings vary between runs [1], and replacing the judge model replaces the grader. Fix: pin the judge with TENSORUP_JUDGE_MODEL, re-run the last passing baseline whenever the judge changes, and check its ratings against a handful of human-labelled cases before trusting a threshold. OpenAI's guide says to scale an LLM judge only once it consistently agrees with human annotations [2].

The cases grade the path. Symptom: a good answer fails because the agent called get_case before resolve_case, or searched twice. Cause: a case that expects an exact tool sequence penalises valid alternatives. Anthropic recommends grading what the agent produced over the path it took [1]. Fix: assert sets, not sequences. ToolPolicy checks only that required tools ran and forbidden ones did not.

One trial hides a coin flip. Symptom: the gate passes on Monday and fails on Tuesday with no change. Cause: a single run per case. At a 75% per-trial success rate, the chance of passing three trials in a row is about 42% [1].

Fix: repeat each case and gate regression checks on every trial passing. In the sample, with_trials() copies each case three times with its own session, and a live run prints any case that passed some trials but not all as FLAKY, with its count, such as 2/3 trials passed. The price is three times the calls: 21 agent runs and 84 judge ratings for seven cases.

Checklist

• Write the cases before the prompt: one per business behaviour, including what the agent must not do.
• Give every side-effecting tool a case where it must not be called.
• Enforce hard limits such as ownership and idempotency in the tools, and use evals for the choices code cannot check.
• Run the deterministic evaluators with a fake model on every commit.
• Run the judges against the real model, several trials per case, on every model, prompt or retrieval change.
• Pin the judge model, and re-baseline when it changes.
• Gate on each evaluator's pass rate, not on an average across all of them.
• Turn every user complaint into a new case before you fix it.

Writing evals first is what lets us scale agents in production and keep the models up to date. After the GPT-5 upgrade, our release process gained a mandatory automated eval run on every pull request, and developers run the same suite on demand while they work. What is the first behaviour of your agent you would write down as an eval case?

References

1. Anthropic, "Demystifying evals for AI agents", Anthropic Engineering, 2026-01-09.
2. OpenAI, "Evaluation best practices", OpenAI API docs, updated on or after 2026-09-03 (inferred from its GPT-6 Astra references; accessed 2026-11-06).
3. Strands Agents, strands-agents/evals, GitHub, release v1.9.0, 2026-11-04.
4. Strands Agents, strands-agents/harness-sdk: harness-py, GitHub, SDK release python/v1.64.0 and strands-harness 0.3.4, 2026-11-04.
5. Strands Agents, "Built-in evaluators" sample, GitHub, updated 2026-07-15.
6. OpenAI, "GPT-6 Luna", OpenAI API docs, updated on or after 2026-05-18 (inferred from its knowledge cutoff; accessed 2026-11-06).
7. OpenAI, "Models", OpenAI API docs, updated on or after 2026-09-03 (inferred from its GPT-6 Astra listing; accessed 2026-11-06).