evals-before-features
Eval-first development for a support agent: cases written before the prompt, a Strands harness agent, Strands Evals scoring with deterministic checks and LLM judges, repeated trials, and a per-evaluator release gate.
cases.py is the spec. Seven Strands Evals Cases describe what a support chatbot must do: answer policy questions from a knowledge base and cite the article, report on a case, and resolve a case only when the customer confirms the fix. Each case also names the tools the agent must and must not call, and three cases cover what it must never do.
The agent is built with create_harness() from the Strands harness, with the built-in shell, file and web tools, memory and sessions switched off. It gets three tools on a SupportDesk: search_kb over a mocked knowledge base, get_case and resolve_case. The desk enforces case ownership and idempotent resolution in code. make_task() runs each case on a fresh desk and agent, captures the OpenTelemetry spans with StrandsEvalsTelemetry and maps them to a Session.
Two deterministic evaluators, ToolPolicy and CitesSource, score every run without a judge. Before a release, four Strands Evals judges add correctness, relevance, helpfulness and faithfulness ratings from a pinned OpenAI judge model. gate() compares each evaluator's pass rate with its threshold in THRESHOLDS and exits 1 below it. A live run repeats every case three times and flags cases that pass only some trials.
The 23 tests run offline with a scripted FakeModel. They include an "upgraded" model script that answers from memory and closes an unconfirmed case, and the gate rejects it.
git clone https://github.com/tensorupai/tensorup-examplescd tensorup-examples/evals-before-features && uv sync && uv run pytestTo run the full suite live, put OPENAI_API_KEY or ANTHROPIC_API_KEY in .env, then run TENSORUP_LIVE=1 uv run python -m supportevals eval. It exits 1 when the gate fails, so it works as a CI step. TENSORUP_MODEL and TENSORUP_JUDGE_MODEL override the agent and judge models.
To use it for your own agent, replace cases.py with your business behaviours, including what the agent must not do. Then swap build_agent() and the SupportDesk tools for your own, and set THRESHOLDS per evaluator.