Best practices

Writing a user scenario

The user_scenario is a brief for the simulated user on how to behave. The simulator improvises within it, so describe the person and their goal rather than the exact words:

The caller has an appointment tomorrow but wants a weekend slot. They are in a hurry, switch to Hindi midway, and ask twice whether a cancellation fee applies.

Keep one intent per scenario. A scenario that asks the user to reschedule, then cancel, then complain produces one long conversation where a Fail tells you little — split it into three test cases.

Writing expected behaviors

An expected behavior is a check the judge grades against the finished transcript. The judge can only check what it can see, so write each behavior as one observable, yes-or-no question:

Use the name as a short label you’ll recognise in results, and the description to tell the judge exactly what counts as a pass.

Behavior or guardrail?

An expected behavior is specific to one test case — it asks “did the agent do its job in this scenario?”. A global guardrail is graded against every test case in the suite — it asks “did the agent stay inside policy, whatever the scenario?”.

For example, “Offers at least two alternative slots” belongs in the rescheduling test case’s expected behaviors, while “Never gives medical advice” belongs in the suite’s guardrails, because it must hold in every conversation.

A simple test: if you find yourself copying the same behavior into every test case, it is a guardrail.

When to use variables and overrides

Use suite variables for values that appear across many test cases — a customer name, an amount due, a preferred language. They let you change a value once instead of editing every scenario. Reference a variable inside a user scenario with double braces, and the value is filled in when the test runs:

The caller owes {{due_amount}} and asks for two more weeks to pay.

Use a test case’s variable overrides when one case needs a different value to trigger the flow it tests. For example, keep due_amount: 4500 at the suite level, and override it to 120000 in the one case that checks whether large amounts are escalated.

Why map suite variables to agent variables

A suite’s variables do not automatically reach the agent. The agent has its own variable names (set under Variables & personalization), and the two are connected by a variable mapping created per app_id and app_version:

{ "app_id": "clinic-front-desk", "app_version": 4,
  "variable_mapping": { "customer_name": "patient_name" } }

Keys are the agent’s variable names; values are the suite’s. Create a mapping whenever the agent’s instruction uses variables and you want your test values to flow into them. Without one, the run still executes, but the agent falls back to its default variable values — your carefully parameterised cases all end up testing the same thing. Recreate the mapping when you commit a new app version.

Suite run or single test-case run?

Run a single test case while you are writing or fixing that case. Run a test suite as your regression check — after committing a new agent version, or before a release. Note that a suite run consumes more credits: every test case executes, multiplied by frequency, so a 50-case suite at frequency: 2 is 100 simulated conversations.

Polling a run

Starting a run returns 202 with a run_id — execution happens in the background, and Get a run is how you follow it:

• Poll every 30–60 seconds until status is completed or failed. A run simulates whole conversations, so it takes minutes; polling faster just spends requests on identical answers.
• completed_executions against total_executions is your progress bar.
• A 402 when starting the run means the workspace is out of credits.

Reading the results

Start from the run summary and drill down only where something failed:

1. Per-case pass counts first. In Get a run, each test case reports passed out of total_executions. An execution counts as passed only when every expected behavior passed. 0/3 is broken; 2/3 is flaky.
2. Then open the failing case. Get test case results returns one entry per execution, each with the full transcript and the judge’s verdict for every behavior.
3. Read the judge’s reasoning. Every Pass or Fail comes with a short explanation of what the agent said. There is also one extra entry, Guardrails, which grades the suite’s guardrails for that conversation.

Next