Run experiments across agent versions

An experiment splits one campaign’s calls across two to five committed versions of the same agent. You choose each version’s share of traffic, so you can try a change on real calls before rolling it out to everyone.

When to use an experiment

Use an experiment when you want to compare versions under the same campaign settings — same audience, schedule, dial rate, and phone numbers. Common cases:

a different greeting or opening line
a prompt or tone change
a new tool or conversation flow
a language or persona change
a safe rollout, where a new version gets 10% of calls and the current one keeps 90%

How it works

Each version in an experiment is a variant, and each variant has a traffic share. When the campaign dials, it picks a variant for that attempt from the split. A retry to the same contact can draw a different variant.

The cohort must include the variables that every variant needs. If any version in the experiment requires a column, that column is required on the CSV.

An experiment is fixed once you create it. To change the split or the versions, create a new experiment.

Experiment vs. a single-version campaign. A normal campaign pins one app_id and app_version. An experiment campaign passes experiment_id instead, and omits app_id and app_version. Dial rate, phone numbers, retries, schedule, and the webhook still live on the campaign.

Before you start

You need two or more committed versions of the same agent. Draft versions are rejected.

You can also create experiments from the dashboard.

Set up an experiment

Commit the versions

Publish each arm as a committed version of the same agent. Draft versions are rejected.

Create the experiment

POST /v1/orgs/{org_id}/workspaces/{workspace_id}/experiments with two to five variants. See Create an experiment.

{
  "name": "greeting-test",
  "description": "Formal vs casual opener",
  "app_type": "agent",
  "variants": [
    {
      "name": "control",
      "traffic_percentage": 50,
      "app_config": { "app_id": "payment-reminder-agent", "app_version": 3 }
    },
    {
      "name": "casual",
      "traffic_percentage": 50,
      "app_config": { "app_id": "payment-reminder-agent", "app_version": 4 }
    }
  ]
}

The response includes experiment_id and a variant_id for each arm.

Create the experiment campaign

Create the campaign with app_config.experiment_id and omit app_id and app_version. Dial rate, phone numbers, retries, schedule, and the webhook stay on the campaign. See Create Experiment Campaign.

Rules

Two to five variants. traffic_percentage is an integer from 1 to 100, and the variants sum to 100.
One agent. Every variant uses the same app_id and a different committed app_version. Variant names are unique.
No extra fields. A variant only names the agent version. Attempts per second, numbers, and retries belong on the campaign.
Fixed campaign binding. After create, the campaign stays on that experiment. A later update can change the dial plan when you resend the same experiment_id. It cannot switch the campaign to a single version, or to a different experiment.
Assignment at dial time. Each attempt draws a variant from the traffic split. A retry of the same contact can draw a different variant.

Tips

Include the current version as a control.
Change one thing per version so you know what moved the result.
Use enough contacts for each variant to see a clear signal.
Start with 50/50 for a head-to-head test, or 90/10 for a cautious rollout.

Cohorts

Upload the cohort the same way you do for a single-version campaign. The file is checked against every variant, so a column required by any version in the experiment is required on the CSV.

Webhooks

After each attempt, the webhook payload includes an experiment object that identifies which variant handled the attempt:

{
  "experiment": {
    "experiment_id": "greeting-te-9f1c2a7b-4d3e",
    "variant_id": "casual-7a8b9c0d-1e2f"
  }
}

app_id and app_version on that payload are the variant that was dialed. experiment is null when the campaign is pinned to one version.

List and fetch

List experiments returns a page of experiments in the workspace. Get experiment returns one experiment, including variant ids.