Skip to main content
An experiment A/B tests more than one version of a message at a single send. A send node runs one experiment holding 2 to 10 variants, each backed by its own journey template. Courier assigns each recipient to a variant deterministically and splits traffic by weight. It tracks delivery and engagement per variant, so you can promote one once you have seen enough data. Use an experiment to compare subject lines, body copy, layouts, or timing without branching the journey or maintaining parallel workflows. Every recipient passes through one send node. Courier decides which variant they receive.

​
How experiments work

  • Variants are templates. Each variant links to a journey template you edit inline, the same way you edit any send node’s template. A send node runs either a single template or an experiment, not both.
  • Traffic splits by weight. Each variant has a relative weight. A variant with weight 3 receives three times the traffic of a variant with weight 1.
  • Bucketing is deterministic. Courier hashes a bucketing key to decide which variant a recipient gets, so the same recipient always lands in the same variant. See Bucketing key and stickiness.
  • Assignments are sticky. Once a recipient is assigned to a variant, they keep it across sends, even if you change the weights later.
  • Metrics are per variant. Courier tracks sent, delivered, opened, clicked, and error or undeliverable counts for each variant so you can compare them.
  • You promote manually. When you’ve seen enough, you promote one variant. Its template becomes the only template on the node and the experiment ends.

​
Create an experiment

1

Add an experiment

Create an experiment at the send step you want to test. Courier starts you with two variants, each backed by its own template.
2

Name the experiment

Give the experiment a name so you can find it later in the results and metrics views.
3

Edit each variant's template

Open each variant and edit its template. Name the variants something you’ll recognize later, for example Short subject and Long subject. Edits to a template show up on its variant right away.
4

Set traffic allocation

Give each variant a weight to control how much traffic it receives. Weights are relative, so 3, 1, 1 sends 60% of traffic to the first variant and 20% to each of the others. A variant with a weight of 0 receives no traffic.
5

Publish

Publish the journey to make the experiment live and start splitting traffic.

​
Variants and traffic allocation

An experiment starts with two variants. Add or remove variants up to a maximum of 10. Each variant links to a full journey template, so anything you can build in a template (subject, body, channel-specific content) can differ between variants. Each variant carries a relative weight that sets its share of traffic. Weights need no particular total: 3, 1, 1 splits traffic 60/20/20, and so does 30, 10, 10. Weights must be zero or greater, and at least one variant must have a weight above zero. A variant weighted 0 receives no traffic, which pauses it without removing it.

​
Keep only this variant

Before an experiment is published and taking live traffic, each variant shows a Keep only this variant control. It collapses the experiment down to that single variant, for example if you change your mind before launching. Once the experiment is live, this control is hidden. To reduce a live experiment to one variant, promote a variant instead.

​
Bucketing key and stickiness

The bucketing key is a path to a value in the invocation, in the recipient’s profile or the data payload, for example user.id, user.email, or data.account_id. It isn’t a literal value. Courier reads the value at that path for each recipient and hashes it, together with the variant weights, to pick a variant deterministically. The same value always maps to the same variant. If the path can’t be resolved for a recipient, Courier falls back to the user ID, then the email. The key must be non-empty with no leading or trailing whitespace. Assignments are sticky: once a recipient is bucketed into a variant, they stay in that variant across sends, even if you change the weights afterward. This keeps each recipient’s experience consistent for the length of the experiment.
Editing a variant’s weight, template, or name keeps existing assignments in place. Two changes reset the experiment and re-bucket everyone:
  • Adding or removing a variant
  • Changing the bucketing key

​
Results and metrics

Courier tracks metrics for each variant: sent, delivered, opened, clicked, and error or undeliverable counts. The Results tab breaks these down per variant, including which template each delivered message used. The Metrics view groups the experiment with its templates alongside the rest of the journey’s analytics.

​
Promote a variant

When you’ve seen enough data, promote the variant you want to keep. Promoting:
  • Replaces the experiment with the promoted variant’s template, so the node sends that single template
  • Removes the other variants from the node
  • Ends the experiment
Publish the journey to apply the change. Runs already in progress finish on the version they started with. Only new entrants get the promoted template. In the metrics view, any variant that received traffic stays visible as inactive, so its historical data isn’t lost. Promotion is manual. Courier never picks or promotes a variant for you.

​
Common patterns

  • Subject line tests - Run two or more subject lines against the same body and promote the one with the stronger open rate.
  • Copy and layout tests - Compare body copy, calls to action, or channel layouts on a single send.
  • Gradual rollout - Weight a new template low to expose it to a small share of traffic, then raise its weight as you gain confidence. Existing recipients keep their assignment, so raising a weight shifts only new entrants.

​
Configure via the API

An experiment lives on a send node in place of message.template. The message object stays required but can be empty when an experiment is present. message.to, message.delay, and message.data still apply to whichever variant is chosen.
For the full field reference, see .