Skip to main content
Experiments let you A/B test more than one version of a message at a single send in a journey. A send node runs an experiment that holds 2 to 10 variants, each backed by its own journey template. Courier assigns each recipient to a variant deterministically, splits traffic across the variants by weight, tracks delivery and engagement per variant, and lets you promote a variant once you’ve seen enough data. Use an experiment to compare subject lines, body copy, layouts, or timing without branching the journey or maintaining parallel workflows. Every recipient passes through one send node; Courier decides which variant they receive.

​
How experiments work

  • Variants are templates. Each variant is linked to a journey template you edit inline, the same way you edit any send node’s template. A send node runs either a single template or an experiment, not both.
  • Traffic splits by weight. Each variant has a relative weight. A variant with weight 3 receives three times the traffic of a variant with weight 1.
  • Bucketing is deterministic. Courier hashes a bucketing key to decide which variant a recipient gets, so the same recipient always lands in the same variant. See Bucketing key and stickiness.
  • Assignments are sticky. Once a recipient is assigned to a variant, they keep it across sends, even if you change the weights later.
  • Metrics are per variant. Courier tracks sent, delivered, opened, clicked, and error or undeliverable counts for each variant so you can compare them.
  • You promote manually. When you’ve seen enough, you promote one variant. Its template becomes the only template on the node and the experiment ends.

​
Create an experiment

1

Add an experiment

Create an experiment at the send step you want to test. Courier starts you with two variants, each backed by its own template.
2

Name the experiment

Give the experiment a name so you can find it later in the results and metrics views.
3

Edit each variant's template

Open each variant and edit its template. Name the variants something you’ll recognize later, for example Short subject and Long subject. Edits to a template show up on its variant right away.
4

Set traffic allocation

Give each variant a weight to control how much traffic it receives. Weights are relative, so 3, 1, 1 sends 60% of traffic to the first variant and 20% to each of the others. A variant with a weight of 0 receives no traffic.
5

Publish

Publish the journey to make the experiment live and start splitting traffic.

​
Variants and traffic allocation

An experiment starts with two variants, and you can add or remove more, up to a maximum of 10. Each variant is linked to a full journey template, so anything you can build in a template (subject, body, channel-specific content) can differ between variants. Each variant carries a relative weight that sets its share of traffic. Weights don’t need to add up to any particular total: 3, 1, 1 splits traffic 60/20/20, and so does 30, 10, 10. Weights must be zero or greater, and at least one variant must have a weight above zero. A variant weighted 0 receives no traffic, which is a way to pause a variant without removing it.

​
Keep only this variant

Before an experiment is published and taking live traffic, each variant shows a Keep only this variant control. Use it to collapse the experiment down to a single variant, for example if you change your mind before launching. Once the experiment is live, this control is hidden. To reduce a live experiment to one variant, promote a variant instead.

​
Bucketing key and stickiness

The bucketing key is a path to a value in the invocation, in the recipient’s profile or the data payload, that Courier uses to assign each recipient to a variant, for example user.id, user.email, or data.account_id. It isn’t a literal value: Courier reads the value at that path for each recipient and hashes it, together with the variant weights, to pick a variant deterministically. The same value always maps to the same variant. If the path can’t be resolved for a recipient, Courier falls back to the user ID, then the email. The key must be non-empty with no leading or trailing whitespace. Assignments are sticky: once a recipient is bucketed into a variant, they stay in that variant across sends, even if you change the weights afterward. This keeps each recipient’s experience consistent for the length of the experiment.
Editing a variant’s weight, template, or name keeps existing assignments in place. Two changes reset the experiment and re-bucket everyone:
  • Adding or removing a variant
  • Changing the bucketing key

​
Results and metrics

Courier tracks metrics for each variant: sent, delivered, opened, clicked, and error or undeliverable counts. The Results tab breaks these down per variant so you can see how each one is performing and which template each delivered message used. The Metrics view groups the experiment with its templates alongside the rest of the journey’s analytics. Use both to compare variants before deciding which one to promote.

​
Promote a variant

When you’ve seen enough data, promote the variant you want to keep. Promoting:
  • Replaces the experiment with the promoted variant’s template, so the node sends that single template
  • Removes the other variants from the node
  • Ends the experiment
Publish the journey to apply the change. Runs already in progress finish on the version they started with; only new entrants get the promoted template. In the metrics view, any variant that received traffic stays visible as inactive, so its historical data isn’t lost. Promotion is a manual decision. Courier doesn’t pick a variant for you or promote one automatically. You choose which variant to keep based on the results.

​
Common patterns

  • Subject line tests - Run two or more subject lines against the same body and promote the one with the stronger open rate.
  • Copy and layout tests - Compare body copy, calls to action, or channel layouts on a single send.
  • Gradual rollout - Weight a new template low to expose it to a small share of traffic, then raise its weight as you gain confidence. Existing recipients keep their assignment, so raising a weight shifts only new entrants.

​
Configure via the API

An experiment lives on a send node in place of message.template. The message object stays required but can be empty when an experiment is present; message.to, message.delay, and message.data still apply to whichever variant is chosen. 
{
  "id": "send-welcome",
  "type": "send",
  "message": {},
  "experiment": {
    "name": "Welcome subject-line test",
    "bucketingKey": "user.id",
    "variants": [
      { "id": "control", "name": "Control",      "weight": 6, "templateId": "ntf_control" },
      { "id": "short",   "name": "Short subject", "weight": 3, "templateId": "ntf_short" },
      { "id": "emoji",   "name": "Emoji subject", "weight": 1, "templateId": "ntf_emoji" }
    ]
  }
}
For the full field reference, see Build journeys via the API.

​
What’s next

Send node

Configure channels, recipients, conditions, and send windows.

Build journeys via API

Define journeys and experiments programmatically.

Metrics

Track journey and message performance.