A/B tests
Split the people who reach a paywall location between paywalls and find out which converts better.
An A/B test shows different paywalls at the same paywall location and measures which one makes more people convert. Each variant is a paywall and a share of traffic. Voidhash picks each person's variant when their app opens the location, so the app needs no changes.
A/B tests are in early access. Ask us on Discord, or email support@voidhash.com on a paid plan, to turn them on for your team.
Set up a test
Create the test
Open A/B testing → A/B tests, click New A/B test, enter a Name of 3–64 characters and click Create A/B test. The test's page opens. A new test is a Draft with Variant A and Variant B at 50% each.
Describe it
On the Overview tab, add a description and a Hypothesis: what you expect to happen, and why.
Add locations and paywalls
The variants grid has a row per variant and a column per location. Click Select location for the first column, and Add location to run the same test at more places. In each cell, click Select paywall and pick the paywall that variant shows there. Every paywall needs a live build; see Build and deploy.
Set the split
Each variant gets a paywall and a share of traffic
Give each variant a weight; weights must add up to 100%. While they don't, Split evenly divides traffic equally. Add variant adds more than two. The first row is the control, the variant the others are compared with.
Choose the goal
In the properties, set the Goal event: what counts as a conversion. Type the event's code exactly as shown:
- For a paywall that sells subscriptions, type
$subscription.created. It counts Subscription started: every new subscription, free trials included. - For one that sells one-time products, or both kinds, type
$purchase.completed. It counts Purchase completed: every new purchase, but not renewals.
Always set the goal before you start. An empty goal counts nothing, and every variant shows 0 conversions.
An event your app records works too; ask your developers for its exact name. Secondary events, comma separated, are recorded for reference. Both lock once the test starts.
Save and start
Click Save changes, then Start. The test turns Running and replaces whatever the locations showed.
A test can't start while a variant's paywall has no live build, or while one of its locations runs another test. Each location runs one test at a time.
Statuses
| Status | What happens at the locations |
|---|---|
| Draft | Nothing changes yet. Everything can be edited. |
| Running | Each person sees their variant's paywall. Variants and locations are locked. |
| Paused | Everyone sees the control's paywall, and nobody is counted. Variants can be edited again. |
| Concluded | The test has ended and each location shows the winner. Final. |
| Archived | Listed under Archived and All. A running or paused test's locations show no paywall. |
Use Start, Pause, Resume, Conclude, Archive and Restore on the test's page. Conclude is available once the test has started. Conclude a test before you archive it, so its locations keep showing a paywall. Restore brings an archived test back. A test that was running then shows everyone the control's paywall and counts nobody until you Pause and Resume it.
The name, description and hypothesis stay editable until the test is concluded or archived. The change history at the bottom of the Overview tab lists every change, with the user ID of whoever made it, or System, and when; Changes shows the details. Members see "Could not load activity." instead.
How people are assigned
Each person is assigned at random the first time they reach the test, and keeps that variant: on every visit, after they sign in, and once signed in on every device.
- A variant always shows its paywall's current live build. Deploying a new build of that paywall changes the variant while the test runs.
- Changing weights while paused only affects people assigned after you resume. Everyone already assigned keeps their variant.
- Removing a variant gives its people a variant again, by the split as it then is.
- A person whose app is too old to show a variant's paywall sees nothing there and isn't counted.
- People who declined analytics still see a paywall but aren't counted.
Put a tester in a variant
To see one variant yourself, or to give it to a teammate, fix a person to it:
- Open the test's Setup tab and find Testers.
- Click Add tester, choose how the person is named (their app user ID, email or person ID) and enter it, pick the variant, and click Add tester.
A tester always sees that variant, in production and in builds that use a test key, whatever the split says. Testers aren't counted in the results. The variants you can pick are the ones last saved, so save a new variant before fixing someone to it. Remove a tester with the bin beside them.
Read the results
The Results tab has a row per variant:
Read the Reading column to see which variant wins
| Column | Meaning |
|---|---|
| People | People who saw the variant, counted in the variant they saw first. |
| Converted | People who reached the goal any time after first seeing it. |
| Rate | Converted divided by People. |
| Revenue | Revenue from those people after they first saw it, in US dollars. |
| Reading | How the variant compares with the control. |
Reading is one of:
- Better than the control or Worse than the control: the difference is unlikely to be chance (95% confidence). With several variants, each needs a bigger difference, so the chance of any false winner stays at 5%.
- No clear difference: there's enough data, but the difference could still be chance.
- Too early to tell: fewer than 100 people in the variant or the control, or fewer than 10 conversions in the two together.
- Nobody yet: nobody has seen this variant, or the control, yet.
Results cover the time since the test last started or resumed, and only production traffic: people who saw a paywall through a test key, and their simulated purchases, don't appear here. Secondary events aren't analyzed. On Voidhash Cloud, the A/B tests list also shows Views, Unique viewers, Purchases and Conversion for each test.
Compare the variants on other numbers
The goal is one number. To see what the test did to anything else, click Compare in Explore under the results. Explore opens split by the test's variants, each named with its test and its variant. Pick any metric, such as MRR or an event of your own, and read it per variant.
The variants are also listed with your cohorts wherever a chart takes one, so you can choose them yourself: see Split by cohort.
- A variant's people are everyone the test gave it, whether or not they opened the paywall, so these counts can be higher than People in the results.
- Each person counts with everything they did in the period you look at. Set the period to start when the test did.
- Testers you fixed to a variant aren't included.
Conclude a test
Click Conclude, pick the winner under Winning variant (optional) or leave it at No winner, and click Conclude in the dialog. The winner's paywall is placed at each location; with no winner, the control's. A concluded test can't restart; create a new one.
If something's off
- Every variant shows 0 conversions: the Goal event is empty or misspelled. It locks once the test runs, so conclude and start a new test.
- Start does nothing: a variant's paywall has no live build, a location runs another test, or you're a Member. Studio shows no message.
- People stays at 0: nobody has opened the locations since the test started, or the Experiment exposure event is off in Settings → Events. It's off by default in self-hosted Studio.
- "Failed to create A/B test" or "Failed to save changes": Members can open tests but not change them. See Members and roles.
- Stuck on Too early to tell: the locations get too little traffic. Run the test longer, or add locations.
For developers: tests can also be managed from the CLI or the API (developer docs).
Next steps
Still stuck? Check Troubleshooting, ask us on Discord, or email support@voidhash.com on a paid plan.
Something wrong or missing on this page? Tell us by email or on Discord.