A/B testing content variants

What you can test

A/B testing is available on:

• Article hero copy (H1 + first paragraph).
• CTA buttons on service / location pages.
• Email subject lines (when email is connected).
• LinkedIn post hooks (Domination only).

You cannot A/B test entire articles. The tooling exists, but the traffic-to-conclusion math doesn't work for most workspaces at article volume.

Starting a test

From any eligible piece of content, click Test variants. The engine:

1. Generates 2–4 variants of the testable element.
2. Lets you review and discard any you don't like.
3. Assigns traffic in equal proportions (or your chosen split).
4. Begins counting events.

Tests target a primary metric (clicks, time on page, conversion) and you choose it at setup. Pick one — multi-metric optimisation produces ambiguous winners.

Variant generation strategy

The engine produces variants that differ on a single axis when possible:

• Length — short vs. long.
• Frame — benefit-first vs. problem-first.
• Specificity — concrete vs. abstract.
• Tone — confident vs. curious.

Knowing the axis helps you reason about the winner. "Variant B won because it leads with the problem" is actionable; "variant B won, who knows why" isn't.

The confidence model

We use a Bayesian model. Two reasons:

1. Continuous monitoring. Bayesian results are valid at any time, unlike frequentist tests where peeking early invalidates the p-value.
2. Decision-friendly output. "Variant B has an 87% chance of being better, with an expected lift of 4.1%" is more actionable than "p=0.043."

A test is flagged conclusive when one variant exceeds 95% posterior probability of being best AND the expected lift exceeds the minimum meaningful effect you set in setup.

Sample-size guidance

Pre-test, the setup view tells you roughly how many events you'll need to reach 95% confidence at the lift you specified. If your traffic doesn't support that volume, the engine suggests:

• Increase the minimum meaningful effect (only care if it's a big difference).
• Reduce the number of variants (2-way tests need fewer events than 4-way).
• Use the test on a higher-traffic page.

Promoting a winner

When a test concludes:

1. The conclusive variant is highlighted.
2. Promote swaps it in as the canonical version. Losing variants are archived (recoverable for 90 days).
3. The test record stays in your history with the final probabilities and event counts.

You can also manually promote a variant that hasn't reached statistical confidence — the engine warns you, but the choice is yours.

Auto-promote

For LinkedIn hook tests (Domination only), you can enable auto-promote: when a variant reaches 95% confidence, the winner ships automatically. Useful for high-cadence accounts where the operator doesn't want to babysit individual tests.

Common pitfalls

• Test running too long. If a test is inconclusive after 4 weeks, the effect is probably real but too small to detect at your traffic. Conclude and move on.
• Multi-variable changes. Testing two changes at once (new headline AND new CTA) means a winner doesn't tell you which change mattered.
• Seasonality. Tests that span a holiday week or a market shock are suspect. Re-run.
• Selection bias. Generated variants are biased by your voice profile. If all four candidates "feel the same," widen the voice settings before re-generating.

Audit log

Every test creation, variant edit, promotion, and auto-promote logs to your audit log. Filter by abtest.* to see your testing history.