Primary KPI: Clips per User
12.8 No Diff
vs. Control / No Send
Secondary KPI: Subscriptions
Flat 0% var
Across all variants
Guardrails
Healthy
No flags on unsubscribes or churn

Hypothesis Not Supported

Tone makes no difference on clip creation. All 8 experimental tones and the control group perform identically.

Performance by Variant

When all bars look the same height, that IS the finding.

14 8 4 0
Control
No Send
Fully Pro
Semi-Formal
Social
Neutral
Informal
Very Inf.

Performance by Persona Segment

Variant performance across the 4 key user segments.

Never Created (21%)

No variance

Simple Mode Only (18%)

No variance

Created, Never Listened (45%)

No variance

Power Users (16%)

No variance

Additional Learnings

  • Tone isn't the lever. The words land the same way because the core message ("go create") is identical.
  • Complexity tax. Running 8 versions that produce an identical result adds reporting overhead without benefit.
  • Opportunity cost. Simplifying frees up engineering and creative bandwidth for experiments that actually move the needle.

Action Points

  • Pick single variant as the sole Welcome Email.
  • Stop splitting traffic across identical emails.
  • Transition to Behavior-based messaging.

Next Tests

Proposed Test 1

Behavior-Based Welcome Series

Hypothesis: Varying the message based on actual behavior (e.g., Never created vs. Power users) will drive higher engagement than unified messaging.

Rationale: Each segment has a specific gap. Matching the email to their friction point addresses their needs directly.

View Test Proposal