Notion A/B testing board with statistical significance

Installation Instructions

1. Click the access template button on the PDF.
2. Navigate to your Notion account.
3. On the top right of the template, press "Start with this template".
4. Follow the instructions provided by Notion.
5. Read the instructions provided on the template.
6. Begin testing! Enjoy.

What to know about A/B testing before proceeding🤓

You should run an A/B test when there’s a clear hypothesis, a measurable and relevant metric, and potential downside that justifies validating your assumptions before rolling out the change. If the change is low risk, the metrics won’t meaningfully affect your decisions, or you’re already certain about the outcome, you can skip the test and proceed.

❓ Why Use This Board?

💡 Benefits of Logging All Experiments Here

• 📚 Single Source of Truth
• Keep all experiments, hypotheses, outcomes, and insights in one central place — no more scattered docs or Slack threads.
• 🔄 Repeatable Process
• The structured workflow ensures every test follows a consistent, thoughtful process — from idea to conclusion.
• 🔍 Clarity on What’s Working
• Track what experiments drove impact, which channels performed best, and where to double down.
• 📈 Data-Driven Decisions
• Easily reference past results to support or challenge new ideas. Reduce gut decisions, increase ROI.
• 🧠 Build Institutional Knowledge
• Over time, your board becomes a powerful learning library — accelerating growth and avoiding repeated mistakes.
• 🔗 Connect Experiments & Learnings
• Link related tests and see how one insight led to the next. Great for storytelling and strategy recaps.
• 🏆 Prioritize with Confidence
• The built-in scoring system helps you focus on the highest-leverage experiments first.

🧪 How to Use the A/B Test Experiment Template

📁 Setup

• 🔄 Duplicate the example from the "No Status" board to create a new experiment.
• ➕ Click "New Hypothesis" to start.
• ✍️ Fill out all qualitative details (context, hypothesis, background info).

⚙️ Configure the Experiment

• 🎯 Set the Status
• (moves through: Brainstorm → Queue → Scheduled → In Progress → Analyze → Conclusion)
• 👥 Assign:
• Team
• Success metric (North Star)
• Goal of the experiment
• Channel
• 📊 Add estimates for:
• Expected impact
• Confidence level
• Ease of implementation
• 🧮 The system will auto-calculate a priority score
• → Use this to organize into Brainstorm or Queue

🚀 Launch

• 📅 Move the card to "Scheduled to Launch"
• ✅ When starting the test, set the Launch Date and move to "In Progress"

📈 Analyze Results

• ⏳ Once the test has run long enough for valid data, move it to "Analyze"
• Log:
• ✅ Yes/No outcome
• 👥 Audience + conversions (Control & Variant)
• 📐 Hypothesis outcome: Under / Accurate / Over
• 🧠 The template will automatically calculate:
• Statistical significance (65%–95% confidence)
• p-value, t-score, and z-score

📝 Wrap-Up & Learnings

• 🗒 Add final notes and learnings (next steps, key insights).
• 🔗 If the experiment leads to follow-ups, link related experiments.
• 🏁 Set the final status:
• ✅ Winner
• ❌ Loser
• 📁 Archived (often used for flat or inconclusive results)

💡 Tip: Avoid inconclusive results by writing clear, actionable hypotheses that truly justify testing.