A/B testing
This notebook visualizes and measures the results of an A/B test.
Measure weekly conversions from visiting the landing page to signing up
Measure weekly retention of users who signed up.
Conversion rate
We need to check whether one variant resulted in a higher conversion rate than another. Let's start with a time series.
Here is the overall conversion rate for each variant:
Significance test
It looks like Variant B has a higher conversion rate than Variant A. We need to make sure these results are significant.
Retention rate
We also want to know whether one variant of the website results in higher retention than another variant. For this, let's look at the weekly sessions of our users.
Significance test
Once again, check that the difference in retention is significance.