A/B Testing
This notebook visualizes and measures the results of an A/B test.
Measure weekly conversions from visiting the landing page to signing up.
Measure weekly retention of users who signed up.
Conversion Rate
We need to check whether one variant resulted in a higher conversion rate than another. Let's start with a time series.
Here is the overall conversion rate for each variant:
variantobject
conversion_ratefloat64
0
variant_a
0.008130040077
1
variant_b
0.01352708068
Significance Test
It looks like Variant B has a higher conversion rate than Variant A. We need to make sure these results are significant.
p-value is 4.536508742253485e-12
The difference in Conversion Rate between Variants A and B is statistically significant.
Retention Rate
We also want to know whether one variant of the website results in higher retention than another variant. For this, let's look at the weekly sessions of our users.
variantobject
Retentionfloat64
0
variant_a
0.1760209235
1
variant_b
0.3770292854
Significance Test
Once again, check that the difference in retention is significance.
p-value is 2.5434010296902054e-12
The difference in Retention between Variants A and B is statistically significant.