Voodoo - Business Case - Pierre Bollack

Packages

Data importation & Exploratory Data Analysis

Description of the data:
'sessionid': Unique number for each sessions.
'platform': Android or IOS. It could be one way to split the analysis to check wether the revenue / session_number is higher on one platform.
'open_at': Timeseries variable at which the application was open.
'user_id: Unique user_id per user. There are 102203 unique user_id.
'country': Code of countries. It could help on visualisations.
'ab_cohort_id': There are 13 different cohorts. Moreover, the cohort 727 only have 1 user_id.
'segment_id': There are 3 different segments. Also the segment 722 have only one user_id.
'session_length': It is the length of the session. It could be interesting to check on which platform it is the higher.
'session_number': User lifetime session count.
'publisher_revenue': Total ad revenue generated during the session. This is one of the most important feature. It will be one of the variable tested for the AB test.
'fs_shown': Full-Screen Ad format. Displayed between games / levels.
'rv_shown': Rewarded Video Ad format.
'game_count': Maybe the number of time a user played the game. We can observe that more than 30000 users didn't played.
'offline_game_count': same as game_count but offline.
'acquired_at':
'segment_name': segmentation per platform. Only for new users.
'ab_name': segmentation per platforms for the ab test.
'ab_cohort_name': Ads frequencies. There are 6 different groups. The control group which remain unchanged. 4 groups (xHigh to xxLow) that have different frequencis. And gameTune...

Summary statistics

The average revenue generated is 0.038 per session.

We can observe that the median is 3. This means that 50% of the users played only 3 times. It is one specificity of the hyper-casual games. Also there could be an outlier as 99% of users played 27 times and the maximum is 137.

We can observe that it:

• Deviate from the normal distribution.
• Have appreciable positive skewness.
• Show peakedness.

We can conclude that the majority of users play between 1 and 7 times. There are also some more 'loyal' customers.

AB test context

A/B testing is a basic randomized control experiment. It is a way to compare different versions of a variable to find out which performs better in a controlled environment.
The goal of this AB test is to find the right ad frequency. There are various frequencies, from xLow to xxHigh. There is also the control group that allow to compare the previous product with our alternatives.

The observations have been splitting randomly and the number of observation had been equally balanced between the frequencies.

What is the success metric of this test?

To me, the success metric of this test is publisher_revenue. It is the main revenue source for Voodoo and is the main KPI we should follow in this test.

We can make several comments here:

• As expected, regarding the means, we can observe that the ad frequencies 'high' (xHigh + xxHigh) generate more revenue than the 'low' frequencies. It is also higher than the 'control' group.
• However, the maximum of revenue generated is made with a 'xxLow' frequency. It could mean that the user had a higher session_time and therefore generate more revenue at once.

Now we have to verify that this hypothesis is statistically verify.

A second metric that could be interesting is the session_number. This is an indicator of how users like the game and therefore of the success of the game.

Some comments:

• Paradoxically, it is not the group with 'xxLow' that has the most sessions.
• Regarding means the 'control' group appears to be the best one in terms of session_number.

Is there a clear winner to this AB test? Why?

We will focus on the 'publisher_revenue' variable.
Our assumption is that when it comes to 'high' ad frequencies, the publisher_revenue is higher than the 'low' groups and also than the 'control' group.

Therefore:

• HO: There is no clear difference in publisher_revenue between the means of 'control' and 'high' groups.
• H1= There is a clear difference in publisher_revenue between the means of 'control' and 'high' groups.

From these results, we can make some comments:

• We can observe that pvalues between 'control' and 'high' groups are below 5%. Therefore, both tests confirm that means revenue generated from the 'xHigh' and 'xxHigh' groups are statistically different from the revenue generated from the 'control' group.
• Moreover, as the pvalue is above 5% while comparing both 'xHigh' and 'xxHigh' groups, there is no statistically difference between the revenue generated from these groups. We can therefore not conclude that there is a clear winner to this AB test.

What would be your business recommendation(s) to the game manager?

My recommendations:

• We should increase the frequency of ads on the Ball Mayhem game. Indeed, there is a significant increase in publisher revenue when we increase ads to 'xHigh' and 'xxHigh' frequencies.
• However, as the difference between 'xHigh' and 'xxHigh' revenue is not significant, I would recommend to only increase the ad frequency to 'xHigh' as too much ads could decrease the users' satisfaction and the sessions lenght.

Do you have any suggestions for future experiments?

My suggestions:

• We could also use this AB test to check the impact of ad frequency on the number of session and sesison_lenght.
• To deepen this study and get more informations, I could split the analysis between both android and ios platforms.
• I could also split the analysis between users that have fs_shown and rv_shown ads.
• Finally, to have a clear view on all the A/B test, I could create a Tableau dashboard with KPIs such publisher_revenue_increase, session_number, D1_retention, D7_retention. This would help teams to follow easily the performance of each cohorts throught time.