ChilliConnect's A/B Testing allows you to run experiments on your game to find out what particular features or configurations of your game are most effective. AB tests can be created to target specific segments of players based on country, platform, device type and device model. For each test, one or more test groups can be created, with players evenly distributed across groups in a round-robin fashion. Tests can also be restricted to a maximum number of players and can be configured to target new players only.
Players are assigned to test groups on login and initial creation. A player is considered to be part of a test group until they log in again, at which point they will be unassigned from the test if it has ended.
To create a new Test in ChilliConnect, select top level the Live Ops menu option followed by the test option. Use the "Add Test" button in the top right to bring up the New Test form.
The details tab captures basic configuration information for the test, including the test name, description and start and end dates.
|Name||A descriptive name for the test|
|Key||A unique Key that will be used to refer to the test in API calls|
|Description||Optional additional text that can be used to capture more information about this test|
|Start Date||The Date the test is due to start|
|End Date||The optional End Date that the test should complete. If not provided, End Date can be changed once the test has started.|
|Total Installs||The total number of installs of your game|
|Target KPI||The KPI that should be tracked for this test. This can be changed after the test has been created.|
The segments tab allows you to filter the A/B Test to specific groups of players.
For more details see Segmentation.
The Test Groups tab allows you to define how many players should be included in your Test, whether or not players should be restricted to new players only, how many players should be included and if the Test should retain it's players regardless of the Tests completion status.
|Player Limit||An optional maximum number of players that should be included in this Test. Setting this limit allows you make the most of your player base and run multiple tests in parallel. If no limit is set, players will be allocated to the test until the test ends.|
|New Players Only||Should the Test be restricted to new players only. When true, players will only be allocated to Test Groups on CreatePlayer calls and not on log in calls. This option is useful when you are testing fundamental changes to the game economy and apply these changes to in progress games would affect Test results.|
|Sticky||When a Test has completed, players are usually unassigned from the Test on their next login call. With the Sticky flag enabled, players will not be unassigned from a Test once it completes - this will also prevent such players from ever being assigned to a new Test. This can be useful in situations where the player has been part of a Test that changes some aspect of your game that you would not want to revert once the Test has completed - for example, changes to IAP prices or game balancing. Note that players will still be removed from a Test if that Test is later deleted. When Sticky flag enabled more data is included into KPIs calculation of player's retention. This could be useful to capture retention periods exceeding the length of the Test.|
|Test Groups||The Test Groups that players should be allocated to. A Test must contain at least one test group. Players are allocated evenly in a round robin fashion. For each Test Group, a descriptive Name and a unique Key can be provided. The Key will be returned to the game when making the GetActiveCampaign call.|
Overrides allow you to easily define changes to the Catalog for different test groups. Items can be added, removed and updated. Overrides are automatically applied for Players when retrieving the item definitions or downloading the Catalog package file. This means that no additional logic is required in your game to support overrides.
For more details see Overrides.
ChilliConnect will compute any possible conflicts with the Overrides and existing Campaign Overrides. If conflicts are found you will be presented with options on prioritising the Campaigns.
For more details see Campaign prioritisation.
Custom Data allows you to define arbitrary, free-form JSON to be attached to the A/B Test definition. This JSON can be retrieved at run time from your game via the GetActiveCampaigns call, providing a convenient place to add any small, extra data to the A/B Test.
All tests, active, upcoming and completed are displayed in the test views. Filters in the top right of the screen can be used to show tests in a specific state. Active tests are always shown in blue.
Details of a specific test can be viewed by selecting the test from the list.
The overrides tab will show you what overrides are defined for the current test.
The KPIs tab will show you all the KPIs collected for the test. The displayed KPIs can be changed to only show the ones that are required to be tracked for this test.
The Players tab can be used to view specific players that have been assigned to any given test group:
The edit button in the top right of the test view can be used to modify test details. Once a test is running, only some fields will be able to be modified.
Rather than manually ending the test by setting an End Date in the past, Active tests can be ended using the End Test button in the top right of the test view.
Changing Player Assignment
Often, it is desirable to be able to change the test group a specific player is assigned to for testing purposes. This can be done from the player view, using the Edit button displayed next to the currently assigned test group:
ChilliConnect will automatically capture and track KPIs for each individual test group you define, making it easy to track how the changes between each test group affect KPIs. To view metrics for test groups, you can use the Segments menu on any of the analytics views:
The two most recent and active A/B Tests are shown in the drop-down with the "More" option allowing you to select recently completed and older A/B Tests.