Firebase Tutorial: iOS A/B Testing

We’ve all heard stories about how some developer changed the label of a button or the flow of their welcome screens and suddenly found their retention rate or their in-app purchase rate increased by some crazy amount.

Maybe your to-do list contains an “experiment with my purchase button” item still waiting to be checked off because you discovered that running these experiments correctly is a lot of work.

The Firebase Remote Config Tutorial for iOS showed you how to use Remote Config to update certain parts of your app on the fly and deliver customized content to users in particular countries.

This follow-up tutorial will explore how to use Firebase Remote Config to conduct A/B testing by experimenting with different values and viewing the results to find out which values work better.

Getting Started

Download the materials by clicking the Download Materials button at the top or bottom of this tutorial. Even if you have the project from the previous Remote Config tutorial, it will be better to start with the new starter project because it uses the latest Firebase version. The necessary libraries were added as packages for Swift Package Manager and might take a few moments to load when opening the project the first time.

Creating a Firebase Project

If you already created a project with Google Analytics for the previous Remote Config tutorial, you can skip this section. If not, follow these instructions to create your project:

1. Go to https://console.firebase.google.com.
2. If this is your first project, click Create a project. Otherwise, click Add project.
3. Name your project Planet Tour and then click Continue:
   
4. Add Google Analytics to your project (switched on by default) and click Continue:
   
5. Select your region, read and accept the terms of use. Then click Create project:
   

Next, add an iOS app to the project. Click the iOS button and follow these instructions:

1. Add the bundle ID of your project — com.raywenderlich.PlanetTour — give it a fun nickname, leave the App Store ID field blank and then click Register app:
   
2. Download the GoogleServices-info.plist file and click Next.
3. Open PlanetTour.xcodeproj in Xcode and drag the downloaded GoogleServices-info.plist file into the Planet Tour project (select Copy Items if Needed).
4. Head back to your browser to the Firebase console. Complete the next few steps (numbers 3 and 4) by clicking the Next button.
5. Press Continue to the console on step number 5:
   

Build and run your app. You should see a lovely tour of our solar system. Click a few planets to see details about them:

If you’re new to the project, review RCValues.swift to understand how Remote Config is being used in the project. Then, go back to the main screen and check out the banner at the bottom where the app asks you to sign up for the newsletter.

Click “Get our newsletter!”. You’re now going to look at optimizing this screen with some A/B testing.

An Introduction to iOS A/B Testing

The higher-ups at Planet Tour Inc. are concerned there aren’t enough people subscribing to the Planet Tour newsletter. After all, how can you build a successful business if you’re not able to email people about exciting new Planet Tour offers?

Any way we can make this Subscribe button more enticing?

Any way we can make this Subscribe button more enticing?

The folks from marketing have a theory. You might be able to get more subscribers if you change the label of the Subscribe button to Continue. While you’re at it, maybe you should try changing the text of the banner from Get our newsletter to Get more facts.

These are easy changes to make now that your app gets its data from Remote Config. It would just be a few seconds of work publishing these new values in the Firebase console. Then after a week, you could see if you get more newsletter subscribers. Simple, right?

Well, hang on. How would you know the changes you made are directly responsible for the results you’re seeing? What happens if some influential blogger mentions your app’s newsletter in their latest post? Or you end up running an ad campaign in another app’s newsletter, thereby bringing in an audience more inclined to subscribe to newsletters in the first place?

You might end up drawing the wrong conclusions if you can’t separate the change from the change in newsletter sign-ups.

Ideally, you’d want to release two versions of your app simultaneously. One random group of users gets to see the new newsletter labels, and the other group gets to see the current ones. As long as these two groups are unbiased, you can compare the results between them. The more users in each group, the more confident you can be that the differences are due to the changes you made and not some external factor.

Well, that’s exactly what A/B testing is, and it’s a common way to run these kinds of experiments. Many larger companies have built their own infrastructure to run and measure these tests. You can join these larger companies with the handy tools that come with Firebase Remote Config.

Adding Analytics to Your App

One of the main steps in creating an A/B test is telling Firebase the goal of your experiment. Sometimes, it might be a high-level goal, such as increasing retention (how many of your users return after a few days) or user engagement (how much time your users spend with your app each day). But other times, it might be a specific goal, such as increasing the number of users who visit your in-app store. Or, in this case, increasing the number of users who sign up for the Planet Tour newsletter.

Of course, the only way Firebase knows whether your user has signed up for your newsletter is if you tell it. To do that, you’ll use Firebase Analytics. So before you start creating an A/B test in your app, you should add some analytics to your app to give you a few goals you can start working toward.