6. TestFlight

Written by Pietro Rea & Keegan Rush

Ad hoc distribution is a good option for distributing internal apps to a limited audience. However, there’s a catch: You can only install ad hoc builds on a maximum of 100 devices. If you work for a large organization with lots of internal users or need to accommodate numerous testers, you can quickly hit this limit.

Historically, you could get around ad hoc distribution’s device limit by applying to be in the Apple Developer Enterprise Program or, more recently, by using Custom Apps distribution. With those distribution methods, you can install internal apps on an unlimited number of devices. However, you can only use them to distribute apps to employees or associates. So how could you get external testers to test your app and not worry about the device limit?

This is where TestFlight comes in. TestFlight is Apple’s solution for testing apps — and it can accommodate both internal and external testers. As of this writing, TestFlight is the only officially-supported way to conduct large-scale external testing, commonly known as a beta test, on iOS.

TestFlight is unique among all other internal distribution methods. Before diving into specific features, it’s worth going over some key differences:

• Users instead of devices: TestFlight follows a user-centric model instead of a device-centric model. In practice, this means you can onboard users directly instead of having to collect and register their device UDIDs. TestFlight keeps track of which users have access to which apps, so you don’t have to spend time creating and managing provisioning profiles, either.
• Two types of tester: TestFlight supports 100 internal testers per app. Each internal tester can test on up to 30 devices. TestFlight also supports 10,000 external testers per app. You can add external testers with an email address — no Apple ID required — or with a shareable invite link.
• System integrations: TestFlight is so deeply integrated with the operating system and the App Store that it’s difficult to know where TestFlight begins and where it ends. At the most obvious level, “TestFlight” refers to the app on the App Store that testers need to install beta apps. TestFlight also covers all related user management, build management and feedback management in App Store Connect. On top of that, TestFlight includes the system integrations that power its user-facing features, like crash reporting and analytics.
• Regular updates: From a developer’s point of view, other types of internal distribution don’t change much year to year. TestFlight is different. Apple wants TestFlight to be the de facto way to test apps. Whenever there’s a new product or platform, Apple is quick to add TestFlight support. You can also expect Apple to improve TestFlight and add new features regularly. The yearly WWDC session named “What’s New in App Store Connect” always covers what changed with TestFlight that year.

For this chapter, you’ll use the raywenderlich.com iOS app to upload a distribution build to App Store Connect.

You did something similar in Chapter 5, “Internal Distribution”, except you used an ad hoc provisioning profile instead of a distribution provisioning profile. The steps are almost identical, so this chapter omits them. You can refer to Appendix A if you need them. Make sure to upload a distribution build before moving on.

Getting started with internal testing

You need to be a registered App Store Connect user to be considered an “internal” tester. Fortunately, you don’t need to register someone new for this section. You can (and should!) always add yourself as an internal tester.

Note that not all App Store Connect users can become internal testers. Only the roles of Account Holder, Admin, App Manager, Developer or Marketing are eligible. To add a new user or change an existing user’s role, refer to App Store Connect’s documentation: https://apple.co/3nV8pUp.

Registering testers

Now, it’s time to add yourself as an internal tester. Log into App Store Connect. Click My Apps. Click the raywenderlich.com app record you created in Chapter 3, “Submitting Your First App for Review”. Next to the main header, click the TestFlight tab.

The TestFlight tab lists all the builds you’ve ever uploaded to App Store Connect, organized by version and build number.

Note: If you’re unsure where version numbers come from, a build’s version and build number come from the target’s Info.plist in Xcode. Version corresponds to the key CFBundleShortVersionString. Likewise, the build number corresponds to CFBundleVersion.

Under Version 1.0.6, click the build number. The version and build number should match the build you uploaded earlier.

External testing supports groups of testers as well as individual testers. But when it comes to internal testing, you can only add testers to the default App Store Connect Users group.

Under Internal Group, click App Store Connect Users. Surprisingly, the group is empty. Shouldn’t you at least see yourself?

Adding internal testers is a two-step process. First, you have to register new users in App Store Connect. Second, you have to add them to the default group of internal testers.

Click Add Testers. On the next screen, select the checkbox next to your name. Click Add.

On success, App Store Connect navigates you back to the previous screen. Here, you can verify that you’re now part of the default App Store Connect Users group.

Testing an app

When you add an internal tester, Apple sends an email invite to the user’s email address. Next to the user’s name, the Status column keeps track of which users haven’t accepted their invites so you can resend the invites to them. For each user, you can also see the number of times they’ve opened this build (sessions), as well as the number of crashes and feedback they’ve sent back.

Under the Builds section, click a build.

On this page, you can add notes for your testers so they know what to test. Under Test Details, enter something short. Click Save.

On your iOS device, open the email invite you received from TestFlight.

Tap View in TestFlight. If you don’t have the TestFlight app, doing so takes you to the App Store. Download it and come back. If you already have the TestFlight app installed, the email link opens TestFlight directly.

Tap Accept. Then tap Install. Tap the Back button to go back to the previous screen. You’ll see the raywenderlich.com app listed on the main screen.

Exit the TestFlight app. Once the raywenderlich.com app finishes installing, the name of the app appears next to an orange dot.

The orange dot means that this is a TestFlight-managed app. If you’ve enabled App Library (iOS 14 and above), apps managed by TestFlight also show up in an app group named TestFlight.

TestFlight features

TestFlight supports different features to help you and your testers, which are all available for both internal and external testers. Before moving on to external testing, here are the most important ones:

1. Redemption codes: In case you can’t access the invite email on your test device, TestFlight also supports accepting invites with a redemption code. To use redemption codes, have the tester open the invite link on a desktop computer. Opening an invite link on a web browser displays its redemption code. The TestFlight app has a Redeem button on its main screen.
2. Using a different Apple ID: The way people set up their devices can get complicated. Some users may have a personal device and a work device. Others may use one device for everything. To accommodate as many setups as possible, testers can redeem TestFlight invites even when they’re logged into a different Apple ID than the one that you invited. As long as they have access to the email link or redemption code, they’re good to go.
3. Automatic updates: In TestFlight, automatic updates are turned on by default. Whenever there’s a new build available, it will download automatically and replace the old one. Testers can turn off automatic updates for a particular app in the TestFlight app.
4. Build notifications: TestFlight supports push and email notifications out of the box. Every time a new build becomes available for testing, testers receive email and push notifications. Testers may change their notification settings in the TestFlight app.
5. Previous builds: TestFlight builds are available for 90 days from the date the developer uploads them to App Store Connect. Testers can install a previous build if it falls within the 90-day window and if the developer hasn’t manually expired it in App Store Connect. This is helpful when trying to determine if a bug is new or if it existed in previous builds.

TestFlight also supports rich features for reporting bugs and submitting feedback. These are also available to internal and external testers. Internal testers, however, often prefer internal channels of communication for submitting feedback. The chapter covers feedback-related features in the next section, along with external testing.

Using TestFlight for internal testing is not radically different from other internal distribution methods, such as ad hoc distribution. If you’re deciding between TestFlight and ad hoc distribution for internal testing, the second to last section of this chapter covers the advantages and disadvantages of each.

Beta testing with TestFlight

In addition to internal testing, TestFlight can also help with external testing, which is also known as beta testing. Beta testers are closer to being “real users” than internal testers. In many cases, they are real customers that you recruited to test a future software release.

Unlike internal testers, who might literally sit next to you, beta testers have less incentive to submit feedback. To make a round of beta testing meaningful, the number of beta testers has to be much greater than the number of internal testers. TestFlight supports 10,000 beta testers per app.

Running a successful beta test is hard work. Furthermore, the technical work is minuscule compared to the coordination and communication work that needs to happen. If you want to run a beta test, expect to spend a lot of time doing the following:

• Recruiting users: If you have an existing customer base, you could put out a call for testers. Otherwise, you can use a public link, covered in the next section, to recruit testers from social media and the general public. Some beta tests use an intake or application process if there’s a lot of interest. It’s not uncommon to offer an incentive for participating, like a free copy of the app.
• Onboarding users: Using TestFlight isn’t straightforward to everyone. Some testers might have trouble accepting the invite or downloading the TestFlight app. Others might be confused by your app. Someone on your team needs to field questions from beta testers and help them get started.
• Triaging feedback: Receiving feedback and bug reports is the whole point of the beta test. Once you start receiving feedback, someone needs to make sense of it. If the feedback isn’t clear, someone needs to follow up with questions. If you’re receiving bug reports, someone needs to reproduce the bugs and communicate with your development team.

Ongoing communication with your beta testers is key to running a successful beta test. You need to answer questions from testers, make sense of incoming feedback and work with your development team to make any improvements.

Finally, you should close the loop and ask testers to retest after you ship improvements.

TestFlight helps you by providing some tools to do this, but it cannot do the work for you.

Onboarding beta testers

After you’ve found a group of interested testers, there’s a series of steps you have to take to give them a build to test. First, you have to add them to App Store Connect — not as full App Store Connect users, but as external testers. Dealing with so many beta testers can get unruly, especially if you’re testing multiple apps at once. You can organize testers into groups or add them individually.

Before adding any beta testers, create a group. In App Store Connect, click the raywenderlich.com app’s TestFlight tab. Next to External Groups in the navigation bar, click + to add a new group. Type Beta Testers as the Group Name. Click Create.

There are two ways to add beta testers. If you have a list of names and email addresses, you can add them yourself in App Store Connect. You can also let testers sign up by themselves with a public invite link.

Adding beta testers in App Store Connect

If you only have a few testers, you can add them manually by entering their names and email addresses in App Store Connect.

As you can imagine, this takes a long time when you have a lot of testers to enter. To help, App Store Connect supports adding multiple testers at once with a CSV upload.

For now, just add yourself. Under the Testers section, click Add Testers.

From the drop-down, select Add New Testers. Fill out the Email field with a secondary email address you can access. If you use a service like Gmail, you could use an email alias like username+ailas@gmail.com. Fill out the First Name and the Last Name fields. Finally, click Add.

Like internal testing, you can tell if a beta tester accepted your email invite. You also have access to basic metrics for sessions, crashes and feedback.

Although the build you uploaded is available to internal testers, the Status column for your beta tester says No Builds Available. Why?

Apple requires beta builds to pass Beta App Review before beta testers can use it. Beta App Review ensures that beta builds adhere to the App Store Review Guidelines.

Furthermore, if you check your secondary email, you won’t find a TestFlight invite. TestFlight doesn’t send email invites to new users until there’s a build available. Once your beta build passes Beta App Review, you’ll attach it to a group and TestFlight will send out invites to any beta user that needs one.

Another significant difference with internal testing is the Apple ID requirement. All internal testers need to first register as App Store Connect users, which requires an Apple ID. You can invite external testers even if they don’t have an Apple ID.

Note: Internal testers and external testers can test different builds at once. While a build goes through Beta App Review, and even after you distribute it to external testers, internal testers in the default App Store Connect Users group will continue to receive the latest builds as you upload them.

Adding beta testers with a public link

In addition to adding beta testers yourself, you can also create a shareable public link that allows anyone to join your beta test. If you don’t already have a pool of users to draw from, opening your beta test to the public can help. Keep in mind that the testers that join your beta program count against your limit of 10,000 external testers.

There are some drawbacks to using a public link. Most obviously, you lose some control. You can’t control how others share the link once you publish it, nor can you control who signs up or vet them in any way before they do.

Additionally, when someone joins a Testflight beta through a public link, they’re added as anonymous testers. You won’t know these testers’ names or email addresses unless they choose to share their contact information when they submit feedback via TestFlight.

Note: You can only create public links if you have at least one build approved by Beta App Review.

Click the Beta Testers group name. Under Public Link, click Enable Public Link.

App Store Connect confirms you want to proceed. Then, under Public Link, a link with this format appears: https://testflight.apple.com/join/{identifier}.

If you’re worried that your beta build might get a lot of attention, you can cap the number of testers that can sign up using the public link by clicking Set limit. Once you have enough beta testers, you can also disable a public link altogether by clicking Disable Link.

Submitting for Beta App Review

Adding beta testers won’t do any good if they don’t have a build to test. A build needs to pass Beta App Review before it becomes available to external testers.

Only the first build of a version needs a review. Subsequent builds don’t need a review as long as the version is the same. For example, if version 1.1 (1) passed Beta App Review, version 1.1 (2) won’t need one. Beta review times vary but they’re generally shorter than going through the full, non-beta App Review.

Submitting a build for Beta App Review is part of adding a build to a group. Back in App Store Connect, navigate to the Beta Testers group. Under the heading Builds, click Add Build.

Select the radio button next to Build 106. Click Next.

Fill out the Test Information form, including the Feedback Email and Contact Information fields. Click Next.

On the next screen, under What to Test, type in short release notes. Leave the checkbox next to Automatically notify testers selected.

If you were doing this for a real app, the next step would be to click Submit for Review. However, real Apple employees go over Beta App Review submissions, so don’t submit this build for review.

Once the app has passed Beta App Review, you’ll receive an email confirmation. If you selected Automatically notify testers, your beta testers will also receive notification emails. If you didn’t select this option, you have to go back to App Store Connect to start testing. Your beta testers will receive an email at that point.

Collecting TestFlight feedback

Collecting tester feedback is the whole point of testing. TestFlight supports different ways to submit feedback, but some of them are so deeply integrated into the operating system that they’re easy to miss. You can save a lot of time by documenting the next few sections in your own words for your testers.

Sending feedback

If you see a bug in an app and want to it show someone else, what’s the first thing you do? You take a screenshot. That’s exactly how the first way to submit feedback from a TestFlight app starts.

Try it now. Open the raywenderlich.com app. Take a screenshot. Then quickly tap the screenshot preview to open it in Markup. Here, you can mark up the screenshot with the built-in tools.

Note: How you take a screenshot depends on the device you have. Some testers might need help figuring out how to do it. It helps to keep Apple’s documentation handy so you can share it with your testers: http://apple.co/3qhXHck.

There are two ways to send feedback from Markup. To see the first way, tap Done in the upper-left corner. Doing so opens an action sheet with options. Tap Share Beta Feedback….

To see the second way to send feedback from Markup, tap the Action button in the upper-right corner. You can submit feedback from here as well. Tap Share Beta Feedback.

Doing so brings up a feedback screen that includes your screenshot. Tap the area above the screenshot to start typing. Enter something short and tap Submit.

If you took a screenshot but forgot to send it, you can do it later from the TestFlight app.

Try that now. Open the TestFlight app. Tap raywenderlich.com. Tap Send Beta Feedback.

TestFlight asks you if you want to include a screenshot. Say yes, and you can choose any photo from your photo library. You can also send feedback without a screenshot. Tap Don’t Include Screenshot in the action sheet to try that.

Enter something short in the text field. Tap Submit.

All the feedback that testers submit via TestFlight ends up in App Store Connect, where you can view it or export it.

Sending crash reports

Despite your team’s best efforts, your app will crash sometimes. Every time this happens, some unlucky user is summarily and unceremoniously kicked out of your app and onto their home screen.

Beta testing is a great way to find crashes that you overlooked during development or internal testing. Fortunately, TestFlight also supports crash reporting.

Unbeknownst to you, the starter project you used to create and upload a build is not the exact copy of version 1.0.6. It has a defect that makes the app crash every time you try to sign in. Whoops!

Try it now. Open the raywenderlich.com app. Tap Sign In. The app exits and you’re faced with an alert asking if you want to share additional information with the developer. Tap Share.

On the feedback screen, enter something short. Tap Submit.

Sharing crashes with developers is optional, so when you onboard beta testers, instruct testers to share crash reports as often as possible.

Collecting feedback

App Store Connect has a section that aggregates all feedback and crash reports. In the previous two sections, you submitted feedback with and without a screenshot, along with a crash report. Now, you’ll see where you can find this information.

Open the app’s page in App Store Connect. Click the TestFlight tab. On the left menu, under Feedback, click Screenshots.

This page aggregates all feedback sent from the TestFlight app as well as feedback sent immediately after taking a screenshot. Note that the name of the page is slightly misleading. Even feedback that doesn’t contain a screenshot ends up here.

Now, turn your attention to the crash you experienced earlier. On the left menu, under Feedback, click Crashes. The crash feedback page is where you can find all crash reports.

Click the single crash report for version 1.0.6. The crash detail page has useful troubleshooting information about the device that sent the crash report. Click the Download button in the top-right corner.

Unzip testflight/_feedback.zip. The resulting folder contains two files: crashlog.crash and feedback.json. Open crashlog.crash.

This is a symbolicated stack trace. It’s like a trail of breadcrumbs that the app left before it crashed. Sending .crash files to your development team helps diagnose and, eventually, fix the crash.

As a developer, you can also open a .crash file in Xcode. Try it now. Right-click crashlog.crash. Select Open With. Select Xcode. Xcode asks which project it should use to open the .crash file. Select emitron.

Doing so opens a debugging session in Xcode’s Debug navigator, as if you had crashed while running the app locally. It even highlights the line where the crash occurred.

You can also download TestFlight and App Store crash reports from Xcode’s Organizer window. As of this writing, downloading them manually from App Store Connect is the most reliable way to collect them.

TestFlight versus ad hoc

TestFlight is the only option if you want to run a large beta test on iOS, but when it comes to internal testing, you have more choices. The previous chapter covered ad hoc distribution and this chapter did the same for TestFlight. If you’re deciding which one to use for internal testing, here are three things to consider:

1. Flexibility: Ad hoc distribution wins on flexibility. With ad hoc distribution you can use a third-party solution like Firebase or AppCenter, which support a wide variety of use cases, or you can build your own over-the-air (OTA) solution tailored to your specific needs. TestFlight has a growing set of features and APIs, but you’re always constrained by what Apple provides.
2. Speed: Ad hoc distribution wins again. With ad hoc distribution, you’re only constrained by Xcode build times. As soon as you have an IPA, you can upload it to a third-party service or host it yourself and you’re done. With TestFlight, there’s an additional processing step once your build reaches Apple’s servers. It usually takes only a few minutes, but it can sometimes take longer. The size of your team and the frequency of your builds will determine how much this matters to you.
3. Simplicity: TestFlight wins the medal for simplicity. It’s simpler to add users to TestFlight than it is to collect, register and manage device UDIDs. TestFlight is also the simplest option for developers. In fact, TestFlight doesn’t require any additional development work to use — not even linking against a new framework in Xcode. This also means TestFlight requires zero ongoing maintenance.

Smaller teams tend to prefer TestFlight for its simplicity. As your internal distribution needs grow, larger organizations tend to prefer third-party ad hoc distribution solutions. The largest organizations typically benefit the most from building and maintaining custom OTA solutions.

Key points

• TestFlight is a good solution if you hit the device limit in ad hoc distribution or want to run a beta test with external testers.
• Internal testers need to be registered App Store Connect users on your account.
• Internal and external testers receive an email invite. Tapping the email’s invite link prompts them to download the TestFlight app and install the build.
• Running a successful beta test requires a lot of time to recruit and onboard testers, triage feedback and manage ongoing communications.
• You can add external testers manually in App Store Connect. Another option is to create and share a public link, which anyone can use to sign up for your beta.
• Unlike internal testing, Apple requires beta builds to pass Beta App Review before external testers can use the build. Beta builds must also adhere to the App Store Review Guidelines.
• Testers can submit feedback by either taking a screenshot while using the TestFlight build or using the TestFlight app. TestFlight also supports sharing crash reports.
• You can use TestFlight or ad hoc distribution for internal testing. Ad hoc distribution is faster and more flexible, but TestFlight is simpler to use, implement and manage.