What’s New in Testing With Xcode 12

Don’t Let Hung Tests Break the Loop

Execution Time Allowance is a new customizable feature you can opt-in to, which enforces a time limit on each test. You can use this option to guard against test hangs and ensure the rest of your tests always complete running.

By default, each test gets ten minutes. You can customize this value in the test plan if you want to apply it to all tests. Use the executionTimeAllowance API if you only need it for specific tests or cases.

The value is provided as TimeInterval but rounded to the nearest minute. Values under 60 seconds are rounded up to one minute.

When a test exceeds the provided limit, Xcode will:

• Capture a spindump and attach it to the test results to shed light on what went wrong. The spindump shows you which functions each thread is spending the most time in and can help you better diagnose app stalls and hangs.
• Kill the test that hung.
• Restart the test runner so the rest of the test suite can execute.

To experiment with execution time allowance, you need to turn this option on:

1. Go to Edit Test Plan.
2. Select the Configurations tab.
3. Find the Test Timeouts option and set it to On.

Now, go back to the first test you wrote, testInitialLetter(), and add these two lines, right before calling XCTAssertEqual:

executionTimeAllowance = 30
sleep(70)

This code sets the execution time allowance for this test to 30, which will be rounded up to one minute, then sleeps for a little over a minute to ensure the test fails. Run the test now and notice it hangs for about a minute before failing.

Once the test finishes with a failure, go to the test report and see the attached spindump. This can be a life-changer in cases of complex tests with multiple threads when you’re trying to figure out what went wrong.

There are other ways to set the execution time allowance, and Xcode has a clear precedence order for this:

1. executionTimeAllowance: You used this in the example above. It takes top priority.
2. xcodebuild option: -default-test-execution-time-allowance.
3. Test plan setting.
4. Default value (10 minutes): Lowest priority.

Speeding Things Up Even More

If you have a large test suite and are running it frequently, which you definitely should, you may find it takes a long time to complete. Xcode 12 introduces another great way to speed things up: Parallel distributed testing.

Parallel distributed testing involves running a test plan on multiple devices in parallel. Once a specific device finishes running a test, Xcode gives it a new test until there aren’t any tests left. Even by adding one additional device, you can get a 30% speed-up compared to testing on a single device.

While a deep dive into implementing Parallel distributed testing is beyond the scope of this article, there are a few principles worth stressing:

• The allocation of tests to run destinations is non-deterministic. If the logic you’re testing is device or OS specific, using this technique can cause issues such as unexpected results or skipped tests.
• Ideally, you should use the same kinds of devices and OS versions per test suite run.
• If you’re using multiple types of devices or OS versions, you should aim to run tests that are device and OS agnostic, for example, pure business logic.

Failure = Success

It’s always fun to see green tests, but great tests catch bugs. You write them once then run them many times throughout the development cycle. You should expect and plan for them to fail.

If written well, failed tests help you improve the quality of your app and the confidence you have in it. New APIs and enhanced test failure reporting UI in Xcode 12 make researching and resolving test failures more efficient.

If the line of failure in the failed test is gray then the failure happened underneath the annotated line, but not at that line itself. If you then go to the test report to explore this further, you’ll see the call stack. Clicking a specific row in the call stack will take you to the exact code location.

If the annotation is red, this is the actual point of failure.

XCTest has always recorded failures as four discrete values:

• Failure message
• File path
• Line number
• Expected flag

Xcode 12 introduces XCTIssue, a new object which encapsulates these four values and adds additional details and capabilities to improve your debugging experience. These include:

• Issue type for classifying different kinds of issues that may occur during testing.
• Detailed description that may include transient data, such as numbers, object identifiers and timestamps, to help diagnose the issue.
• An optional associated error.
• An array of attachments, such as files, images, screenshots, data blobs or ZIP files, to make it possible to assess customer diagnostics for test failures.
• sourceCodeContext, which captures and symbolicates call stacks to provide additional context for failures in complex test code.

Setting Up and Tearing Down

Xcode 11.4 provides new APIs for throwing from setUp and tearDown: setUpWithError() and tearDownWithError(). You’ll find both of these methods in the template for new tests.

These new APIs let you have a simplified and tailored flow instead of boilerplate code. Source code locations are included as part of thrown errors starting with iOS and tvOS 13.4 and macOS 10.15.4, saving you the extra handling code for attaching these details as parameters.

Use setUpWithError() to state the assumptions needed for your tests to run properly and set the app’s state to match those assumptions. Previous tests may have changed the app’s state or modified data the test relies on, so this is a good opportunity to start with a clean slate.

You can leverage launchArguments and launchEnvironments to set the needed state. For example, you might bypass 2FA during testing or bypass the main tab to start with the menu tab. This is also a good opportunity to adopt product changes to keep your tests focused.

tearDownWithError() is also a good opportunity to take advantage of the new error management. You can use it to collect additional logging during tear-down, including some analysis of the failures, while the data is still available. You can also use this as an opportunity to reset the environment from the changes made during the most recent test run.