What’s New in Testing With Xcode 12

The idea behind test-driven development is simple: Think about what you want to accomplish, break that down into use cases and write tests to verify those cases. Only then do you get to implementation, with the goal of getting your tests to pass. As a result, you avoid unintentionally matching your tests to the code you’ve already written and force yourself to think about what you want to achieve before writing a single line of code.

Xcode 12 introduces improvements to enhance the testing experience, including:

• Improved test results with the new XCTIssue type.
• More robust test error handling.
• Breadcrumbs for test failures which point you directly to the point of failure.
• Streamlined test initialization code.
• The ability to skip tests instead of having to comment them out using XCTSkip.
• Execution Time Allowance to better handle hung tests.
• XCTest improvements.

It may be a bit painful to admit, but everyone writes buggy code sometimes. :] In this article, you’ll learn how to use these Xcode 12 testing topics to enhance your app’s quality and your development velocity.

Sound like a plan? Time to get started!

Getting Started

To demonstrate the testing concepts and tools discussed in this article, you’ll work on a Scattergories client app. Like the original game, you score points by naming objects from a set of categories, all of which have to start with a randomly selected letter.

Currently, the app only supports randomly selecting a letter. This is a great time to add tests, before you get to the heavy-lifting part of implementing gameplay.

Use the Download Materials button at the top or bottom of this article to download the starter and final Scattergories app used throughout this article. If you prefer, follow along using your own app.

Adding Testing Targets and a Test Plan

First, add two new testing targets: one for unit tests and another for UI tests. If you’re using your own app and already have test targets and a test plan, feel free to skip this section.

To add a testing target:

1. Click the project file.
2. Click + in the bottom left corner.
3. Select Unit Testing Bundle.
4. Click Next.

Leave all of the pre-filled fields on the next screen as-is. But if any are missing, fill them in making sure that the Language is set to Swift. Then click Finish.

Repeat the same steps described above to create a UI Testing target by selecting the UI Testing Bundle option when needed.

Next, create a test plan by selecting the current scheme and then Edit Scheme…

To add the testing targets, select the Test menu on the left side panel and make sure that the Info tab is selected along the top. Now do the following:

1. Click Convert to use Test Plans….
2. Select Create empty Test Plan.
3. Click Convert….
4. When prompted, leave the suggested values unchanged and click Save. Click Replace if prompted to replace the existing file.

Click Close to dismiss the window. Now add your test targets to the test plan:

1. Click the Test navigator.
2. Click the current test plan, Scattergories (Default).
3. In the menu that appears, click Edit Test Plan.

Now do the following:

1. Click the + in the bottom left corner.
2. In the dialog that appears, choose both ScattergoriesTests and ScattergoriesUITests.
3. Click Add.

That’s it! Now that everything is fully wired, you’re ready to dive into Xcode 12’s testing enhancements. So, what are you waiting for?

The Testing Feedback Loop

The testing feedback loop is a simple concept. It states you should:

1. First, write tests.
2. Then, run them.
3. Finally, interpret the results.

You should rinse and repeat these three steps until you gain enough confidence in both your tests and your app code to move on. Xcode 12 provides tools to ensure you always get faster feedback and your testing loop doesn’t break due to hung tests.

Writing Your First Test

Each test should focus on performing a single action and then asserting when the action completes. The assertion message should be specific, but not too specific. People, or scripts, reviewing the results should easily recognize multiple tests failing due to the same reason.

Go to ScattergoriesUITests.swift and add a new test at the end of the class:

func testInitialLetter() {
  let app = XCUIApplication()
  app.launch()

  XCTAssertEqual(
    app.staticTexts.count, 
    1, 
    "There should only be one text view - the letter")
}

This is a simple test with the sole goal of ensuring there’s exactly one label in your app.

Use the small Play button next to the test name to run it. It’s an empty diamond until you hover your mouse over it.

You’ll see the simulator open in the background, and the test run until it passes. The status indicator next to the test will turn to a green checkmark.

Isn’t it fun to see a green test? Don’t get used to it. You’ll start seeing red tests and learning how to fix them soon. :]

Getting Feedback Faster

One pitfall of asserting is asynchronous events. XCTest has built-in re-tries, but depending on the code you’re testing, it may not be enough. You’ll add a new test to demonstrate this.

Add the following code to ScattergoriesUITests.swift:

func testGetLetter() {
  let app = XCUIApplication()
  app.launch()

  app.buttons.element.tap()
  XCTAssertEqual(
    app.alerts.count, 
    1, 
    "An alert should appear once a letter is selected")
}

In this new test, you simulate a button click, which will iterate through the letters for a randomly pre-selected number of milliseconds. When the app reaches the final letter it displays an alert. You’re testing for the existence of this alert.

Run the test by clicking the small Play button next to the test name. When the test finishes running, you’ll notice it’s failing.

Open the Report navigator by typing Command-9. Select the failing test to display the associated test report in the center panel and expand the report by clicking the disclosure indicator. You’ll see the description you provided earlier for the assert call. Expanding this line exposes an Automatic Screenshot that was captured and attached to the assert message. Open the screenshot and you’ll see that there wasn’t an alert, so the test did actually fail.

The alert isn’t showing up because the test checks for its existence right after simulating the button click instead of giving the app time to run through the letters. The app takes a few milliseconds to run through the letters before selecting the last one. At that point, the user can get the selected letter and start playing.

You could mitigate this by waiting the maximum number of milliseconds the app can take to iterate through the letters and then checking for the alert. This lets the test pass or fail deterministically and in an environment you designed. However, in most cases, the alert presents much sooner than that, so why wait?

A better solution is to use waitForExistence(timeout:). If the expectation is true before the timeout, you save waiting time.

Replace the existing XCTAssertEqual line in testGetLetter() with:

XCTAssertTrue(
  app.alerts.element.waitForExistence(timeout: 10), 
  "Letter should be selected within the provided timeframe")

Run the updated test and see it now passes. More than that, look at what’s happening in the simulator while the app runs and see the test is waiting for the app to stop iterating through the letters before checking for the alert. In the test report, you can see how long the test waited for the result.