2. What Is a Test?

Written by Fernando Sproviero

A test is a manual or automatic procedure used to evaluate if the System Under Test (SUT) behaves correctly.

The SUT may be a method, an entire class, a module or even a whole application.

From now on, when mentioning anything related to writing a test this book will be referring to the automatic procedure form of a test.

To write a test, you need to understand the feature, specification or requirement of the component you are implementing. That component may be an Activity, Fragment, View Model or several of these components working together. These may come in different forms, such as user stories, use cases or some other kind of documentation.

Testing is an important part of software development. By including tests along with your code, you can ensure that your code works and that later changes to the code won’t break it. Tests can give you the peace of mind you need to develop quickly and catch bugs before they’re released.

Essentially, there are two approaches to writing tests:

• Write tests before you write the feature.
• Write tests after you write the feature.

This book primarily focuses on writing tests first versus writing them after a feature has been implemented.

Why should you test?

Writing tests can take more time up front, and it is code you write that the client won’t “see”, which is why tests are sometimes skipped by developers. However, having tests can speed up development down the road and it presents some advantages.

Change/refactor confidence

You have probably run into a scenario in which you have a section of your application that works correctly before adding new functionality to the application. After adding new functionality, either in Quality Assurance (QA) or after it is released to customers you discover that this new functionality broke the previously working section. That is called a regression.

Having good, reliable, effective tests would have caught that at the moment the bug was introduced saving time in QA and preventing preventable bugs from making it to your users. Another related scenario is where you have a section of your application that is working correctly, but could use some refactoring to use a new library, break things up to follow a more readable architectural pattern, etc. A good test suite will provide you with the confidence to make those changes without having to do time consuming manual QA regression test cycles to ensure everything is still working correctly.

However, you should always bear in mind that this is not a 100% “insurance”. No matter how many tests you write, there could be edge cases that the tests don’t catch. Even so, it’s absolutely safer to have tests that catch most issues than not having them at all!

Usually, you will write tests for the most common scenarios your user may encounter. Whenever someone finds a bug that your tests didn’t catch, you should immediately add a test for it.

Documentation

Some companies and developers treat tests as a complementary documentation to explain how the implementation of a feature works. When you have well-written tests, they provide an excellent description of what your code should do. By writing a test, its corresponding implementation and repeating this until a feature is completed, bearing in mind that these tests can be treated as specifications, will help you and your team when a refactor or a modification of the feature is required.

When you’re working on a piece of code, you can look at the tests to help you understand what the code does. You can also see what the code should not do. Because these are tests rather than a static document, as long as the tests are passing you can be sure this form of documentation is up-to-date!

How to write a test

There are many things to bear in mind when writing a test. You’ll understand them by reading this book and practicing writing tests. However, the most important aspects of writing a test are as follows:

• Naming: You should give a meaningful name to each test so that it is clearly identifiable in code and in subsequent reports.

For example, consider a quiz game:

fun whenAnsweringCorrectly_shouldIncrementCurrentScore() {
  ...
}

This test’s name represents the state of what you are testing and the expected behavior. This is useful when looking at the report after running a test suite.

• Short and simple: You should aim to write tests that focus on a narrow piece of functionality. As a rule of thumb, if your test methods get long, and have multiple assertion statements to check conditions of the system, it may be trying to test too many things. In that scenario it may be a good idea to break up that test into multiple, more narrowly focused tests. Take a look at this test:

fun whenIncrementingScore_shouldIncrementCurrentScore() {
  val score = Score(0)

  score.increment()

  if (score.current == 1) {
    print("Success")
  } else {
    throw AssertionError("Invalid score")
  }
}

The test only has seven lines of code to bring the SUT in the desired state and check the expected behavior.

• Check one single thing: Check one thing at a time. If you need to test multiple things, write an additional test similar to the one you’ve just previously run, but change the check:

fun whenIncrementingScore_aboveHighScore_shouldAlsoIncrementHighScore() {
  val score = Score(0)

  score.increment()

  if (score.highest == 1) {
    print("Success")
  } else {
    throw AssertionError("Invalid high score")
  }
}

As you can see, this test is very similar to the previous one; however, the check is different. In the first test, you checked that the score of the quiz game incremented correctly. Now, you check that the highest score also increments along with the score.

• Readable: Anyone in the team should be able to read and understand what is going on in your test and/or what is the purpose of the test. Consequently, you should pay attention to the naming of each variable or method used and the logic sequence of the test. If you don’t, then the tests will become difficult to maintain and keep up-to-date.

What should you test?

You should test code that is related to the logic of your app. This may include code that you have to write to:

• Show the UI and navigate between the screens of your app.
• Make network requests to an API.
• Persist data.
• Interact with device sensors.
• Model your domain.

Having tests for the logic of your application should be your main goal, however, bear also in mind the following:

Code that breaks often

If you have a legacy project without tests, and it breaks often whenever you modify its code, it’s useful to have tests for them, so that the next time you make a modification you will be sure that it won’t keep breaking.

Code that will change

If you know that some code will be refactored in the near future, tests will be useful here, too, because if you wrote tests for this feature, you can support on them to refactor the code and be sure you don’t break anything.

What should you not test?

External dependencies

You should assume that all dependencies (libraries and frameworks, including those from the Android SDK) have been tested. Thus, you shouldn’t test functionality of external libraries because the goal is to test things that you control, not a third party tool created by someone else.

Note: In the real world, sometimes some of those dependencies are not tested. So, as a rule of thumb, when you have to choose between two or more libraries that have the same functionality, you should go with the one that has tests. This assures you that the features of the library work as expected and that the library developers won’t break the features when adding new features and releasing new versions.

Autogenerated code

You shouldn’t write tests for autogenerated code. Following the previous principle, it’s supposed to be that the library or tool that generates code is tested properly.

When should you not test?

Throwaway/prototype code

Usually, when writing a Minimal Viable Product (MVP), you should focus on just writing the features so the client can get a feeling of what the final product could be.

However, all the stakeholders need to understand that all the code (or almost everything) you wrote will be thrown away. In this case, it doesn’t make sense to write any kind of tests.

Code you don’t have time to test

This is a controversial topic. Often, developers get stuck in a rut wherein they are fighting fires instead of proactively writing quality code, and they are not given the time to address code quality.

If you are working on a cash-strapped startup, where requirements are changing rapidly, that extra time to test could cause this fledgling company to miss key deadlines, not iterate fast enough, fail to raise it’s next round of funding and go out of business.

On a new greenfield project, writing tests can double the amount of time to get features out in the short term. But, as the project gets larger, the tests end up saving time. Writing tests has its benefits; however, it’ll take time to write and maintain tests. You and your team will need to make sure that you understand the trade-offs when determining which path you want to take.

A Note on Technical Debt

When you take out a financial loan, you get the benefit of an immediate infusion of cash. But a lender charges you interest on the loan in addition to the principal, all of which you will need to pay back. If you take on too much debt, you can end up in a situation where it is impossible to pay back the loan. In this case, you might have to declare bankruptcy.

Technical debt has many parallels to financial debt. With technical debt you make trade offs in your code, such as not writing unit tests, not refactoring, having less stringently quality standards, etc. to get features out quicker. This is analogous to getting a financial cash infusion. But as the code base grows, the lack of tests increase the number of regressions, bugs, time it takes to refactor and QA time. This is analogous to interest on a financial loan. In order to pay off that debt you start to add unit tests to your code. That is analogous to paying down the principal on a loan. Finally, if too many shortcuts are taken for too long, the project may reach a point where it is more advantageous to scrap the entire project and start with a clean slate. That is the same as declaring bankruptcy to get relief from too much financial debt.

Code spikes

At some point, you may find yourself working with a new library or, perhaps, you may realize that you aren’t sure how to implement something. This makes it very difficult to write a test first because you don’t know enough about how you are going to implement the functionality to write a meaningful failing test. In these instances, a code spike can help you figure things out.

A code spike is a throwaway piece of untested code that explores possible solutions to a problem. This code should not be considered shippable. Once you have a solution, you will want to delete your spike and then build up your implementation using TDD.

What is test coverage?

You can measure how many lines of code of your app have been executed when you run your tests. An app with a high test coverage percentage “suggests” that it works as expected and has a lower chance of containing bugs.

You may have asked yourself how many tests should you write. As mentioned before, you should at least write those that cover the most common scenarios.

In general, you can think of this metric as follows:

Criterion

To measure, there are several coverage criterion that you may choose. The most common are:

• Function/method coverage: How many functions have been called?
• Statement coverage: How many statements of each function have been executed?
• Branch coverage: Has each branch in an if or a when statement been executed?
• Condition coverage: Has each subcondition in an if statement been evaluated to true and also to false?

For example, suppose that the following code is part of a feature of your app:

fun getFullname(firstName: String?, lastName: String?): String {
  var fullname = "Unknown"
  if (firstName != null && lastName != null) {
    fullname = "$firstName $lastName"
  }
  return fullname
}

Having at least one test that calls this function would satisfy the function/method coverage criteria.

If you have a test that calls getFullname("Michael", "Smith") you would satisfy the statement coverage criteria, because every statement would be executed.

If you also have a test calling getFullname(null, "Smith"), now it complies with branch coverage criteria, because the line inside the if is not executed and the previous test that called getFullname("Michael", "Smith") executes the line inside the if statement.

To satisfy the condition coverage criteria, you need tests that call getFullname(null, "Smith") and getFullname("Michael", null) so that each subcondition, firstName != null and lastName != null would evaluate to true and false.

Tools

There are tools that can assist you to measure the test coverage metric.

JaCoCo (Java Code Coverage Library) is one of them. Don’t worry, it handles Kotlin as well!

This library generates a report for you to check which lines were covered by your tests (green) and which ones were not (red or yellow).

Android Studio also comes with a built-in feature to run tests with Coverage.

100% coverage?

In real-world apps, reaching a test coverage of 100%, no matter which criterion you use, is almost impossible to achieve. It often doesn’t add value to test all methods, of all the classes, all of the time.

For example, suppose you have the following class:

data class Pet(var name: String)

You shouldn’t write the following test:

fun whenCreatingPetWithName_shouldTheNameSetFromTheConstructor() {
  val aName = "Rocky"
  val aPet = Pet(aName)

  if (aPet.name == aName) {
    print("Success\n")
  } else {
    throw AssertionError("Invalid pet name")
  }
}

In this case, you are testing a feature (getting and setting a property) of a Kotlin data class that is auto-generated for you!

Test coverage gives you an exact metric of how much of your code has not been tested. If you have a low measure, then you can be confident that the code isn’t well tested. The inverse however is not true. Having a high measure is not sufficient to conclude that your code has been thoroughly tested.

If you try to reach 100% test coverage, you’ll find yourself writing meaningless, low-quality tests for the sake of satisfying this goal.

Neither you nor any team member should be obsessed with a test coverage of 100%. Instead, make sure you test the most common scenarios and use this metric to find untested code that should be tested.

If you feel that writing a particular test is taking too long, you might want to take a step back and evaluate if that test is adding enough value to justify the effort. Also, if a simple fix is causing a lot of changes to your tests, you may need to look at refactoring your tests or implementation to make them less brittle.

At the end of the day, your goal is to create software that provides value to its users. If you are doing TDD well, as your project gets larger, the total amount of effort spent on tests, implementation and QA should be the same or less than if you were creating the same product, with the same level of quality without doing TDD. That said, a project that is doing a good job at TDD may still take more development effort than a project that is not because the project with TDD will have a higher level of quality. The key is finding the right balance for your project.

Key points

• A test is a procedure used to evaluate if a method, an entire class, a module or even a whole application behaves correctly.
• This book focuses on writing tests before implementing the features.
• You should write tests to have confidence when refactoring.
• Tests also act as complementary documentation of the application features.
• The tests you write should be short, simple to read and easy to follow.
• You should only write tests related to the logic of your application.
• You can use test coverage tools to find untested code that should be tested.

Where to go from here?

Congratulations! Now you should understand what a test is, why it matters and the coverage metric.

In the next chapter, you’ll find out what Test Driven Development (TDD) is and what the benefits are of writing tests before writing the feature. In the following chapters, you’ll also start writing apps with their corresponding tests.