You (probably) don’t need to test that

Like many mentors, instilling best practices is an important part of what I do. And I encourage my mentees to follow best practices, including testing. For example, I encourage them that code isn’t done until it’s been properly tested, and that the higher the test coverage, the better the outcome overall.

Yet even as an advocate for testing best practices, I’ve steadily learned that “well-tested” or even “completely tested” doesn’t always mean “100% test coverage.” And while this might seem a paradox, it’s not.

Consider the purpose for writing a test: you want to verify some predictable behavior, so that you can expect that behavior to continue well into the future of your application, regardless of changes you might make. For example, you know that adding two figures together should always produce a predictable sum; therefore, if you’re developing the PHP language, you want to know that 2+2 always equals 4. Moreover, you want to know if someone borked the math engine, and 2+2 now equals 65.

This is a valid goal, and the output is easily verifiable. Either the answer is 4, or it is not. Period. End of discussion.

Where this gets much more complex is when the output of a particular bit of code is variable, based on circumstance or input. For example, submtting a form can have any number of consequences, from validation errors to a successful submission. The faithful tester is responsible for knowing, and testing, all of these scenarios. And one of the easiest ways to do this is to use a mock.

A “mock” is an object or bit of code that we use to test discrete functionality, taking uncertain situations and making them certain. That way, we can know whether or not our code functioned under the expected conditions presented. This is really important, because we’re not asserting that our code works in every circumstance. We’re now asserting that it works when given the right inputs, and that it produces a consistent output.

This is a noble goal as well, because we can effectively test one unit of the system without invoking all the other units and dependencies of the system, thus ensuring that each unit works correctly. The result is a dramatic reduction in bugs and wasted developer time.

Where does this run into trouble? Most often we run into trouble when we choose to over-test. For example, imagine you have a value object, and this value object has a number of getters and setters. Each getter and setter simply sets or reads a property on the object, something that is verifiable simply by reading the code.

If we opt to test each getter and setter, we’re adding tons of tests to our code without adding much value. After all, what we’re really testing isn’t the behavior of our code; it’s the behavior of PHP. It can be argued that we are in fact verifying that we have the API we specified in the requirements; this is a reasonable point of view. But instead of testing each setter and getter’s behavior, we can test for the existence of the methods. Until we make the methods do something other than set a property, we’re not doing anything worth testing.

To make matters worse, imagine that for every use of our value object, we decide to use a mock in our tests. This carries with it a whole new set of problems. Mockery, the most popular framework for mocking objects in tests, rightly complains (and fails your test) if you call a method on the mock that you have not defined (unless you explicitly ask it not to). The idea is to have clear insight into what you’re testing. But changing your value object, and the subsequent code, will require a significant change to your tests to update your mocks. And if you miss even one, your tests will fail.

This is not desirable behavior. The real nirvana of unit testing is that we should be able to write our tests, have them pass, refactor our code, and have them pass again (without making changes to the tests). This is because we care that the output is consistent, regardless of our code changes. Having to change the tests every time we change the code doesn’t result in verifiable applications; they’re just as unstable as before, we’ve just added a layer of false security. And while we likely need to add tests for new features (in order to provide appropriate test coverage), the goal is that our other tests won’t require modification just because we’ve added some new feature. This is the ultimate goal of test-driven design and unit testing.

So what does this mean for your tests? First, it’s worth dispelling the notion that 100% code coverage is your ultimate goal. It’s not. 50% coverage with exceptional tests is far better than 100% coverage with crappy tests. Even The League of Extraordinary Packages, to which I have contributed code, aims for 80% code coverage in v1. It can go up from there, but 100% is not the stated goal.

In addition, it’s worth considering carefully how and when you make use of mock objects. Unit testing works best inside a vacuum where the only true dependency is the unit under test. In the real world, this isn’t always practical. And that’s okay. Integration tests can be powerful, too! Testing two objects in concert can reveal useful results. Where most developers run into problems with integration tests is by hitting their data store or other bottlenecks (a subject for another post), causing slow test results. Mocks have their place; use them! But think carefully about it.

Finally, consider whether or not the code you’re writing needs tests. A complex algorithm absolutely needs testing. A simple value object? Probably not as much (and what parts of it you need to examine can be done through integration testing anyway). Identifying which tests absolutely MUST be written can be challenging; a good book on the subject has been written by Chris Hartjes called Minimum Viable Tests and is worth a read for determining the most crucial elements of your application to test.

Testing is important, and doing it right can be challenging. But the reward for doing it well can be incredible.

Brandon Savage is the author of Mastering Object Oriented PHP and Practical Design Patterns in PHP

Steven Attevelt wrote at 2/17/2016 10:46 am:

Hi Brandon,

This quote resonated pretty for with me: ‘After all, what we’re really testing isn’t the behaviour of our code; it’s the behaviour of PHP.’. Thank you for sharing this.

I’d be looking forward to a blog post on considering where and when to use mocks in your tests and what kind of valuable information we can get from testing two objects in concert.

Thanks!

gggeek (@gggeek) wrote at 2/17/2016 3:37 pm:

Absolutely agree on the uselessness of testing all atomic setters and getters.

The crux of the matter is: a test should be written whenever a ‘simple’ get/setter changes its implementation, to make sure both that the results are still correct and that there are no unwanted side effects.
But making sure that devs add tests when refactoring is hard to extremely hard.
So the high cost of writing many useless tests upfront is often preferred…

Luca Tumedei (@lucatume) wrote at 2/18/2016 4:00 am:

Hi Brandon,

thanks for the article and the explicit statement that 100% coverage is more of a manic effort then a useful goal.

I got some mental guidelines I try to stick to when writing tests, beside the good principles you write about, and one is not to test output but to test the relations that will build it.

What’s your thought about it? Or anyone’s really.

Copyright © 2023 by Brandon Savage. All rights reserved.