Salesforce developers, step up your game: how to improve test quality with mutation testing
Quite often I encounter low quality test classes where the developer is clearly just trying to achieve the coverage limit of Salesforce. Many great developers feel disappointed or even disgusted when working in such environment.
Test classes are very useful for the next developer who needs to maintain your program. Tests help to understand the requirements in a better way. Additionally, they help verifying that minor changes do not break functionalities. This concept is straightforward: the better the quality of your test suite, the better the chances of preventing a new bug.
A crucial problem occurs when a developer only tries to reach the highest coverage and forgets that tests are about the requirements. We write tests to automate the verification of programs. The program must behave as expected in all situations described in requirements. Therefore, test classes are written to avoid manually checking this.
You have probably found unverified and partially verified test classes before as well. If not, you are in an extreme rare and amazing environment. This year I presented mutation testing for the Belgian Developer Group and French Touch Dreamin’, where indeed most of the developers seem to recognize the problem.
If you are a Salesforce developer and you care about the quality of the work you deliver, this blog post is definitely worth reading.
Mutation testing is the process of modifying source code, in small ways with the purpose to find issues in tests.
Mutation testing is the way to stop bad tests from going to production. It is based on the hypothesis that developers create bugs by doing small syntactical errors. According to this hypothesis the types of changes introduced during this process are divided in three groups:
- Decision (changing comparison operators <, >, ==, >=, <= or logical operators ||, &&)
- Value (changing all “magical” numbers to some other random numbers)
- Statement (add, remove, or change statement order)
Mutation testing follows this process:
- Copies with small changes are generated from the source class.
- The test classes are run against these changed (mutated) code.
- If any of the tests fail, it means that the mutant is killed.
- If all tests pass, the mutant is still living. This means that the test suite is ineffective to find code bugs.
- The first and second steps are run for each class.
- At the end, the living mutants are analyzed and:
- The developer edits the tests so that the mutant is killed.
- Or chooses to ignore because it could either be an equivalent (the mutant does the same functionality as the original code) or the mutation is not very relevant. The developer thinks this kind of bug will never be introduced or has already reached the “mutation coverage” (a threshold percentage of mutants killed over the total number of mutants)
- Start again from 1 until quality threshold is met.
Mutation in Salesforce
Mutation testing is a computational expensive process but with the recent advance in computer power it is now easily performed. In Salesforce however, everyone is assigned certain “cloud power” governed by the Salesforce Limits. To run tests and change the class, Salesforce Tooling API must be used and there’s a Salesforce limitation on it. The number of mutants depends on the Lines Of Code (LOC), how much code is present compared to comments, number of operands and how many other operands it could be potentially mutated to.
Nevertheless, most mutation analysis will be done on projects smaller than 800 thousand classes. In the developed prototype we calculate 6 API calls for each mutant (call mutation engine, run test towards original, substitute class, run tests towards mutant, retrieve original and substitute back to original). Some of these API calls can be removed in a bulk call (for example the substitution to the original source code can be done only once after all mutants are tested). Since most companies have a limit of 5 million API calls on their sandboxes, mutation testing becomes possible in Salesforce too.
We surveyed the 4C developers and they all agree that a mutation testing would be very useful. You could also show your interest by voting on this idea. Meanwhile, we at 4C are working to make an application for mutation testing available for everyone. If you want more information, you can always contact us here.
And now let’s start slaying mutants ;)