At Yumasoft, we believe that testing is an inseparable part of software development. Mutation testing, which recently got a bit more attention in programming communities, is a great way of improving our tests. In today’s article, we’re exploring what mutation testing is and why everyone should be using it ?

What is mutation testing?

To make it clear, mutation testing isn’t just a new type of tests you can write. At least not in a casual sense of a test ?

Mutation testing is a technique for validating your already-existing tests. They basically allow to test our tests.

Mutation tests are not for people who don’t write tests (go away, we don’t like you!). They are for people who care about testing and want to make their tests better.

How does mutation testing work?

Mutation tests introduce mutations into our source code under tests (you can call it production code). Mutations can be any changes in the source code you can think of. Let it be a modification of a condition within an if statement, replacement of a string value or change of an arithmetic operator – you name it. All you need to know is that mutation testing is based on automatically introducing changes to your source code.

Our codebase with such a change (a single one) introduced is called a mutant. It makes sense – our production code is now somehow mutated.

What happens with such a mutant then? All our existing tests are executed against it. Such tests’ execution may produce two outcomes:

• all tests have passed – that’s a bad scenario. It means that our mutant has survived. In other words, it means that a modification of our production source code wasn’t detected by our tests suite
• at least one test has failed – that’s a good scenario. It means that our mutant has been killed. This is what we want, because it means our tests are well-covering and detect changes in the production code.

You can see the flow of mutation testing on the following schema:

Example of mutation tests

To make mutation tests more practical, let’s see an example. We’ll be using StrykerJS for production code written in TypeScript.

This is our production code:

We have of course written some tests for this code. When we execute these tests with jest, we can see the following result:

So we have 100% coverage for DiscountApplier class! What’s more, we have 100% statements, branch, functions and line coverage! This sounds perfect, right? Why should we bother more? ?

Let’s deep dive and run StrykerJS mutator against our app. After installing and configuring Stryker properly, we can execute npx stryker run command. That’s the final result:

Whoooaa… we see here some 82.14% value, not 100% anymore ? This is what I call mutant coverage.

Remember what we said above about survived and killed mutants? Pay attention to our mutations results above – we have 46 mutants killed, but 10 of them survived. It means that we have a potential to add more tests to cover these scenarios.

We will not be going through each mutant and adding the test for it in this article, but I can tell you what these results can give you. In the beginning, when our mutant coverage was 82.14% with 10 mutants survived, we had 6 unit tests written for DiscountApplier class. In the end, after working out and killing each mutant one-by-one, we finished with 12 unit tests and 100% mutant coverage.

In 10 minutes (time needed to add these 6 tests), I got twice as many unit tests as in the beginning. Now, both “traditional” and mutant coverages are 100%. My confidence in refactoring or modifying the production code is much higher now. Notice that we got this all “for free”. Mutation testing tool (StrykerJS in that case) did everything for us. This is huge! ?

How to start writing mutation tests

Just find a proper mutation testing tool for the framework/programming language you work with. Here are a few most popular examples:

• JavaScript/TypeScript/C#/Scala: Stryker mutator
• Ruby: mutant
• Java: Pitest
• PHP: Infection

Go straight to the docs of your tool of choice and start mutating ? The messages about each mutant are very clear, so adding new tests to kill them will be very easy. It’s like someone sitting next to you and telling you exactly what test you should add for your code. That’s actually a pretty decent solution for anyone who doesn’t like writing tests ?

I don’t see any reason why anyone would not use mutation testing. It’s such a great tooling that I don’t see any disadvantages in using it. It can greatly help you with TDD as well as an additional “layer” of inventing new test cases and patching your code. Believe me, you’ll see the benefits straightaway.

So – download one of the mutation tools and make your tests bulletproof today ? Or at least see how bad they are now ?

The post Make Your Tests Bulletproof With Mutation Testing appeared first on Yumasoft.

Jest config

First, make sure that your jest configuration is correct. If you’re with TypeScript (and you should ?), I recommend using ts-jest configuration preset. Our config looks as follows:

If everything is fine, you should be able to execute npm test command (assuming it’s defined in package.json as "test": "jest") without issues. All your tests should be executed.

We now need to make StrykerJS working with that.

StrykerJS with jest configuration

If you install StrykerJS as described here, you’d expect that npx stryker run command runs mutations and everything is just fine. However, if you’re having wierd issues and errors with that, it probably means that Stryker is not properly configured to execute tests with jest.

When you run npx stryker run, it creates mutants (modifications in your source code) and runs tests against them. It means that StrykerJS must run your tests in the same way you run them. So, if you run tests with jest, Stryker must be able to do the same.

We struggled with many issues related to running StrykerJS with jest. Some of them were related purely to jest, but some of them to TypeScript as well. After numerous trial-and-errors and consulting with StrykerJS community members, we worked out a configuration that finally works fine for us.

Going straight to the point, here’s the stryker.conf.json file that would make StrykerJS work fine with jest and TypeScript:

To make it work, install the following npm packages: @stryker-mutator/core, @stryker-mutator/jest-runner and @stryker-mutator/typescript-checker.

Notice few important settings here:

• "checkers": ["typescript"] – this enabled a TypeScript checker plugin, which makes mutations being checked for types before running tests on them. It means that if a mutation introduced by Stryker doesn’t compile because of types issues, it won’t be taken into account for mutations (which makes perfect sense)
• “disableTypeChecks” – you still need to tell StrykerJS to not check types when compiling your code with type errors (mutants). It will compile them without types checking, but thanks to TypeScript checker plugin these ones will be filtered out for mutations
• "tsconfigFile": "tsconfig.json" – make sure to point to the TS config file you’re using here
• "concurrency": 4: this is another setting we experimented with to make Stryker working correctly. Without it, we were getting weird errors about lack of memory when running mutations (FATAL ERROR: Committing semi space failed. Allocation failed - JavaScript heap out of memory)
• "testRunner": "jest" – this one enables the jest as our tests runner
• "jest" – here you tell Stryker to use your jest.config.js file we talked about at the beginning

With these settings, I hope that you are able to see this beautiful output of npx stryker run in your console:

StrykerJS with jest and TypeScript – summary

I hope this is helpful for you. StrykerJS is a developing framework, so some things are still hard to figure out. We believe mutation tests are a great way of improving our tests, so we use them on a daily basis. If you ever have any issues with StrykerJS, I encourage you to reach out to the community on Slack. They are extremely helpful there ?

The post How To Use StrykerJS With jest And TypeScript appeared first on Yumasoft.