Mutation testing for Go source code
go-mutesting is a framework for performing mutation testing on Go source code. Its main purpose is to find source code, which is not covered by any tests.
The following command mutates the go-mutesting project with all available mutators.
go-mutesting github.com/zimmski/go-mutesting/...The execution of this command prints for every mutation if it was successfully tested or not. If not, the source code patch is printed out, so the mutation can be investigated. The following shows an example for a patch of a mutation.
for _, d := range opts.Mutator.DisableMutators {
pattern := strings.HasSuffix(d, "*")
- if (pattern && strings.HasPrefix(name, d[:len(d)-2])) || (!pattern && name == d) {
+ if (pattern && strings.HasPrefix(name, d[:len(d)-2])) || false {
continue MUTATOR
}
}The example shows that the right term (!pattern && name == d) of the || operator is made irrelevant by substituting it with false. Since this change of the source code is not detected by the test suite, meaning the test suite did not fail, we can mark it as untested code.
The next mutation shows code from the removeNode method of a linked list implementation.
}
l.first = nil
- l.last = nil
+
l.len = 0
}We know that the code originates from a remove method which means that the mutation introduces a leak by ignoring the removal of a reference. This can be tested with go-leaks.
The definition of mutation testing is best quoted from Wikipedia:
Mutation testing (or Mutation analysis or Program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants.
-- https://en.wikipedia.org/wiki/Mutation_testing
Tests can be created to verify the correctness of the implementation of a given software system, but the creation of tests still poses the question whether the tests are correct and sufficiently cover the requirements that have originated the implementation.
-- https://en.wikipedia.org/wiki/Mutation_testing
Although the definition states that the main purpose of mutation testing is finding implementation cases which are not covered by tests, other implementation flaws can be found too. Mutation testing can for example uncover dead and unneeded code.
Mutation testing is also especially interesting for comparing automatically generated test suites with manually written test suites. This was the original intention of go-mutesting which is used to evaluate the generic fuzzing and delta-debugging framework Tavor.
go-mutesting includes a binary which is go-getable.
go get -t -v github.com/zimmski/go-mutesting/...The binary's help can be invoked by executing the binary without arguments or with the --help argument.
go-mutesting --helpNote: This README describes only a few of the available arguments. It is therefore advisable to examine the output of the
--helpargument.
The targets of the mutation testing can be defined as arguments to the binary. Every target can be either a Go source file, a directory or a package. Directories and packages can also include the ... wildcard pattern which will search recursively for Go source files. Test source files with the suffix _test are excluded, since this would interfere with the testing process most of the time.
The following example gathers all Go files which are defined by the targets and generate mutations with all available mutators of the binary.
go-mutesting parse.go example/ github.com/zimmski/go-mutesting/mutator/...Every mutation has to be tested using an exec command. By default the built-in exec command is used, which tests a mutation using the following steps:
Alternatively the --exec argument can be used to invoke an external exec command. The /scripts/exec directory holds basic exec commands for Go projects. The test-mutated-package.sh script implements all steps and almost all features of the built-in exec command. It can be for example used to test the github.com/zimmski/go-mutesting/example package.
go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/exec/test-mutated-package.sh" github.com/zimmski/go-mutesting/exampleThe execution will print the following output.
Note: This output is from an older version of go-mutesting. Up to date versions of go-mutesting will have different mutations.
…5b1ca0cfedd786d9df136a0e042df23a
The blacklist file, which is named `example.blacklist` in this example, can then be used to invoke go-mutesting.
```bash
go-mutesting --blacklist example.blacklist github.com/zimmski/go-mutesting/exampleThe execution will print the following output.
Note: This output is from an older version of go-mutesting. Up to date versions of go-mutesting will have different mutations.
PASS "/tmp/go-mutesting-208240643/example.go.0" with checksum b705f4c99e6d572de509609eb0a625be
PASS "/tmp/go-mutesting-208240643/example.go.1" with checksum eb54efffc5edfc7eba2b276371b29836
PASS "/tmp/go-mutesting-208240643/example.go.2" with checksum 011df9567e5fee9bf75cbe5d5dc1c81f
--- example.go 2014-12-29 23:37:42.813320040 +0100
+++ /tmp/go-mutesting-208240643/example.go.3 2014-12-30 00:49:33.573285038 +0100
@@ -16,7 +16,7 @@
}
if n 2 | The mutation produced an unknown exit code which might be a flaw in the exec command. |
Examples for exec commands can be found in the [scripts](/scripts/exec) directory.
## Which mutators are implemented?
### Branch mutators
| Name | Description |
| :------------ | :------------------------------------------------- |
| branch/case | Empties case bodies. |
| branch/if | Empties branches of `if` and `else if` statements. |
| branch/else | Empties branches of `else` statements. |
### Expression mutators
| Name | Description |
| :-------------------- | :--------------------------------------------- |
| expression/comparison | Searches for comparison operators, such as `>` and `` is replaced by `>=`. |
| expression/remove | Searches for `&&` and \|\| operators and makes each term of the operator irrelevant by using `true` or `false` as replacements. |
### Statement mutators
| Name | Description |
| :------------------ | :--------------------------------------------- |
| statement/remove | Removes assignment, increment, decrement and expression statements. |
## How do I write my own mutators?
Each mutator must implement the `Mutator` interface of the [github.com/zimmski/go-mutesting/mutator](https://godoc.org/github.com/zimmski/go-mutesting/mutator#Mutator) package. The methods of the interface are described in detail in the source code documentation.
Additionally each mutator has to be registered with the `Register` function of the [github.com/zimmski/go-mutesting/mutator](https://godoc.org/github.com/zimmski/go-mutesting/mutator#Mutator) package to make it usable by the binary.
Examples for mutators can be found in the [github.com/zimmski/go-mutesting/mutator](https://godoc.org/github.com/zimmski/go-mutesting/mutator) package and its sub-packages.
## Other mutation testing projects and their flaws
go-mutesting is not the first project to implement mutation testing for Go source code. A quick search uncovers the following projects.
- https://github.com/darkhelmet/manbearpig
- https://github.com/kisielk/mutator
- https://github.com/StefanSchroeder/Golang-Mutation-testing
All of them have significant flaws in comparison to go-mutesting:
- Only one type (or even one case) of mutation is implemented.
- Can only be used for one mutator at a time (manbearpig, Golang-Mutation-testing).
- Mutation is done by content which can lead to lots of invalid mutations (Golang-Mutation-testing).
- New mutators are not easily implemented and integrated.
- Can only be used for one package or file at a time.
- Other scenarios as `go test` cannot be applied.
- Do not properly clean up or handle fatal failures.
- No automatic tests to ensure that the algorithms are working at all.
- Uses another language (Golang-Mutation-testing).
## Can I make feature requests and report bugs and problems?
Sure, just submit an [issue via the project tracker](https://github.com/zimmski/go-mutesting/issues/new) and I will see what I can do. Please note that I do not guarantee to implement anything soon and bugs and problems are more important to me than new features. If you need something implemented or fixed right away you can contact me via mail to do contract work for you.No open issues yet, or sync has not completed.