Incrementally add mutation testing to SFML workflows
Prerequisite Checklist
- I searched for existing issues to prevent duplicates
- I searched for existing discussions on the forum to prevent duplicates
- I understand SFML's scope and believe the feature fits it
Describe your feature request here
Mutation testing is a new testing technique brought by Mull, an LLVM-based tool. It basically modifies the source code (through the AST) in order to check if tests fail.
In the case that all tests still pass for some mutation, that means a mutant has escaped and a new test should be added to kill that mutant. See their docs for more info.
This feature currently doesn't require modifications to the build system, as all you need to do is add a few compiler flags to Clang (only compiler that works with this feature). Most of the work would be done in the GitHub actions, but that shouldn't require a lot of work.
I've done some tests. For instance, if we use Mull on the test test-sfml-system, we get the following output:
$> mull-runner-15 ./bin/test-sfml-system
[info] Using config /home/francisco/Mis_Programas/SFML/mull.yml
[warning] Could not find dynamic library: libstdc++.so.6
[warning] Could not find dynamic library: libm.so.6
[warning] Could not find dynamic library: libgcc_s.so.1
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 121ms
[info] Filter mutants (threads: 1)
[################################] 1/1. Finished in 1ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 121ms
[info] Running mutants (threads: 15)
[################################] 15/15. Finished in 141ms
[info] Survived mutants (8/15):
/home/francisco/Mis_Programas/SFML/include/SFML/System/Angle.inl:65:54: warning: Survived: Replaced + with - [cxx_add_to_sub]
return radians(priv::positiveRemainder(m_radians + priv::pi, priv::tau) - priv::pi);
^
/home/francisco/Mis_Programas/SFML/include/SFML/System/Utf.inl:322:52: warning: Survived: Replaced + with - [cxx_add_to_sub]
output = ((first - 0xD800u) << 10) + (second - 0xDC00) + 0x0010000;
^
/home/francisco/Mis_Programas/SFML/include/SFML/System/Utf.inl:322:72: warning: Survived: Replaced + with - [cxx_add_to_sub]
output = ((first - 0xD800u) << 10) + (second - 0xDC00) + 0x0010000;
^
/home/francisco/Mis_Programas/SFML/src/SFML/System/String.cpp:168:37: warning: Survived: Replaced + with - [cxx_add_to_sub]
m_string.reserve(length + 1);
^
/home/francisco/Mis_Programas/SFML/src/SFML/System/String.cpp:178:42: warning: Survived: Replaced + with - [cxx_add_to_sub]
m_string.reserve(ansiString.length() + 1);
^
/home/francisco/Mis_Programas/SFML/src/SFML/System/String.cpp:191:37: warning: Survived: Replaced + with - [cxx_add_to_sub]
m_string.reserve(length + 1);
^
/home/francisco/Mis_Programas/SFML/src/SFML/System/String.cpp:201:42: warning: Survived: Replaced + with - [cxx_add_to_sub]
m_string.reserve(wideString.length() + 1);
^
/home/francisco/Mis_Programas/SFML/src/SFML/System/String.cpp:379:35: warning: Survived: Replaced + with - [cxx_add_to_sub]
pos = find(searchFor, pos + step);
^
[info] Mutation score: 46%
[info] Total execution time: 386ms
[info] Surviving mutants: 8We can do the same with the rest of the testers to obtain multiple results.
It's important to note that this tool is still quite experimental and the results can be overwhelming if not done incrementally (which is also supported), but it's an interesting thing to keep in mind in order to improve the robustness of the library.
Use Cases
New workflows could be added to the CI to incrementally check for new mutants that have escaped, which would hint new test cases to be added.
API Example
No response
Source: SFML/SFML