make check-zenmap reports success regardless of test results

make check-zenmap reports success regardless of test results

Component: Zenmap / build system Affects: all platforms Tested at: master b97fabd935362f85ad0575632e6d887f4d778b59

Summary

zenmap/test/run_tests.py runs the suite but never propagates the result to its exit status, so make check-zenmap exits 0 even when tests fail or modules fail to import.

To be precise, since the file does contain a sys.exit: line 10 calls sys.exit(0) inside the if not hasattr(unittest.defaultTestLoader, "discover") branch, which is unreachable on any Python 3. The final line is:

unittest.TextTestRunner().run(suite)      # test/run_tests.py:17

The returned TestResult is discarded, so the script falls off the end and exits 0.

Makefile.in:459-460 wires this into the target:

check-zenmap:
	@cd $(ZENMAPDIR)/test && $(PYTHON) run_tests.py

Reproduction

On a tree configured with Zenmap enabled (Configured with: ndiff zenmap zlib lua):

$ make check-zenmap
Ran 71 tests in 0.075s
FAILED (errors=44)
$ echo $?
0

Adding a deliberately failing test changes the report but not the status:

Ran 72 tests in 0.076s
FAILED (failures=1, errors=44)
$ echo $?
0

Exit 0 was observed for assertion failures, uncaught exceptions, ModuleNotFoundError and SyntaxError alike.

Fix

result = unittest.TextTestRunner().run(suite)
sys.exit(0 if result.wasSuccessful() else 1)

Scope, stated honestly

• No CI is being falsely greened. .github/workflows/build.yml never runs make check on any of its 14 legs; only the inactive .travis.yml does. The people affected are developers and packagers running make check by hand.
• check-zenmap only enters make check when Zenmap configures in, which requires the build interpreter to import both setuptools and build.
• Worth noting what the fix immediately surfaces: on a host without PyGObject the discovery step reports 44 import errors, which are silently ignored today. Fixing the exit status makes that visible, which is the point, but it does mean the target will start failing on machines where it previously "passed".
• There is one genuinely useful thing being thrown away at line 17: discovery imports every zenmapGUI, zenmapCore and radialnet module, so this is already a real import smoke test whose verdict is discarded.

Prior art

The project has already accepted this exact fix elsewhere. CHANGELOG:

o [Ncat] Made test-addrset.sh exit with nonzero status if any tests fail. This in turn causes "make check" to fail if any tests fail. [Andreas Stieger]

(commit 0cd8c9d6e579.) So "make check should fail when tests fail" is stated project intent, not my opinion. The sibling target already behaves correctly: ndiff/ndifftest.py ends in unittest.main(), which does propagate.

This also appears to be a regression with an identifiable origin: commit b75df9d8f9cb (2014-08-23) replaced python -m unittest discover — which exits nonzero — with run_tests.py, which does not.

Filed under the umbrella of #85 (Zenmap unit tests), which tracks test discovery rather than exit status. A comment on #85 shows a traceback ending in make: *** [zenmap_check] Error 1, which confirms make propagates the script's status correctly — the script is the only broken link.

Searched issues, PRs and commits for run_tests, TextTestRunner, wasSuccessful, zenmap unit test exit, test suite exit code, make check zenmap; and GitHub-wide code search for wasSuccessful path:zenmap. No existing report, and no distro appears to carry a downstream patch.

Tested on macOS 26 (arm64) only.