Coverage job reports a number that varies by ~2000 lines between runs of the same code

Author: DennisOSRMCreated Aug 27, 2026Updated Aug 27, 2026

Issue

The coverage job reports a number that moves by about 2000 lines between runs of the same code. #7705 tried to fix this by running the cucumber suite serially where coverage is measured. That did not work, and the variance is still there.

Data

Lines covered, as reported by lcov at the end of clang-20-debug-cov:

Commit Cucumber Lines covered Of
adf0e04c parallel 77716 96461
adf0e04c re-run, identical tree parallel 75623 96461
0adc8be parallel 75608 96461
0adc8be re-run, identical tree parallel 75622 96461
6214e910 serial 75600 96461
6214e910 re-run, identical tree serial 75597 96461
932a3aa (#7704) serial 77934 96748
985f45c0 (#7704) serial 75934 96795

Most runs land near 75600. Occasionally one lands near 77800, about 2100 lines higher. That happens under both settings, so parallelism was not the cause.

The last two rows are not quite the same tree, they differ by 58 lines of unit tests, but that cannot account for 2000 lines and the run with more tests is the lower one.

What has been ruled out

  • Parallel writes to shared .gcda files. This was the theory behind #7705. Serial runs still produce both the low and the high value.
  • A different amount of work between runs. Both #7704 runs completed all six matrix combinations with identical scenario counts, 1482 each, and ran all 14 unit test binaries.
  • Processes dying before flushing coverage. features/lib/osrm_loader.js:63 sends SIGTERM and waits for close, and src/tools/routed.cpp:365 blocks and handles SIGTERM, so osrm-routed exits cleanly.

Why it matters

codecov/project compares against whichever sample the base commit happened to draw. When the base drew a high one, every later pull request looks like a 2.9% regression. This happened on #7702, where the only change in the commit that "regressed" was a 25 line cucumber feature file containing no C++ at all.

The status is informational since #7705, so it no longer blocks. It should stay that way until the number is reproducible.

Suggested next step

Take a high run and a low run and diff their per-file coverage, rather than the totals. Roughly 2100 lines is a large and specific amount, so it is likely one or two files or one subsystem appearing and disappearing rather than noise spread thinly. That would localise it.

Also worth considering: the serialisation added in #7705 costs wall clock on the coverage job and has not been shown to buy anything. Unless it turns out to help for another reason, it can be reverted.

Requirements / Relations

Follows #7705. Noticed on #7702 and again on #7704.

Source: Project-OSRM/osrm-backend