Test suite: reduce runtime — 52 % is Unicode byte sweeps, plus redundant re-parsing in binary roundtrip tests
Description
Measured with GCC 13, default (unoptimized) build, ctest --no-skip (what the Linux gcc/clang/coverage/sanitizer CI jobs run): 562 s CPU total, extremely concentrated:
| test | time | share |
|---|---|---|
| test-unicode4_cpp11 | 178 s | 32 % |
| test-msgpack (cpp11+cpp17) | 118 s | 21 % |
| test-cbor_cpp11 | 54 s | 10 % |
| test-unicode3_cpp11 | 54 s | 10 % |
| test-unicode5_cpp11 | 42 s | 7 % |
| test-binary_formats_cpp11 | 30 s | 5 % |
| test-ubjson + test-bjdata | 30 s | 5 % |
Note doctest::skip() only helps JSON_FastTests=ON runs (Windows/macOS/Cirrus); the Linux CI jobs pass --no-skip and run everything, so only real iteration cuts help there.
Where the time goes, and proposed reductions
1. Unicode "wrong Nth byte" sections sweep all combinations of the other bytes. unicode4 runs 5,517,507 iterations (its own progress counter, unit-unicode4.cpp:126); e.g. "wrong 2nd byte" is 3·192·64·64 = 2.36 M iterations. Each iteration calls both check_utf8string (1 × json::parse) and check_utf8dump (8–9 × dump() across error-handler/ensure_ascii combinations) — roughly 50 M dump() calls suite-wide. The property tested ("byte N out of range ⇒ error") is independent of the other trailing bytes. Pinning unrelated bytes to one representative value removes ~73 % of the 8.9 M unicode iterations; calling check_utf8dump only on well-formed sweeps (or a sample of ill-formed ones) cuts ~8× more. Must be coordinated with the byte3/byte4 typo fix (#5416), which would otherwise add 3.1 M iterations.
2. The binary * roundtrips test cases parse every corpus file 4×. In unit-cbor.cpp, unit-msgpack.cpp, unit-ubjson.cpp, unit-bjdata.cpp, the per-file loop body has four sibling blocks (std::vector<uint8_t> / std::ifstream / uint8_t* and size / output adapters) and each re-opens and re-parses the same JSON file, and re-reads the binary 3×. Hoisting const json j1 = json::parse(f_json); and the binary read above the four blocks is mechanical and cuts ~75 % of the parse work. (all_unicode.json, a 1.1 M-element file, is currently parsed 9× across the suite.)
3. Exhaustive 16-bit integer sweeps run in every configuration, including FastTests. for (i = 256; i <= 65535; ++i)-style loops at unit-cbor.cpp:293,481,616,721, unit-msgpack.cpp:258,443,649, unit-ubjson.cpp:267,427,633, unit-bjdata.cpp:420,580,913 — ~750 k iterations, each doing encode + decode ×2 + ~8 CHECKs; plus two 65,536-key object builds with a std::stringstream per key (unit-cbor.cpp:1426, unit-msgpack.cpp:1142). Boundary values plus a stride (e.g. i += 7, which still hits every low-byte residue) prove the same encoding-width selection; the exhaustive versions could live in skip()-decorated sibling TEST_CASEs if desired. Also worth checking: unit-msgpack.cpp gained JSON_HAS_CPP_17 and therefore runs (and compiles) twice — the 65 k sweeps probably don't need to re-run at C++17.
4. Smaller items
unit-unicode1.cpp:26:codepoint_to_unicodeconstructs astd::stringstreamper call — ~1.16 M constructions across the 0..0x10FFFF sweep. Manual hex formatting cuts this section several-fold. The escape/unescape loop overall_unicode.json(line 171) testsdetail::escape/unescape, not Unicode — sampling every 64th element gives identical coverage.unit-large_json.cpp:20:depth = 5000000builds and destroys a ~5 M-node tree, unskipped in every run (6.7 s). 500,000 proves the same #1419 non-recursion property at a tenth of the cost.unit-binary_formats.cpp:136: the 52 MBjeopardy.jsonblock produces ~500 MB of serialization output; splitting it into its ownskip()ed TEST_CASE lets the four cheap corpus files keep running everywhere.
Items 1–3 together should cut the --no-skip suite runtime by well over half without losing meaningful coverage.
Source: nlohmann/json