About 1.3-1.5% additional successful-write time after b7a9d414 in a manual-WAL workload

I examined commit b7a9d414, comparing the unmodified commit with its immediate parent. In these measurements, a successful single-record write workload with manual WAL flushing took 1.315% longer [95% CI: 1.048%, 1.582%]. A separate check with hardware counters disabled in the same executables measured +1.466% [1.013%, 1.922%]. This is a constructed workload, not an exact reproduction of the original per-write sync=true case.

Environment and measurement

The machine has two Intel Xeon Gold 6430 CPUs, 64 physical cores and 128 logical CPUs, and runs Ubuntu 22.04, Linux 5.15.130-0515130-generic, glibc 2.35 and GCC 11.4. Data directories were on NVMe. The measured thread was pinned to logical CPU 17; the orchestration process used CPU 13. The governor was performance, Turbo was enabled, and frequency was not locked. This was a shared host, not an isolated machine; a running rshim service consumed approximately one CPU and was not stopped or repinned.

Both sides used unmodified static libraries built with the same Make options:

make -C "$SOURCE" -j32 static_lib DEBUG_LEVEL=0 PORTABLE=1 \
  DISABLE_WARNING_AS_ERROR=1 CC=gcc-11 CXX=g++-11

The library's release optimization level was -O2; the test programs used -O3 -DNDEBUG -fno-rtti -std=c++17 and linked against their respective revision's headers and static library. The tested snapshots identify themselves as 10.2.0. These are tested source revisions, not a claim that every release with those version numbers is affected.

After screening, the selected workload received 16 new confirmation pairs. Within each pair, the versions ran serially in alternating before/after and after/before order, each with a fresh database. Screening samples were not pooled with confirmation samples. Means below are arithmetic means; percentage changes are geometric means of paired after/before ratios, with two-sided 95% Student t intervals calculated on the log ratios. These intervals describe within-host repeatability, not variation across machines. All confirmation pairs are included in the attached CSV files.

Workload and results

• Before: be99011f08a042ad76859476b2263ce108ec3be8
• After: b7a9d414c8bbcc7a5dd9329e344a0451770ab555 (Fix WriteBatch atomicity and WAL recovery for some failures, #13489)
• Component: the successful DBImpl::WriteImpl path with WAL enabled.

The program prebuilds 1,024 single-record WriteBatch objects with k%09d keys and empty values, then cyclically submits them through 2,097,152 successful DB::Write calls. It uses VectorRep, allow_concurrent_memtable_write=false, a 1 GiB write buffer, no compression, disabled automatic compaction, manual_wal_flush=true and sync=false. There is one writer, no write pipeline and no separate WAL-flusher thread.

Timing includes the writes and a final FlushWAL(false). Batch construction, final FlushWAL(true), database close/reopen, WAL recovery and data verification are excluded. The original issue used sync=true on each Put; this constructed case changes the durability timing to expose the successful-write overhead, and the percentage should not be applied to the original synchronous workload.

Elapsed time and cycles increased in 16/16 pairs. The differences correspond to approximately seven additional instructions and 24 additional user-space cycles per write. Generic cache-miss and branch-miss counters did not show a corresponding consistent increase.

The hardware events were collected using a per-thread perf_event_open group around the timed workload, excluding kernel and hypervisor execution. Runs with a counter running/enabled ratio below 99% were rejected. In a separate experiment, the same executables ran with counters disabled: before averaged 1.143227 s, after 1.159999 s, for +1.466% [1.013%, 1.922%], with after slower in all ten blocks. This experiment used ten balanced blocks of five versions, including three diagnostic builds, with rotated and reversed positions. The stock-version comparison consists of 20 of those runs; it is not ten adjacent AB/BA pairs and is not pooled with the 16-pair confirmation.

Every trial checked all write statuses, exactly 2,097,152 active-memtable entries and sequence number 2,097,152, no immutable memtable, and the complete final set of 1,024 key/value pairs. The program then synchronized the WAL, closed without flushing the memtable, reopened, and rechecked the data and sequence number. Recorded WAL size was 67,108,864 bytes in every original-version trial. No timed memtable flush was allowed.

Preliminary attribution

The commit fixes important failure semantics, but it also changes the successful-write path by saving the previous WAL size and conditioning sequence-number publication on successful insertion. In the tested assembly, GetFileSize() is an ordinary atomic acquire load compiled to a memory mov, not a filesystem query or a new lock. The old status-check helper was already inlined, so an added out-of-line helper call is not the explanation here.

Separate diagnostic builds removed the size read or restored unconditional sequence publication. Removing the size read eliminated about four instructions per write but did not produce a reliable elapsed-time improvement. Restoring unconditional publication reduced user-space cycles by 0.403% [0.295%, 0.510%] relative to a no-change rebuild, but its elapsed-time interval included zero; with counters disabled, its elapsed-time interval also included zero. The no-change rebuild had identical object disassembly and identical final executable .text to the original after build.

This supports a contribution from the changed status-check/publication sequence in the cycle metric, not a complete explanation of the elapsed-time regression. The remaining cause is unresolved, and the diagnostic changes also alter code layout. Both diagnostic variants change failure semantics and are not proposed fixes or deployable versions. No diagnostic build is used for the stock-version regression estimates above.

The exact tested program is upstream-attachments-20260913/write_counters.cc, with its adjacent common.h. After building, these are separate single-trial commands; use a fresh $RUN for each:

# Hardware counters enabled; requires permission to open user-space perf events.
"$BINARY" "$RUN" 2097152 1024 0 1 0 manual 0 1

# Same executable, hardware counters disabled.
"$BINARY" "$RUN" 2097152 1024 0 1 0 manual 0 0

The program pins its writer to logical CPU 17 internally. CPU 17 must be available in the process's allowed CPU set. If a different CPU must be selected in common.h/the caller, that adaptation and the new environment should be recorded with the new measurements.

Building the attached test program

Use separate clean source trees for the before and after commits listed above. Set $SOURCE to the relevant checkout, $HARNESS to the attached .cc file and $BINARY to a distinct output executable. Keep common.h next to write_counters.cc. The tested link command was:

g++-11 -std=c++17 -O3 -DNDEBUG -fno-rtti -I"$SOURCE/include" \
  "$HARNESS" "$SOURCE/librocksdb.a" \
  -pthread -ldl -lrt -lsnappy -lz -lbz2 -lzstd -o "$BINARY"

Use matching development libraries on both sides; build metadata in the attachment records the detected features. Run versions serially and counterbalance their order, rather than comparing one before run with one after run. These commands reproduce the workload; recompilation on another machine is not expected to reproduce the archived executable hashes or identical percentages.

Attachments

rocksdb-b7a9d414-attachments-20260913.tar.gz

contains this commit's standalone reports and supporting material:

• The exact tested write_counters.cc and common.h, copied without modification.
• All 32 write-confirmation runs and all 100 five-version diagnostic runs with/without counters: 132 native trial records, including all 20 counter-disabled stock-version runs.
• Per-pair CSV files, original summary objects, independently recalculated statistics, commit/source evidence, original assembly, diagnostic build scripts and records, build provenance and environment information.
• A commit-specific provenance record, checksum manifest and evidence verification record.

The historical-scripts directory preserves original orchestration scripts as provenance; their historical paths are not prerequisites for running the C++ workload described here. This standalone bundle does not require another report or attachment, and excludes database directories and binaries. No sampled flamegraphs were collected for this case.