Question: configuration, tooling and reproduction script behind the Performance documentation numbers
Hi FoundationDB team,
First, thank you for maintaining such a well-documented project. We've been benchmarking FoundationDB on our own hardware and are trying to line our results up against the numbers published on the official Performance page (https://apple.github.io/foundationdb/performance.html), but we're having trouble reproducing them and would be grateful for some clarification.
The page documents the hardware clearly (e.g. 8.2M ops/sec on a 24-machine EC2 c3.8xlarge cluster, 384 server processes, 90% read / 10% write, 16-byte keys and 8-100 byte values; and the latency figures on a 12-machine E3-1240 / SATA SSD cluster with 48 processes). However, a few details that matter for reproduction don't seem to be covered:
- Benchmark tool - Which tool was used to generate the load? Was it a workload defined inside FDB's simulation/test harness,
mako, a custom client, or something else? - Client-side configuration - How many load-generating client machines/processes/threads were used, and what concurrency (in-flight transactions) per client? Ops/sec is very sensitive to this and it isn't stated.
- Cluster configuration - What redundancy mode and storage engine (ssd/ssd-2/redwood) were used, and were any non-default knobs set?
- FDB version - Which release were these numbers measured on?
- Reproduction script - Is there a script or a documented procedure we could run to reproduce these results end-to-end? If one exists in the repo, a pointer would be very helpful.
Our goal isn't to dispute the numbers - we just want to make sure we're measuring the same thing so our own comparisons are fair. Any pointers would be much appreciated. Thanks!
Source: apple/foundationdb