Recorder stores each mock's payload twice — mocks files are ~2.7× larger than the traffic
Summary
Every recorded mock stores its payload twice, so mocks files are roughly 2.7× larger than the traffic they represent. On the recording that triggered keploy/api-server#2136, a 10.44 MB Mongo message became a 28.31 MB line on disk.
Measured
Test set test-set-597c645d86-api-server-v2-staging, line 66 (mock-65):
Mongo wire message (header.length) = 10,435,560 bytes (10.44 MB) ← what the app actually sent
stored JSON line = 28,308,812 bytes (28.31 MB)
blowup = 2.71×The duplication is exact:
spec.requests[0].message = 12,581,163 bytes
spec.metadata.operation = 12,581,157 bytes ← the same payload, stringifiedspec.metadata.operation is a byte-for-byte copy of the request message, held as a string. The remaining ~2.7 MB is JSON-escaping of BSON.
The mock itself is ordinary: one OP_MSG, an update on entity_versions carrying 71,483 statements ($max: {current_version}, all entity_type: test_suite) — one round trip instead of 71k, which is correct Mongo usage.
Why it matters
Halving this roughly halves every production mocks bundle, which is the single largest lever on:
- object-storage cost and egress
- upload time from the customer's cluster
- download and parse time at replay
- how close a bundle gets to the nginx
proxy-body-size: 100mingress ceiling
It is also what pushed that recording past the old 10 MB per-line scanner limit. The app was only 4% over; our own storage format tripled it. The streaming work in #2136 removes the limit, so this is no longer a correctness issue — it is now purely cost and latency.
Scope
Recorder-side, independent of the upload path. The consumers of spec.metadata.operation need auditing first: if it exists only for display or debugging, it can be derived from spec.requests[0].message on read rather than stored. If something depends on the stringified form, the cheaper fix is to stop storing the parsed copy instead.
Not urgent
No correctness impact once keploy/api-server#2136 lands. Filing it so the measurement isn't lost — it was found while diagnosing that bug and is easy to re-discover the hard way.
Source: keploy/keploy