[BUG] Replay cannot reproduce concurrent address updates or mandate.created_at
Two values a replay cannot reproduce against the build that recorded them. Neither is a clock or entropy read that a seam was missing; both are places where the router leaves a fact implicit that a comparison has to be able to see.
1. Concurrent forks share a span path
Several payment operations spawn futures side by side and join them — the try_join! groups the code itself labels "Parallel calls - level 1" and "level 2". All were instrumented .in_current_span(), so every fork shares the caller's span path.
That matters because several forks call the same helper. Shipping and billing address updates both enter the same function, from the same call site, under the same span path — separable only by the order the scheduler happened to choose.
A record/replay comparison keys calls on span path. Two calls from different forks therefore collapse to one key, and the only thing left to pair them by is position. Position is exactly what is not stable here. When the two land in the other order, each address update is read against the other's row and a single transposition is reported as two behaviour changes, against a candidate that did nothing wrong.
Naming each fork states the concurrency that is already there. It does not change what the code does; it makes the existing parallelism visible to anything reading the trace.
2. mandate.created_at is left to the database
MandateNew.created_at is Option<PrimitiveDateTime>, and the mandate table carries DEFAULT now().
diesel_models/src/mandate.rs has a fallback that looks like it covers this:
impl From<&MandateNew> for Mandate {
created_at: mandate_new.created_at.unwrap_or_else(common_utils::date_time::now),It does not, on the path that matters. router/src/db/mandate.rs branches on the storage scheme:
MerchantStorageScheme::PostgresOnly => mandate.insert(&conn), // MandateNew goes to diesel directly
MerchantStorageScheme::RedisKv => { /* builds Mandate::from(&mandate), running the unwrap_or_else */ }MandateNew::insert() calls generics::generic_insert(conn, self), which hands the MandateNew straight to diesel. The From impl — and its unwrap_or_else — runs only on the Redis KV branch.
So a KV-scheme merchant gets a timestamp the application chose and a PostgresOnly merchant gets one postgres chose, from the same code. On the Postgres path the value is a clock read that happens inside the database after the statement leaves the process, which no application-side instrumentation can observe and no replay can reproduce. A fallback inside one storage path is not a default for the type.
Why this is worth fixing rather than tolerating
Both surface as value divergences on a self-replay — the same image replaying its own recording, which should be zero-diff. Every such divergence that is not a real behaviour change is noise a reviewer has to learn to ignore, and the ones worth acting on get harder to see for it.
The same shape exists elsewhere and is not covered here: generic_link.created_at / last_modified_at (set via ..Default::default() in payouts and payment-method collect links) and file_metadata.created_at (absent from the insert struct entirely) are the same defect on paths that a replayed payment does not currently exercise.
Source: juspay/hyperswitch