Key becomes unreachable after recovery when its record was reused via revivification
Describe the bug
After a restart, a key can end up in a state where KEYS and SCAN enumerate it and DBSIZE counts it, but EXISTS returns 0 and GET returns nil.
It happens when the key's record was reused via revivification and a checkpoint was taken afterwards:
- write a key
- delete the key
- checkpoint
- write key again
- checkpoint
- restart Garnet
- get key returns nil
So, reproducible after five commands from the CLI and a restart.
Steps to reproduce the bug
Environment
- Garnet 2.1.6, 64-bit, standalone mode
- Persistence enabled with revivification on (config below)
Configuration
{
"EnableAOF": true,
"EnableRevivification": true,
"Recover": true,
"CheckpointDir": "C:/Program Files/Garnet/checkpoints",
}The following steps conducted with an empty cache:
Reproduction
> SET a 1
"OK"
> DEL a
(integer) 1
> SAVE
"OK"
> SET a 2
"OK"
> SAVE
"OK"
> EXISTS a
(integer) 1Restart Garnet, then:
> EXISTS a
(integer) 0
> GET a
(nil)
> KEYS *
1) "a"
> DBSIZE
(integer) 1EXISTS returns 1 before the restart and 0 after it. KEYS and DBSIZE disagree with EXISTS and GET on whether the key exists.
Writing the key again repairs it. So, a subsequent SET a 3 followed by GET a returns the new value normally.
Reviv Info shows the record was reused
Immediately after SET a 2 in the repro sequence above:
> INFO STOREREVIV
# StoreDeletedRecordRevivification_DB_0
Successful Adds: 1
Failed Adds: 0
Successful Takes: 1
Failed Takes: 0
Empty bins: 0
Address limit: 0
Record size limit: 0
Successful In-Chain: 0
Failed In-Chain: 0Successful Adds: 1 from the DEL and Successful Takes: 1 from the SET, so the new record occupies the freed record via the free record pool.
The key also survives the restart if you don't do the second SAVE (after the re-add); as follows:
SET a 1
DEL a
SAVE
SET a 2
[no checkpoint here]
>>> restart <
> EXISTS a
(integer) 1So recovery from AOF replay alone reconstructs the key correctly. The problem appears only when a checkpoint is taken after the record has been reused.
Also observed with two different keys
The same issue occurs when the reused record belongs to a different key:
SET a 1
DEL a
SAVE
SET b 2
SAVE
>>> restart <
> EXISTS b
(integer) 0
> KEYS *
1) "b"Expected behavior
After recovery, a key that existed at checkpoint time is reachable by GET, EXISTS, and other point lookups, consistent with KEYS, SCAN, and DBSIZE.
Screenshots
No response
Release version
v2.1.6
IDE
No response
OS version
Windows Server 2025
Additional context
In our actual deployment AofSizeLimit is configured, so checkpoints fire automatically every few minutes, and the workload deletes and recreates keys continuously, so records are reused constantly. Any restart in that state leaves keys unreachable, and so we hit this issue routinely while testing recovery. We had assumed it was a consequence of #2101, but the repro steps above show these are distinct issues.
Source: microsoft/garnet