Replicated servers mode resends the RLock acquire script after a response timeout; one tryLock ends with hold count 2 and the lock is never released
Redis version
7.2.16
Redisson version
3.52.0
Redisson configuration
Config config = new Config();
config.useReplicatedServers()
.addNodeAddress("redis://<endpoint>:6379")
.setReadMode(ReadMode.MASTER)
.setTimeout(3000)
.setRetryAttempts(4);
// lockWatchdogTimeout default 30000
RLock lock = redisson.getLock("mylock");
if (lock.tryLock(10, TimeUnit.SECONDS)) {
try { ... } finally { lock.unlock(); }
}What is the Expected behavior?
The lock acquire script runs at most once on Redis per tryLock() call. If Redis answers after timeout, either the call fails with RedisResponseTimeoutException (as it does in single server mode) or it succeeds with getHoldCount() == 1. One unlock() then releases the lock.
What is the Actual behavior?
In replicated servers mode, when Redis answers the acquire EVALSHA later than timeout, Redisson cancels the attempt and sends the same EVALSHA again after the retry delay. Redis executes the script twice. The second execution finds the hash field <clientId>:<threadId> already present and takes the reentrant hincrby branch. One tryLock() returns true with getHoldCount() == 2.
The caller calls unlock() once, the hash field stays at 1 and the key stays locked. Every later tryLock() by the same thread is reentrant and refreshes pexpire, so as long as the owning thread keeps using the lock (a scheduler thread in our case) the phantom hold never expires. All other threads time out on tryLock() until the key is deleted by hand. We had this in production for several hours.
redis-cli MONITOR during the reproducer below, one tryLock() on the Java side:
1790065841.836655 [0 172.17.0.1:56760] "EVALSHA" "149c5a8bf18676fd99e9e6e24923bd64d9b62f94" "1" "[test]f20c96e9-..." "30000" "92c2ec07-...:1"
1790065843.153746 [0 172.17.0.1:56760] "EVALSHA" "149c5a8bf18676fd99e9e6e24923bd64d9b62f94" "1" "[test]f20c96e9-..." "30000" "92c2ec07-...:1"Same script, same key, same <clientId>:<threadId>, same connection, 1.3 s apart (timeout 1000 ms + retry delay 200 ms + round trip). HGETALL of the key afterwards returns the field with value 2.
The same test with useSingleServer() behaves as expected: tryLock() throws RedisResponseTimeoutException, the EVALSHA is sent once, the hash holds at most 1.
Additional information
Where I believe the resend comes from. RedissonLock.tryLockInnerAsync goes through RedissonBaseLock.evalWriteSyncedNoRetryAsync → CommandAsyncService.syncedEvalNoRetry, so the intent is clearly that the acquire script is never resent. The retry=false flag is honoured in the single server branch but lost in the batch branch:
CommandAsyncService.syncedEval3.52.0, L1062-L1073:isSingleConfig()→evalWriteNoRetryAsyncwhenretryis false.- same method, L1164-L1165: all other modes create a
CommandBatchServiceand callexecutorService.evalWriteAsync(...);retryis not passed on. CommandBatchService3.52.0, L482-L483:new RedisCommonBatchExecutor(..., referenceType, false),noRetryhard-coded tofalse.RedisExecutor.isResendAllowed3.52.0, L485-L489:attempt < attempts && !noRetry && (command == null || ...). The batch executor hascommand == null, so the per-commandisNoRetry()check is skipped as well.RedisExecutor.scheduleResponseTimeout3.52.0, L453-L467: completes the attempt withCancellationExceptionand callsexecute()again afterretryInterval.
Same on master: syncedEval L1095-L1106 / L1198-L1199, CommandBatchService L481-L482, isResendAllowed L505-L509.
The acquire script is not idempotent (hincrby), so a client-side resend after a late reply doubles the count. The same path can also leave an orphan when every attempt times out: the caller gets RedisResponseTimeoutException, but the attempts did run on Redis and the hash field stays for lockWatchdogTimeout with no owner to release it.
Suggested fix. Pass noRetry = !retry from syncedEval into the CommandBatchService / RedisCommonBatchExecutor instead of the hard-coded false, or make the batch executor honour RedisCommand.isNoRetry() for its entries.
Workaround we use. On release, drain getHoldCount() down to the number of acquisitions the calling code itself made on that thread, and log when the two differ.
Source: redisson/redisson