Silent, intermittent cache block corruption with S3 UFS (correct length, head/tail identical, bounded middle region differs)

Alluxio version: 2.9.5 (OSS)

UFS: S3-compatible cloud object storage via the s3a under-filesystem. Access pattern: warm a block into the worker cache once (a sync CacheRequest), then serve subsequent reads via short-circuit local reads.

Describe the bug Occasionally, after a block is cached, the cached block file on the worker does not match the UFS object, even though:

• the cached file size equals the UFS object size;
• the beginning and the end are byte-identical to the UFS object;
• only a bounded region in the middle differs — an in-place substitution with no shift (everything after the differing region is byte-identical and at the same offsets).

It is not always reproducible. During the fill there are no errors/warnings in the worker log — only Received sync cache request: ... (CacheRequestManager). The fill returns success but writes wrong bytes. Once a block is cached this way, all subsequent short-circuit reads return the corrupted bytes (downstream decompression then fails). md5 of the cached block != md5 of the object downloaded directly from the store (same length).

Impact: silent data corruption served from cache.

Suspected stock code paths

1. UfsInputStreamCache pools and reuses seekable UFS input streams per fileId, calling seek(offset) on reuse (for object stores: abort + reopen the HTTP stream at a new range). Reusing a stale-positioned stream could silently return bytes from a wrong offset. (UfsInputStreamCache.acquire, gated by alluxio.worker.ufs.instream.cache.enabled.)
2. CacheRequestManager deduplicates concurrent cache requests for the same blockId (mActiveCacheRequests.putIfAbsent), but this does not coordinate with the passive-cache-on-read fill path (UnderFileSystemBlockReader via the data server). A sync CacheRequest fill and a concurrent passive-cache fill of the same block appear to have no mutual exclusion, so writes to the same (temp) block could interleave and produce an in-place mid-block overwrite — consistent with the intermittency and the bounded-middle signature.

Questions

• Is this a known issue?
• Is mutual exclusion expected between a CacheRequest-driven fill and a passive-cache-on-read fill of the same block?
• Any relevant fixes in 2.10.x / master (around UfsInputStreamCache stream reuse or block-fill concurrency)?