ACL cache may reference an uncommitted ACL after metadata transaction retry

What happened

When creating a file under a directory with a non-minimal Default ACL, a metadata transaction retry can leave the new inode referencing an ACL ID whose ACL record was never committed.

On the first attempt, insertACL stages the ACL record in the current metadata transaction and updates the process-local aclCache before the outer transaction commits. If the transaction conflicts and is rolled back, the cache entry remains. During the retry, insertACL finds the same ACL in aclCache and skips staging the ACL record again, while the child inode is still written with that cached AccessACL ID.

After clearing the ACL cache or restarting the metadata client, reading the ACL fails because the inode references a missing ACL record.

This was reproduced on an unmodified upstream/main at commit 3038748936a4076a89390c59a657a4c2d5ea39cd using the TiKV metadata backend. The reproduction uses ordinary file creation and is independent of #7523.

What you expected to happen

The ACL record and the inode AccessACL reference should be committed consistently. After clearing the process-local ACL cache or restarting the client, the newly created file ACL should remain readable.

How to reproduce it (as minimally and precisely as possible)

The most reliable reproduction is a package-level test against TiKV. A test-only synchronization barrier is useful to make the transaction interleaving deterministic.

Prerequisites:

• A TiKV/PD instance reachable at 127.0.0.1:2379.
• ACL support enabled with format.EnableACL = true.
• A non-minimal Default ACL, containing at least one named user or group entry.
• Two JuiceFS metadata clients connected to the same TiKV namespace.

Steps:

1. Create a parent directory and assign it a non-minimal Default ACL. The named entry is important because it causes file creation to materialize a separate access ACL record.

   go复制

   defaultRule := &aclAPI.Rule{
       Owner: 7,
       Group: 5,
       Mask:  5,
       Other: 0,
       NamedUsers: []aclAPI.Entry{
           {Id: 1001, Perm: 4},
       },
   }
   
   m.Mkdir(ctx, RootInode, "parent", 0770, 0, 0, &parent, nil)
   m.SetFacl(ctx, parent, aclAPI.TypeDefault, defaultRule)

2. Start an ordinary file creation under the parent directory:

   go复制

   m.Create(ctx, parent, "child", 0666, 022, 0, &child, nil)

3. In a test-only hook, pause the first transaction in kvMeta.insertACL immediately after the ACL record is staged and the cache is updated:

   go复制

   tx.set(m.aclKey(aclId), rule.Encode())
   m.aclCache.Put(aclId, rule)
   // pause here before the outer metadata transaction commits

4. While the file-creation transaction is paused, use the second metadata client to modify the parent directory GID and commit the update:

   go复制

   updater.SetAttr(ctx, parent, SetAttrGID, 0, &Attr{Gid: 2469})

   Changing the GID keeps the parent Default ACL unchanged, so the retry generates the same child ACL rule and the same cache lookup can be observed.

5. Release the first transaction. Because it read the parent inode before the concurrent GID update, TiKV detects a write conflict and JuiceFS retries the transaction.

6. During the retry, insertACL hits aclCache and skips writing the ACL record again. The retry nevertheless writes the child inode with a non-zero AccessACL ID.

7. Clear the ACL cache or restart the metadata client:

   go复制

   m.getBase().aclCache.Clear()

8. Read the child inode and its ACL:

   go复制

   var attr Attr
   m.GetAttr(ctx, child, &attr)
   
   got := &aclAPI.Rule{}
   status := m.GetFacl(ctx, child, aclAPI.TypeAccess, got)

Observed result:

• GetAttr succeeds.
• attr.AccessACL is non-zero.
• GetFacl returns EIO because the referenced ACL record is missing from TiKV.
• The transaction log reports that the metadata transaction succeeded after one or more retries.

Without the synchronization barrier, the same issue can be observed with repeated concurrent file creation and parent-directory metadata updates, but the exact interleaving is timing-dependent.

Anything else we need to know?

The likely cause is that the process-local ACL cache is updated before the enclosing metadata transaction commits successfully. The cache is not rolled back when the transaction is retried.

A possible fix is to publish new ACL entries to the in-memory cache only after the outer transaction commits, or to ensure that the ACL record is staged again on every transaction retry.

The same cache-before-commit pattern also appears in the Redis metadata backend and should be reviewed there.

Environment

• JuiceFS version: upstream/main, commit 3038748936a4076a89390c59a657a4c2d5ea39cd
• Metadata engine: TiKV
• TiKV endpoint: 127.0.0.1:2379
• OS: [to be filled]
• TiKV version: [to be filled]