Hadoop SDK: Kerberos AP-REQ fails intermittently when static LoginUser is overwritten by another component

What happened

JuiceFileSystemImpl#buildAuthCredential and io.juicefs.kerberos.KerberosUtil#genApReq both derive the authenticating principal from UserGroupInformation.getLoginUser().

UserGroupInformation.loginUserRef is a static field on the UGI class. In a normal deployment, the class is loaded once by the AppClassLoader, so this static reference is shared across the entire JVM. Any component in the same JVM can overwrite it — for example via UserGroupInformation.loginUserFromSubject(new Subject()) — replacing the "real" login user with one whose Subject carries no KerberosTicket.

When the JuiceFS Hadoop SDK is later invoked in that JVM:

• getLoginUser().hasKerberosCredentials() still returns true (because authMethod is only an enum on the User principal and isn't reset when Subject content changes), so buildAuthCredential routes into the Kerberos branch and calls KerberosUtil.genApReq.
• Inside genApReq, loginUser.doAs(...) executes under a Subject that has no TGT, so JGSS Krb5InitCredential.getInstance fails with GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt).

Stack trace (abridged):

java.lang.reflect.UndeclaredThrowableException at ...KerberosUtil.lambda$genApReq$0(KerberosUtil.java:...) at UserGroupInformation.doAs(...) at io.juicefs.kerberos.KerberosUtil.genApReq(...) at io.juicefs.JuiceFileSystemImpl.buildAuthCredential(...) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(...) at sun.security.jgss.GSSContextImpl.initSecContext(...)

Reproduction

A very common trigger is a service that logs in its own principal via loginUserFromKeytabAndReturnUGI and then wraps all downstream work in catalogUgi.doAs(...). The private UGI stays in the current AccessControlContext (observable via getCurrentUser()), but never becomes the global LoginUser — so relying on getLoginUser() picks up the wrong identity.

Minimal reproducer:

java // (1) Some other component in the JVM has already installed a Subject-less UGI // as the global LoginUser. Anything that calls loginUserFromSubject() does this. UserGroupInformation.loginUserFromSubject(new javax.security.auth.Subject()); // getLoginUser() now returns a UGI whose Subject has no KerberosTicket, // but authMethod is still KERBEROS from an earlier login, // so hasKerberosCredentials() lies and returns true.
// (2) The application does its own private Kerberos login and drives all work // through doAs — the standard AndReturnUGI + doAs pattern. UserGroupInformation catalogUgi = UserGroupInformation.loginUserFromKeytabAndReturnUGI( "hadoop/host@REALM", "/path/to/host.keytab");
catalogUgi.doAs((PrivilegedExceptionAction ) () -> { FileSystem fs = FileSystem.get(new URI("jfs://vol1/"), conf); fs.listStatus(new Path("/")); // ← throws GSSException: No valid credentials provided return null; });

Both UGIs are authMethod=KERBEROS and hasKerberosCredentials()=true, but only catalogUgi actually owns a TGT.

Why the code is fragile

Hadoop mainline itself does not use getLoginUser() to build the SASL/Kerberos identity for outbound RPCs. It uses getCurrentUser():

• org.apache.hadoop.ipc.Client#setupIOstreams puts the SASL handshake inside ticket.doAs(...), where ticket = remoteId.getTicket() = getCurrentUser(), and switches to ticket.getRealUser() if it is a PROXY user.
• org.apache.hadoop.security.SaslRpcClient#createSaslClient decides "am I allowed to use KERBEROS?" with ugi.getRealAuthenticationMethod().getAuthMethod() == KERBEROS, so PROXY users still route through Kerberos as long as their RealUser is Kerberos-authenticated.

getLoginUser() in Hadoop is only used to trigger relogin (e.g. handleSaslConnectionFailure), not to pick the authenticating identity.

The JuiceFS SDK's current use of getLoginUser() therefore diverges from Hadoop mainline and breaks whenever the static LoginUser doesn't match the true caller — which is the common case for services that use AndReturnUGI + doAs (Impala/Kyuubi/Hive Metastore/Ranger-embedded workloads, Flink UserCodeClassLoader isolation, etc.).

Proposed fix

Align the two entry points with Hadoop mainline:

1. In KerberosUtil.genApReq, derive the authenticating UGI from getCurrentUser(), fall back to getRealUser() when it is PROXY, and do checkTGTAndReloginFromKeytab against the UGI that actually owns the keytab.
2. In JuiceFileSystemImpl.buildAuthCredential, judge Kerberos eligibility with getRealAuthenticationMethod() == KERBEROS, and select delegation tokens from getCurrentUser().getCredentials() instead of the this.ugi captured at initialize() time.

A PR implementing this is on the way and links back to this issue.

Scenarios verified end-to-end (against a Kerberized deployment)

No behavior change for callers that were already correct (LoginUser == CurrentUser).

Environment

• JuiceFS Hadoop SDK: main (verified against commit c9a67b23)
• Hadoop client: 3.x
• JDK: 8/11/17
• Deployment: KDC + keytab-based Kerberos, running the SDK from inside a multi-tenant JVM (service embeds its own loginUserFromKeytabAndReturnUGI + doAs flow)