kube-apiserver fails to start when --kube-apiserver-arg=service-account-signing-endpoint is set due to default SA key flags
Environmental Info: K3s Version:
k3s version v1.36.4+k3s1 (eb9f87c3)
go version go1.25.8Node(s) CPU architecture, OS, and Version:
Linux 6.12.27-amd64 #1 SMP PREEMPT_DYNAMIC x86_64 GNU/LinuxCluster Configuration: 1 server
Describe the bug:
When configuring kube-apiserver to use an external ServiceAccount token signer (ExternalServiceAccountTokenSigner, KEP-740) via --kube-apiserver-arg=service-account-signing-endpoint=/path/to/signer.sock, kube-apiserver fails to start because K3s unconditionally sets default --service-account-signing-key-file and --service-account-key-file flags in apiServer() (pkg/daemons/control/server.go#L225 and pkg/daemons/control/server.go#L244).
Upstream Kubernetes kube-apiserver enforces mutual exclusivity between --service-account-signing-endpoint and the static key-file flags in three places:
validateServiceAccountTokenSigningConfig(pkg/controlplane/apiserver/options/validation.go#L99-L101)Options.Complete(pkg/controlplane/apiserver/options/options.go#L305-L306)BuiltInAuthenticationOptions.ToAuthenticationConfig(pkg/kubeapiserver/options/authentication.go#L620-L622)
Furthermore, because --service-account-key-file is registered with pflag.StringArrayVar (pkg/kubeapiserver/options/authentication.go#L430-L435) (which appends to KeyFiles rather than overwriting), users cannot clear the default --service-account-key-file flag via --kube-apiserver-arg.
Steps To Reproduce:
- Installed K3s with
--kube-apiserver-arg=service-account-signing-endpoint=/run/k3s-jwt-signer.sock(where/run/k3s-jwt-signer.sockserves the KubernetesExternalJWTSignerv1 gRPC service).
Expected behavior:
When service-account-signing-endpoint is provided in ExtraAPIArgs (--kube-apiserver-arg), K3s omits its default service-account-signing-key-file and service-account-key-file flags in apiServer()—matching the existing pattern for authorization-config and authentication-config in pkg/daemons/control/server.go#L215-L224—allowing kube-apiserver to start with the external JWT signer.
Actual behavior:
K3s passes both --service-account-signing-key-file=<data-dir>/tls/service.current.key and --service-account-key-file=<data-dir>/tls/service.key alongside --service-account-signing-endpoint, causing kube-apiserver to exit during flag validation (pkg/controlplane/apiserver/options/validation.go#L99-L101):
can't set `--service-account-signing-key-file` and/or `--service-account-key-file` with `--service-account-signing-endpoint` (They are mutually exclusive)Additional context / logs:
In pkg/daemons/control/server.go (apiServer()):
if util.ArgValue("service-account-signing-endpoint", cfg.ExtraAPIArgs) == "" {
argsMap["service-account-signing-key-file"] = runtime.ServiceCurrentKey
argsMap["service-account-key-file"] = runtime.ServiceKey
}Guarding these two defaults when util.ArgValue("service-account-signing-endpoint", cfg.ExtraAPIArgs) is non-empty resolves the conflict while still allowing users to explicitly pass either flag if needed.
Source: k3s-io/k3s