`KubeAPIServerConfig` is missing `extraVolumes` from the deprecated `.cluster.apiServer`

Bug Report

Note: I used AI for research, summarizing and writing parts of this issue. I read and reviewed everything and removed things I think are not important for this issue.

It is still quite long, sorry for that. I preferred to explain the whole problem and possible solutions instead of just asking to bring extraVolumes back, because I understand that arbitrary extraVolumes might not be a good idea and might also not be the direction Talos wants to move in.

Description

.cluster.apiServer was deprecated in v1.14 in favor of KubeAPIServerConfig. But the new KubeAPIServerConfig does not have an equivalent for .cluster.apiServer.extraVolumes.

The old .cluster.apiServer and the new KubeAPIServerConfig are mutually exclusive (V1Alpha1ConflictValidate), so we also cannot use the new config and keep extraVolumes from the old one.

This means functionality that was available with the deprecated config is currently missing from its replacement.

We ran into this because we want to use aws-iam-authenticator for kubectl access to our Talos clusters. We already use this for our other Kubernetes clusters and would like to keep authentication the same.

For this, kube-apiserver needs:

--authentication-token-webhook-config-file=/path/to/kubeconfig.yaml

Since v1.14, EtcFileConfig can be used to write arbitrary files under /etc, so at first this looked like it should solve the problem.

But while we can write the file and set the kube-apiserver flag, there is no way to make the file visible inside the kube-apiserver static pod.

authorizers:
  - webhook:
      connectionInfo:
        type: KubeConfigFile

KubeAuthorizerConfig supports KubeConfigFile, but kubeConfigFile is a path (KubeConfigFile *string in k8s.io/apiserver/pkg/apis/apiserver/v1). So the file somehow needs to exist inside the kube-apiserver container.

The examples in pkg/machinery/config/types/k8s/authorization.go only use InClusterConfig.

Why EtcFileConfig does not solve this

The obvious idea is:

1. Write the kubeconfig using EtcFileConfig.
2. Set --authentication-token-webhook-config-file using KubeAPIServerConfig.extraArgs.

The problem is that EtcFileConfig writes to the host /etc, while kube-apiserver only gets a few specific host directories mounted:

This comes from k8stemplates/apiserver.go and KubeAPIServerConfigV1Alpha1.ExtraVolumes() using:

filepath.Dir(constants.DefaultTrustedCAFile)

which results in /etc/ssl/certs.

But validateEtcFilePath rejects the ssl/ prefix, together with pki/, ca-certificates/ and kubernetes/ (pkg/machinery/config/types/runtime/etc_file.go).

So there is currently no directory where EtcFileConfig can write a file that is also visible inside kube-apiserver.

/etc/kubernetes and reboot

Before v1.14 we could use a path like:

/etc/kubernetes/aws-iam-authenticator/kubeconfig.yaml

and mount it into kube-apiserver using .cluster.apiServer.extraVolumes.

But there is another change in v1.14 which affects this setup.

Commit f146c6a18 (feat: refactor /etc mounts) changed /etc/kubernetes to a tmpfs. This means files stored there are lost after a reboot.

This is especially a problem for aws-iam-authenticator.

If the kubeconfig is created by the authenticator DaemonSet or another Kubernetes workload, it cannot recreate the file after reboot because kube-apiserver already needs that file to start:

kube-apiserver needs kubeconfig
        ↓
kubeconfig is created by Kubernetes workload
        ↓
workload needs kube-apiserver

So after a reboot with an empty /etc/kubernetes, the cluster cannot recover this by itself.

Previously extraVolumes together with a persistent host file gave us a way to handle this. With KubeAPIServerConfig, we cannot configure that mount anymore.

What changed in v1.14

So there are two changes which together break this use case:

1. /etc/kubernetes became a tmpfs, so files there no longer survive a reboot.
2. .cluster.apiServer was deprecated in favor of KubeAPIServerConfig, but KubeAPIServerConfig does not have extraVolumes.

The second one is the main reason for this issue: the deprecated config supports something that its replacement currently cannot express.

There were also similar issues with options missing from the new config documents:

• #14279: --oidc-signing-algs was not usable with KubeAuthenticationConfig
• #14030: authentication-config was rejected in .cluster.apiServer.extraArgs

Both were closed as completed.

Possible solutions

I see two possible ways to solve this, and wanted to ask which direction you prefer before opening a PR.

1. Add extraVolumes to KubeAPIServerConfig

This would basically restore what .cluster.apiServer.extraVolumes already provides.

Probably the same should then also be done for KubeControllerManagerConfig and KubeSchedulerConfig, as their old configs also support extraVolumes.

This would solve the aws-iam-authenticator case, but also other cases where kube-apiserver needs an additional file, for example:

• authentication webhook kubeconfigs
• AuthorizationConfiguration with KubeConfigFile
• custom audit policy files

The downside is that this allows arbitrary host mounts into the static pod again. I understand this was one of the concerns in #13916.

There is also a problem when combining this with EtcFileConfig: it supports name, mode and contents, but no uid/gid.

kube-apiserver runs as uid/gid 65534, so a credential written by EtcFileConfig would probably need to be 0444 to be readable by kube-apiserver. This makes the credential world-readable on the host.

Users would also need to choose a path that survives reboot, so /etc/kubernetes would no longer be a good location.

2. Let Talos manage the file for kube-apiserver

The other option would be to let Talos own the whole thing.

For example, a config document could contain the file contents and Talos could render it into KubernetesAPIServerConfigDir, which is already mounted into kube-apiserver.

Talos already does something similar for:

• authentication-config.yaml
• authorization-config.yaml
• admission-control-config.yaml
• auditpolicy.yaml

This would also allow Talos to create the file with proper permissions, for example 0400 owned by uid/gid 65534, instead of making the credential world-readable on the host.

It also avoids adding arbitrary host mounts again.

The main question is how general this should be.

It could either be something specific like KubeAuthenticationWebhookConfig, or a more general way to provide additional files to kube-apiserver.

If it is general, there would probably also need to be a way for other configs, for example KubeAuthorizerConfig.kubeConfigFile, to reference such a file.

This solution also would not help with non-file use cases like the KMS socket discussed in #13916.

We already have a working implementation of the narrow version: a KubeAuthenticationWebhookConfig document which contains the kubeconfig inline. Talos renders the file with the correct permissions and also manages the kube-apiserver flag.

We can open this as a PR, but before doing that I wanted to check which direction you would prefer: adding extraVolumes to the new config, or letting Talos manage additional kube-apiserver files directly.

Environment

• Talos: v1.14 / v1.15 (main)
• Kubernetes: any
• Platform: any