helm upgrade rotates crossplane-root-ca and breaks mTLS to installed packages
mTLS breaks on every helm upgrade with GitOps (Flux/Argo CD)
What happens
Crossplane pod is rolled out on every helm release change (chart bump,
values tweak, even unrelated reconciliations). The init container inside
the new pod treats the crossplane-root-ca Secret as a new resource,
the CA rotates, and all installed provider/function packages lose mTLS
trust. Composition pipelines fail with:
transport: authentication handshake failed: tls: failed to verify
certificate: x509: certificate signed by unknown authority
'Crossplane' Reproduction
- Install Crossplane via Helm in a GitOps setup (Flux HelmRelease works well as a reproducer).
- Install a provider or function (any).
- Have a composition that uses it.
- Bump the chart version (or change any value).
- Watch the composition pipeline break; check init container logs for
"TLS CA secret is empty or not complete, generating a new CA..."
on every upgrade.
Expected
The root CA should be stable across upgrades. mTLS trust to installed
packages should not break.
Proposed fix
Move Secret ownership from declarative chart resources to pre-install/ pre-upgrade hook jobs that create the placeholders only if missing, and a pre-delete hook to clean up on uninstall. Draft PR incoming.
Source: crossplane/crossplane