Discussion: crd.projectcalico.org/v1 vs projectcalico.org/v3

This issue comes up frequently enough that I think it warrants its own parent issue to explain and discuss. I'll try to keep this up-to-date with the latest thinking and status should it change.

The problem generally manifests itself as one of the following:

1. no matches for kind "X" in version "projectcalico.org/v3" when attempting to apply a resource.
2. Applying a resource with apiVersion: crd.projectcalico.org/v1 and Calico not behaving as expected.

TL;DR

Don't touch crd.projectcalico.org/v1 resources. They are not currently supported for end-users and the entire API group is only used internally within Calico. Using any API within that group means you will bypass API validation and defaulting, which is bad and can result in symptoms like # 2 above. You should use projectcalico.org/v3 instead. Note that projectcalico.org/v3 requires that you install the Calico API server in your cluster, and will result in errors similar to # 1 above if the Calico API server is not running.

Ok, but why do it that way?

Well, it's partly because of limitations in CRDs, and partly due to historical reasons. CRDs provide some validation out of the box on their own, but can't do some of the more complex cross-field and cross-object API validation that the Calico API server can perform. For example, making sure that IP pools are consistent with the IPAM block resources within the cluster is a complex validation process that just can't be expressed in an OpenAPI schema. Same goes for some of the defaulting operations (e.g., conditional defaulting based on other fields).

As a result, Calico uses an aggregation API server to perform these complex tasks against projectcalico.org/v3 APIs, and stores the resulting validated and defaulted resources in the "backend" as CRDs within the crd.projectcalico.org/v1 group. Prior to the introduction of said API server, all of that validation and defaulting had to be performed client-side via calicoctl, but data was still stored in the "backend" as CRDs, for Calico itself to consume.

CRD validation has come a long way since Calico initially started using them way back in beta when they were actually called ThirdPartyResources. However, they still don't (and probably won't ever) support the types of validation that Calico currently enforces via its API server.

Pain points

Yes, this model is not perfect and has a few known (non-trivial) pain points that I would love to resolve.

• Having two APIs that look very similar (but in fact have a few small, key differences) is very confusing for users.
• Relying on an aggregation layer to serve the APIs introduces a more complex ordering dependency, since the Calico API server requires Calico in order to run, meaning projectcalico.org/v3 isn't usable until after the API server image is pulled and running on the cluster. This can cause issues for GitOps tools like Flux, which don't have a great way of understanding that multi-layer dependency.
• Aggregate API servers are not commonly used in other projects, whereas CRDs are, causing additional confusion.

Can we make it better?

Maybe. I hope so! But the solutions are not simple. We'd need to do at least some combination or the following, based on my current best guesses.

• Add validation and defaulting to crd.projectcalico.org/v1 and make it supported. We can't do this without introducing a webhook, which is not really desirable. We can do maybe 25% of our validation via CRD server-side validation, but we'd be losing a lot of our current validation and defaulting if we go this route.
• Allow the API server to run as host network, prior to installing Calico. This solves some problems, but not all of them (still have two APIs that are confusing and users still need to install a pod on the cluster to use our API).
• Remove crd.projectcalico.org/v1 altogether, instead back the aggregation layer in Kubernetes' etcd instance. This solves the "two APIs" problem, but would be a rather cumbersome data migration project and doesn't remove the need for a Calico-specific API server.
• Introduce a new API group (or groups) - e.g., policy.projectcalico.org/v3 and write all new CRD-based APIs within it, with a focus on making the syntax and semantics 100% compatible with what CRD validation and defaulting provides.