Multi-site admin/public RGWs leak all the way to the application

This is a followup to #14544 and #15043, both of which were resolved by #15064.

I need to have publicly-routable S3 buckets. However, in the interest of minimizing my attack surface, that's all I'd like to expose publicly: S3. I don't want to expose Swift etc., and I certainly don't want to expose the admin API. I used the documented multi-site approach (even though I just have the one site) to deploy a pair of shared-pool RGW instances:

1. An internal RGW with the admin API and a storage class associated with it for provisioning ObjectBucketClaims (OBCs)
2. An external RGW with only the s3 API exposed, with no storage class, and an HTTPRoute allowing public access

My hope was that this would be an implementation detail, and ultimately would change nothing from the consuming application's perspective: I create an OBC and voila, have a secret and configmap that allow me to access it via BUCKET_HOST and so on. That's where things go wrong: the BUCKET_HOST is pointing at (1). Well, the entire point of having (2) was to be able to hand pre-signed URLs to (2) back to the users of the app so they could access it from their own devices. Handing it URLs to (1) is pretty pointless since it's an internal service. In fact, the app itself only talks S3, it doesn't really need to know about (1) at all beyond the initial OBC!

The "obvious" solution to this is to use advertiseEndpoint in the spec.hosting config for the internal RGW to point to the external RGW, which (I believe) would set BUCKET_HOST appropriately. However, as specified by the docs, Rook itself uses that advertised endpoint for management operations, i.e. hits it and expects to find the admin API. Well, the entire point of having (2) was to not have the admin API. Clearly this is not a solution.

As a result, today, any application that uses this needs to actually understand that it should NOT use the BUCKET_HOST from the OBC configmap, and actually use a completely different URL for accessing S3. In other words, this configuration leaks the fact that there are two RGWs, one for provisioning, and one for public access. I think in an ideal world this would be entirely transparent to the application, and there would be a way in the private ObjectStore CRD for a cluster maintainer to specify that there's another URL that consuming applications should use to access it without interfering with Rook implementation details.

My naive suggestion is to add another key to spec.hosting for the ObjectStore CRD that gives us the ability to point the BUCKET_HOST in the resulting configmap elsewhere while not interfering with Rook's internal operations. Maybe bucketHostEndpoint or something. Basically: advertiseEndpoint but not something that Rook itself uses beyond simply defining the configmap.

Another naive option is to go the other way entirely: add the storage class on the public ObjectStore, but grow a key in spec.hosting for overriding where admin operations should go. Then Rook can use that value for provisioning the bucket/users, while the rest of the OBC provisioning looks just like only the public RGW exists. Applications can use BUCKET_HOST, and so on, and not need to care about the private RGW even in its OBCs. Nice separation of concerns there.

I say "naive" because there seems to be an issue with that kind of approach, too. What if, in the future, I have an application that wants to use this same public/private RGW pair, but actually DOES need to do its own admin operations. Well, we just removed its ability to know (via the OBC) where to hit the RGW that DOES have the admin API, and thus it will now need to know that there are two RGWs. To be clear I have no intention of writing such an app, I'm using this as a strawman to point out that this is probably not the proper solution. It almost feels like we need to be able to specify preferred URLs per API, honestly. Then they can be exposed in the configmap as separate services. S3_HOST, SWIFT_HOST, ADMIN_HOST, and so on. By default all of those variables can hold the same value, but cluster maintainers can override each of them with particular keys in the ObjectStore CRD's spec.hosting config. Or... maybe an application that needs something beyond S3 shouldn't be using an ObjectBucketClaim and this entire strawman is silly?

Anyway, let's discuss! If we agree that a feature is missing here, I'd be happy to add it once we arrive at what exactly we want.