How to properly expose Dolt remotesapi behind Kubernetes + Istio VirtualService to support clone / pull / push
Background
We are running Dolt on Kubernetes and would like to expose Dolt's remotesapi (remotesapi.port: 22222) to external clients through an Istio Gateway + VirtualService, so that dolt clone, dolt pull, and dolt push can be used against it.
In our current deployment, remotesapi is exposed through a Service on port 22222 (port name grpc-dolt-restapi, targetPort: dolt-restapi), backed by a Service named dolt-remotesapi. We tried adding a routing rule to the Istio VirtualService so that external requests could reach remotesapi through a URL with a path prefix (e.g. http://example.com/dolt/clone/mydb).
Current Dolt configuration
remotesapi:
port: 22222
read_only: false
listener:
host: 0.0.0.0
port: 6033
cluster:
remotesapi:
port: 50051
standby_remotes:
- name: standby
remote_url_template: http://dolt-1.dolt-headless.default.svc.cluster.local:50051/{database}VirtualService configuration we tried
We first added the following rule under spec.http, hoping to match on the URL prefix and rewrite the path:
- match:
- uri:
prefix: /dolt/clone/
rewrite:
uri: /
route:
- destination:
host: dolt-remotesapi.default.svc.cluster.local
port:
number: 22222The client then ran:
dolt clone http://example.com/dolt/clone/mydb -urootwhich failed with:
error: failed to get remote db
cause: could not access dolt url 'http://example.com/dolt/clone/mydb':
rpc error: code = Unknown desc = unexpected HTTP status code received from server:
405 (Method Not Allowed); transport: received unexpected content-type "text/html; charset=utf-8"From the gRPC requests observed on the client side, dolt clone uses gRPC over HTTP/2 under the hood, and the request path is a standard gRPC method path, for example:
POST /dolt.services.remotesapi.v1alpha1.ChunkStoreService/GetRepoMetadatarather than the URL path /dolt/clone/mydb or the rewritten /mydb. We suspect the previous rewrite.uri: / broke the gRPC method path, causing the backend to return 405.
What we are trying to solve
We would like a reference configuration or best practice for correctly exposing Dolt remotesapi behind Kubernetes + Istio VirtualService, specifically:
- Gateway side: What protocol should the port exposing remotesapi declare (
GRPC/HTTP2/HTTP)? Are any additional Server settings required? - VirtualService side: What should
match.urimatch — the gRPC method path prefix (e.g./dolt.services.remotesapi.v1alpha1.ChunkStoreService/) or the business URL prefix? Is arewriteneeded? - Service side: How should the port name or
appProtocolbe set so that Istio correctly recognizes it as gRPC? - Client URL: How is the path portion of the URL used by
dolt clone(e.g./dolt/clone/mydb) handled in gRPC communication? Is it passed to the backend as:path, or is it only used to construct request parameters? - DestinationRule: Is a DestinationRule required for the backend Service — for example,
h2UpgradePolicyor other traffic policies?
What we are hoping to get
Either of the following would be helpful:
- A reference Istio
Gateway+VirtualServiceconfiguration showing how to expose remotesapi under a URL with a path prefix; or - A clear statement that remotesapi does not support being exposed through a reverse proxy with a path prefix, along with the recommended alternative (e.g. using a dedicated hostname or port without a path prefix).
Additional information
- Dolt version: 2.2.1
- Within the cluster, remotesapi is directly reachable at
dolt-remotesapi.default.svc.cluster.local:22222; the SQL port6033and the cluster remotesapi50051both work fine. - The only problem is on the
22222path exposed through the Istio VirtualService.
Any guidance on correctly exposing remotesapi would be greatly appreciated.
Source: dolthub/dolt