VTOrc: partition shard monitoring across instances with a consistent hash ring
Overview
Every VTOrc instance currently watches the entire topology — all keyspaces and shards. For large fleets this is a scaling ceiling: each instance processes topology events and health checks for every shard, and adding instances gives HA but not horizontal scale.
--clusters_to_watch allows static partitioning, but it requires enumerating
keyspaces/keyranges and editing config whenever keyspaces are created or
deleted — impractical for fleets with high keyspace churn.
Proposal
Add an optional consistent hash ring so a pool of VTOrc instances can split shard-monitoring responsibility deterministically, with no keyspace lists:
--vtorc-ring-size N(total instances in the ring) and--vtorc-ring-index i(this instance's 0-based position).- Each shard's primary owner is
fnv32a(keyspace + "/" + shard) % ring_size. - The two ring-adjacent instances also watch each shard, giving 3-way HA coverage per shard (a primary crash/restart always leaves two instances already watching).
- Fully backward compatible: default
ring-size=1watches everything (today's behavior). Ring sizes ≤3 are a deliberate no-op (every instance is primary and both neighbors for every shard), so partitioning takes effect atring-size >= 4.
Even distribution (optional)
Raw hash % N can be uneven depending on keyspace naming. An optional
--vtorc-ring-assignments-file maps virtual hash buckets to ring partitions
for even load, without listing keyspace names (the file is just
{"num_buckets": …, "bucket_assignments": […]}). Unset falls back to direct
hash modulo, which is fine for small rings.
Backward compatibility
Opt-in and off by default; unset flags = current behavior. Composes with
--clusters_to_watch (ring gating is applied on top of the existing filter).
Alternatives considered
- Static
--clusters_to_watchpartitioning — operationally painful with ephemeral keyspaces.
An implementation PR is opened alongside this issue. Feedback on the design and flag naming is very welcome before it's reviewed in depth.
Source: vitessio/vitess