cluster: use persistent unique vmstorage IDs for consistent sharding across vmstorage nodes

Is your feature request related to a problem? Please describe

In cluster version of VictoriaMetrics the ingested time series are consistently sharded by vminsert nodes across configured vmstorage nodes. The sharding function is applied for each received time series individually and takes as arguments the following:

• metric name
• metric labels
• the list of vmstorage TCP addresses passed to -storageNode command-line flag

The result of the function is one of the vmstorage nodes from the list. This is how vminsert decides where each specific time series will be sent. Taking into account that all vminsert nodes in the cluster are supposed to be configured equally - the hashing result remains consistent. This means samples for each unique time series ends up on the same vmstorage node, doesn't matter which vminsert received it initially.

But this approach doesn't work well when TCP addresses of existing vmstorage nodes change frequently in dynamic environments, due to re-deploys across multiple hosts with different IP addresses. Such events may cause vminsert to get a new list of vmstorage addresses. Because of this, the distribution of incoming time series can be re-shuffled across the existing vmstorage nodes. This may load to resource usage spikes at vmstorage nodes, since they need to register new time series in the inverted index. See the related issue.

Describe the solution you'd like

The problem can be solved if every vmstorage could identify itself via an unique persistent ID. This ID must remain the same after vmstorage restart or the migration to another host with new IP. For example, storage-1 will remain storage-1 despite its IP address change. On the first start, with empty -storageDataPath vmstorage can generate an unique ID and store it in -storageDataPath (to retain it on restarts or backup restores). Each time vminsert performs a handshake, vmstorage can communicate its ID back to vminsert. This ID can then be used for consistent hashing instead of TCP address of vminsert node.

Advantages:

• No manual settings, ID is automatically created and communicated by vmstorage to vminsert
• ID persists on disk, so re-deployments, backups and restores will retain the same ID.

Disadvantages:

• For vmstorage to communicate its ID the communication protocol need to be changed between vminsert and vmstorage.

Describe alternatives you've considered

No response

Additional information

No response