vmagent: allow dropping persistent queue data during startup

Is your feature request related to a problem? Please describe

vmagent stores pending data that cannot be sent to the configured remote storage systems in a timely manner under -remoteWrite.tmpDataPath. The maximum amount of data that can be stored under -remoteWrite.tmpDataPath per configured -remoteWrite.url can be limited via -remoteWrite.maxDiskUsagePerURL command-line flag. When this limit is reached, vmagent drops the oldest data from disk in order to store newly ingested data.

There are cases where users may prefer to drop the queue instead of waiting for it to be drained. For example, it is recommended to configure vmagent to write to two independent vmcluster deployments to achieve HA, as described here. In this setup, if one of the vmclusters is unavailable due to an incident and the corresponding persistent queue in vmagent fills up and starts dropping data, data completeness can no longer be guaranteed on that vmcluster even if the queue is eventually drained, since part of the data has already been dropped. In such cases, it is usually recommended to backfill the affected period from the healthy vmcluster using vmctl. Since backfilling is required anyway after the incident, the buffered data in vmagent is no longer particularly useful. In addition, draining the buffered queue may increase load on the recovering vmcluster and prolong incident recovery time.

Describe the solution you'd like

Provide an option to allow dropping persistent queue data during startup. Ideally, this should be a one-time operation, so users do not need to roll out vmagent twice just to add and later remove the option.

One idea is to add a new file such as clean_on_start, or introduce a new field in the existing metainfo.json under the remote write queue directory <remoteWrite.tmpDataPath>/persistent-queue/1_B9EB7BE220B91E9D. At startup, vmagent would check for this marker, clean the queue data for the corresponding remote write destination, and then reset or remove the clean_on_start marker so it is not triggered again on the next restart and requires no further action after the incident.

One downside is that users would need to add this marker manually for all vmagent instances.

Describe alternatives you've considered

Currently, there are a few workarounds to achieve similar behavior:

1. Temporarily remove the affected vmcluster from -remoteWrite.url. vmagent will clean persistent queues when there is no matching -remoteWrite.url, as long as -remoteWrite.keepDanglingQueues is not enabled. However, if there are additional remote write settings associated with the same -remoteWrite.url, such as -remoteWrite.relabelConfig, they must also be removed.

2. Reduce -remoteWrite.maxDiskUsagePerURL to a very small value. vmagent will delete old blocks while writing new blocks until the queue size becomes smaller than -remoteWrite.maxDiskUsagePerURL. The procedure can be relatively slow, since vmagent needs to pause writes to the queue and clean queue blocks one by one in order to avoid block corruption.

Both approaches require users to update vmagent configuration again once the queue is cleaned or the incident is resolved, which is more error-prone.

Additional information

No response