vmselect: update vm_metric_rows_skipped_total cause high cpu contention

Describe the bug

vm_metric_rows_skipped_total{name="vmselect"} counts rows skipped by vmselect while unpacking blocks returned from vmstorage.

During query execution, vmselect reads blocks and then applies the query time range filter:

sb.Timestamps, sb.Values = tmpBlock.AppendRowsWithTimeRangeFilter(...)
skippedRows := tmpBlock.RowsCount() - len(sb.Timestamps)
metricRowsSkipped.Add(skippedRows)

So this metric increases when a block contains samples outside the requested query time range. These samples are decoded/read as part of the block, but then skipped by vmselect because their timestamps are out of the requested query time range.

For instant queries, vmselect usually needs data for only one evaluation timestamp, many returned blocks may contain only a small number of useful samples for the requested timestamp, while the rest of the rows in those blocks are outside the required time range and are counted as skipped.

As a result, vm_metric_rows_skipped_total{name="vmselect"} can grow much faster for workloads dominated by instant queries, which is triggered by vmalert

In this case, updating this metric can become a noticable CPU hotspot under high query concurrency. vmselect has multiple layers of concurrency to updating this metric:

• A: -search.maxConcurrentRequests, defaulting to 2 * available CPUs
• B: -search.maxWorkersPerQuery, defaulting to min(available CPUs, 32)
• C: when a series has more than 1000 blocks, each worker will start extra workers to unpack this timeseries, the number is up to available CPUs

The max concurrency is A*B*C. For example on a host with 100 CPU cores, the max concurrency to update the metric can be (2*100)*32*100 = 640000

Additional information

vmselect_cpu_profile.zip

Version

latest