Thanos Store Bucket Operation Latency since v0.33.0

Author: Pidu2Created Jan 17, 2024Updated Sep 9, 2026

Labelscomponent: storepackage: objstore

Thanos, Prometheus and Golang version used:

Thanos v0.33.0
Prometheus v2.48.1

Object Storage Provider: Cloudscale.ch

What happened: Since upgrading thanos to v0.33.0 from v0.32.5, we are running into the ThanosStoreObjstoreOperationLatencyHigh alert and the corresponding metric shows an increase of the 99th percentile by 3-6 times.

What you expected to happen: No significant increase in latency.

How to reproduce it (as minimally and precisely as possible): Screenshot of our production environment (the increase corresponds exactly to the upgrade time): Screenshot from 2024-01-17 13-55-14 Screenshot of our testing environment (same setup, less load) after we downgraded back to 0.32.5 (the decrease corresponds exactly to the downgrade time):

Full logs to relevant components:

Some logs AFTER upgrade

ts=2024-01-17T13:06:13.343281201Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.903627604s duration_ms=1903 cached=904 returned=594 partial=0
ts=2024-01-17T13:09:13.762500525Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.322083442s duration_ms=2322 cached=904 returned=594 partial=0
ts=2024-01-17T13:12:13.239154955Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.798598508s duration_ms=1798 cached=905 returned=595 partial=0
ts=2024-01-17T13:15:13.725025768Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.285317139s duration_ms=2285 cached=905 returned=595 partial=0
ts=2024-01-17T13:18:13.574275454Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.134137495s duration_ms=2134 cached=905 returned=595 partial=0
ts=2024-01-17T13:21:13.266295648Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.826640613s duration_ms=1826 cached=905 returned=595 partial=0
ts=2024-01-17T13:24:13.620279818Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.179145307s duration_ms=2179 cached=905 returned=595 partial=0
ts=2024-01-17T13:27:13.411437655Z caller=fetcher.go:529 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.971582294s duration_ms=1971 cached=905 returned=595 partial=0

Some logs BEFORE upgrade

ts=2024-01-09T22:11:42.717109681Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=407.652353ms duration_ms=407 cached=626 returned=326 partial=0
ts=2024-01-09T22:10:53.586476697Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.943787221s duration_ms=1943 cached=846 returned=546 partial=0
ts=2024-01-09T22:08:42.481675919Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=172.403147ms duration_ms=172 cached=626 returned=326 partial=0
ts=2024-01-09T22:07:53.490727796Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.848686114s duration_ms=1848 cached=846 returned=546 partial=0
ts=2024-01-09T22:05:42.608920073Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=299.362529ms duration_ms=299 cached=626 returned=326 partial=0
ts=2024-01-09T22:04:53.447103042Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.804983733s duration_ms=1804 cached=846 returned=546 partial=0
ts=2024-01-09T22:02:42.469191789Z caller=fetcher.go:487 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=159.853342ms duration_ms=159 cached=626 returned=326 partial=0

Anything else we need to know: args

    - store            
    - --data-dir=/data  
    - --index-cache-size=500MB    
    - --chunk-pool-size=500MB
    - --objstore.config-file=/etc/thanos/thanos-s3-credentials/thanos-s3-credentials.yaml
    - --store.caching-bucket.config-file=/etc/thanos/cache/bucket-cache-config.yaml

caching-bucket.config-file:

type: MEMCACHED # Case-insensitive
config:
  addresses:
    - dns+thanos-store-memcached-headless:11211
  timeout: 450ms~ $

Source: thanos-io/thanos

View original on GitHub View discussion on GitHub