Baike.dev
Log in
Back to tool/Back to issues
#6033·thanos

Large amount of S3 calls of compactor in small Thanos instance

Author: albgusCreated Jan 10, 2023Updated Sep 14, 2026

Thanos, Prometheus and Golang version used: Installed by Helm chart: bitnami/thanos-11.6.8

#values.yml
existingObjstoreSecret: thanos-objstore
clusterDomain: cluster.local

query:
  enabled: true
  replicaLabel:
    - replica
    - receive

queryFrontend:
  enabled: true

compactor:
  enabled: true
  retentionResolutionRaw: 0s
  retentionResolution5m: 0s
  retentionResolution1h: 0s
  persistence:
    enabled: false

storegateway:
  enabled: true
  persistence:
    enabled: false

ruler:
  enabled: false

receive:
  enabled: true
  mode: standalone
  tsdbRetention: 30d
  replicationFactor: 1
  persistence:
    enabled: true
    existingClaim: thanos-receive

receiveDistributor:
  enabled: false

metrics:
  enabled: true

minio:
  enabled: false
thanos, version 0.30.1 (branch: HEAD, revision: 27e4714174e2b7d4542ce9a723716134afba1579)
  build user:       root@90643487b1ff
  build date:       20230104-18:37:40
  go version:       go1.19.4
  platform:         linux/amd64
Containers:
  compactor:
    Container ID:  containerd://1b2c5ad8552d156f6947ba691490404cd6718c1b16bac6dbac706fef768185b2
    Image:         docker.io/bitnami/thanos:0.30.1-scratch-r0
    Image ID:      docker.io/bitnami/thanos@sha256:c35abdf94a0d111a2e710b08e47a41d4a5ee58ac36a545192c44fdf2edb212ec
    Port:          10902/TCP
    Host Port:     0/TCP
    Args:      compact
      --log.level=info
      --log.format=logfmt
      --http-address=0.0.0.0:10902
      --data-dir=/data
      --retention.resolution-raw=0s
      --retention.resolution-5m=0s
      --retention.resolution-1h=0s
      --consistency-delay=30m
      --objstore.config-file=/conf/objstore.yml
      --wait

Object Storage Provider: AWS S3

What happened: I am seeing a very large amount of requests billed as Amazon Simple Storage Service EUN1-Requests-Tier2 in a test account with a small Thanos installation. I'm seeing about 40 Million requests per month, which comes down to around 60,000 requests per hour, or 16-20 requests / second.

What you expected to happen: I would have expected a small instance to perform significantly less requests against S3.

This is with a very small instance of Thanos, the S3 bucket having 1.5GB of storage in about 550 files, containing a bit over 2 years of data at fairly low resolution (1-5 minutes)

Full logs to relevant components:

Looks like almost all of the requests comes from the compactor, in an about 50/50 mix between HEAD [...]/index.json, or GET requests for deletion-mark.json or no-compact-mark.json

$ wc -l s3.log
1 338 023 s3.log
$ grep -c 'HEAD.*/meta.json' s3.log
649 613
$ grep -c 'NoSuchKey' s3.log
610 258

Example logs from S3

thanos.i.example.com [09/Jan/2023:21:46:22 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos XCJRGVE9EF1QQVQC REST.GET.OBJECT test-infra/01GJG6H63WAJ3CJ8WH20SYVK5R/deletion-mark.json "GET /thanos.i.example.com/test-infra/01GJG6H63WAJ3CJ8WH20SYVK5R/deletion-mark.json HTTP/1.1" 404 NoSuchKey 327 - 7 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:21:46:22 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos XCJT5KFN7GSKH45Z REST.GET.OBJECT test-infra/01F6G9S4JBT0YXB6N22WW1845C/deletion-mark.json "GET /thanos.i.example.com/test-infra/01F6G9S4JBT0YXB6N22WW1845C/deletion-mark.json HTTP/1.1" 404 NoSuchKey 327 - 7 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:21:46:22 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos XCJTQMSGR2R7QDTH REST.GET.OBJECT test-infra/01GJJB5XCCB5ZMGHSSVZDFWVD3/deletion-mark.json "GET /thanos.i.example.com/test-infra/01GJJB5XCCB5ZMGHSSVZDFWVD3/deletion-mark.json HTTP/1.1" 404 NoSuchKey 327 - 7 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:21:56:24 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 2QR25GBJDTZGA6J4 REST.GET.OBJECT test-infra/01GJG6NQHP1GCXYV9TGMNESC3D/no-compact-mark.json "GET /thanos.i.example.com/test-infra/01GJG6NQHP1GCXYV9TGMNESC3D/no-compact-mark.json HTTP/1.1" 404 NoSuchKey 329 - 12 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:21:56:24 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 2QR2ASW479QYA3R7 REST.GET.OBJECT test-infra/01GPC73YN26C7893Q3VAA28D6B/no-compact-mark.json "GET /thanos.i.example.com/test-infra/01GPC73YN26C7893Q3VAA28D6B/no-compact-mark.json HTTP/1.1" 404 NoSuchKey 329 - 12 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:21:56:24 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 2QR2C2NVJDEBKRMZ REST.GET.OBJECT test-infra/01GJG4K4480ES027J2KC4SFV7J/deletion-mark.json "GET /thanos.i.example.com/test-infra/01GJG4K4480ES027J2KC4SFV7J/deletion-mark.json HTTP/1.1" 404 NoSuchKey 327 - 8 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:20:59:54 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 65Z1XC39562G1968 REST.HEAD.OBJECT test-infra/01F9RBZ8KX4DFZA43M8BHY62JC/meta.json "HEAD /thanos.i.example.com/test-infra/01F9RBZ8KX4DFZA43M8BHY62JC/meta.json HTTP/1.1" 200 - - 9907 9 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:20:59:54 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 65Z22HFBACCH2ADC REST.HEAD.OBJECT test-infra/01GGW00S98GF4DZZE580RPDZXF/meta.json "HEAD /thanos.i.example.com/test-infra/01GGW00S98GF4DZZE580RPDZXF/meta.json HTTP/1.1" 200 - - 9045 17 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:20:59:54 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 65Z2PV1CY6R20BNN REST.HEAD.OBJECT test-infra/01GP9C2JW7BFPVGRB20JCBW8HE/meta.json "HEAD /thanos.i.example.com/test-infra/01GP9C2JW7BFPVGRB20JCBW8HE/meta.json HTTP/1.1" 200 - - 691 13 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
thanos.i.example.com [09/Jan/2023:20:59:54 +0000] 1.2.3.4 arn:aws:iam::012345678910:user/test-infra-thanos 65Z2RFKHSXGP9KCJ REST.HEAD.OBJECT test-infra/01GJG6HYWK0RAQJW524JAJ6YR4/meta.json "HEAD /thanos.i.example.com/test-infra/01GJG6HYWK0RAQJW524JAJ6YR4/meta.json HTTP/1.1" 200 - - 9049 11 - "-" "MinIO (linux; amd64) minio-go/v7.0.45 thanos-compact/0.30.1 (go1.19.4)"
Compactor logs 

level=info ts=2023-01-09T21:36:22.629309533Z caller=compact.go:1291 msg="start sync of metas"
level=info ts=2023-01-09T21:36:23.085786189Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=456.549601ms duration_ms=456 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:23.42711341Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=341.112912ms duration_ms=341 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:23.427256864Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2023-01-09T21:36:23.427281465Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2023-01-09T21:36:23.42729718Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2023-01-09T21:36:23.42733271Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"
level=info ts=2023-01-09T21:36:23.795860475Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=368.522075ms duration_ms=368 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:23.795976484Z caller=compact.go:1296 msg="start of GC"
level=info ts=2023-01-09T21:36:23.796886112Z caller=compact.go:1319 msg="start of compactions"
level=info ts=2023-01-09T21:36:23.797733551Z caller=compact.go:1355 msg="compaction iterations done"
level=info ts=2023-01-09T21:36:23.797798598Z caller=compact.go:430 msg="start first pass of downsampling"
level=info ts=2023-01-09T21:36:24.143234162Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=345.401832ms duration_ms=345 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:24.152232175Z caller=compact.go:444 msg="start second pass of downsampling"
level=info ts=2023-01-09T21:36:24.532001531Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=379.733126ms duration_ms=379 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:24.54048509Z caller=compact.go:451 msg="downsampling iterations done"
level=info ts=2023-01-09T21:36:24.877461273Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=336.938885ms duration_ms=336 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:24.87758081Z caller=retention.go:32 msg="start optional retention"
level=info ts=2023-01-09T21:36:24.877619402Z caller=retention.go:47 msg="optional retention apply done"
level=info ts=2023-01-09T21:36:25.237112881Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=359.471303ms duration_ms=359 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:36:25.237238629Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2023-01-09T21:36:25.237260988Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2023-01-09T21:36:25.237276747Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2023-01-09T21:36:25.237311343Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"
level=info ts=2023-01-09T21:36:54.197922581Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=253.098354ms duration_ms=253 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:37:54.206961741Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=262.236529ms duration_ms=262 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:38:54.237132756Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=292.379752ms duration_ms=292 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:39:54.189723553Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=244.982464ms duration_ms=244 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:40:54.281415238Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=336.728403ms duration_ms=336 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:41:22.629232573Z caller=compact.go:1291 msg="start sync of metas"
level=info ts=2023-01-09T21:41:23.113351584Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=484.134113ms duration_ms=484 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:23.464824382Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=351.250383ms duration_ms=351 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:23.464940114Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2023-01-09T21:41:23.464973383Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2023-01-09T21:41:23.464988945Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2023-01-09T21:41:23.465023218Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"
level=info ts=2023-01-09T21:41:23.821646543Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=356.623108ms duration_ms=356 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:23.821837441Z caller=compact.go:1296 msg="start of GC"
level=info ts=2023-01-09T21:41:23.822956468Z caller=compact.go:1319 msg="start of compactions"
level=info ts=2023-01-09T21:41:23.824211984Z caller=compact.go:1355 msg="compaction iterations done"
level=info ts=2023-01-09T21:41:23.824288458Z caller=compact.go:430 msg="start first pass of downsampling"
level=info ts=2023-01-09T21:41:24.155779485Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=331.465285ms duration_ms=331 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:24.164656225Z caller=compact.go:444 msg="start second pass of downsampling"
level=info ts=2023-01-09T21:41:24.515630779Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=350.937276ms duration_ms=350 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:24.523994286Z caller=compact.go:451 msg="downsampling iterations done"
level=info ts=2023-01-09T21:41:24.882721047Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=358.677568ms duration_ms=358 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:24.882834518Z caller=retention.go:32 msg="start optional retention"
level=info ts=2023-01-09T21:41:24.882872977Z caller=retention.go:47 msg="optional retention apply done"
level=info ts=2023-01-09T21:41:25.280126209Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=397.23131ms duration_ms=397 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:41:25.280246097Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2023-01-09T21:41:25.280269503Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2023-01-09T21:41:25.280285496Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2023-01-09T21:41:25.280319508Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"
level=info ts=2023-01-09T21:41:54.234273488Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=289.38488ms duration_ms=289 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:42:54.172009137Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=227.668901ms duration_ms=227 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:43:54.257231606Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=313.268026ms duration_ms=313 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:44:54.232058255Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=287.306826ms duration_ms=287 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:45:54.187292048Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=242.468381ms duration_ms=242 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:46:22.629135101Z caller=compact.go:1291 msg="start sync of metas"
level=info ts=2023-01-09T21:46:23.085725635Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=456.655381ms duration_ms=456 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:23.430794619Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=344.854733ms duration_ms=344 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:23.430910434Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2023-01-09T21:46:23.430933074Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2023-01-09T21:46:23.430948122Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2023-01-09T21:46:23.430986987Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"
level=info ts=2023-01-09T21:46:23.791041727Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=360.049239ms duration_ms=360 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:23.791156348Z caller=compact.go:1296 msg="start of GC"
level=info ts=2023-01-09T21:46:23.792092458Z caller=compact.go:1319 msg="start of compactions"
level=info ts=2023-01-09T21:46:23.79293398Z caller=compact.go:1355 msg="compaction iterations done"
level=info ts=2023-01-09T21:46:23.792995905Z caller=compact.go:430 msg="start first pass of downsampling"
level=info ts=2023-01-09T21:46:24.153357808Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=360.337512ms duration_ms=360 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:24.162179161Z caller=compact.go:444 msg="start second pass of downsampling"
level=info ts=2023-01-09T21:46:24.50385105Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=341.635578ms duration_ms=341 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:24.512246975Z caller=compact.go:451 msg="downsampling iterations done"
level=info ts=2023-01-09T21:46:24.848390243Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=336.102565ms duration_ms=336 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:24.848528041Z caller=retention.go:32 msg="start optional retention"
level=info ts=2023-01-09T21:46:24.848572253Z caller=retention.go:47 msg="optional retention apply done"
level=info ts=2023-01-09T21:46:25.184809391Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=336.214489ms duration_ms=336 cached=170 returned=139 partial=0
level=info ts=2023-01-09T21:46:25.184934782Z caller=clean.go:34 msg="started cleaning of aborted partial uploads"
level=info ts=2023-01-09T21:46:25.184957378Z caller=clean.go:61 msg="cleaning of aborted partial uploads done"
level=info ts=2023-01-09T21:46:25.18497269Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion"
level=info ts=2023-01-09T21:46:25.185006189Z caller=blocks_cleaner.go:58 msg="cleaning of blocks marked for deletion done"
level=info ts=2023-01-09T21:46:54.216566449Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=272.255358ms duration_ms=272 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:47:54.189112261Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=244.772637ms duration_ms=244 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:48:54.200035653Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=255.722853ms duration_ms=255 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:49:54.19546239Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=251.121839ms duration_ms=251 cached=170 returned=170 partial=0
level=info ts=2023-01-09T21:50:54.192010211Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=248.048608ms durat

Source: thanos-io/thanos

View original on GitHubView discussion on GitHub