小型 Thanos 实例中的压缩机 S3 调用量较大

Object Storage Provider: AWS S3 What happened: I am seeing a very large amount of requests billed as "Amazon Simple Storage Service EUN1-Requests-Tier2" in a test account with a small Thanos installation. I'm seeing about 40 Million requests per month, which comes down to around 60,000 requests per hour, or 16-20 requests / second. What you expected to happen: I would have expected a small instance to perform significantly less requests against S3. This is with a very small instance of Thanos, the S3 bucket having 1.5GB of storage in about 550 files, containing a bit over 2 years of data at fairly low resolution (1-5 minutes) Full logs to relevant components: Looks like almost all of the requests comes from the compactor, in an about 50/50 mix between "HEAD [/]index.json", or GET requests for "deletion-mark.json" or "no-compact-mark.json"

$ wc -l s3.log
1 338 023 s3.log
$ grep -c 'HEAD.*/meta.json' s3.log
649 613
$ grep -c 'NoSuchKey' s3.log
610 258

Example logs from S3

thanos.i.example.com [09/Jan/2023:21:46:22 +0000] 1.2.3.4 arn:AWS:iam::012345678910:user/test-infra-thanos XCJRGVE9EF1QQVQC REST.GET.OBJECT test-infra/01GJG6H63WAJ3CJ8WH20SYVK5R/deletion-mark.json "GET
…