Retention/TTL management unusable on v0.137.1 self-hosted; one path silently deletes logs (ttl=0 written to Distributed table defaults)

Retention/TTL management is unusable on v0.137.1 (self-hosted), and one path silently deletes logs

Version: SigNoz v0.137.1, chart signoz-0.137.1, otel-collector v0.144.8 Deployment: self-hosted, Kubernetes (EKS), ClickHouse via clickhouse-operator, S3 cold storage enabled

Summary

There is no working way to configure log retention with S3 cold storage on this version. Three separate paths fail, and one of them permanently deleted ~15 minutes of production log rows and ~3.5 hours of resource-attribute mappings — while returning HTTP 200 with a success message.

Goal was ordinary: logs hot 7 days on local disk, then S3, delete at 30 days.

1. Settings UI submits negative durations, so Save always 400s

Setting the total retention period without also setting the cold-storage field submits toColdDuration=-1h. Entering "1 day" in the cold-storage field submits -24h — the value with the sign inverted.

Backend log:

POST /api/v1/settings/ttl → 400
{"errors":[{"code":400,"msg":"not a valid toCold TTL duration -1h"}]}
{"errors":[{"code":400,"msg":"not a valid toCold TTL duration -24h"}]}

GET /api/v1/settings/ttl?type=logs returns logs_move_ttl_duration_hrs: -1 when unset, and the UI appears to resubmit that sentinel for whichever field the user does not touch. The validator then rejects it. Net effect: retention cannot be saved from the UI at all.

2. POST /api/v1/settings/ttl refuses any valid cold-storage duration

POST /api/v1/settings/ttl?type=logs&duration=720h&coldStorage=s3&toColdDuration=168h
→ 500 {"errors":[{"code":500,"msg":"SetTTLV2 only supported"}]}

Note the asymmetry: omitting toColdDuration gets past this and fails at validation instead. So v1 accepts requests without cold storage and hard-refuses ones with it, pointing at v2.

GET /api/v1/disks correctly returns [{"name":"default","type":"Local"},{"name":"s3","type":"ObjectStorage"}], so the backend does see the disk.

3. POST /api/v2/settings/ttl returns 200, changes nothing, and causes log deletion

This is the serious one.

POST /api/v2/settings/ttl
{"type":"logs","default_ttl_days":30,"cold_storage_ttl_days":7}
→ 200 {"message":"custom retention TTL has been successfully set up"}

Nothing was applied. signoz_logs.logs_v2.storage_policy stayed default, no TO VOLUME rule appeared, and the s3 disk stayed empty. Backend logged:

No valid conditions found, returning default TTL

Neither payload field was mapped. In ttl_setting, the inserted rows show ttl = 0 (the column's DEFAULT 0) and cold_storage_ttl = -1 — not 30 and 7:

table_name           ttl  cold_storage_ttl  condition  created_at
signoz_logs.logs_v2  30   -1                []         2026-07-30   <- pre-existing, correct
signoz_logs.logs_v2  0    -1                null       2026-08-26   <- created by the POST

Note the pre-existing rows have condition = '[]' where the new ones have null.

Consequence: log deletion. logs_v2 TTL is toDateTime(timestamp/1e9) + toIntervalDay(_retention_days). The _retention_days column default went 30 → 0, and the writer began stamping 0 on every new row — TTL expired on write, and TTL merges dropped the parts within minutes. Roughly 15 minutes of logs were permanently lost before it was caught.

GET endpoints continued reporting "status":"success" throughout. Nothing surfaced the problem.

Expected

Either reject the payload, or apply it. A 200 with a success message that (a) applies nothing and (b) sets a retention value causing immediate deletion is the worst combination.

Suggested fixes

• Validate/require the fields the v2 handler actually reads, and 400 on unknown/missing ones.
• Never let a retention write result in ttl = 0 — clamp or reject.
• Guard the TTL expression against _retention_days = 0 so a bad config cannot expire rows on write.

4. Mechanism: the ttl = 0 is written to FOUR tables' column defaults, including Distributed ones

This is what turns #3 from a config bug into data loss, and it took a while to pin down.

The ttl_setting row with ttl = 0 is applied by SigNoz as a column DEFAULT on the log tables. We found it on four:

The collector never sets the column. system.query_log shows the real insert:

INSERT INTO signoz_logs.distributed_logs_v2 (
    ts_bucket_start, resource_fingerprint, timestamp, observed_timestamp, id,
    trace_id, span_id, trace_flags, severity_text, severity_number, body,
    attributes_string, attributes_number, attributes_bool, resources_string,
    resource, scope_name, scope_version, scope_string, inserted_at
) FORMAT Native

_retention_days is absent, so the DEFAULT supplies it — and because the target is a Distributed table, the Distributed table's default materialises before the row is forwarded. The local table's default is never consulted.

Practical consequence for anyone recovering from this: fixing ALTER TABLE signoz_logs.logs_v2 MODIFY COLUMN _retention_days UInt16 DEFAULT 30 alone has no effect. You must fix distributed_logs_v2 (and distributed_logs_v2_resource, logs_v2_resource) too.

The resource table amplifies this badly

logs_v2_resource holds the fingerprint -> resource-attribute mapping, and its TTL is:

TTL (toDateTime(seen_at_ts_bucket_start) + toIntervalDay(_retention_days)) + toIntervalSecond(1800)

At _retention_days = 0 that evaluates to seen_at + 30 minutes. Every resource row written after the POST expired almost immediately. Hourly fingerprint counts showed zero rows for three consecutive hours, with the newest surviving ret = 30 row dating from just before the POST.

Log rows in that window survive, but their resource attributes can no longer be resolved — so filtering or grouping logs by service.name silently returns incomplete results for that period. That is arguably worse than losing the log rows outright, because nothing surfaces it as missing.

Suggested fixes

• When applying retention, validate the value before writing it as a column DEFAULT; 0 should never be accepted for a TTL-driving column.
• Apply DEFAULT changes to the Distributed tables too, or the local-table change is a no-op on the insert path.
• Guard both TTL expressions against 0, e.g. toIntervalDay(if(_retention_days = 0, 30, _retention_days)). This alone would have prevented all data loss here.

Workaround, for anyone hitting this

Configure tiering directly in ClickHouse, bypassing SigNoz:

ALTER TABLE signoz_logs.logs_v2 MODIFY SETTING storage_policy = 'tiered';

SET materialize_ttl_after_modify = 0;
ALTER TABLE signoz_logs.logs_v2 MODIFY TTL
  toDateTime(timestamp / 1000000000) + toIntervalDay(7) TO VOLUME 's3',
  toDateTime(timestamp / 1000000000)
    + toIntervalDay(if(_retention_days = 0, 30, _retention_days)) DELETE;

Also fix the column DEFAULTs on the tables that actually receive inserts:

ALTER TABLE signoz_logs.distributed_logs_v2          MODIFY COLUMN _retention_days UInt16 DEFAULT 30;
ALTER TABLE signoz_logs.distributed_logs_v2_resource MODIFY COLUMN _retention_days UInt16 DEFAULT 30;
ALTER TABLE signoz_logs.logs_v2_resource             MODIFY COLUMN _retention_days UInt16 DEFAULT 30;

-- and the same guard on the resource table
SET materialize_ttl_after_modify = 0;
ALTER TABLE signoz_logs.logs_v2_resource MODIFY TTL
  (toDateTime(seen_at_ts_bucket_start)
     + toIntervalDay(if(_retention_days = 0, 30, _retention_days))) + toIntervalSecond(1800);

The if(_retention_days = 0, 30, ...) wrappers are what stop #4 from deleting data. Any subsequent use of SigNoz's retention API overwrites them, re-exposing the bug.

This is verified working — 154 GiB tiered to S3, queries served transparently from object storage.

Environment notes

Cold storage itself is fine. S3 writes and reads both verified (MOVE PARTITION ... TO VOLUME 's3', objects confirmed present in the bucket, rows read back intact). The tiered storage policy rendered by the chart is correct. The problem is entirely in retention configuration.