Repeated delete-marker HEAL/service error in 1.0.0; live worker may miss HEAD 405 handling from #7756

Summary and environment

Two independent SNSD instances use bidirectional native SiteReplication (A ↔ B), with direct peer endpoints and delete-marker replication enabled in both directions. Ordinary versioned deletes can produce repeated HEAL / service error events even when both sites already have the same delete marker.

• rc.6: reproduced with the official Linux x86_64 musl binary in an isolated fixture, without cleanup scripts, resync schedules, Loki/Tempo, or OTLP exporters. Separate tmpfs volumes used the disk-check bypass; scanner settings were default.
• 1.0.0: repeated HEAL observed again after redeployment on physical XFS hosts. This environment includes Loki/Tempo, a version-cleanup timer, and OTLP logging. The isolated fixture has not yet been rerun on 1.0.0.

Reproduction (rc.6)

1. Join two empty SNSD instances using bidirectional SiteReplication.
2. Create a versioned bucket; write four keys twice each on A and wait for replication to B.
3. Delete two keys on A using ordinary DeleteObject without VersionId. Do not configure lifecycle expiry or purge historical versions.
4. Verify that both sites have the same latest delete-marker VersionId. HeadObject for that marker version returns HTTP 405 on both.
5. Observe multiple scanner passes: the deleted keys repeatedly produce events like this (nonessential fields omitted):

{
  "event": "replication_object_failed",
  "bucket": "plain-versions",
  "object": "key1",
  "op_type": "HEAL",
  "replication_status": "FAILED",
  "error": "service error"
}

Before adding any lifecycle rules, we counted 10 HEAL failures on A and 9 on B.

Evidence from 1.0.0

On September 17, 2026 (UTC+08:00), both instances started at 10:27 and remained ready without automatic restarts during inspection:

• Site B logged 8 HEAL failures for loki/index/delete_requests/delete_requests.sqlite.gz between 10:30 and 10:40, with increasing intervals. Ordinary replication failures continued through 11:05.
• At 11:04, complete version listings matched across both sites: Tempo had 122 data versions and 10 delete markers; Loki had one data version.
• Three recently failed Tempo objects had identical data-version and latest marker IDs on both sites. HEAD for each marker returned 405, x-amz-delete-marker: true, and the matching VersionId.

The Tempo samples are ordinary replication failures, distinct from the Loki HEAL events. The earlier Loki marker was no longer in the listing when inspected, so its HEAD response was not captured. These observations support the same classification hypothesis but do not prove every event has the same cause.

Suspected code path

The following excerpts are from 1.0.0.

The live worker's replicate_delete_to_target checks only modeled SDK fields after the marker-version HEAD:

let non_retryable = matches!(
    e.as_ref(),
    SdkError::ServiceError(service_err)
        if is_retryable_delete_replication_head_error(
            service_err.err().is_not_found(),
            service_err.err().code(),
        )
);
if non_retryable {
    rinfo.replication_status = ReplicationStatusType::Failed;
    rinfo.error = Some(e.to_string());
    return rinfo;
}

Its helper is:

pub fn is_retryable_delete_replication_head_error(is_not_found: bool, code: Option<&str>) -> bool {
    !(is_not_found || matches!(code, Some("MethodNotAllowed" | "405")))
}

For a bodiless 405 with is_not_found() == false and code() == None, this returns true, causing the worker to record failure.

By contrast, #7756 addresses this in verify_resync_head_result using the raw HTTP status:

let not_found = is_not_found || has_raw_status(&err, 404);
if roi.version_purge_status.is_empty() {
    let code = code.or_else(|| has_raw_status(&err, 405).then_some("405"));
    is_retryable_delete_replication_head_error(not_found, code)
} else {
    !not_found
}

Expected behavior and scope

Could equivalent raw-status handling be applied to the live delete worker, with regression coverage for ordinary replicated deletes followed by scanner HEAL? An already-propagated marker should not repeatedly fail because the SDK omits its error code. Marker-version purge must remain distinct: 405 means the marker still exists, not that a purge succeeded. Other ambiguous errors must not become blanket successes.

Separately, site B showed superseded scanner backoff (1280 seconds) and stale usage statistics. Allocator reclaim was progressing on both 1.0.0 instances, and OTLP logs reached Loki. The short observation window does not establish a memory leak or a causal link between HEAL and the scanner symptoms; this issue focuses on delete-marker replication errors.