#8235·rustfs

Heal Scheduler continuously reports "File not found" and fails to self-heal after the object is deleted.

Author: aubdiyCreated Sep 29, 2026Updated Sep 30, 2026

LabelsS-confirming

Supported storage confirmation

I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.

Describe the bug

问题描述 (Description) RustFS 集群中 Heal scheduler 持续打印 task_failed 错误，提示 Storage error: File not found。关键异常：即使通过 rc object remove 手动删除了对应的 .incomplete 临时对象，Heal 调度器仍然不断重试该任务并报错，未能正确感知对象已删除并清理任务队列，导致日志刷屏。同时，伴随出现大量 http_request_inflight_slow 警告，ReadVersion RPC 调用耗时超过 5s 阈值，怀疑与 Heal 任务阻塞或元数据不一致有关。
环境信息 (Environment) 组件: rustfs-server / heal-scheduler 部署方式: Kubernetes (bd-rustfs-4 namespace) 受影响 Bucket/Object: dl-ods/adt_callback/.../.incomplete/... 时间范围: 2026-09-24T09:49:13Z 起持续发生

Steps to reproduce

在 oracle 云上使用 OKE( K8S集群, v1.34.2) , rustFS v1.0.0, 上传镜像到私有镜像服务 Tenant文件如下

yaml

apiVersion: rustfs.com/v1alpha1
kind: Tenant
metadata:
  name: tenant-4
  namespace: bd-rustfs-4
  annotations:
    operator.rustfs.com/runtime-default-image-ack: "harbor.yowin.mobi/bd/rustfs@sha256:1486862ee7211988ead3afd7afb188ece8fd48e2f6626c0af0c95f45133f7305"
spec:
  image: harbor.yowin.mobi/bd/rustfs@sha256:1486862ee7211988ead3afd7afb188ece8fd48e2f6626c0af0c95f45133f7305
  env:
    - name: RUSTFS_OBS_METRIC_ENDPOINT
      value: "http://vminsert-vm-stack.monitor.svc:8480/insert/0/opentelemetry/v1/metrics"
    - name: RUSTFS_OBS_METRICS_EXPORT_ENABLED
      value: "true"
    - name: RUSTFS_OBS_TRACES_EXPORT_ENABLED
      value: "false"
    - name: RUSTFS_OBS_LOGS_EXPORT_ENABLED
      value: "false"
    - name: RUSTFS_OBS_SERVICE_NAME
      value: "tenant-4"
    - name: RUSTFS_OBS_METER_INTERVAL
      value: "10s"
  credsSecret:
    name: rustfs-tenant-creds
  pools:
    - name: pool-0
      resources:
        requests:
          cpu: "6"
          memory: "24Gi"
        limits:
          cpu: "7"
          memory: "26Gi"
      servers: 4
      persistence:
        volumesPerServer: 1
        volumeClaimTemplate:
          storageClassName: oci-bv
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
      nodeSelector:
        name: sr-cn-np
      tolerations:
        - effect: NoSchedule
          key: starrocks
          operator: Equal
          value: iscn
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/name: rustfs
              topologyKey: kubernetes.io/hostname

通过 flink2.3.0 的flink-s3-fs-native-2.3.0.jar 向 rustfs 持续写入数据
手动kill 掉一个 rustfs pod, 此时可以正常写入
再次 kill 掉另一个rustfs pod(此时集群只有 2 个 rustfs pod) , flink 端提示不可写入, taskmanager 反复重启
等 rustfs pod 4 个节点完全恢复后, flink 任务恢复正常
查看 rustfs log, 持续报错 "File not found"，集群中出现未完成的上传对象（.incomplete 目录下的临时文件）

bash

{"timestamp":"2026-09-24T09:49:13.033843858Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-88d644b6","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}
{"timestamp":"2026-09-24T09:49:14.889623555Z","level":"ERROR","message":"Heal object recovery failed","event":"heal_object_result","component":"heal","subsystem":"object","task_id":"106887d4-def5-4b05-b58a-a1b49a2b8bf4","bucket":"dl-ods","object":"adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f","result":"recreate_failed","error":"Storage error: File not found","target":"rustfs::heal::task","filename":"crates/heal/src/heal/task/heal_object.rs","line_number":457,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:14.889679562Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"106887d4-def5-4b05-b58a-a1b49a2b8bf4","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Heal task execution failed: Failed to recreate missing object dl-ods/adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f: Storage error: File not found","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:18.998316016Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-c3b0b619","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:19.002503686Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-b075ae89","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5000,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:19.00360811Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-fffab15d","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:20.388922137Z","level":"ERROR","message":"Heal object recovery failed","event":"heal_object_result","component":"heal","subsystem":"object","task_id":"c1610074-74dc-4079-90ac-fb3901abe66f","bucket":"dl-ods","object":"adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f","result":"recreate_failed","error":"Storage error: File not found","target":"rustfs::heal::task","filename":"crates/heal/src/heal/task/heal_object.rs","line_number":457,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}
{"timestamp":"2026-09-24T09:49:20.388979877Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"c1610074-74dc-4079-90ac-fb3901abe66f","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Heal task execution failed: Failed to recreate missing object dl-ods/adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f: Storage error: File not found","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}

Heal 调度器尝试恢复该对象，但因底层存储文件缺失而失败。这个对象 ui 可见,
使用管理员命令手动删除该对象, 删除对象后无法自愈(这个对象是flink checkpoint 创建的)

bash

rc object remove local/dl-ods/adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/bb9a06f6-5d73-4f95-a058-3cf5d50b3105
Removed: local/dl-ods/adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/bb9a06f6-5d73-4f95-a058-3cf5d50b3105
✓ Removed 1 object(s).

确认对象已被移除（✓ Removed 1 object(s).）。
观察： Heal scheduler 仍然以相同 task_id 或新 task_id 反复尝试 heal 该路径，继续报 File not found。

Expected behavior

当对象被手动删除后，Heal 调度器应检测到对象不存在，自动将该任务标记为完成或取消，不再重试。不应产生无限循环的错误日志。

RustFS version

1.0.0

Deployment method

Kubernetes / Helm

Data directory filesystems and mounts

pod中

Source:                                                                                                                   
    Type:              CSI (a Container Storage Interface (CSI) volume source)                                            
    Driver:            blockvolume.csi.oraclecloud.com                                                                    
    FSType:            ext4                                                                                               
    VolumeHandle:      ocid1.volume.oc1.phx.abyhqljtt76bl5beg2e5pbmznmdjgylzlszqy4lbxne4mnsmytlgs2upghbq                  
    ReadOnly:          false                                                                                              
    VolumeAttributes:      attachment-type=                                                                               
                           needResize=false                                                                               
                           newSize=                                                                                       
                           storage.kubernetes.io/csiProvisionerIdentity=1790665326694-1109-blockvolume.csi.oraclecloud.com
                           vpusPerGB=10                                                                                   
Events:                <none>

Server operating system and architecture

Linux 5.15.0-317.197.5.1.el8uek.x86_64 AMD EPYC 7J13 64-Core Processor

Client environment

No response

Relevant logs

bash

{"timestamp":"2026-09-24T09:49:13.033843858Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-88d644b6","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}
{"timestamp":"2026-09-24T09:49:14.889623555Z","level":"ERROR","message":"Heal object recovery failed","event":"heal_object_result","component":"heal","subsystem":"object","task_id":"106887d4-def5-4b05-b58a-a1b49a2b8bf4","bucket":"dl-ods","object":"adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f","result":"recreate_failed","error":"Storage error: File not found","target":"rustfs::heal::task","filename":"crates/heal/src/heal/task/heal_object.rs","line_number":457,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:14.889679562Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"106887d4-def5-4b05-b58a-a1b49a2b8bf4","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Heal task execution failed: Failed to recreate missing object dl-ods/adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f: Storage error: File not found","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:18.998316016Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-c3b0b619","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:19.002503686Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-b075ae89","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5000,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:19.00360811Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-fffab15d","trace_id":"unknown","span_id":"unknown","peer_addr":"10.244.20.132","method":"POST","uri":"http://tenant-4-pool-0-3.tenant-4-hl.bd-rustfs-4.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(9)"}
{"timestamp":"2026-09-24T09:49:20.388922137Z","level":"ERROR","message":"Heal object recovery failed","event":"heal_object_result","component":"heal","subsystem":"object","task_id":"c1610074-74dc-4079-90ac-fb3901abe66f","bucket":"dl-ods","object":"adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f","result":"recreate_failed","error":"Storage error: File not found","target":"rustfs::heal::task","filename":"crates/heal/src/heal/task/heal_object.rs","line_number":457,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}
{"timestamp":"2026-09-24T09:49:20.388979877Z","level":"ERROR","message":"Heal scheduler task failed","event":"heal_scheduler_state","component":"heal","subsystem":"manager","task_id":"c1610074-74dc-4079-90ac-fb3901abe66f","heal_type":"object","set":"pool_0_set_0","state":"task_failed","error":"Heal task execution failed: Failed to recreate missing object dl-ods/adt_callback/20260924/16/p-8f4e89ed-2c14-4ce5-a5ba-fd918f57caa5-5/.incomplete/OTJhMzFmYmEtMTU5OC00OGUyLWI4NTUtZDhkMjFjZTIyM2IxLjgwZjkxOGIzLWQyNzItNDczYy1iZTg3LTJhN2VkMWFkMTY1NXgxNzkwMjM4ODY1NTI2MDY2NzU3/2cce3f4e-b97f-4e76-9dbc-b08dadf4c89f: Storage error: File not found","target":"rustfs::heal::manager","filename":"crates/heal/src/heal/manager/scheduler.rs","line_number":269,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}

Screenshots

No response

Additional context

No response

Source: rustfs/rustfs

View original on GitHub View discussion on GitHub