accounting: deglobalize errors in concurrent stats groups
Before you start
- I have searched the forum and the existing issues and this hasn't already been requested.
- I have checked the latest beta and this feature doesn't already exist.
Associated forum post URL
No response
rclone version
rclone v1.75.1
- os/version: darwin 26.6.2 (64 bit)
- os/kernel: 25.6.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.26.8
- go/linking: dynamic
- go/tags: cmount
What problem are you trying to solve?
Several places in rclone decide whether it's safe to delete by checking the stats group's error count (accounting.Stats(ctx).Errored()). On the CLI that's fine, because each run gets its own stats. But under rcd, a stats group can be shared by many operations (an explicit _group, or several runs in one group), so an error from one run can block deletes in another. We now have a whole family of reports and PRs for this, each fixing one gate at a time:
- #9634 / #9635 (approved), #9777 and #9815 (proposed as duplicates of #9635) --
deleteEmptyDirectories - #9717 --
deleteFiles - #9824 / #9825 -- bisync's "errors while building listings" abort
The natural fix -- and the one I've been suggesting on the PRs -- is to have each run check only its own errors, i.e. err / currentError(), and stop reading the group counter. But I don't think we can safely do that yet, because some errors are reported only through the counter and never returned to the run:
- local
List(local.go#L676-L681, #L728, #L759) -- a directory we don't have permission to open, a failedLstaton an entry, or a broken symlink with-L: the error is counted and the listing returns without it. Anything wrapping local (crypt, union, combine, etc.) inherits this. lib/transform.Path(transform.go#L43) -- called by march to build sort keys, and has no error return at all. (This one was largely my fault.)filter.MakeListR(filter.go#L657) -- with--files-from --no-traverse, a file that can't be read is counted and skipped.
From the run's point of view, an unreadable directory therefore just looks empty -- and the group counter is the only thing standing between that and deleting its contents on the other side.
The local behavior was very reasonable when it was added (9c6f3ae82, for #3179): keep going like rsync does, but set the error flag so the command fails and sync won't delete. That was written when there was one global counter, and stats groups came along a couple of months later. I think it's the same kind of problem we had with bisync's globals in 9073d17 -- a design that assumed one run per process, which rc quietly broke.
How do you think rclone should be changed to solve that?
The goal would be to get these errors back to the run that caused them, so that err / currentError() really does include everything, and the deletion gates can stop reading shared state. Once that's done, #9717 and #9825 (and #9635) become straightforward.
An alternative might be a per-run error count in fs/accounting that Error() increments alongside the group's count. That's smaller, but it would need the run identity to travel in the context, and I'm not sure it's cleaner than just returning the errors.
Getting involved
- I'm willing to help implement, test or fund this feature.
Source: rclone/rclone