vacuum never runs once volumes are marked read-only by low disk space, so a full store cannot reclaim deleted data

Version

4.48 (530be3e37337488ecc34d58441e0bc476e121c93), reproduced on a stock weed server with no special configuration.

What happens

Once a volume server's disk drops to the point where its volumes are marked read-only, vacuum stops running entirely — the automatic sweep and an explicit volume.vacuum both do nothing, no matter how much garbage there is. Deleting every object in the store does not get the space back, so a store that fills up cannot recover on its own.

The same rig reclaims perfectly when the volumes are not read-only, which is what makes the contrast clear:

How to reproduce

A small dedicated filesystem makes the state easy to reach:

dd if=/dev/zero of=disk.img bs=1M count=2048
mkfs.ext4 -q -F disk.img
mount -o loop disk.img /mnt/w

weed server -dir=/mnt/w/data -ip=127.0.0.1 \
  -master.volumeSizeLimitMB=32 \
  -volume.max=100 -volume.minFreeSpacePercent=20 \
  -filer -s3

Then, with the aws CLI against the S3 port:

1. aws s3 mb s3://floor
2. PUT 8 MiB objects in a loop until one fails (here: 241 objects, then 500 InternalError). The disk is now full and every volume reports ReadOnly:true.
3. aws s3 rm s3://floor --recursive — all 241 objects gone, garbage ratio 1.00.
4. Wait. Nothing is reclaimed, through t+16min — well past the 840s sweep interval.
5. Restart weed, then lock + volume.vacuum -garbageThreshold 0.01. Still nothing.

Observed via the volume server's /status:

t+14min  1.9G used, 0 free (100%) | volumes=70 read-only=70 garbage=1928 MiB
t+15min  1.9G used, 0 free (100%) | volumes=70 read-only=70 garbage=1928 MiB
t+16min  1.9G used, 0 free (100%) | volumes=70 read-only=70 garbage=1928 MiB
restart + volume.vacuum
after:   1.9G used, 0 free (100%) | volumes=70 read-only=70 garbage=1928 MiB

Control, identical except the volumes never went read-only (stopped filling at 12% free): the automatic sweep reclaimed everything on schedule, 1.6G used → 7.1M used, garbage 1584 MiB → 0, at exactly t+14min. So the sweep itself is healthy; it is the read-only state that stops it.

Why this matters

This is the one situation where reclaiming garbage matters most — the store is full — and it is exactly when it stops working. The operator deletes everything, watches the space not come back, and has no way forward from the S3 API.

On a larger store I saw the intermediate case: at the minFreeSpacePercent floor (9% free, not 0), vacuum was equally dead for 40+ minutes, but restarting the volume server revived it and a manual volume.vacuum then reclaimed 837 GB → 1 GB immediately. With truly 0 bytes free, as above, even the restart does not help — presumably because compaction has nowhere to write its scratch copy.

Expected

A volume that is read-only because the disk is low on space is precisely the one that should be vacuumed — compacting it is what frees the space. Either the sweep should include read-only volumes when their garbage ratio exceeds the threshold, or there should be a documented way to reclaim without restarting or growing the disk.

At minimum, volume.vacuum returning silently with no output and no effect makes this very hard to diagnose — an error or a "skipped N read-only volumes" line would help a lot.