Is SeaweedFS suitable as a transparent S3 read cache for Milvus (vector DB) ?

Use Case

We are using Milvus (vector database, v2.6.21) for biometric search (1:N face/fingerprint matching). Milvus stores segment data in Ceph RGW S3 object storage. On each search, Milvus loads a partition (~512MB segment) from S3 into GPU VRAM, searches, then releases it.

We want to add a read cache layer between Milvus and Ceph to speed up repeated partition loads. We deployed SeaweedFS 4.47 as an S3 cache proxy with tmpfs (RAM) storage.

Current Configuration

SeaweedFS deployment: Single-pod (master + volume + filer + S3 gateway) on Kubernetes, pinned to one node with 1TB RAM.

Startup command:

weed server -s3 -filer -dir=/data \
  -master.volumeSizeLimitMB=30000 \
  -master.defaultReplication=000 \
  -volume.max=100 \
  -volume.allowUntrustedRemoteEndpoints \
  -filer.allowUntrustedRemoteEndpoints \
  -s3.config=/data/s3_config.json

Remote mount:

remote.configure -name=ceph -type=s3 \
  -s3.endpoint=https://10.10.103.14:443 \
  -s3.region=us-east-1 \
  -s3.access_key=$CEPH_ACCESS_KEY \
  -s3.secret_key=$CEPH_SECRET_KEY

remote.mount -dir=/buckets/abis-milvus-storage \
  -remote=ceph/abis-milvus-storage

Storage: tmpfs (RAM-backed emptyDir), 400Gi limit

S3 credentials: Static JSON config file loaded via -s3.config

Milvus S3 config: Points to seaweedfs-s3:8333 (HTTP, no TLS) instead of Ceph directly

What Works

• S3 API works — Milvus can read/write/delete through SeaweedFS
• First read fetches from Ceph, caches in tmpfs
• Repeat reads hit tmpfs cache (faster than Ceph direct)

Problems We Are Seeing

1. Unreliable under sustained load: When Milvus issues back-to-back load+search+release cycles (6 workers, 13 partitions per batch), batches fail with context deadline exceeded on partition loads. 10 out of 13 consecutive batches failed. With Ceph direct (no SeaweedFS), all batches succeed.

2. Cache invalidation on deletes: Milvus deletes old WAL segments (S3 DeleteObject) as part of its garbage collection. SeaweedFS forwards these deletes to Ceph, but we're unsure if the local tmpfs cache is properly invalidated. Could stale cache entries cause read failures?

3. remote.sync fragility: We added remote.sync to upload locally-written files back to Ceph and evict local copies. It required a tail -f /dev/null | hack to keep the weed shell process alive in the background. Is there a better way to handle write-back + local eviction?

4. Vacuum interference: SeaweedFS volume vacuum runs periodically and we're concerned it may evict cached data that's still needed. Is there a way to exclude remote-mounted paths from vacuum?

Questions

1. Is this use case supported? Using SeaweedFS as a transparent S3 read cache (remote.mount + tmpfs) in front of another S3-compatible store (Ceph RGW)?

2. Is there a recommended configuration for read-heavy cache workloads? Our workload is: ~90% reads (partition loads), ~10% writes/deletes (WAL truncation, compaction).

3. How does cache invalidation work with remote.mount? When a DeleteObject comes in, does SeaweedFS invalidate the local cache entry, or just forward the delete to Ceph?

4. Is there a way to make the cache read-only? We want writes to go directly to Ceph, but reads to go through SeaweedFS cache. (We know Milvus uses a single S3 endpoint, so we can't split reads/writes at the application level.)

5. Any tuning for sustained concurrent reads? 6 workers issuing concurrent GetObject requests for ~512MB objects. Are there connection pool or concurrency settings we should tune?

Environment

• SeaweedFS: 4.47 (single-pod, master+volume+filer+S3)
• Storage: tmpfs (RAM), 400Gi
• Backend: Ceph RGW S3 (HTTPS, self-signed cert)
• Kubernetes: single node, 80 CPU, 1TB RAM
• Workload: Milvus 2.6.21-gpu, 13 collections, ~512MB segments, page-mode (load→search→release per batch)

seaweedfs-github-issue.yaml