[Bug]: standalone panics with fatal error: concurrent map iteration and map write shortly after startup, no query traffic required

Is there an existing issue for this?

• I have searched the existing issues (gh issue list --repo milvus-io/milvus --search "PopulateEvents" / "file_source.go" / "refreshPeriodically" — no matches as of 2026-09-17)

Environment

- Milvus version: v2.5.6 standalone, reproduced identically against every later release in
  the 2.5.x and 2.6.x lines as of 2026-09-17 (see "Anything else?" below) — this is not
  version-specific
- Deployment mode: standalone (single container, `command: ["milvus", "run", "standalone"]`)
- MQ type: N/A (standalone, no external MQ configured)
- SDK version: N/A — the crash requires no client connection, see Steps To Reproduce
- OS: Ubuntu (host), official `milvusdb/milvus` container image
- CPU/Memory: `mem_limit: 4g` in the reproducing docker-compose service; `docker inspect`
  confirms `OOMKilled=false` on every crash — not a memory-pressure issue
- Others: `COMMON_STORAGETYPE: local`, `ETCD_ENDPOINTS` pointed at a sibling `etcd` container
  in the same compose network

Current Behavior

The standalone process crashes with a Go fatal error: concurrent map iteration and map write (SIGABRT, process exit code 134) within about two minutes of startup, with no query traffic and no client connection required — the crash is triggered purely by Milvus's own internal periodic config-reload.

Confirmed goroutine frames from the crash dump (docker logs):

goroutine ... [running]:
runtime.mapiternext(...)
	.../runtime/map.go:862
github.com/milvus-io/milvus/pkg/v2/config.PopulateEvents({0x..., 0xa}, 0x..., 0x...)
	/workspace/source/pkg/config/event.go:47
github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).update(0x..., 0x...)
	/workspace/source/pkg/config/file_source.go:170
github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).loadFromFile(0x...)
	/workspace/source/pkg/config/file_source.go:159
github.com/milvus-io/milvus/pkg/v2/config.(*FileSource).loadFromFile-fm()
	<autogenerated>:1
github.com/milvus-io/milvus/pkg/v2/config.(*refresher).refreshPeriodically(0x..., {0x..., 0xa})
	/workspace/source/pkg/config/refresher.go:71-73
github.com/milvus-io/milvus/pkg/v2/config.(*refresher).start.func1.gowrap1()
	/workspace/source/pkg/config/refresher.go:52
runtime.goexit({})

i.e. the panic fires inside config.PopulateEvents (pkg/config/event.go:47, the for key, value := range updatedConfig loop), reached via FileSource.update → FileSource.loadFromFile, driven by refresher.refreshPeriodically (pkg/config/refresher.go:71-73) — the periodic (default 5-second) config-file reload path every standalone instance runs by default.

Expected Behavior

The periodic config-reload should not crash the process. Either the map passed into PopulateEvents is being mutated concurrently by another goroutine while this loop iterates it (a genuine data race — go build -race against a reproduction would confirm), or PopulateEvents/its callers need defensive copying before iterating a map that may be written elsewhere without holding the source's own lock.

Steps To Reproduce

1. docker compose up -d etcd milvus   # standalone mode, local storage, no query traffic
2. Wait ~2 minutes.
3. `docker ps` shows the milvus container `Exited (134)`.
4. `docker logs <container>` shows the goroutine dump above.

Reproduced identically on two independent restarts in the same environment, each crashing within approximately two minutes of a clean start — this is not a one-off, and not related to prior container state (docker start on a fresh container each time).

Milvus Log

See the goroutine dump under "Current Behavior" above (trimmed to the relevant frames; full capture available on request).

Anything else?

Confirmed NOT version-specific, checked against source, not assumed: pkg/config/file_source.go (the file at the top of this crash's call stack, containing update()/loadFromFile()) is byte-identical between v2.5.6 and v2.5.27 (the newest 2.5.x release as of this report) — fetched both via the GitHub Contents API and diffed directly, zero output. Comparing v2.5.6 against v2.6.24 (the newest 2.6.x release) shows only cosmetic differences (error-wrapping style via errors.Wrapf, and fs.RWMutex.Lock() → fs.Lock(), the same promoted method spelled two ways) — update()'s locking is functionally unchanged. This race appears to be present in every currently published Milvus release, 2.5.x and 2.6.x alike.

Root-cause hypothesis, offered as a starting point, not a proven diagnosis: FileSource.update() and the structurally identical EtcdSource.update() (pkg/config/etcd_source.go) both take their own updateMu+Lock() before calling PopulateEvents(source, currentMap, newMap), and look correctly synchronized in isolation — currentMap (fs.configs/es.currentConfigs) is protected by that lock, and the new map is a fresh local variable built by loadFromFile/refreshConfigurations with no other reference. The concurrent writer was not identified from the post-mortem panic trace alone — that would need a live goroutine/pprof dump captured at the moment of the crash (not available to us in this environment without root access inside the container), so we deliberately don't assert a specific fix. A reasonable next step for maintainers: check whether another ConfigSource implementation, or a manager-level cache-eviction/event-firing path, ever reads fs.configs/es.currentConfigs by direct reference (rather than through GetConfigurations()'s already-copying accessor) without holding that source's own lock.

This report was drafted by an AI coding agent (Claude, via Anthropic's Claude Code) investigating a CI/test-infrastructure flake in an unrelated downstream project. Every claim above about Milvus's own source was verified directly against the linked file paths at the cited tags via the GitHub API, not assumed from memory or documentation. Filed for maintainer review — happy to provide the full log capture or any additional detail on request.