Windows: transient TSDB staging-file rename failure aborts WAL recovery

Describe the bug

Loki fails to start during TSDB WAL recovery on Windows when rebuilding TSDB files from WALs. During startup, Builder.Build() creates a temporary .staging TSDB file in the scratch directory and then attempts to rename it into the multitenant index directory. In a very rare occurrence, the rename operation fails with:

Access is denied.

This causes buildTSDBFromWALs() to fail, store initialization to abort, and the entire Loki process to exit.

The failure occurs after the staging file has been opened through index.NewMmapFileReader(). Although the reader is closed before os.Rename(), Windows may still consider the file in use briefly, causing os.Rename() to return ERROR_ACCESS_DENIED. The staging file is subsequently removed by deferred cleanup, making post-mortem investigation difficult because the source file no longer exists after the failure.

To Reproduce

Steps to reproduce the behavior:

1. Start Loki v3.6.15 on Windows.

2. Configure TSDB storage on a filesystem path.

3. Allow Loki to generate WALs.

4. Restart Loki so it performs WAL recovery.

5. Observe startup occasionally fail during buildTSDBFromWALs() with: rename : Access is denied

6. Loki exits with:

error initialising module: store

The issue is extremely rare but results in a complete failure of Loki startup.

Expected behavior

A transient filesystem rename failure should not cause Loki startup to fail permanently.

Possible expected outcomes:

• Retry the rename operation.
• Handle transient ERROR_ACCESS_DENIED errors gracefully.
• Successfully initialize the store after WAL recovery.
• At minimum, emit additional filesystem diagnostics before terminating.

Environment:

• Infrastructure: Windows Server VM
• Deployment tool: Native executable
• Loki version: 3.6.15
• Storage: TSDB filesystem storage
• Single Loki instance

Logs:

rename \loki\data\tsdb-shipper-active\scratch\filesystem_2020-10-24\index-3600768d42cd7949.staging \loki\data\tsdb-shipper-active\multitenant\index_20693\1790672493-50c5718a9ca3-1787320194439188800.tsdb: Access is denied. building tsdb from WALs github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*HeadManager).buildTSDBFromWALs /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/head_manager.go:322 github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*HeadManager).Start /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/head_manager.go:283 github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*store).init /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:130 github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.NewStore /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:59 github.com/grafana/loki/v3/pkg/storage.(*LokiStore).storeForPeriod /src/loki/pkg/storage/store.go:294 github.com/grafana/loki/v3/pkg/storage.(*LokiStore).init /src/loki/pkg/storage/store.go:210 github.com/grafana/loki/v3/pkg/storage.NewStore /src/loki/pkg/storage/store.go:189 github.com/grafana/loki/v3/pkg/loki.(*Loki).initStore /src/loki/pkg/loki/modules.go:930 github.com/grafana/dskit/modules.(*Manager).initModule /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 github.com/grafana/dskit/modules.(*Manager).InitModuleServices /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 github.com/grafana/loki/v3/pkg/loki.(*Loki).Run /src/loki/pkg/loki/loki.go:531 main.main /src/loki/cmd/loki/main.go:129 runtime.main /usr/local/go/src/runtime/proc.go:283 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1700 building tsdb from old WAL files github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*HeadManager).Start /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/head_manager.go:284 github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*store).init /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:130 github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.NewStore /src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:59 github.com/grafana/loki/v3/pkg/storage.(*LokiStore).storeForPeriod /src/loki/pkg/storage/store.go:294 github.com/grafana/loki/v3/pkg/storage.(*LokiStore).init /src/loki/pkg/storage/store.go:210 github.com/grafana/loki/v3/pkg/storage.NewStore /src/loki/pkg/storage/store.go:189 github.com/grafana/loki/v3/pkg/loki.(*Loki).initStore /src/loki/pkg/loki/modules.go:930 github.com/grafana/dskit/modules.(*Manager).initModule /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 github.com/grafana/dskit/modules.(*Manager).InitModuleServices /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 github.com/grafana/loki/v3/pkg/loki.(*Loki).Run /src/loki/pkg/loki/loki.go:531 main.main /src/loki/cmd/loki/main.go:129 runtime.main /usr/local/go/src/runtime/proc.go:283 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1700 error initialising module: store github.com/grafana/dskit/modules.(*Manager).initModule /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138 github.com/grafana/dskit/modules.(*Manager).InitModuleServices /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 github.com/grafana/loki/v3/pkg/loki.(*Loki).Run /src/loki/pkg/loki/loki.go:531 main.main /src/loki/cmd/loki/main.go:129 runtime.main /usr/local/go/src/runtime/proc.go:283 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1700 level=error ts=2026-09-29T09:07:59.8952898Z caller=log.go:223 msg="error running loki" err="rename \loki\data\tsdb-shipper-active\scratch\filesystem_2020-10-24\index-3600768d42cd7949.staging \loki\data\tsdb-shipper-active\multitenant\index_20693\1790672493-50c5718a9ca3-1787320194439188800.tsdb: Access is denied.\nbuilding tsdb from WALs\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*HeadManager).buildTSDBFromWALs\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/head_manager.go:322\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*HeadManager). Start\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/head_manager.go:283\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*store).init\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:130\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.NewStore\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:59\ngithub.com/grafana/loki/v3/pkg/storage.(*LokiStore).storeForPeriod\n\t/src/loki/pkg/storage/store.go:294\ngithub.com/grafana/loki/v3/pkg/storage.(*LokiStore).init\n\t/src/loki/pkg/storage/store.go:210\ngithub.com/grafana/loki/v3/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:189\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:930\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:531\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nbuilding tsdb from old WAL files\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*HeadManager).Start\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/head_manager.go:284\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.(*store).init\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:130\ngithub.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb.NewStore\n\t/src/loki/pkg/storage/stores/shipper/indexshipper/tsdb/store.go:59\ngithub.com/grafana/loki/v3/pkg/storage.(*LokiStore).storeForPeriod\n\t/src/loki/pkg/storage/store.go:294\ngithub.com/grafana/loki/v3/pkg/storage.(*LokiStore).init\n\t/src/loki/pkg/storage/store.go:210\ngithub.com/grafana/loki/v3/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:189\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:930\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:531\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:531\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"