[Bug]: config.json gets reset to defaults if the disk fills up during startup

Before you file a bug report

• I have checked the issue tracker and have not found an issue that matches the one I'm filing.
• This issue is not a troubleshooting question.
• This issue is not a feature request.
• This issue reproduces on one of the currently supported server versions.
• I have read the contribution guidelines.

Mattermost Server Version

11.8.2 (Enterprise build, no license). The code involved is unchanged on current master.

Operating System

Debian 13 (trixie), arm64. Installed from the tarball, running under systemd.

Steps to reproduce

1. Run Mattermost with the default file-based config (config/config.json).
2. Fill up the disk that config/ lives on.
3. Restart Mattermost (or let systemd restart it).
4. Free up some disk space and start Mattermost again.

Expected behavior

My existing config.json should survive. If Mattermost can't write the config, it should leave the old file alone. If it finds an empty config file on startup, it should fail and tell me, not assume it's a new install.

Observed behavior

All of my config was gone. Mattermost started up with a fresh default config: DataSource pointed back at mmuser:mostest@localhost/mattermost_test, SiteURL was empty, plugin settings were gone, and so on. There was no backup, so I had to rebuild it by hand and reset the database password.

Here's what I think happens, from reading the code:

1. On every startup, Store.Load writes the config back to disk. On the first load it compares against an empty model.Config{}, so hasChanged is always true: https://github.com/mattermost/mattermost/blob/d16665723616ee8b63a127816cbad2022668354f/server/config/store.go#L320
2. FileStore.persist writes with os.WriteFile, which truncates the file before writing. If the write fails with ENOSPC, the truncate has already happened and config.json is left empty: https://github.com/mattermost/mattermost/blob/d16665723616ee8b63a127816cbad2022668354f/server/config/file.go#L120
3. On the next start the file is zero bytes. Store.Load doesn't treat that as an error. It just builds a default config and writes that out: https://github.com/mattermost/mattermost/blob/d16665723616ee8b63a127816cbad2022668354f/server/config/store.go#L259

So a full disk and a restart at the wrong moment is enough to silently reset the whole config.

Log Output

# first start after the disk filled up
Sep 14 17:19:10 mattermost[843]: Error: failed to load configuration: failed to create store: unable to load on store creation: failed to persist: failed to write file: write /opt/mattermost/config/config.json: no space left on device
Sep 14 17:19:22 mattermost[3159]: Error: failed to load configuration: failed to create store: unable to load on store creation: failed to persist: failed to write file: write /opt/mattermost/config/config.json: no space left on device

# later starts, once there was some space again: running on the default config
{"level":"error","msg":"Failed to ping DB","caller":"sql/sql_utils.go:51","database":"master","dataSource":"postgres://****:****@localhost/mattermost_test?binary_parameters=yes&connect_timeout=10&sslmode=disable","error":"pq: password authentication failed for user \"mmuser\""}

Additional Information

Two changes would each have prevented this:

• Write the config atomically: write to a temp file in the same directory, fsync, then rename it over config.json. A failed write then leaves the old file as it was.
• Refuse to start if config.json exists but is empty, instead of treating it like a missing file.

Not skipping the write-back on every startup would also shrink the window, but the first two are the real fix.

#27299 ("config.json wiped out") might be the same bug. It was closed because nobody could reproduce it.