Config syncing still stops entirely when any config file fails to load
Describe the bug
#5020 reports that a single bad connection block stops every connection from syncing until the service is restarted. #5021 addresses the two connection-level triggers (a duplicate connection name and an invalid schema name) by skipping the offending connection with a warning.
The mechanism that causes the freeze is still in place. handleFileWatcherEvent
(pkg/connection/connection_watcher.go:63) returns at line 78 when the config load reports an
error, before reaching RefreshConnections at line 121. Since LoadConnectionConfig re-parses
the entire config folder on every event, any persistent error in that folder means no file event
ever reaches RefreshConnections again.
Triggers that still fail the whole load:
- Malformed HCL in any
.spcfile in the config folder —loadConfigreturns atpkg/steampipeconfig/load_config.go:250or:255. - A duplicate plugin block label —
addPlugin,load_config.go:281. - A disallowed or duplicated options block —
optionsBlockPermitted,load_config.go:302.
The symptom is identical to #5020 and just as hard to diagnose: the file watcher keeps running,
existing connections keep answering queries, connections added afterwards never get a schema or a
steampipe_connection_state row, and rewritten credentials are never applied. A half-written
.spc file from an automated generator is an easy way to hit this.
Expected behavior
One bad file should disable what is in that file, not stop the config from syncing at all. The watcher should refresh with whatever loaded successfully, and report the parts it could not load.
Proposed fix
- Have
handleFileWatcherEventproceed toRefreshConnectionswith the config that did load, rather than returning early on any load error. - Treat per-file and per-block failures as warnings against the offending item, as #5021 does for connections, so the rest of the folder still loads.
- Decide what should happen when the whole folder fails to parse, where there is nothing to refresh with.
- Add a regression test over the watcher path — the tests added in #5021 call
loadConfigdirectly, so nothing currently covers the point where the freeze occurs.
Source: turbot/steampipe