Secrets from `env_file` are persisted unredacted in Update logs since 2.3.0 (Compose Config stdout)
Filing this publicly because there's no SECURITY.md or private vulnerability reporting enabled on the repo. It involves credentials being stored where they shouldn't be, so if you'd rather this lived somewhere private, say the word and I'll pull it down and refile. Happy to go through Discord instead.
What I run
A home lab. Komodo Core on one VM, Periphery on five others, about nine stacks.
I don't keep application secrets in Komodo. They live in 1Password, and a
pre-deploy hook on each Periphery renders *.env.tpl files into .env files
next to the compose file, which the service picks up with env_file:.
That's deliberate. Each VM only holds the secrets for the services running on it,
so if the media box is compromised it doesn't give up the LDAP bind password.
Rotation is also just "change it in 1Password and redeploy", since the hook
re-reads on every deploy. Komodo's [secrets] block is a single flat namespace on
the Core host, so putting everything there would centralise the blast radius and
turn rotation into a config-management run. I do use [secrets] for the things
Komodo itself needs, like Pushover and registry credentials, which matters below.
After upgrading 1.19.5 -> 2.3.2 I found all of those secrets sitting in plaintext in Komodo's Update history.
What happens
Since 2.3.0, bin/periphery/src/api/compose.rs pushes the Compose Config log
into res.logs on the success path:
config_log.sanitize(&replacers);
res.logs.push(config_log);In 1.19.5, 2.0.0 and 2.2.0 that push only existed inside if !config_log.success.
On success the output went to res.merged_config and was never persisted as a log.
Counting the pushes in that file across tags:
| version | res.logs.push(config_log) |
|---|---|
| v1.19.5 / v2.0.0 / v2.2.0 | 1 (failure only) |
| v2.3.0 / v2.3.2 / v2.3.3 | 3 (failure, parse error, and success) |
That change came in with "Fix compose config sanitization ordering" (re #868), and
it's clearly meant to be safe, since sanitize(&replacers) was added right before
each push. That's the second half of the problem: replacers comes from
interpolator.secret_replacers, so it only holds values Komodo interpolated
itself. docker compose config resolves env_file: natively, those values never
went through the interpolator, so there's no replacer for them and sanitize does
nothing.
Neither change is a problem alone. Together, every successful deploy writes the
fully resolved compose config, including every value from every env_file, into a
stored Update.
Worth heading off one thing: the failure path persisted the log before 2.3.0 too,
so in principle this could have leaked on a failed deploy. In practice it didn't.
Across 161 pre-upgrade deploys of mine, 53 failed, only 3 persisted a Compose
Config log, and all 3 had zero bytes of stdout with the error on stderr. When
compose config fails it never produces the resolved document, so there was
nothing to leak. The success path is what turned this into real exposure.
Evidence
Same stack, same compose file, same secrets, before and after the upgrade:
- 1.19.5 (Jan-Aug): no credential values in any stored Update
- 2.3.2 (Sep): 8-9 credential values in the
Compose Configlog of every deploy
Across the fleet that's 17 real credentials in 123 stored Updates, the oldest from the day I upgraded.
The control case is in the same logs. One stack uses Komodo [secrets] with
[[VAR]] interpolation, and it renders as it should:
APPRISE_STATELESS_URLS: pover://<PUSHOVER_USER_KEY>@<PUSHOVER_APP_TOKEN>So the sanitiser works. It just can't see anything Komodo didn't put there.
This will likely impact others
compose_cmd_wrapper documents these examples:
/// - "op run -- [[COMPOSE_COMMAND]]" (1password CLI)
/// - "sops exec-file --no-fifo /path/to/secret.env '[[COMPOSE_COMMAND]]'" (sops)and the field below it says:
/// Set to ["config", "build", "pull", "up"] for sops exec-file with {} placeholder.config is in that list, so anyone following it runs docker compose config
inside the secret-injecting wrapper and the resolved secrets get stored the same
way. The include list defaults to ["up"], so whether this happens depends on a
setting most people would change for an unrelated reason.
Affected: anyone whose secrets reach the container through env_file, or through
the wrapper with config included. Not affected: Komodo Variables and [secrets]
users, and anyone whose app pulls from Vault at runtime.
Suggested fix
Don't persist the success-path Compose Config log. That restores pre-2.3.0
behaviour, and res.merged_config already carries the output for the UI.
Extending the sanitiser can't fully solve this, since it can only redact values
Komodo already knows about.
If the success log is useful for debugging, putting it behind a per-stack opt-in would at least make it a choice.
#1601 doesn't cover this, and says so: "Literal secrets in Compose file contents or external env files are outside this key/value seam."
Versions
Komodo 2.3.2, upgraded from 1.19.5. Still present in 2.3.3.
Source: moghtech/komodo