#12109·temporal

Invalid system.secondaryVisibilityWritingMode value silently fails every visibility write

Author: tsurdiloCreated Sep 16, 2026Updated Sep 16, 2026
Labelsenhancement

system.secondaryVisibilityWritingMode accepts exactly three values — off, on, dual

So a typo — "duel", or "dual " with a trailing space — is accepted everywhere it is checked, and only fails at write time, on every write, in defaultManagerSelector.writeManagers()

Since this is a cluster wide global setting, a small typo currently can cause a lot of harm:

  • It can end up in visibility tasks being sent to DLQ The error is a serviceerror.Internal. With the default history.TaskDLQInternalErrors=false it is not treated as terminal, so the task retries as an unexpected error — and at history.TaskDLQUnexpectedErrorAttempts (default 70, roughly 70 minutes) the visibility task is sent to the DLQ. Users would need to use tdbg workflow refresh tasks to rebuild which is a pain.

  • Symptom does not point at the cause. Workflow execution is unaffected and visibility reads keep working normally, so the cluster looks healthy. No real way to alert on this, and server starts and operates fine.

Ask:

  • Validate the value where it is loaded, so a bad value is reported by validate-dynamic-config and logged as a warning at startup.
  • Treat an unrecognized value as off and log a warning, so a typo degrades to "primary only" instead of "no visibility writes at all". This keeps the cluster functional and is recoverable without a DLQ replay.