dispatch-upstream-timeout is never applied; outbound dispatches always use the 60s default

Summary

--dispatch-upstream-timeout (SPICEDB_DISPATCH_UPSTREAM_TIMEOUT) has no effect. The configured value never reaches the outbound dispatch client, which falls back to the hardcoded 60s default. Setting the flag to any value is silently a no-op.

Affects v1.56.0, v1.56.1, and main.

Details

The flag is parsed correctly into Config.DispatchUpstreamTimeout, but there are two option setters named RemoteDispatchTimeout and neither delivers it to the code that applies the wire deadline.

1. The combined (outbound) dispatcher never receives it.

combineddispatch.NewDispatcher at pkg/cmd/server/server.go#L345-L369 is passed 14 options; combineddispatch.RemoteDispatchTimeout is not among them. That option has no non-test caller anywhere in the tree:

$ grep -rn "combineddispatch.RemoteDispatchTimeout\|combined\.RemoteDispatchTimeout" --include='*.go' .
(only internal/dispatch/combined/combined_test.go:149)

So opts.remoteDispatchTimeout keeps its zero value, is passed as DispatchOverallTimeout: 0 at combined.go#L308-L311, and is replaced by the 60s fallback at remote/cluster.go#L127-L130:

dispatchOverallTimeout := config.DispatchOverallTimeout
if dispatchOverallTimeout <= 0 {
	dispatchOverallTimeout = 60 * time.Second
}

That value is then stamped onto every outbound dispatch at remote/cluster.go#L280.

2. The one call site that does pass it targets a field that is never read.

server.go#L400 calls clusterdispatch.RemoteDispatchTimeout(c.DispatchUpstreamTimeout). That assigns optionState.remoteDispatchTimeout at internal/dispatch/cluster/cluster.go#L71-L76, but the field (declared at :25) has no reader — cluster.NewClusterDispatcher builds only graph.NewDispatcher and caching.NewCachingDispatcher, no remote client, so it has nothing to apply a wire timeout to. This call site appears to be the reason the omission has gone unnoticed: it looks like the flag is wired.

Why the deadline is observable as 60s rather than min(caller, 60s)

resenje.org/singleflight v0.4.3 runs the leader's function under context.WithoutCancel, so the caller's remaining deadline is stripped before remote/cluster.go:280 applies its own WithTimeout. Singleflighted unary DispatchCheck / DispatchExpand / plan-check therefore advertise ~60s on the wire regardless of what the caller had left. Streaming and the singleflight bypass paths (empty TraversalBloom, loop detection, Debug != NO_DEBUG) correctly carry min(caller, 60s).

Impact

Deployments that set a low --dispatch-upstream-timeout to bound peer failures do not get it. When one peer stops responding but its TCP connection stays open — a frozen node rather than a crash — dispatches to it hold a 60s budget instead of the configured one. Callers with shorter deadlines give up first, but the peer-side work and the singleflight leader persist, so a single unresponsive node degrades checks fleet-wide for far longer than the configuration implies.

Suggested fix

Add the missing option:

 dispatcher, err = combineddispatch.NewDispatcher(
 	combineddispatch.UpstreamAddr(c.DispatchUpstreamAddr),
 	combineddispatch.UpstreamCAPath(c.DispatchUpstreamCAPath),
+	combineddispatch.RemoteDispatchTimeout(c.DispatchUpstreamTimeout),
 	combineddispatch.SecondaryUpstreamAddrs(c.DispatchSecondaryUpstreamAddrs),

and separately either remove cluster.RemoteDispatchTimeout or document it as inert.

Prior art

#1719 lists internal/dispatch/combined/combined.go:119:6: unreachable func: RemoteDispatchTimeout in its dead-code inventory, and #2157 proposed deleting the option — closed unmerged. Worth noting the right resolution is wiring it rather than removing it.

One reason the signal went quiet: combined_test.go:149 (added between v1.53.0 and v1.54.0) now calls RemoteDispatchTimeout, which makes the symbol statically reachable, so deadcode no longer flags it while production wiring is still absent.

Environment

• SpiceDB v1.56.0 (also verified present in v1.56.1 and main)
• Kubernetes, 8 replicas, --dispatch-cluster-enabled, --dispatch-upstream-addr=kubernetes:///...