v2.0.4 giving i/o deadline reached errors

Overview of the Issue

After upgrading for v2.0.3 to v2.0.4 on a small cluster, the servers were getting errors about i/o deadline reached:

2026-09-12T11:56:16.991-0400 [ERROR] agent.server.rpc: RPC error: conn=from=10.5.15.213:44803 error="rpc: server cannot decode request: i/o deadline reached"

Clients were reporting these errors:

Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: 2026-09-12T11:56:35.072-0400 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF" Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: 2026-09-12T11:56:35.072-0400 [ERROR] agent: Coordinate update error: error="rpc error making call: EOF" Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF" Sep 12 11:56:35 consul-client.dev.dimins.com consul[1873]: agent: Coordinate update error: error="rpc error making call: EOF" ... Sep 12 12:01:31 consul-client.dev.dimins.com consul[1873]: agent.client: Retrying RPC to server: method=Catalog.NodeServiceList server=10.5.15.212:8300 error="rpc error making call: EOF"

Reproduction Steps

I created a test bed with just one server and one client, and compiled different versions for the server between 2.0.3 and 2.0.4 to bisect the issue. commit 22c5407203 shows the errors, previous commit 230e22ccba does not.

Client had briefly joined the bigger cluster earlier, but did remove /var/cache/consul/serf before joining test cluster.

I tried increasing rpc_handshake_timeout, but that did not solve the problem.

As this is a testbed running off of source, I can easily run test code.

Consul info for both Client and Server

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 3
build:
	prerelease = 
	revision = a3eb721c
	version = 2.0.4
	version_metadata = 
consul:
	acl = disabled
	known_servers = 1
	server = false
runtime:
	arch = amd64
	cpu_count = 32
	goroutines = 48
	max_procs = 2
	os = linux
	version = go1.26.7
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 9
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 262680
	members = 2
	query_queue = 0
	query_time = 1

{
  "datacenter": "markley",
  "bind_addr": "{{ GetInterfaceIP \"eth0\" }}",
  "data_dir": "/var/cache/consul",
  "encrypt": "<redacted, same as above>",
  "log_level": "INFO",
  "disable_update_check": true,  
  "enable_syslog": true,
  "retry_join": [
    "consul-server.dev.dimins.com"
  ]
}

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease = dev
	revision = 22c54072
	version = 2.0.2
	version_metadata = 
consul:
	acl = disabled
	bootstrap = true
	known_datacenters = 1
	leader = true
	leader_addr = 10.5.15.212:8300
	server = true
raft:
	applied_index = 262
	commit_index = 262
	fsm_pending = 0
	last_contact = 0
	last_log_index = 262
	last_log_term = 9
	last_snapshot_index = 0
	last_snapshot_term = 0
	latest_configuration = [{Suffrage:Voter ID:4f0999e0-4c72-0a51-7ddb-c077545363b2 Address:10.5.15.212:8300}]
	latest_configuration_index = 0
	num_peers = 0
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 9
runtime:
	arch = amd64
	cpu_count = 32
	goroutines = 192
	max_procs = 32
	os = linux
	version = go1.27.1
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 9
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 262680
	members = 2
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1

datacenter = "markley"
data_dir = "/data/consul"
encrypt = "<redacted>"

disable_update_check = true

server = true

bootstrap_expect = 1

Operating system and Environment details

Debian bookworm

Log Fragments

Server at TRACE: 2026-09-12T12:14:21.871-0400 [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=markley locality=local 2026-09-12T12:14:22.637-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true 2026-09-12T12:14:24.634-0400 [TRACE] agent.server.usage_metrics: Starting usage run 2026-09-12T12:14:24.636-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true 2026-09-12T12:14:26.637-0400 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true 2026-09-12T12:14:26.718-0400 [ERROR] agent.server.rpc: RPC error: conn=from=10.5.15.213:59507 error="rpc: server cannot decode request: i/o deadline reached"

Client at TRACE: Sep 12 12:17:07 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:07.272-0400 [DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: consul-server.dev.dimins.com 10.5.15.212:8301 Sep 12 12:17:07 consul-client.dev.dimins.com consul[3767]: agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: consul-server.dev.dimins.com 10.5.15.212:8301 Sep 12 12:17:20 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:20.442-0400 [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state Sep 12 12:17:20 consul-client.dev.dimins.com consul[3767]: agent.proxycfg.agent-state: syncing proxy services from local state Sep 12 12:17:21 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:21.911-0400 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.5.15.212:46248 Sep 12 12:17:21 consul-client.dev.dimins.com consul[3767]: agent.client.memberlist.lan: memberlist: Stream connection from=10.5.15.212:46248 Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:25.073-0400 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF" Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: 2026-09-12T12:17:25.073-0400 [ERROR] agent: Coordinate update error: error="rpc error making call: EOF" Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: agent.client: RPC failed to server: method=Coordinate.Update server=10.5.15.212:8300 error="rpc error making call: EOF" Sep 12 12:17:25 consul-client.dev.dimins.com consul[3767]: agent: Coordinate update error: error="rpc error making call: EOF"