#35713·go-ethereum

engine_forkchoiceUpdated applies the reorg before returning -38002 (forkchoice state not updated atomically)

Author: CPerezzCreated Sep 15, 2026Updated Sep 15, 2026

Context

We are consolidating the reorg tests that live in each execution client's repository into one declarative cross-client suite run through hive, so the same payload DAGs and Engine API step lists execute against every client:

Running it against go-ethereum, reth, nethermind, besu and erigon surfaced this.

Finding

Canonical a1 <- a2 <- a3 (head a3); a sibling pair b1 <- b2 delivered with newPayload only, both VALID. Then:

engine_forkchoiceUpdated(head = b2, safe = a2, finalized = a1)

a2 and a1 are not on b2's chain, so the spec requires -38002:

Client software MUST return -38002: Invalid forkchoice state error if the payload referenced by forkchoiceState.headBlockHash is VALID and a payload referenced by either forkchoiceState.finalizedBlockHash or forkchoiceState.safeBlockHash does not belong to the chain defined by forkchoiceState.headBlockHash.

and, in the same section:

All updates to the forkchoice state resulting from this call MUST be made atomically.

geth returns -38002 and has already applied the reorg:

after the call go-ethereum v1.17.6-unstable-02872e9e nethermind / besu / erigon
eth_getBlockByNumber("latest") b2 a3 (unchanged)
heights 1-2 b1 / b2 a1 / a2

reth shows the same behaviour (see https://github.com/paradigmxyz/reth/issues/27242).

What the suite sees (workspace/logs/details/*.log):

step 7/9 newPayload(b1): status=VALID lvh=0x800c1d...3f3a96 -> 'valid'
step 8/9 newPayload(b2): status=VALID lvh=0xee685e...8534d4 -> 'valid'
step 9/9 forkchoiceUpdated(head=b2, safe=a2, fin=a1): error -38002: Invalid forkchoice state
  assertHead: 'latest' is b2 (0xee685e...8534d4), expected a3 (0x8cdbb1...eab1f2)   <-- FAIL

What geth logs for that one call (workspace/logs/go-ethereum/client-*.log):

INFO Chain head was updated      number=3 hash=8cdbb1..eab1f2      <- a3, before the call
INFO Chain reorg detected        number=0 hash=a92463..8b54dd drop=3 dropfrom=8cdbb1..eab1f2 add=2 addfrom=ee685e..8534d4
INFO Chain head was updated      number=2 hash=ee685e..8534d4      <- b2 is now the head
WARN Final block not in canonical chain   number=1 hash=9f5212..05a504
WARN Served engine_forkchoiceUpdatedV3    err="Invalid forkchoice state" errdata="{\"err\":\"final block not in canonical chain\"}"

The reorg is detected, committed and the head updated; the consistency check then fails and the call is answered with -38002, with the new head left in place. For contrast, nethermind on the same fixture logs Inconsistent ForkChoiceState - finalized block hash and latest stays a3.

Reproduce

Requires go + docker only; the fixtures come from a preview release and the simulator is built from the PR branch:

git clone -b reorg-suite-pr https://github.com/CPerezz/hive
cd hive && go build .

./hive --sim ethereum/eels/consume-reorg \
       --sim.buildarg repo=https://github.com/CPerezz/execution-specs \
       --sim.buildarg branch=reorg-suite-pr \
       --sim.buildarg fixtures=https://github.com/CPerezz/execution-specs/releases/download/reorg-preview%40v1/fixtures_reorg.tar.gz \
       --client go-ethereum --sim.limit ".*test_forkchoice_state_unchanged_on_invalid_state.*" --sim.loglevel 3

# results: workspace/logs/<ts>-<id>.json ; per-step trace in workspace/logs/details/*.log
# client output: workspace/logs/go-ethereum/client-*.log

To regenerate the fixtures instead of using the preview release (needs uv):

git clone -b reorg-suite-pr https://github.com/CPerezz/execution-specs
simulators/ethereum/eels/consume-reorg/stage.sh ../execution-specs   # fills and stages fixtures.tar.gz
./hive --sim ethereum/eels/consume-reorg \
       --sim.buildarg repo=https://github.com/CPerezz/execution-specs \
       --sim.buildarg branch=reorg-suite-pr \
       --sim.buildarg fixtures=/fixtures \
       --client go-ethereum --sim.limit ".*test_forkchoice_state_unchanged_on_invalid_state.*" --docker.nocache consume-reorg