My determinism test passed for months while the two builds played different games
I compiled the rules engine of a shipped Android game to the browser. Same Java, two compilers. Then I checked whether the two agreed. They did not — and the test I already had for exactly this had been green the whole time. The same command twice: green against the current engine, then against the committed recording of the broken build. Play it as a terminal session if you want to select the text. The setup The rules live in one module with no Android on its classpath, which is what let me compile them a second time with TeaVM and run the same logic on a canvas in a browser tab. A seeded run should be reproducible. Give the engine seed 42 and a fixed sequence of inputs, and you should get the same game every time — that is what makes a run replayable and two builds comparable. Here is...
I compiled the rules engine of a shipped Android game to the browser. Same Java, two compilers. Then I checked whether the two agreed. They did not — and the test I already had for exactly this had been green the whole time. The same command twice: green against the current engine, then against the committed recording of the broken build. Play it as a terminal session if you want to select the text. The setup The rules live in one module with no Android on its classpath, which is what let me compile them a second time with TeaVM and run the same logic on a canvas in a browser tab. A seeded run should be reproducible. Give the engine seed 42 and a fixed sequence of inputs, and you should get the same game every time — that is what makes a run replayable and two builds comparable. Here is what I actually got, same seed, same inputs: JVM browser first obstacle x, frame 60 405.426 304.426 still alive at frame 360 yes no final score 9 6 Not a rounding difference. A different game. The cause is boring. The test failure is not. used . Its algorithm is specified down to the constants — you can read the exact linear congruential generator in the Javadoc. So a seed ought to name exactly one sequence. But my code was not running that algorithm. It was running whichever implementation the runtime supplied, and TeaVM's is not the JVM's. The specification describes what does; it does not force a foreign runtime's reimplementation to match. The fix took ten minutes: write the LCG out longhand so both builds execute the same arithmetic instead of trusting that they will. The interesting part is the test. The test that could not have caught it I had a test called . It ran the engine twice, with the same seed, and asserted the results matched. It passed on every commit, including every commit during which the browser build was playing a different game. It had to pass. It runs the engine twice in the same runtime. A test shaped like that cannot observe a disagreement between runtimes