Flaky TestRegisterAndUnregister due to asynchronous ATS goroutine shutdown
Observed behavior
Running go test -run TestRegisterAndUnregister -count=5 in server/ats/ats_test.go can fail:
--- FAIL: TestRegisterAndUnregister (0.50s)
ats_test.go:72: Expected same number of go routines after removing all registered: 3 vs 2
--- FAIL: TestRegisterAndUnregister (0.50s)
ats_test.go:72: Expected same number of go routines after removing all registered: 3 vs 2
FAIL
exit status 1
FAIL github.com/nats-io/nats-server/v2/server/ats 3.112sExpected behavior
This test should pass without asssertion failure
Server and client version
v2.14.3
Host environment
Not relevant to the bug.
Steps to reproduce
Running go test -run TestRegisterAndUnregister -count=100 in folder server/ats can reproduce the failure.
The bug is caused by the ATS goroutine shutdown being asynchronous. Unregister() only sends a stop signal to the global done channel, but it does not wait for the ticker goroutine to actually receive the signal and exit. As a result, when Register() and Unregister() are called repeatedly or concurrently, an old ticker goroutine may still be alive while a new one is started. This can temporarily leave multiple ATS ticker goroutines running, and in tests such as TestRegisterAndUnregister -count=100, a leftover goroutine from the previous iteration can be counted by runtime.NumGoroutine(), causing flaky failures.
Source: nats-io/nats-server