race condition setting ZMQ_LINGER

Issue description

race condition setting ZMQ_LINGER

Environment

• libzmq version (commit hash if unreleased): 4.2.5 (8fb5b10d8a60e06b9adebd22ecc118f13580375c)

• (Note that the 4.2.5 release does not build on Linux without subsequent patch)

• OS: CentOS 6.9

Minimal test code / Steps to reproduce the issue

Repro code is at: https://github.com/WallStProg/zmqtests.git

The test code for this issue is in the 3186 directory (see the Readme.md file in that directory for more information).

See below for instructions on reproducing the issue.

What's the actual result? (include assertion message & call stack if applicable)

Process hangs in zmq_ctx_term:

[/home/btorpey/work/zmqtests/3186] pstack 15612
Thread 3 (Thread 0x7f0ee74bd700 (LWP 15615)):
#0  0x00000033bfee91c3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f0ee75521bd in zmq::epoll_t::loop() () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#2  0x00007f0ee7583ccc in thread_routine () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#3  0x00000033c0607aa1 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033bfee8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0ee6abc700 (LWP 15616)):
#0  0x00000033bfee91c3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f0ee75521bd in zmq::epoll_t::loop() () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#2  0x00007f0ee7583ccc in thread_routine () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#3  0x00000033c0607aa1 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033bfee8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f0ee74bf7a0 (LWP 15612)):
#0  0x00000033bfedf383 in poll () from /lib64/libc.so.6
#1  0x00007f0ee756f556 in zmq::signaler_t::wait(int) () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#2  0x00007f0ee7556f35 in zmq::mailbox_t::recv(zmq::command_t*, int) () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#3  0x00007f0ee75485f0 in zmq::ctx_t::terminate() () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#4  0x00007f0ee758c8a8 in zmq_ctx_term () from /build/share/libzmq/4.2.5/lib64/libzmq.so.5
#5  0x0000000000405a0e in main ()
[/home/btorpey/work/zmqtests/3186] 

What's the expected result?

zmq_ctx_term returns successfully

Notes

Note that this is not the same as #2910 -- the tested version of libzmq includes the fix for that bug, but there is still a race condition. We suspect the race condition is between the application thread and the IO thread, and has to do with when process_commands is run on the socket.

Reproducing the issue

See the Readme.md file in the 3186 directory for detailed instructions, including how to build the example code.

To reproduce the hang problem:

./runme.sh -send -seconds 2

This will cause one or more of the background peer processes to hang indefinitely in zmq_ctx_term, typically in under a minute.

The hang appears to be caused by the ZMQ_LINGER setting not having taken effect at the point where zmq_close is called on the dataPub socket.

At the point where the process hangs, the dataPub socket has the following values:

_owner = 0x0
_owned = std::set with 0 elements
_term_acks = 2

I would expect the term_acks to be 0 if we no longer have any owned objects, as per http://zeromq.org/whitepapers:architecture.

Avoiding the problem

Running the following command avoids the problem:

./runme.sh -send -seconds 2 -early

The "-early" flag causes the ZMQ_LINGER setting to be set at socket creation time, as opposed to immediately prior to calling zmq_close. When ZMQ_LINGER is set at socket creation time the processes don't hang in zmq_ctx_term. (Our latest test ran for ~18 hours with no hangs).

Root Cause Analysis

There appears to be a race between setting ZMQ_LINGER on the socket and calling zmq_close. All by itself, this wouldn't be a major issue -- we could simply set the desired linger value at socket creation time and all would be well.

What this issue does demonstrate, however, is a general race condition between the application thread and the IO thread where certain operations on sockets are not processed in a deterministic fashion -- i.e. the socket operations are either not processed when expected, or in some cases (as in this example) are never processed at all.

This is clearly unacceptable for production code, so we are trying to understand the root cause so we can fix, avoid or work around the problem(s).