Backpressure Propagation via zmq_proxy using ZMQ_XPUB_NODROP
Hello,
I am exploring a service architecture built around zmq_proxy and I have a few questions regarding flow control and congestion management.
I want to ensure that congestion from a slow consumer propagates all the way back to a publisher, slowing it down when it publishes in blocking mode, by applying strict backpressure instead of causing silent message loss.
graph TD
Publisher("Application publisher (XPUB or PUB)") --> ProxyXSUB
subgraph Proxy
ProxyXSUB("XSUB") -- "zmq_proxy()" --> ProxyXPUB("XPUB")
end
ProxyXPUB --> ConsumerSUB("Consumer SUB")This mirrors the standard Weather Update Proxy architecture from the ZeroMQ Guide. For reference, here are the standard examples that match this topology:
Current Approach: To achieve this, I have configured the proxy XPUB and publisher XPUB sockets with:
ZMQ_SNDHWMlimitsZMQ_XPUB_NODROP = 1
My understanding is that when a consumer is slow, the proxy's XPUB socket will stop forwarding, which in turn prevents the proxy's XSUB socket from accepting new publications. This cascading queue saturation seems to be exercised by the official test_proxy_hwm.cpp test.
Given this context, I would greatly appreciate your insights on the following points:
- Architectural Validity: Is relying on
XPUBwithZMQ_XPUB_NODROP = 1solely to achieve lossless backpressure a supported and recommended pattern? - Does this inherently risk system-wide blocking where one slow consumer halts data delivery to all other fast consumers? It seems to me that this is not the case, as there is one separate queue for each producer-subscriber pair?
- Notification Consumption: In this setup, is the publishing application explicitly required to actively consume (read and discard)
XPUBsubscription notifications? - Buffer Growth: If the application does not read these subscription notifications, can they grow indefinitely and cause a memory leak, cause system-wide blocking, or will they simply saturate
ZMQ_RCVHWMand be dropped?
Thank you in advance for your time and guidance.
Source: zeromq/libzmq