Question: Does a Topic buffer messages in memory for future Ephemeral Channels when no channels exist?
Hi NSQ Maintainers,
I have a question regarding the behavior of Topics and Ephemeral Channels. I read the design docs stating that ephemeral channels drop messages when the buffer is full and disappear when the last client disconnects.
However, during my testing, I observed a scenario where a newly created ephemeral channel received a message that was published while no channels existed.
Here is my reproduction process:
Start Consumer A: I started a consumer with the ephemeral channel name lcs-push-lcs-0b79fcd1-80_2421441#ephemeral. Test Connectivity: I published a message to the topic, and Consumer A received it immediately. Stop Consumer A: I stopped the consumer process. Verify State: I checked curl http://127.0.0.1:4151/stats?format=json. The response showed that the topic had 0 channels. Wait: I waited for 1 minute to ensure cleanup. Publish Message (The Key Step): I published one new message to the topic. I checked /stats again immediately. Result: The topic showed depth=1, paused=false, and channels=[] (empty). Note: This is where I am confused. It seems the Topic held the message in depth even with no channels. Wait: I waited for another minute. Start Consumer B: I started the consumer program again. Note: The channel name is generated dynamically, so this time it was lcs-push-lcs-0b79fcd1-80_2421540#ephemeral (a different ephemeral channel name). Observation: Consumer B received the message that was published in Step 6. My Hypothesis: Based on the official documentation:
"a topic/channel whose name ends in the string #ephemeral will not be buffered to disk and will instead drop messages after passing the mem-queue-size."
My understanding from the test result is: Because the number of messages did not exceed mem-queue-size, the Topic itself buffered the message in memory (even though there were no registered channels at that moment). When a new channel (Consumer B) connected later, the Topic flushed this buffered message to the new channel.
My Questions:
Is my hypothesis correct? Does a Topic retain messages in memory (up to mem-queue-size) when there are zero channels attached? Or, strictly speaking, should the Topic drop the message immediately if no channels exist, regardless of the memory queue size? I would appreciate any clarification on this internal mechanism.
My Env: Ubuntu 20.04.6 with NSQ 1.3.0 consumer: go(1.20.3) program, using go-nsq/1.1.0
Thanks!
Source: nsqio/nsq