Topic to Queue Mapping and Queue Full
In a setup where we are using Topic to Queue Mapping, lets say the queue is full and can not receive any new messages, what happens to the messages that are being posted to the Topic and are subscribed by the Queue (that is full and unable to receive any new message) .
Are these messages lost ?
Comments
-
Hi @vn01,
that depends….
In queue config, you can find a setting "Reject message to sender on discard".
If this is set, the publisher will receive a NACK, if the queue has to discard (= can not store) that message.This setting can be set per queue:
Imagine you have 2 queues with identical subscriptions, one queue for a productive workflow, one queue only for reporting statistics.
Maybe you want to set "reject message to sender on discard" for the productive workflow queue (because the publisher needs to know, when that does not work), but not for the reporting queue.In this case: the publisher would receive an ACK, even if the reporting queue is full.
Does that answer your question ?
Uli
2 -
I'll note here that "reject message to sender on discard" is enabled by default… the queues will automatically NACK a publisher if its full. Turning this off means the queue will silently discard new messages if it is full.
One thing to keep in mind with the publish-subscribe paradigm in a Guaranteed messaging context: if any queue that is subscribed to the same topic is full (unless configured not to NACK the publisher, as above), then the publisher will get a NACK and none of the queues will get the message. It is an "all or nothing" kind of situation. So, for example, you have three applications consuming some event stream, each with their own queue subscribed to the correct topics, and one of them dies and their queue fills up without anyone noticing, then eventually the publisher will start getting NACKs and messages won't be delivered to any of the queues, even if they have ample room.
This is why sizing your queues correctly is very important, and also proper monitoring of queue depths: listening to Syslog events when the queues hit a HIGH threshold. Sizing of queues takes into account avg and burst message rates, avg message size, expected outage time for a consuming app (how long it should take someone to restart it), etc. Each queue has their own max spool capacity, but also the VPN as a whole has a set limit as well (that all the queues in it share). And of course the broker/system has a limit as well.
One last thing I'll mention here for completeness: this behaviour can have a knock-on effect. For example, let's say you also have a VPN bridge sending the same data over to another broker (and the bridge also is using a queue to feed it, as this is Guaranteed traffic). And on the remote broker you have random app that is listening to that bridged data, also using a queue because it's Guaranteed traffic. IF that app dies, and its queue fills up, then eventually the BRIDGE will start getting NACKs and it's queue in the source VPN will start to fill up, and then eventually the bridge's queue will be full and the original publisher will start getting NACKs. Again, queue sizing and proper monitoring is important, but also if this "remote app" is not very important (e.g. reporting statistics as Uli mentioned above), then perhaps either:
- its queue can be configured not to "reject sender on discard". (or perhaps the bridge's queue?)
- the remote monitoring app could use Direct messaging topic subscriptions instead, which will cause the broker to demote the messages and not have any Guaranteed messaging impact.
0 -
Thank you for raising this important question about message handling in a Topic to Queue Mapping setup. It’s essential to understand how message flow is managed, especially when dealing with full queues.
In scenarios where a queue is full and unable to receive new messages, the behavior typically depends on the messaging system being used. In many systems, messages that are published to the topic will either be discarded or placed into a holding state until the queue can process them, depending on the configuration. For instance, some systems may offer options like message persistence, which ensures that messages are stored and not lost even if the queue is full.
To ensure that messages are not lost, consider implementing strategies such as increasing the queue's capacity, utilizing message batching, or leveraging dead-letter queues to handle messages that cannot be processed. This way, you can maintain message integrity and ensure that your system can handle spikes in traffic effectively.
If your system supports it, you might also look into configuring message retention policies that fit your operational needs. Let’s continue discussing this topic so we can explore the best solutions for your setup!
0
