How to achieve message flow control in the node.js consumer for guaranteed messages?
We have a setup where our Node.js consumer app listens on a queue in a Solace PubSub+ broker. It forwards any messages received to another system in batches. As such, we do not have access to modify the queue configurations on the broker, and only the message consumer and session properties are under our control.
Since the Node.js consumer is based on broker-push events, we have observed that the app may run out of memory in case there is a high number of messages accumulated in the queue when the consumer connects to it - since it tries to send them all at once. We have tried changing the windowSize property on the message consumer to the minimum value of 1, however that only slows down the consumption marginally and the app continues to crash eventually.
Hence, could you help us with these queries:
- Does the message consumer have any way of 'pulling' the messages from the broker synchronously, rather than a broker push (in the pubish-subscribe mode itself)?
- Is there any other way by which we can achieve flow control/throttling in the consumer, based on the rate of flow or size of the messages in the queue?
- Is there a way to instruct the broker to stop sending messages if they are unacknowledged, from the msg consumer (the max delivered unacked msgs property on the queue would hence be unsuitable in this case, since it is a queue config)
We have considered implementing a sleep function in the message listener, however it would not be the best solution since the message flow rate from the broker is variable, and we will be unnecessarily sleeping in most cases.
Best Answer
-
Hi @soham,
Rather than change the window size or implement a pull pattern, the best approach is probably to use the message consumer stop method. This stops delivery of messages to your application and implements the flow control in the application rather than shifting it to the broker or API.
I don't know your application logic or requirements, but one approach may be to accept one downstream batch size of messages, call stop, then start when you're ready to receive another batch.
This is a good use of the broker, acting as a kind of buffer between two systems - we sometimes call this the "shock absorber" pattern.
1
Answers
-
Hi @soham,
Rather than change the window size or implement a pull pattern, the best approach is probably to use the message consumer stop method. This stops delivery of messages to your application and implements the flow control in the application rather than shifting it to the broker or API.
I don't know your application logic or requirements, but one approach may be to accept one downstream batch size of messages, call stop, then start when you're ready to receive another batch.
This is a good use of the broker, acting as a kind of buffer between two systems - we sometimes call this the "shock absorber" pattern.
1
