How to achieve message flow control in the node.js consumer for guaranteed messages?
We have a setup where our Node.js consumer app listens on a queue in a Solace PubSub+ broker. It forwards any messages received to another system in batches. As such, we do not have access to modify the queue configurations on the broker, and only the message consumer and session properties are under our control.
Since the Node.js consumer is based on broker-push events, we have observed that the app may run out of memory in case there is a high number of messages accumulated in the queue when the consumer connects to it - since it tries to send them all at once. We have tried changing the windowSize property on the message consumer to the minimum value of 1, however that only slows down the consumption marginally and the app continues to crash eventually.
Hence, could you help us with these queries:
- Does the message consumer have any way of 'pulling' the messages from the broker synchronously, rather than a broker push (in the pubish-subscribe mode itself)?
- Is there any other way by which we can achieve flow control/throttling in the consumer, based on the rate of flow or size of the messages in the queue?
- Is there a way to instruct the broker to stop sending messages if they are unacknowledged, from the msg consumer (the max delivered unacked msgs property on the queue would hence be unsuitable in this case, since it is a queue config)
We have considered implementing a sleep function in the message listener, however it would not be the best solution since the message flow rate from the broker is variable, and we will be unnecessarily sleeping in most cases.