Message throttling on Solace Queues

sjerin

Hi Team,

We have a requirement where the application wants to control the number of messages being pushed to a queue

Application A sends a batch of million messages on a queue in a minute. However the consumers(B,C....X) should be able to process only 2-3k messages at a time or else they will get memory exceptions.

Is there a way the application can control the number of messages being published?

marc

Hi @sjerin,

If you want to use guaranteed messaging with a queue then you probably want to look at the "reject message to sender" feature. It will basically punt it back to the sender and make them slow down sending so your consumers can keep up. Take a look at this blog and see if it will give you what you want: https://solace.com/blog/pubsub-message-handling-features-reject-message-to-sender-on-discard/

But, sometimes you don't want to slow down your senders. In that case you need to either scale up your consumers to keep up or if you are okay with these specific consumers missing messages then you can consider using direct messaging which can drop messages when your consumers can't keep up.

Hope that helps!

Aaron

Thread.sleep(1); 😅

Haha, but seriously.

First question: are you using Direct messaging or Guaranteed? If Guaranteed, your queue on the broker should be acting like a shock absorber to buffer all those messages. This way your publisher can be bursty, and your consumers can work at whatever pace they work at. Obviously your average/sustained publish rate has to be less than what your consumers can process, or you'll need to scale out the consumers.

If using Direct messaging, then you'll probably just have to slow down your publisher unless you can parallelize your consumers. Check out Shared Subscriptions.

If your consumers are blowing up their memory, it sounds like the API is pulling data off the broker faster than they can process. If using Guaranteed messaging, your consumers can stop() and start() the Flow to control messages being received. Or switch to a "pull" type of message receipt (using receive() and not specifying a Flow callback).