How to slow down consumption rate in my Solace Consumer Application

rohit0903

I have a Spring Boot based Java application which reads messages from a Solace queue(let's say A) using javax.jms. Then my application applies a 10 seconds delay to each messages individually(using DelayedQueue). Once that 10 seconds have been passed, I push the message to another Solace Queue(let's say B). It all works fine, even I can have proper 10 seconds delay for each message if we have less initial messages in queue A.

But the challenge for me is that if I have huge number of messages in queue A(generally happens in my use case), and when I start the application, it pulls messages very fast(10-15K messages per second). This makes my application go out of memory(GC overhead limit exceeded).

Question: My question is how to slow down the consumption speed in my application. Do we have a way by which I can have around 2-3K messages per second. Do I need to do some configuration changes in my code. Please advise.

Below are the settings I tried, it doesn't help. Even after this my application pulls 12-15K messages per second:

environment.put(SupportedProperty.SOLACE_JMS_CONSUMER_DEFAULT_FLOW_CONGESTION_LIMIT, 2000);
environment.put(SupportedProperty.SOLACE_JMS_CONSUMER_DISPATCHER_QUEUE_SIZE, 2000);

Even I tried to increase heap memory(initial:2GB and max: 4GB) but it doesn't help.

Kindly help. Thanks!

uherbst

Hi @rohit0903,
Do I understand you correctly : You just have an application that reads messages from a queue, give them a 10s delay and publish them back to some other queue ?

If yes: You don't need any application for that, you can do that with just a clever configuration in your Solace broker:
For your queue A, configure:

• a Dead Message Queue (call it "B") (that is a simple queue, where all discarded messages are moved to)
• a TTL of 10 sec.
• a max-spool-usage that is large enough to hold 10sec of messages in that queue. (max-spool-usage = peak message rate * max message size)

With that setup, all messages from queue A are automatically moved to queue B after 10 sec. Your final application can consume them there.

rohit0903

@uherbst Thanks for your reply.

Yes, you understand the problem statement correctly.
But, due to certain situations I can't use this approach. The messages has to follow this flow only(Queue A --> my application (10 sec delay) --> Queue .

rohit0903

Hey guys, Any update on this one.
Do we have some sort of solution that I can use. Please help!

Aaron

Hey @rohit0903 ..! Welcome to the community! 🎉

So as @uherbst already suggested, you could have the broker delay the delivery of messages for you automatically by using some combination of max-ttl and a DMQ, but if that doesn't work for you, ok. Also, Solace is working on some specific "delayed delivery" features, so Solace users don't need to use TTL and DMQs in the future. So stay tuned for that!

in the meantime, if you really want your app to do the 10 second delay, but don't want to blow up your memory, then there are some options and things you could do that could slow down the delivery of messages into your app. Here are some thoughts, in no particular order:

1. You could switch your app to a blocking receive() call, by not specifying a callback handler on your consumer. That way, your app controls how often it pulls data off the queue. https://docs.solace.com/Solace-JMS-API/Receiving-Messages-Synch.htm
2. If you want to stick with async/callback delivery, you can adjust some parameters that affect how fast messages are pulled from the queue and sent to your app. Namely:

• Guaranteed Message Window Size: a JMS Connection Factory property, this is how many messages can be "in-flight" from the broker to the API; by setting this to a very low number, it will slow down how many messages can be streamed across to the receiving application from the broker. https://docs.solace.com/Solace-JMS-API/Message-Delivery-Propert.htm#jms_properties_39190069_303737
• Max Permitted Number of Delivered Unacked Messages: a queue property, this is how many messages can be "outstanding" from the queue, messages that have been delivered but not ACKnowledged by the client. Default is 10,000. If you set this to say 1000, then the broker will send you 1000 and then block until you start acknowledging the messages. https://docs.solace.com/Configuring-and-Managing/Configuring-Queues.htm#managing_guaranteed_messaging_1810020758_455709

Now, I'm assuming that you're using Persistent messages, and using Client ACK mode, and only ACKing the messages once they have been sent "downstream" by your app after 10 seconds? Because if not, and your app crashes, then there might be some messages loss, right? Or are you using non-persistent?

One other thing. You're using Spring, right? Are you aware we have a JCSMP Spring binder as well? JCSMP is our "native Solace" Java API, and our JMS API is a wrapper around that. JCSMP gives you more control over how you handle messages as it's not constrained by the JMS spec. With JCSMP, you create a FlowReceiver to get data off a queue, and you can start() and stop() that Flow anytime you want, which is an effective tool in controlling message rates into the application.

Last thoughts: how accurate does this 10 second delay have to be? Because if (for example) you did get a huge flood of messages on your queue, and you couldn't process them as fast as they were arriving, then you'd need to delay for less than 10 seconds for messages that were backlogged onto the queue. One hint to help could be: the sender/publisher application could insert a timestamp in the message header to give your "10-sec delay" app a better indication of how long it should actually delay each message.

Sorry for the long post! Hope that helps, let us know..! 👍

kimmieer

Thanks for share this information. Very smart and interesting.

rohit0903

Thanks for the best explanation.
Yes, we are using persistent and client ack mode.

And jfyi, we are using DelayedQueue internally to manage 10 sec delay.

I got fare understanding about the solution. Thanks again 😊

marc

Thanks for the thorough response @Aaron!

KumarAbhishek

Hi @Aaron , Thanks for the detailed explanation, We are internally using JCSMP, where I want to handle the back pressure, in case if legitimate messages comes on the server in a huge un seen quantity I do not want to flood other resources.

Internally we can handle through spring-boot by allocating threads, however, is there a configuration which can limit number of messages in flight at a given time.

Copied from above, this is certainly a way to control, In a distributed environment it can be a bottleneck if one node is faulty however others are processing.

"Max Permitted Number of Delivered Un-Acked Messages: a queue property, this is how many messages can be "outstanding" from the queue, messages that have been delivered but not Acknowledged by the client. Default is 10,000. If you set this to say 1000, then the broker will send you 1000 and then block until you start acknowledging the messages. https://docs.solace.com/Configuring-and-Managing/Configuring-Queues.htm#managing_guaranteed_messaging_1810020758_455709"

Aaron

Hi there @KumarAbhishek, sorry for the delay! Wondered if you figured this out yet?

Yes the "max unacked messages per flow" setting on the queue will limit the number of outstanding messages sent to any one consumer (note: per flow, i.e. for each queue bind) before the broker blocks and starts waiting for acknowledgments from that consumer. In the JCSMP API, you can also call stop() and start() on the Flow to control messages being received. I'm not sure if these is exposed in the Spring Boot binder though.

Let us know if you resolved and how..!