Try PubSub+

How to slow down consumption rate in my Solace Consumer Application

rohit0903rohit0903 Member Posts: 4

I have a Spring Boot based Java application which reads messages from a Solace queue(let's say A) using javax.jms. Then my application applies a 10 seconds delay to each messages individually(using DelayedQueue). Once that 10 seconds have been passed, I push the message to another Solace Queue(let's say B). It all works fine, even I can have proper 10 seconds delay for each message if we have less initial messages in queue A.

But the challenge for me is that if I have huge number of messages in queue A(generally happens in my use case), and when I start the application, it pulls messages very fast(10-15K messages per second). This makes my application go out of memory(GC overhead limit exceeded).

Question: My question is how to slow down the consumption speed in my application. Do we have a way by which I can have around 2-3K messages per second. Do I need to do some configuration changes in my code. Please advise.

Below are the settings I tried, it doesn't help. Even after this my application pulls 12-15K messages per second:

environment.put(SupportedProperty.SOLACE_JMS_CONSUMER_DEFAULT_FLOW_CONGESTION_LIMIT, 2000);
environment.put(SupportedProperty.SOLACE_JMS_CONSUMER_DISPATCHER_QUEUE_SIZE, 2000);

Even I tried to increase heap memory(initial:2GB and max: 4GB) but it doesn't help.

Kindly help. Thanks!

Tagged:

Comments

  • uherbstuherbst Member, Employee Posts: 38 Solace Employee

    Hi @rohit0903,
    Do I understand you correctly : You just have an application that reads messages from a queue, give them a 10s delay and publish them back to some other queue ?

    If yes: You don't need any application for that, you can do that with just a clever configuration in your Solace broker:
    For your queue A, configure:

    • a Dead Message Queue (call it "B") (that is a simple queue, where all discarded messages are moved to)
    • a TTL of 10 sec.
    • a max-spool-usage that is large enough to hold 10sec of messages in that queue. (max-spool-usage = peak message rate * max message size)

    With that setup, all messages from queue A are automatically moved to queue B after 10 sec. Your final application can consume them there.

  • rohit0903rohit0903 Member Posts: 4

    @uherbst Thanks for your reply.

    Yes, you understand the problem statement correctly.
    But, due to certain situations I can't use this approach. The messages has to follow this flow only(Queue A --> my application (10 sec delay) --> Queue B).

  • rohit0903rohit0903 Member Posts: 4

    Hey guys, Any update on this one.
    Do we have some sort of solution that I can use. Please help!

  • AaronAaron Member, Moderator, Employee Posts: 226 Solace Employee

    Hey @rohit0903 ..! Welcome to the community! 🎉

    So as @uherbst already suggested, you could have the broker delay the delivery of messages for you automatically by using some combination of max-ttl and a DMQ, but if that doesn't work for you, ok. Also, Solace is working on some specific "delayed delivery" features, so Solace users don't need to use TTL and DMQs in the future. So stay tuned for that!

    in the meantime, if you really want your app to do the 10 second delay, but don't want to blow up your memory, then there are some options and things you could do that could slow down the delivery of messages into your app. Here are some thoughts, in no particular order:

    1. You could switch your app to a blocking receive() call, by not specifying a callback handler on your consumer. That way, your app controls how often it pulls data off the queue. https://docs.solace.com/Solace-JMS-API/Receiving-Messages-Synch.htm
    2. If you want to stick with async/callback delivery, you can adjust some parameters that affect how fast messages are pulled from the queue and sent to your app. Namely:

    Now, I'm assuming that you're using Persistent messages, and using Client ACK mode, and only ACKing the messages once they have been sent "downstream" by your app after 10 seconds? Because if not, and your app crashes, then there might be some messages loss, right? Or are you using non-persistent?

    One other thing. You're using Spring, right? Are you aware we have a JCSMP Spring binder as well? JCSMP is our "native Solace" Java API, and our JMS API is a wrapper around that. JCSMP gives you more control over how you handle messages as it's not constrained by the JMS spec. With JCSMP, you create a FlowReceiver to get data off a queue, and you can start() and stop() that Flow anytime you want, which is an effective tool in controlling message rates into the application.

    Last thoughts: how accurate does this 10 second delay have to be? Because if (for example) you did get a huge flood of messages on your queue, and you couldn't process them as fast as they were arriving, then you'd need to delay for less than 10 seconds for messages that were backlogged onto the queue. One hint to help could be: the sender/publisher application could insert a timestamp in the message header to give your "10-sec delay" app a better indication of how long it should actually delay each message.

    Sorry for the long post! Hope that helps, let us know..! 👍

  • kimmieerkimmieer Member Posts: 5

    Thanks for share this information. Very smart and interesting.

  • rohit0903rohit0903 Member Posts: 4

    Thanks for the best explanation.
    Yes, we are using persistent and client ack mode.

    And jfyi, we are using DelayedQueue internally to manage 10 sec delay.

    I got fare understanding about the solution. Thanks again 😊

  • kimmieerkimmieer Member Posts: 5
    edited July 27 #8

    B2B lead nurturing companies usually have a team of professionals who focus on generating leads. This team of professionals will work with you, track your return on investment (ROI) and other key factors, such as lead conversion https://yoggysmoneyvault.co.uk/?thread=i-will-create-a-lead-generation-vape-or-cbd-website-for-you. This team of professionals can analyze your marketing strategy, identify the loopholes in your strategy, rectify these loopholes and then improve your b2b lead generation process. Once you identify the loopholes, you can quickly improve your b2b lead conversion process by adding or removing features such as lead capture forms, follow up calls and auto responders. If you hire a B2B lead generation company, then it will make sure that you have an effective and efficient b2b lead generation system.

  • marcmarc Member, Administrator, Moderator, Employee Posts: 469 admin

    Thanks for the thorough response @Aaron!

Sign In or Register to comment.