Reprocess messages from Dead Message Queue
Hi Team,
We have developed a code which consumes messages from a Solace Queue and publishes to a weblogic JMS Queue after conerting the JSON message to XML. For the error handling part, we are publishing the failed message to a Dead message queue when the target is unreachable. Once the target is up and running, we want to reprocess the message from the Dead message queue to the main queue. We tried to achieve this by configuring the main queue as the DMQ for our DMQ. But since the message is not marked as setDMQEligible true while getting published to DMQ, the message was not reprocessed.
Is there any other possible way to achieve this?
Thanks,
Namrata Singh
Comments
-
Hi @namsing1, if I understand this correctly: you have an application that consumes the message from PubSub+, converts the JSON to XML, and then attempts to publish it to WMQ. If the publish to WMQ fails, the message is then published to the PubSub+ DMQ. Is that correct?
If so, there's a different and more robust way to deal with this situation. In your publishing application, do not acknowledge the message to PubSub+ until you receive the acknowledgement back from WMQ. That way, the message flow will stop until WMQ is reachable again. If your application crashes, messages can then end up on the DMQ (assuming you've set DMQeligible when they were sent to PubSub+ in the first place.)
1 -
@TomF Your understanding is correct. We have auto acknowledge enabled so the messages are not acknowledged till they are sent to Weblogic JMS Queue. If there is an error with the target server or queue, message is sent to DMQ after 3 retries which is configured in the Solace Queue itself (not sent to DMQ manually). Our requirement is that, when the target server is up, we want to move the messages from DMQ to the solace queue so that the messages get processed again. Is that doable?
0 -
@namsing1, I've done a quick check - the DMQ eligible flag on messages is stripped off once the message expires to its DMQ. I imagine this is to stop a loop developing where the message constantly expires back and forth. Can you have your application detect WMQ is up again, and read from the DMQ?
0 -
@TomF My application cannot detect if WMQ is up and again. It will keep on retrying if there is an unforeseen situation. In case of a planned downtime, we can stop our application till the target is up.
We cannot make the application read from DMQ, because if it is configured to read from both main queue and DMQ, it will go to an infinite loop which we don't want. We can definitely have another application to read from DMQ and post to main Solace Queue. But this won't be an automated process. Is there any way to keep the DMQ eligible flag when it goes to DMQ? Or any other way, so that we avoid writing a new application, otherwise that will be our last option.0 -
Hi @namsing1,
(@TomF also interested in your thoughts here!)
Let me ask another question. Do messages all get published to the same weblogic jms queue? Or is there any reason that message 1 would fail and message 2 would succeed the being published to weblogic part (besides JSON to XML conversion issues)? If not, then I would consider just keeping the messages on the original Solace queue and implementing retry/backoff capabilities in your publishing app so your app continues retrying until it can connect to weblogic.This approach would also help maintain message order (if you care about that). By having messages drop to a DMQ and then in some way being put back onto the original queue order will not be maintained. (e.g: messages 1, 2 go to DMQ after failing retries, message 3 gets processed successfully, then message 1, 2 get processed.
As a side note: I assume you're publishing to WMQ for other reasons, but just in case I wanted to make sure you were aware that Solace supports JMS and you could have your JMS app(s) connect directly to Solace if that helps simplify things.
-Marc
0 -
Hi @marc,
To answer your questions here:
1. All the messages are not getting published to the same weblogic JMS Queue. The weblogic server is same. If there is an issue with the server, all the queues will be affected.
2. If there is a problem with one queue, this might happen that one message meant for Queue A got successfully published and the other meant for Queue B failed due to some issue with the queue (like the queue got corrupted though very unlikely to happen).
3. If the message remains in the solace queue and retries from there, server logs will be full of errors till the message gets processed successfully.
4. Solace supports JMS but conversion of JSOM to XML is not possible in Solace, so we have to have this application.0 -
-
@marc, @TomF , We are using JmsTemplate for sending the message to WMQ.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jms/core/JmsTemplate.html0 -
Hi @namsing1,
Thanks for the info in the thread above!
On the retry/backoff front, since you're using Spring, I'd agree with the person that answered this question over on stackoverflow and go with the Spring Retry library to configure retries & backoff. It will also allow you to make decisions based on what exception occurs.Hope that helps and let us know what you come up with
It would be great to have a Spring Retry example using JmsTemplate as I was surprised I couldn't find one via google!-Marc
0 -
Hi @namsing1, we had an interesting out-of-band discussion on this topic today and I wanted to summarise to close the loop here in the dev forum (and maybe someone else will find this useful as well):
Requirements/use case:- Your application is bridging messages between several Solace queues and several Weblogic queues. The mapping of Solace queue to Weblogic queue is 1:1.
- You want to avoid high resource consumption in your application caused by retrying messages in case of a downstream issue with a Weblogic queue and want to "park" those messages for later processing.
Possible options:
- We discussed options to "pause" the processing of one queue bridge by temporarily closing the connection to the source queue, pausing the queue listener or egress disabling the source queue and setting a timer to re-enable the bridge later, which would have the advantage of preserving order of messages. Also, in your use case, it is fair to assume that if one message can't be processed because the target is down, then ALL messages in that queue won't be able to be processed, so pausing the entire processing of that queue would be a reasonable thing to do. But you weren't sure, if your environment would allow you enough control over the listener etc to try this route.
- As an alternative approach involving a DMQ like you asked about, we established that you can't use the DMQ mechanism twice (to move messages from q1 to its DMQ, then move them back to q1 after a timeout) as the broker resets the DMQ flag on the message to prevent a loop. What would be possible is to programatically move a message from q1 to a "parking" queue in your application when there's a problem downstream. You could set a timer on that message (and set the DMQ flag) to have it expire, which would then trigger the DMQ mechanism to move the message back to q1. Q1 would be configured as a DMQ of the "parking" queue. You decided to try this approach.
Looking forward to hear from you once you had a chance to try these options out!
Best regards,
Christian1 -
Thanks @ChristianHoltfurth for keeping us in the loop and added context!
1 -
@ChristianHoltfurth ,@marc, I tried to implement the parking queue in my code (in case of target error) according to the below sample:
https://github.com/SolaceSamples/solace-samples-spring/blob/master/spring-boot-autoconfig-sender/src/main/java/com/solace/samples/spring/boot/SpringBootSender.java
It didn't seem to work for me as I am using the JmsTemplate to send message to the weblogic JMS Queue. Using the above snippet for sending to parking queue, my application is not able to understand which JmsTemplate to use. I will have to look for another alternative.0 -
Hi @namsing1, I'm not a Spring developer, but I'm wondering if you have created a JMSConnectionFactory for both Solace and Weblogic? You would have to make the distinction in your code which factory and queue template to use depending on whether you are sending to the intended target (Weblogic) or the exception handling queue (Solace).
0 -
Hi @ChristianHoltfurth , I have created separate connectionFactory for both, still spring is searching for the solace queue in weblogic server. Trying to figure out something, will update if it works.
1
