Message redelivery on Exception

Ludo

Hi,
I'm evaluating a move to Solace by playing with a PubSub+ Cloud free instance.
I'm not sure to understand the message redelivery policy.
My implementation is based on spring-jms backed by qpid amqp.
I was expecting that an exception thrown from a @JmsListener annotated method would reject the message which then would be moved (according to the configured number of re-delivery) to the #DEAD_MSG_QUEUE I created.
What happens instead is that the message is simply discarded.
What did I miss ?
Thanks,

TomF

Hi @Ludo, I'm not able to give a full answer at the moment, but for a message to land on the Dead Message Queue it needs to have the DMQ Eligible flag set in the message header. This is something that the producer of the message needs to set. How are you sending messages to the queue?

Ludo

I read the doc about the DMQ-eligibility:

• for AMQP 1.0, every message is set to DMQ Eligible
• for the JMS API, refer to : https://docs.solace.com/Solace-JMS-API/Setting-Message-Properties.htm#Dead
  So here's the producer code:
  jmsTemplate.convertAndSend(jmsDestination, message, new MessagePostProcessor() { @Override public Message postProcessMessage(Message message) throws JMSException { message.setBooleanProperty("SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE", true); return message; } });
  I did try to set a TTL on the Queue Im sending messages to and they're properly redirected to the DMQ

Ludo

What the doc says:

• for AMQP 1.0, every message is set to DMQ Eligible
• for the JMS API refer to: https://docs.solace.com/Solace-JMS-API/Setting-Message-Properties.htm#Dead

Here's the producer code (damn, hard to format !?)
jmsDestination = new JmsQueue(destination);
jmsTemplate.convertAndSend(jmsDestination, message, new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws JMSException {
message.setBooleanProperty("SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE", true);
return message;
}
});
I tried so set a short TTL on the Queue I'm sending messages to and the messages properly go to the DMQ after expiration.

Ludo

I did put the code on GitHub to help the readability:
https://github.com/the8tre/solace-experiment/blob/master/src/main/java/org/ludo/experiment/solace/config/JmsConfig.java
https://github.com/the8tre/solace-experiment/blob/fbaffec7a28cc4338acad3e67e63b6f37b1315d9/src/main/java/org/ludo/experiment/solace/controller/JmsTestController.java#L55

TomF

OK, let's try to separate a code problem from a broker configuration problem. Let's get messages moved to the DMQ to check that works. The easiest way to do that would be to use our test harness, sdkperf. If you can download and run that, try something like:
-pql -mn 1 -mt persistent -mtl 1 -mdq
This will send a message to the queue with a TTL of 1ms and set the DMQ eligibile flag.
You should see a message on the DMQ. If you don't, it's a configuration error. Ensure the queue has the "Respect TTL" setting enabled.

Ludo

Hi TomF,
I managed to run SDKPerf with the suggested command line and the message properly goes to the DMQ.
I got the same behaviour programmatically when putting a short TTL on the queue (see my second post).

MarkyMark

Hi,

I do not have control of the message producer's code. We are validating a message (json text message) but when it fails validation it will go back to the queue but it will not go to the DMQ after the reprocessing attempts have been reached. We are using JMS and a listener on our queue. I've tried to set the SupportedProperty.SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE to true on the incoming message but it throws an error as it cannot be changed.

We could get them to set the header of a json message but this will be a headache as we have many producers pushing json messages to our queue. Is there a way to set the DMQ Eligibility on messages returned to the queue for reprocessing?

TomF

@MarkyMark when you say "when it fails validation it will go back to the queue" can you be more specific:

• the message is not acknowledged by the validation code so it remains on the queue?
• the validator detects the problem and sends the message back to the original queue using a send() call?

If you're using JMS for the publisher, there is a JNDI option to set publisher messages DMQ eligible by default. This is available on the WebUI:

Or via the CLI through:

enable → configure → jndi message-vpn <vpn>

connection-factory <connection factory>

property-list messaging-properties

property default-dmq-eligible

MarkyMark

Hi @TomF thanks for the reply. I have a JMSListerner listening to the queue and I'm using JSR303 to validate incoming objects (after the text message in json format has been marshalled into a Java object). JSR303 validation failures will throw an exception resulting in the message not being acknowledged.

We don't know if our clients pushing to the queue are using JMS…. We have multiple clients that could be using any sort of programming language to push json objects in text form. The reasons for this are complicated.

If validation fails I want to retry the message once or twice (just to try validation again) by pushing the message back to the queue. Then I want it to go to a DMQ. I've seen the config you pasted above and set it up but I believe each failed message itself needs SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE set to true. Is it possible to do that when the message has not been not acknowledged and sent back to the queue?

At the moment I think I may have to resort to pushing the object back to the queue directly using a send call or just push it directly to a DMQ on validation failure. Another option would be a GCP bucket.

TomF

@MarkyMark ah I see. You're reading via JMS but have (as you said before) no control over the publishers.

If you read the message, that message object is immutable - because it represents what has been sent to you from the queue. That's why you can't change the DMQ property. We need another approach.

I'm not sure what benefit re-trying validation will offer? Re-sending messages like this is typically useful in the case of transient failures such as application failure and restart.

If you push the message directly back to the same queue, be aware that this will change message ordering. Sending the message to a GCP bucket would complicate your architecture, but I'm not sure what you do with messages on the DMQ.

I would suggest pushing the message directly to the DMQ itself, because it's easy to trace what happens, the business logic is clear, and it can also be clear in your comments that you're doing this because publishers are not setting the DMQ flag. That's just my 2p, though 😁

MarkyMark

Thanks again for the reply @TomF and for the advice. Yeah the idea for sending messages back to the queue is 'just in case' something else caused the failure. We'll go with pushing directly to the DMQ.

Stay healthy and wealthy.