How to set Message TTL when using Spring Cloud Stream Binder Solace
Hi there,
I am wondering whether it is possible to set a Message's TTL if you are on the Spring Cloud Stream layer.
I wrote a message listener using Spring Cloud Stream, and at most I get access to a GenericMessage from the Spring Cloud Stream framework - but that class does not offer any methods to set a TTL.
Is there any header I can set to specify a TTL that will end up in the final Solace-specific message?
Thanks!
Answers
-
Hi,
The documentation says it is a producer property you can set.
"msgTtl
The number of milliseconds before messages are discarded or moved to a Solace-internal Dead Message Queue."
See - https://github.com/SolaceProducts/spring-cloud-stream-binder-solace#configuration-options
This sets in on a per producer basis - is this sufficient for you?
Not sure if it can also be set in code somehow.0 -
Thanks for your reply!
I was aware of this flag, but unfortunately it is not quite what I need.
The problem is, that this ttl is static. Instead, I would like to achieve a dynamic TTL to implement a kind of retry-backoff.
For that, I would have to modify the message's TTL in code, dependent on the number of retries it has already gone through.
For background: I am implementing the retry-pattern (using DMQs) described here.
I am aware of a Rabbit MQ header ("x-message-ttl" or so) which is used there to set a TTL. I was hoping, there is something equivalent in Solace.1 -
Right, I suspected you wanted to something like that. Looking at the code of the Cloud Stream binder I don't think it's possible at this stage to set the TTL per message.
Looking at the link you provided - it seems like you are actually concerned about poison message handling and need to have retry at certain intervals and eventually re-queueing messages to a DMQ.
I'm curious if you alternatively would be able to attract messages into a queue and use the Solace queue's retry interval, retry count and DMQ settings to achieve what you need?
Maybe this could help ... https://solace.community/discussion/8/how-to-delay-sending-out-messages-in-solace-pubsub
It uses the DMQ mechanisms to delay message delivery, in your case you may have a cascading set of DMQs with different default TTLs?0 -
Thanks for the link. I am using exactly that trick.
I think, what you are suggesting is to have several retry-queues (whose DMQs all point back to the original input queue) that each have a different TTL.
Though I could probably do that, it seems awfully complicated. I know that for RabbitMQ such a pattern is frequently used, since RabbitMQ tends to look at the TTL of messages in a retry queue only sequentially. I.e. if the head of the queue has a TTL of 10 minutes, and the message behind head has a TTL of 5 minutes, the message at the head will block TTL expiry of the message behind. So there you have no other choice.
I was hoping Solace would be smarter there. The Solace Spring Cloud Stream Binder has also some nasty bugs, which sometimes limit the options we are left with (issues have been filed for that).0 -
Hi @FWinkler79,
Good to hear from you again! A few notes here:
1. I agree that it would be useful to set TTL (and other headers/properties) on a per message basis. This is near the top of the list of enhancements for a future binder release. I don't have an exact date (and no promises
) but I'm hoping that's sometime in the November time frame.
2. On top of what Swen recommended, another option that you might be able to leverage is the Retry Template that Cloud Stream is using internally. It allows you to configure the number of retries and a backoff interval/multiplier. More info in the reference guide.0 -
Hi Marc!
Thanks for your answer. Great that you already have it on your list. I am very much looking forward to that feature.
Thanks also for the hint of using retry template. I am already using that - and for short-lived problems (that are easily resolved by a quick retry) that's fine. However, the retry template occupies an application thread (so it's application-level retry) and that's not an option for retry cycles in the scale of 30 sec to several minutes.Currently our setup is as follows:
- we receive an event from Solace using the
@StreamListener-annotated message handler method. - if the message fails to be processed for an anticipated reason (downtime of another system, temporary issues) we retry first using retry template.
- if after 3 times retrying with retry template the event could still not be processed, we dump it into a "retry-queue" with the TTL as configured in
application.yml(using Solace'smsgTtlparameter). The retry queue has its (Solace-internal) DMQ configured to be our original input queue. Thus messages that need to be retried after a longer time (given bymsgTtl) will be delayed by the retry queue and re-queued onto the input queue when the TTL has expired. So they travel around in a circle until hopefully they could be processed. - for message processing errors that are unforeseen or unrecoverable (e.g. as a result of a wrong message format, parsing errors or simply a bug in our code) we have added an application-level DMQ (SCSt-level DMQ) which will collect all the messages that could not be processed and for which retry would not be a fix. These messages will not be removed from the DMQ, but need to be inspected by a service technician and removed (or corrected) manually, when error analysis is done.
Now ideally, we would not like to use a fixed TTL for the retry cycles of messages, but a TTL (i.e. delay) that is increasing (non-linearly) with every new retry attempt. However, that also means that the broker should not "starve" messages in the retry queue, if there are messages in front of them who have a longer TTL. Apparently that starvation happens in RabbitMQ (at least so I was told...).
1 - we receive an event from Solace using the
-
Hi @FWinkler79, could you describe exactly what you mean by "starvation?" A queue is by definition in-order, so a consumer will of course get the message with the longer TTL first - but there's no consumer on the retry queue. With Solace messages are expired due to TTL asynchronously and independently of position in the queue, so if by starvation you mean will a long TTL message prevent the expiration of messages with the shorter TTL then the answer is no.
1 -
Hi Tom,
exactly, by starvation I meant that for RabbitMQ messages are NOT expired due to TTL asynchronously and independently of position in the queue. At least so I was told.
But it's good to hear that for Solace that is not the case!1 -
Hi @FWinkler79 ,
I decided to make absolutely sure of this myself, so I ran Solace's all-singing, all-dancing test harness, sdkperf (find it at the bottom of the downloads page). First, I put a message on my queue "test" with a very long TTL:
./sdkperf_java.sh -cip 127.0.0.1 -cu default@test -pql test -mtl 999999999 -mt persistent -mn 1.
Then, I put a message on the same queue with a TTL of 2s, flagged that it should be expired to the DMQ:
./sdkperf_java.sh -cip 127.0.0.1 -cu default@test -pql test -mtl 2000 -mt persistent -mn 1 -mdq
You'll see that second message disappear after 2 seconds from test and appear on the DMQ.
Et viola!3 -
@FWinkler79 I figured you would have already seen the Spring RetryTemplate but wanted to make sure. Thanks for sharing your error handling strategy. Based on your explanation and available options I think it makes a lot of sense.
And thanks @TomF for verifying that Solace won't starve the messages!
Also +1 for sdkperf being super useful!
1 -
I know this is an old thread, but in case others come across it just a heads up that as of v2.0.0 of
solace-spring-cloudthis is now possible!You can add several Solace headers/properties. A list is available in SolaceHeaders.java
return MessageBuilder.withPayload(payload).setHeader(SolaceHeaders.EXPIRATION, 1000).build();
1
