Exponential retry for messages
1) We have a microservice consuming message and post the data to another microservice via http
2) Lets say if either the database or microservice is down, we like to push the message back to the queue (this we can do with JMSListener by throwing the exception). And we like to consume the message with exponential backoff...
Eg:
consume message every sec -> exception -> push it back to the queue
consume after 2 sec -> exception -> push it back to the queue
consume after 1 min -> all success, proceed with the next message.
How to achieve this with solace?
Comments
-
Hi @minisha, don't "push the message back to the queue" - that's the wrong way of thinking about it. Make sure you use CLIENT_ACK, and simply don't .acknowledge() the message until it's successfully consumed.
What you're doing at the moment is using AUTO_ACK, blocking in the receive callback while you try the other microservice. You're throwing an exception in the receive callback to indicate a temporary failure. You're also looking to the "dumb pipe" messaging layer to do some business logic for you (exponential backoff). This ticks several "please don't do this" boxes
I'll deal with the code piece first, as there are some important points there, but this architecture has some problems which I talk about below. Before changing your code, please think about refactoring your architecture.
A better way to do this is in code:
1) Receive the message. Pop it on to a "pending" Java list/queue or similar, and return from the callback. Make sure you're using CLIENT_ACK;
2) Pick the message off the pending queue in another thread, do your REST call;
3) If you get an OK back, call a method that traverses your list/queue and determines if all messages up to this point have been received - if so, call acknowledge[1]. Otherwise set a flag in your list/queue to say this message has been successfully sent.
4) If not, increment a delivery try counter, and schedule a retry REST call using your exponential backoff.[1] You have to do this "check I've successfully sent all my previous messages" cruft because JMS has a nasty habit of acknowledging all messages up to the current message. It's nasty, but it's part of the JMS spec.
A better way to do this architecturally:
- Can you re-factor your downstream service to accept JMS? That way you've got a single communication mechanism;
- If the downstream microservice must use REST, send a JMS message back to Solace and have a REST Delivery Point in the broker do all the REST stuff for you. This makes your microservice a lot simpler, and removes the unpleasant inter-service coupling you have at the moment. You can then also have other services listen to what you're sending in a proper decoupled microservice pattern. The RDP deals with all the problems of contacting the downstream service. You have a much simpler bit of code to maintain, with much better decoupling, and better microservice characteristics and architecture.
2
