Pattern for handling RejectedMessageError events with guaranteed messages?
Suppose I am publishing guaranteed messages. I want to ensure messages are guaranteed-in-order with at-least-once-delivery all the way from the publisher which is generating the message data, to the subscribers consuming that data.
On the publisher side, I can ensure that messages have made it to the appliance by listening to the callbacks with an event type
SessionEvent.Acknowledgement, allowing the publisher to move on and publish the next message.
But I'm not sure what a publisher should do if it receives a
SessionEvent.RejectedMessageError. Nominally we don't want to move on with publishing the next message, because that might imply breaking message order. On the other hand, if the failing message can never be published then that would of course block the entire publishing pipeline.
So I guess this question boils down to the semantics of a
SessionEvent.RejectedMessageError event. Should this be interpreted as
- a transient failure, in which case we might implement some kind of retry mechanic? Or
- a permanent failure for the whole publisher, in which case we should "log and throw", terminate the publisher pipeline and send an alert to an administrator, or
- a permanent failure but only for that message, in which case we can either discard the message, or alert an administrator who can intervene to check whether the message should be discarded, or massaged and then re-attempted?
Or is the answer "it depends", and if so, is there some data supplied in the event that might help to decide which approach to take (maybe