Mechanics of guaranteed publish callbacks when using batched send

allmhhuran

Suppose I am publishing guaranteed messages. I build a buffer of 50 messages and use the Send(IMessage[]...) method to send all of them at once. I have a few questions about message acknowledgement:
1. Will I receive 50 acknowledgement events, or just one?
2. If I receive 50, are they guaranteed to come back in the same order as the order of the messages in the batch - ie, from message[0]...message[49]?
3. Is it possible to get a mixture of RejectedMessageError and Acknowledgement events for the messages in such a batch, or will the broker always reject or accept every message in a single batch? The latter would be preferable where message order must be preserved. Can transacted sessions help in this regard? The documentation only seems to refer to transacted sessions in the JMS section.
4. More generally, if using non-batched Send(), are callbacks guaranteed to come in the order in which messages were sent?

Aaron

Hi @allmhhuran ..! You ask detailed questions! :-)

First off, using the vectored send option is really only useful at the very far end of the performance scale. Are you intending to build the fastest publisher possible? What kind of message rates / bandwidth rates are you considering? I've written "normal" (non-vectored multi send) publishers

1. You will receive individual ACKs in the callback. Note that on-the-wire, publisher ACKs are often batched or windowed, but will always be dispatched up to the application individually.
2. Yes. ACK order is always guaranteed to be in publish order, at least per-publisher-thread.
3. Definitely... each message will be treated individually. And yes, you might want to consider session transactions if you want an all-or-nothing style of delivery. Interestingly, you may get a (small) performance increase when using transactions as the individual publish ACKs don't need to be sent back. The amount of performance gain depends on your Publisher AD Window size.
4. Yup.

Aaron

Hi @allmhhuran ..! You ask detailed questions! :-)

First off, using the vectored send option is really only useful at the very far end of the performance scale. Are you intending to build the fastest publisher possible? What kind of message rates / bandwidth rates are you considering? I've written "normal" (non-vectored multi send) publishers

1. You will receive individual ACKs in the callback. Note that on-the-wire, publisher ACKs are often batched or windowed, but will always be dispatched up to the application individually.
2. Yes. ACK order is always guaranteed to be in publish order, at least per-publisher-thread.
3. Definitely... each message will be treated individually. And yes, you might want to consider session transactions if you want an all-or-nothing style of delivery. Interestingly, you may get a (small) performance increase when using transactions as the individual publish ACKs don't need to be sent back. The amount of performance gain depends on your Publisher AD Window size.
4. Yup.

allmhhuran

Thanks for addressing all of my questions in one hit!
I was looking into the batched Send API only because that is the "natural" form in which my messages are likely to be generated. Most producers are generating message data by scraping SQL Server change data capture, which will naturally produce blocks of messages, which I run through a System.Threading.Tasks.Dataflow pipeline, and then push through to the broker. But from your answer it sounds like I can save myself some code by simply iterating over the batch at the last step and sending the messages one at a time. I think it's highly unlikely that our message rate would be up at the "very far end of the performance scale", so not having to do transactional sends seems like a good option.

As a bit of a followup - are there any "standard patterns" for handling messages for which the callback to Send is a RejectedMessageError event? Is it worth retrying these, or perhaps retrying them after a few seconds, or retrying them after receiving some session event, or should this generally be considered fatal to a publishing pipeline, requiring some kind of human intervention?

Aaron

I just noticed that I didn't finish writing my 2nd para, just kinda trailed off there. I meant to say that I've written single-thread publishers that can do 10's of 1000's of messages per second (Guaranteed/persistent) with the standard "send one message at a time" way. So yeah, not really much use for the extra hassle of building the vectored send objects unless you are going for absolute max throughput. Obviously, latency would hurt a bit with vectored send as you are buffering (to some degree) your messages before sending them.

Perhaps I'll also mention that the API tries to be network-efficient, in that if you have a bunch of really small messages to send, it can try to put them into the same TCP packet to save the additional bandwidth of the TCP/IP headers. Check the "TCP_NO_DELAY" session property, or Nagle's algorithm. But again, only kinda really useful towards the max-throughput use cases.

For your follow-up, that is a spectacular question and (if we don't already) should have some blogs and documentation and guidance on exactly that. Can you repost as another question to leave this one for discussion on vectored sends? Thanks!

allmhhuran

Will do, thanks again! I hope I'm helping build up your knowledge pool for devs with these low level questions