Using an ITransactedSession and handling errors with the .net API

As a possible answer to my own question, "is there any way to tell, from a subsequent operation, which it was?", is a Last Value Queue an appropriate solution here? Presumably, if the commit actually succeeded the LVQ should have the last message in the batch, if it does not then the operation was rolled back.

It would be nice if this mechanism to "double-check-in-case-of-failure" could be made a feature of the API itself, without having to manually write up all of the logic for having the publisher also create an LVQ, and a subscription to that queue, and "manually" get messages from that subscription in case of a SOLCLIENT_FAIL response to a Commit(). Something along the lines ofITransactedSession.EnableLastMessageCheck() and ITransactedSession.GetLastCommittedMessage() would be nice.

Hi Nicholas.

On the topic of "exactly once vs at least once" - yes, "at least once" is fine - "exactly once" being technically impossible. When I wrote that "I want to avoid sending duplicate messages", what I really meant was something like this: Exactly-once-delivery is not a strict requirement, but I don't want to consciously add logic of my own which will significantly increase the likelihood of duplicate delivery. In other words, in the case of a SOLCLIENT_FAIL on a Commit(), I would not want to always redo the entire batch send-and-commit-unit, since the batch may have been committed, which could easily result in many dozens or even hundreds of duplicate messages, not just one or two which may result from technical realities.

"First, I wanted to better understand the reasoning behind using transactions for order guarantees"

Suppose I have a publisher that has a batch of messages ready to send from its outbox. It calls Send for each message without waiting for the acks (it processes the acks asynchronously. When an ack is received the message can be deleted from the outbox). If we waited for the acks and serialised the Sends then the send rate would at most be the inverse of the ping time… which would be on the order of 100-200 messages per second, which is not nearly fast enough.

You identified the issue in your post, but just to make the scenario explicit, suppose we have the following timeline:

T=0, Send (message0)

T=1, Send (message1)

T=2, ACK  (message0)

T=3, Send (message2)

T=4, NACK (message1), // nack'd for whatever reason

T=5, ACK  (message2)

T=6, Send (message1)  // re-sent due to NACK

T=7, Send (message3)

T=8, ACK  (message1)

// …

Message 1 must be re-sent. But message 2 has already been received and ACK'd. So the order in which messages arrive at the broker, and thus arrive at subscribers, is 0, 2, 1, …, which is not correct.

In general any mechanism which sends messages individually, and processes acks asynchronously, will have this problem.

Transactions seem to provide a solution to absolutely guarantee receive order while significantly increasing throughput, since (from the way I read the documentatation) the Commit is like "sending a bunch of messages all at once, with order preserved". If we send, say, 200 messages before the commit, then wait for the commit response before sending the next 200, the message rate is now at most 200 times the inverse of the ping time, or something on the order of 20000 - 40000 messages per second.

Of course, if the transaction fails then that would mean having to re-send 200 messages, not just one or two, which is expensive. But given the assumption that the vast majority of messages will be successfully pushed to the broker, an expensive but very rare failure is not a significant problem.

There is another way to deal with this problem, which is to have logic on the subscriber side which reorders messages correctly in the case of out-of-order delivery. We might be able to make claims like "we know that a message will never be more than X messages out of order, because the publisher has logic to ensure that Sends don't get too far ahead of acks".

There are two issues with that approach. First, it means subscriber logic is now tightly coupled to publisher logic (it has to know to reorder, and it has to know how out-of-order something could maximally be), which to some degree undermines one of the principles of a pub-sub model.

But the second and more compelling objection is that this is simply quite difficult to do. Something like TCP can do it easily because sessions don't have to persist through host restarts and so on. But publications and subscriptions can run for years. Hosts get restarted. Things crash. This requires a message sequence number generator which is durable, it can't simply be an in-memory increment. But generating durable sequences, quickly, with no gaps, is far from trivial.

I like the idea of publishing to an LVQ without requiring a topic subscription for the same topic being published to, because one of the premises of our current architecture is "a publisher should never subscribe to what it is publishing". This prevents possible infinite-loop scenarios. I enforce this with publish and subscribe permissions and exceptions on a client ACL. If I had to create a topic subscription for the LVQ then I would not be able to enforce those permissions anymore. Your solution is therefore objectively better for our situation, thanks.

I think it's correct to consider the nack "improbable but not impossible", and therefore have a robust solution like this in place. Principally because the most likely scenario which causes the out of order delivery is precisely the one you identified before - a queue fills up.

I don't think this scenario is dependent on the relative size of messages, although per your example that would certainly make the scenario more likely. But really, it seems safe to assume that if a queue fills up it will eventually not be full anymore (it is being drained at the other end), and the scenario you mentioned in your first response - "the queue is full and simultaneously being read from [and Sends are in flight]" - inevitably arises when that does eventually happen. Even in such a scenario the messages that are re-sent due to being nacked could, theoretically, all end up being re-sent at their correct position in the overall sequence, but that is unlikely.