ClientAck behavior in case ack is not sent (timeout)

daniele

Hello everyone,
I'm trying to implement the transactional client with timeout-based delivery in Solace leveraging the ClientAck mode but I'm struggling to understand what is the timeout setting used by Solace to redeliver a message in case the consumer that receives the message does not send the Ack within a specified time.
The scenario I'm working with is a pool of X .NET Core consumers subscribed to a non-exclusive durable queue in which a producer publishes persistent messages.
Consumer queue setup:

IQueue queue = ContextFactory.Instance.CreateQueue(queueName);
                Flow = session.CreateFlow(new FlowProperties()
                {
                    AckMode =  MessageAckMode.ClientAck ,
                    AckTimerInMsecs = 500,
                    MaxUnackedMessages = -1,
                    WindowSize = 1 //set a persist message as one unit
                },
                queue, null, HandleMessageEvent, HandleFlowEvent);
Flow.Start();

// block the current thread until a confirmation received
WaitEventWaitHandle.WaitOne();

//dispose flow then restart the flow in the next loop
Flow.Stop();
Flow.Dispose();
GC.Collect();

Here the consumer code in which I added a "sleep" to simulate an high processing time. I expect at some point the message to be redelivered to another instance because the Ack has not been sent within N seconds:

 // Received a message
Console.WriteLine("Solace: Received message.");
using (IMessage message = args.Message)
{
   var msg = 
   Encoding.UTF8.GetString(Compression.Decompress(message.BinaryAttachment));
   // process the message
   Console.WriteLine("Processing message.. Is redelivered: " + message.Redelivered);
   MessageEventHandle(message.Destination.Name, msg);
   // process completed, we can Ack
   if (Flow != null)
   {
      // to test how long it waits before redelivering to another consumer
      Console.WriteLine(string.Format("Wait {0} millisec before sending ack", _delayMillisec));
      Thread.Sleep(_delayMillisec);
      Flow.Ack(message.ADMessageId);
      Console.WriteLine("Message Ack at " + DateTime.Now.ToString());
   }
}

I executed this code changing the _delayMillisec to simulate short and long processing times.
By running those experiments I was able to determine that:

• when the processing completes in less than 20 seconds the message is not redelivered to another instance
• when processing takes more than 30 seconds the message is always redelivered to another instance
• between 20 to 30 seconds it's inconsistent: sometimes it's redelivered sometimes not.

In summary this experiment shows that the internal timeout setting that Solace uses to decide when to redeliver a message is somewhat between 20 to 30 seconds.

However I was not able to find this number in the documentation nor where I could change it.

Could you please clarify how Solace decides when to redeliver the message when ClientAck mode is enabled in case the Ack is not received? Is it by a fixed timeout or is there any different logic it gets applied?
If it's by fixed timeout, can this setting be changed and how?

Thanks,
Daniele

allmhhuran

I think there's a lot of confusion in this space - I certainly had some for a while, and I can't give the authoritative answer, but here is my understanding of what you're seeing.

First, there is no timeout for acknowledgement. That is to say, redelivery is never triggered as a result of *slow* acknowledgement. Redelivery is only triggered if there's no acknowldgement. That happens when a subscriber receives a message and then their flow is disconnected before acknowledgement happens (eg, the subscriber process crashes between receipt and acknowledgement). When the client reconnects, any messages that have already been sent, but for which no acknowledgement was received before the flow disconnected, will be retransmitted. So it's the disconnection of the flow that matters, not how long you held the message for.

I think the reason you are seeing unpredictable behaviour is because of your Thread.Sleep(). It appears as though the code you have in your question is running inside the MessageEventHandler of the Flow. That means that you have a blocking wait inside your message event handler. But you cannot block the message event handler, because doing so blocks the internal processing going on in the Solace libraries. So what you're probably seeing is an underlying TCP timeout based on loss of communication between the client and broker, and it's that loss of connection which causes any sent-but-unacknowledged messages to be retransmitted to a different consumer.

See this thread for more details: https://solace.community/discussion/961/applying-backpressure-to-a-flow-by-blocking-in-the-messageeventhandler#latest

marc

Thanks for taking the time to answer this @allmhhuran. Given the details in this thread, I can confirm that your understanding of acks/redeliveries is correct and your intuition at the reason they are seeing unpredictable behavior also seems spot on to me!

daniele

Hi @allmhhuran

that explains the behavior! Thanks for sharing the thread with more details, appreciate it.