Expected behaviour if unplug the network cable to a consumer

yangsen
yangsen Member, Employee Posts: 26 Solace Employee
in General Discussions #1

Hi, there are multiple consumers to a non-exclusive queue, if after a message has been delivered to a consumer and before the consumer ACK the message, the network cable of the consumer is unplugged, what is the expected behaviour? Will the message be delivered to other consumer after a certain period of time? And if yes, what is the period?

Another question: when using the -oc option with SDKPerf, what are the properties being used for order checking?

Best Answer

Answers

  • TomF
    TomF Member, Employee Posts: 412 Solace Employee
    #3

    Hi @yangsen,

    Interesting question! Here's how I believe this will work:

    1. Message delivered to your consumer. No ack generated.
    2. Cable unplugged.
    3. Keepalives will fail, causing the connection to be marked as down. The API goes in to reconnection logic. The broker marks the connection as lost and the consumer as unbound.
    4. The message hasn't been acknowledged, and the consumer flow has now been destroyed. The broker will look at the redelivery properties on the queue and message and decide if the message should be redelivered, moved to the DMQ or dropped.
    5. Assuming the message hasn't exceeded its redelivery count, the message will be redelivered to the next available consumer on the non-exclusive queue. The message will be tagged as redelivered.
  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    edited October 2022 #4

    Nice answer, @TomF ! And yeah, no delay... once broker detects connection to client lost, message will be redelivered.

    For SdkPerf question (also: please only one question-per-thread): if you use the -md option on the consumer, you can see that there is simply an increasing number put in the first 16 bytes of the payload of each message. Even if you don't specify a payload size (-msa or other), a minimum payload of 16 bytes is added. The consumer (who also specifies -oc) will check for gaps and dupes and generate a report when finished:

    Message Order Check Summary:
Total Msgs Order Checked        : 10
Total Out of Order Msgs         : 0
Total Missing Msgs              : 0
Total Duplicate Msgs            : 0
Total Redelivered Msgs          : 0
Total Redelivered Duplicate Msgs: 0
Total Republished Duplicate Msgs: 0
Total Replication Group Message Id Duplicate Msgs: 0
Total Replication Group Message Spool Id Changes: 0


    Don't use -md if doing any performance/load testing... anything echoed to console will slow things down a lot.

  • yangsen
    yangsen Member, Employee Posts: 26 Solace Employee
    #5

    Thanks @TomF @Aaron

    with regards to the SDKPerf -oc option, this is copied from doc page:

    "With order checking enabled, published messages will have a Publisher-ID and a Message-ID encoded within the Solace message. We refer to the set of all messages with the same Publisher-ID as a “publisher stream”.

    Consumers will decode this information during message processing and validate that Message-IDs from each publisher stream are received in sequential order. The first Message-ID received in each publisher stream will serve as the starting point for that stream."

    I know the message-ID is used, but what is the Publisher-ID? I tried to debug and there is a producerId, whose value is alwasy -1 no matter whether I apply -oc or not. Which is the correct Publisher-ID?

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    #6

    That might be out of date? Just tried running a test for both Direct and Guaranteed messaging and neither had a publisher ID automatically added. Tried running multiple publishers together, separately, and each time it got confused with OoO and dupes. So yeah, I don't think it works..?

  • yangsen
    yangsen Member, Employee Posts: 26 Solace Employee
    #7
    1. I run the same test as Aaron and it does not work when there are multiple publishers, so the publisher ID is not effective...?
    2. According to https://docs.solace.com/API-Developer-Online-Ref-Documentation/java/com/solacesystems/jcsmp/XMLMessage.html#getMessageId--, the message ID is set by JCSMP, is there any way to overwrite the value programmly? I couldn't find one. If so then there is no way to propagate the message ID across a connector, such as IBM MQ connector and Kafka connector. Which means the -oc is meaningless after the message has gone through a connector. Is my understanding correct?


  • yangsen
    yangsen Member, Employee Posts: 26 Solace Employee
    #8

    Update on the network unplug issue:

    the partner has integrated Solace into their data diode using JCSMP, and they ack messages in a batch.

    Now we are running a POC with a prospect to test the failover of the diode. the setup is as below:

    image.png

    the diode is in active-standby pair, when the SDKPerf is publishing messages, unplug the network cable of the active diode, a batch of messages delivered to the active diode(Not ACKed) will remain in the queue forever, even though the standby diode become active and continue to process all the other messages in the queue. If we connect back the network cable after a period, the stuck messages will be processed and sent over.

    I tried to simulate using 2 consumers from different machines(one host and the other is VM), the consumer will sleep for a period(30 seconds) before ACK the received messages so I have time to do the simulation. When a message is delivered to the consumer on the VM, I tried:

    1. Disable the network interface of the VM
    2. Power off the VM

    In both cases, the un-ACKed message will be re-delivered to the other consumer in a few seconds.

    What is the possible reason for the different behaviour? The broker is able to detect the disconnect from consumer running on VM, but unable to detect the unplug of network cable

  • TomF
    TomF Member, Employee Posts: 412 Solace Employee
    #9

    @yangsen, it sounds like the data diode does something clever to keep the TCP session intact when the failover to the standby diode occurs - is that the case? If so, the broker cannot detect the failover has occured. In this case the diode would have to tear down the TCP connection (or call close on the queue flow and then rebind).

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 538 admin
    #10

    Hey @yangsen - following up on this and checking to see if your issue is resolved?

  • yangsen
    yangsen Member, Employee Posts: 26 Solace Employee
    #11 Answer ✓

    The problem was resolved by enabling the minimum keep-alive setting for the client's client-profile, following this guide: https://docs.solace.com/Security/Configuring-Client-Profiles.htm#Configur4

    This setting is disabled by default and it applies to SMF client only.

    Thanks everyone for your help. Cheers!

Categories

This Month's Leaders

This Week's Leaders