Disable message redelivery in Solace container

Aitamrane
Aitamrane Member Posts: 11 ✭
in General Discussions #1

Hello,

Some integration tests fail randomly because the message is redelivered multiple times and we couldn't find why. We can see in Solace UI that the queue only received 10 messages but we can see that the message was redelivered 20 times even though we are acknowledging the message in the consumer.

Is there a way to disable Message Redelivery using Java or with Docker ?

Thank you.

Tagged:

Best Answers

Answers

  • amackenzie
    amackenzie Member, Employee Posts: 268 Solace Employee
    #4

    The ability to disable message redelivery is available, however, it is a Controlled Access feature. If you wish to use this feature, you should discuss your use case with Solace Support (assuming you are a Solace supported customer).

    You can set the redelivery to a lower number than the default of 0 (which means retry forever). So you could set it to 1 to minimize the retries without turning it off altogether.

    https://docs.solace.com/Configuring-and-Managing/Configuring-Queues.htm

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    #5
    https://solace.community/discussion/comment/4594#Comment_4594

    We want to disable it only for the tests. Should I contact them for that?

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #6

    but we can see that the message was redelivered 20 times even though we are acknowledging the message in the consume

    This sounds like a bug to me. If the message is being acknowledged it should not be re-delivered. Before you disable redeliveries I would try to figure out why that is happening. Any chance your app is logging that it acknowledged a message or something and isn't actually doing it?

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 538 admin
    #7

    To add to Marc's point - may I ask how is the message being acknowledged? And is this a regression bug that you found? Meaning that did you have the tests passing before and after a broker update the tests started failing? Also, what API/language is your consumer written in?

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    #8
    https://solace.community/discussion/comment/4600#Comment_4600

    Here are the logs of my app


    image.png

    As you can see, the messages are dispatched with different sequence id and also acknowledged but after getting acknowledged, we can see other messages to be dispatched too for another consumer (we have 4 consumer connections).


    image.png
  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    edited April 2022 #9
    https://solace.community/discussion/comment/4601#Comment_4601

    This is how we acknowledge a message

    image.png

    And using this for acknowledgment 

    flowProp.setAckMode(JCSMPProperties.SUPPORTED_MESSAGE_ACK_CLIENT);

    This is the publisher

    image.png

    We are using the latest version of the image solace/solace-pubsub-standard:latest using Java and this issue is random, the test fails from time to time (it's frustrating in CI when the tests fails and we need to rerun them multiple times).

    I hope I answered your question.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #10

    Hey @Aitamrane,

    Hmm. The code & logs don't seem to match up. You have a JCSMP FlowReceiver that logs `Consumer received message on queue....` before the ACK but that never shows up in the log. Instead it looks like you might have some other code in the app that is using the qpid amqp jms library to consume the messages?

    Also note that the responseReceivedEx callback is just getting called when the broker acknowledges that it got the published message. To avoid confusion for others in the future you might want to change that message to say "Received ACK for msg..."

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    edited April 2022 #11

    Thank you for your reply @marc. Honestly, I didn't understand what you meant with 

    Instead it looks like you might have some other code in the app that is using the qpid amqp jms library to consume the messages?

    This is the code for the consumer

    image.png

    And this is for the code for the publisher

    image.png

    We are using a beam pipeline to process the messages.

    PS: This is code is for the integration test.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #12

    Hi @Aitamrane,

    I'm saying your `onReceive` code, which is where you are acknowledging the message, has a log message here that starts with "Consumer received". You do not see that in your logs which likely means this code is not being called.

    image.png


  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    #13
    https://solace.community/discussion/comment/4611#Comment_4611

    No, the code is been called. Here is a screenshot of the new logs (I replaced Publisher received ... with Received ACK for msg...)

    image.png


  • amackenzie
    amackenzie Member, Employee Posts: 268 Solace Employee
    #14

    @Aitamrane according to the logs, your consumer isn't the JCSMP consumer above, it's a QPID JMS AMQP consumer.

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    #15

    @amackenzie But we can see that the code of the consumer JCSMP that I created is listening to the messages. As you can see in the logs down

    [Context_1_ConsumerDispatcher] INFO com.test.pipeline.PipelineIT - Consumer received com.solacesystems.jcsmp.impl.JCSMPGenericXMLMessage[messageId=184,ackMessageId=0,prevId=0,CID_count=0,
  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #16

    @Aitamrane,

    Yes, I now see this part which is consuming the messages using JCSMP. But I don't see any repeated messageids being consumed with that code. Correct?

    image.png

    So it seems like the messages being received multiple times is using QPID JMS AMQP, which is code we haven't yet seen on this thread.

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    #17

    @marc & @amackenzie Just to explain, the QPID consumer is lunched by Apache beam in the pipeline. The JCSMP consumer is only for reading from the output queue to verify if the output is correct.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #18

    So my understanding is this is where you are at:

    1. Publish msg to topic using JCSMP code you shared in this thread
    2. Message put onto Solace queue that is subscribed to that topic
    3. Apache Beam pipeline using QPID AMQP JMS reads from that Solace queue, processes the msg, publishes a different msg to a Solace topic or queue, and ACKs the original msg
    4. JCSMP code in this thread consumes the new message that was published by the Beam pipeline

    And my understanding is that the message read in step 3 is the one that is sometimes redelivered. If that is the case then it's the apache beam pipeline that needs to be investigated further. I also want to note that we have an Apache Beam Connector for Solace which you could opt to use if you prefer.

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    #19

    @marc with SolaceIO (Apache beam connector) we are getting two issues:

    • Getting duplicate messages (instead of getting 3 messages we got 4)
    image.png
    • Message not delivered and getting a timeout (3 messages are acknowledged but only 2 were received)
    image.png

    When the tests using QPID AMQP JMS were failing, we found that messages were redelivered in Solace UI.

    stats.png


  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 634 admin
    #20

    I'm still trying to wrap my head around the whole flow of messages & APIs. But one quick thought I had: are the connections being churned at all? As in, as soon as a test finishes are the connections torn down and re-established for the next test? Thinking here: the ackMessage() is not synchronous to the broker... it is an async call that will be transmitted back to the broker to remove the message off the queue on the next "transport ack" by the API, which should be at least once a second. I've run into issues before where a Guaranteed consumer will receive a message off a queue, ACK it, and then disconnect right away... and the ACK hasn't propagated out of the API to the broker, so the message is still sitting on the queue. So a well behaving app should stop consuming and pause for a sec before disconnecting from the broker.

    Any chance that's what could be happening?

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    edited April 2022 #21

    @Aaron Each test case is closing the session, creating a new queues with different names and deleting the old ones. So normally, we don't have this issue.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 634 admin
    #22

    @Aitamrane looking more closely at the screenshots you provided, both of your issues occurred immediately after the connection to the broker got re-established. There was a disconnect (`CONNECTING`) and then a reconnect (`CONNECTED`) in the logs. And quite possibly when duplicate messages or ACKs were received b/c the previous ACK had been lost due to network disconnection.

    Are you specifically doing testing around message/dupe handling when encountering broker connectivity issues?

  • Aitamrane
    Aitamrane Member Posts: 11 ✭
    edited April 2022 #23 Answer ✓

    @Aaron, @marc, @amackenzie We can close this discussion. We figure out why the messages were redelivered by solace. The messages weren't acknowledged by apache beam. We opened a bug issue with DirectRunner https://issues.apache.org/jira/browse/BEAM-14313

    Thanks for your comments and advices.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 634 admin
    #24 Answer ✓

    Nice. Well, not nice really. But glad to know it isn't Solace! Good luck.

Categories

This Month's Leaders

This Week's Leaders