Disable message redelivery in Solace container
Hello,
Some integration tests fail randomly because the message is redelivered multiple times and we couldn't find why. We can see in Solace UI that the queue only received 10 messages but we can see that the message was redelivered 20 times even though we are acknowledging the message in the consumer.
Is there a way to disable Message Redelivery using Java or with Docker ?
Thank you.
Best Answers
-
@Aaron, @marc, @amackenzie We can close this discussion. We figure out why the messages were redelivered by solace. The messages weren't acknowledged by apache beam. We opened a bug issue with DirectRunner https://issues.apache.org/jira/browse/BEAM-14313
Thanks for your comments and advices.
0
Answers
-
The ability to disable message redelivery is available, however, it is a Controlled Access feature. If you wish to use this feature, you should discuss your use case with Solace Support (assuming you are a Solace supported customer).
You can set the redelivery to a lower number than the default of 0 (which means retry forever). So you could set it to 1 to minimize the retries without turning it off altogether.
https://docs.solace.com/Configuring-and-Managing/Configuring-Queues.htm
1 -
but we can see that the message was redelivered 20 times even though we are acknowledging the message in the consume
This sounds like a bug to me. If the message is being acknowledged it should not be re-delivered. Before you disable redeliveries I would try to figure out why that is happening. Any chance your app is logging that it acknowledged a message or something and isn't actually doing it?
0 -
To add to Marc's point - may I ask how is the message being acknowledged? And is this a regression bug that you found? Meaning that did you have the tests passing before and after a broker update the tests started failing? Also, what API/language is your consumer written in?
0 -
This is how we acknowledge a message
And using this for acknowledgment
flowProp.setAckMode(JCSMPProperties.SUPPORTED_MESSAGE_ACK_CLIENT);
This is the publisher
We are using the latest version of the image solace/solace-pubsub-standard:latest using Java and this issue is random, the test fails from time to time (it's frustrating in CI when the tests fails and we need to rerun them multiple times).
I hope I answered your question.
0 -
Hey @Aitamrane,
Hmm. The code & logs don't seem to match up. You have a JCSMP FlowReceiver that logs `Consumer received message on queue....` before the ACK but that never shows up in the log. Instead it looks like you might have some other code in the app that is using the qpid amqp jms library to consume the messages?
Also note that the
responseReceivedExcallback is just getting called when the broker acknowledges that it got the published message. To avoid confusion for others in the future you might want to change that message to say "Received ACK for msg..."1 -
Thank you for your reply @marc. Honestly, I didn't understand what you meant with
Instead it looks like you might have some other code in the app that is using the qpid amqp jms library to consume the messages?
This is the code for the consumer
And this is for the code for the publisher
We are using a beam pipeline to process the messages.
PS: This is code is for the integration test.
0 -
Hi @Aitamrane,
I'm saying your `onReceive` code, which is where you are acknowledging the message, has a log message here that starts with "Consumer received". You do not see that in your logs which likely means this code is not being called.
0 -
@Aitamrane according to the logs, your consumer isn't the JCSMP consumer above, it's a QPID JMS AMQP consumer.
1 -
@amackenzie But we can see that the code of the consumer JCSMP that I created is listening to the messages. As you can see in the logs down
[Context_1_ConsumerDispatcher] INFO com.test.pipeline.PipelineIT - Consumer received com.solacesystems.jcsmp.impl.JCSMPGenericXMLMessage[messageId=184,ackMessageId=0,prevId=0,CID_count=0,
0 -
Yes, I now see this part which is consuming the messages using JCSMP. But I don't see any repeated messageids being consumed with that code. Correct?
So it seems like the messages being received multiple times is using QPID JMS AMQP, which is code we haven't yet seen on this thread.
0 -
@marc & @amackenzie Just to explain, the QPID consumer is lunched by Apache beam in the pipeline. The JCSMP consumer is only for reading from the output queue to verify if the output is correct.
0 -
So my understanding is this is where you are at:
- Publish msg to topic using JCSMP code you shared in this thread
- Message put onto Solace queue that is subscribed to that topic
- Apache Beam pipeline using QPID AMQP JMS reads from that Solace queue, processes the msg, publishes a different msg to a Solace topic or queue, and ACKs the original msg
- JCSMP code in this thread consumes the new message that was published by the Beam pipeline
And my understanding is that the message read in step 3 is the one that is sometimes redelivered. If that is the case then it's the apache beam pipeline that needs to be investigated further. I also want to note that we have an Apache Beam Connector for Solace which you could opt to use if you prefer.
1 -
@marc with SolaceIO (Apache beam connector) we are getting two issues:
- Getting duplicate messages (instead of getting 3 messages we got 4)
- Message not delivered and getting a timeout (3 messages are acknowledged but only 2 were received)
When the tests using QPID AMQP JMS were failing, we found that messages were redelivered in Solace UI.
0 -
I'm still trying to wrap my head around the whole flow of messages & APIs. But one quick thought I had: are the connections being churned at all? As in, as soon as a test finishes are the connections torn down and re-established for the next test? Thinking here: the
ackMessage()is not synchronous to the broker... it is an async call that will be transmitted back to the broker to remove the message off the queue on the next "transport ack" by the API, which should be at least once a second. I've run into issues before where a Guaranteed consumer will receive a message off a queue, ACK it, and then disconnect right away... and the ACK hasn't propagated out of the API to the broker, so the message is still sitting on the queue. So a well behaving app should stop consuming and pause for a sec before disconnecting from the broker.Any chance that's what could be happening?
1 -
@Aitamrane looking more closely at the screenshots you provided, both of your issues occurred immediately after the connection to the broker got re-established. There was a disconnect (`CONNECTING`) and then a reconnect (`CONNECTED`) in the logs. And quite possibly when duplicate messages or ACKs were received b/c the previous ACK had been lost due to network disconnection.
Are you specifically doing testing around message/dupe handling when encountering broker connectivity issues?
0 -
@Aaron, @marc, @amackenzie We can close this discussion. We figure out why the messages were redelivered by solace. The messages weren't acknowledged by apache beam. We opened a bug issue with DirectRunner https://issues.apache.org/jira/browse/BEAM-14313
Thanks for your comments and advices.
0
