What is the best practice to avoid a situation where the consumer reaches the maximum number of unacknowledged messages, causing Solace to stop sending messages and leaving the client uncertain whether to continue listening or restart the application? Should the consumer application periodically restart the client or periodically terminate and restart the messaging service to ensure automated and reliable message consumption?
Working environment: Python
Hi @hobart . Your consumer application should be acknowledging messages as it finishes with them: finished processing them, storing them, whatever it is your app is doing with the data. Call the ack() function on each message when the message data has been safely processed. Typically apps don’t hold onto messages for very long (unless maybe doing some big batch processing), but still once you’ve batched/committed them someplace safe, you’d acknowledge all those messages and continue receiving new ones.
The default is 10,000 unacked messages per flow. Are you constantly running into that limit with your app, and the broker stops sending you messages? Why do you need to hold onto so many messages?
Hi Aaron,
In the application layer, there are no issues with performing ACK or NACK operations. The problems arise from the connections between the Solace consumer and broker. We have many nodes between the Solace consumer and broker, each with its own maintenance schedule. Due to this, underlying signaling may be lost, resulting in the consumer being unaware that the broker is still waiting for an ACK or NACK.
To address this, we are looking for an automation mechanism that can help us prevent this situation.
I hope this helps clarify the situation. Please let me know if you need any further information.
Hi @hobart , sorry I never got back to you on this.
Ideally, your queue would be configured with a large enough “max unACKed messages per flow” to never hit it in normal operations. It is an imposed limit to prevent a runaway app from using up all of the unACKed message resources on the broker (which is a limited resource).
For example: I’ve seen misbehaving apps at a customer that have one thread (API) pulling messages off the queue, and then another thread (application) for processing and ACKing the messages, and that thread dies… and so the app just keeps pulling down tons of messages. With really high unACKed limits, and enough of these apps, it can hurt the broker. Hence why the default is 10k per flow.
Anyhow: the number of unACKed messages you need is a direct calculation of message rates, processing time, and round-trip-time between the broker and the client app. Most of my simple Guaranteed messaging apps are processing messages right away, and ACKing right away, and even under high load I rarely go above 100-200 outstanding unACKed messages on that flow.
But if you have a long chain of applications (not necessarily all through Solace), like:
SOLACE -> YOUR-APP -> ANOTHER-APP -> ONE-MORE -> DATABASE
or something, then your first YOUR-APP will need to hold onto that message until the ONE-MORE app has successfully written things to the database… and it will need some sort of signalling mechanism to pass back through the chain that it has read that message. Like:
SOLACE -(SMF)> YOUR-APP -(HTTP)> ANOTHER-APP -(HTTP)> ONE-MORE -(JDBC)> DATABASE
Then when the database write call succeeds, the ONE-MORE app could reply to the HTTP request with a 200 OK , similarly from the ANOTHER-APP , and then YOUR-APP would at that point ACK the message to Solace. This would prevent any message loss.
However, if you’re sending these messages downstream asynchronously , then ONE-MORE app is going to need some way of signalling to YOUR-APP that the message(s) have been stored/processed successfully, at which point it can ACK. This could be done with another HTTP call in the reverse direction, or (my recommendation) via a Solace message. Or, more ideally: the communication between each app would be through Solace (going in-and-out of the broker) to reduce this “chaining” type dependency.
YOUR-APP must continue to ACK all the messages it receives so that it can continue receiving new ones… this is the only way a properly functioning Solace consumer can work. It should never ideally leave unACKed messages sitting on the queue… because all of those would be redelivered again if YOUR-APP ever crashed and restarted, or even just reconnected following a temporary network outage.
Hope that helps. Let me know!