How to make discard notifications reliable?

radekm

Hi,

we have a producer and a consumer which are using direct messaging. The producer starts sending an infinite stream of messages after the consumer subscribes.

Can the consumer reliably detect that some messages in the middle of the message stream were discarded?

Is it enough to

(1) send all messages with the same priority
(2) and turn off message eliding
(3) and avoid using topic dispatch

to ensure that discard indications are reliable?

Thanks

Aaron

Hey @radekm . How's it going?

TLDR: have the publisher insert a sequence number.

Ok, so a couple things / other things to consider:

1. Same "priority" doesn't matter for Direct messaging. Messages can only be re-ordered based on priority inside Solace Queues and Topic Endpoints, meaning: Guaranteed messaging. So priority doesn't/shouldn't matter.
   a) What does/could matter is sending different Class Of Service (1-3), as those can be re-ordered by the broker for delivery to a Direct consumer. The default is all 1, so just leave it that way.
2. Def turn off eliding. Or: it's off by default... leave it that way.
3. Topic dispatch wouldn't hurt much (are you using C API?). That is all local inside the consumer application, which callback gets run.
4. Other places you can lose messages are: TCP / network disconnect... if either app disconnects from the broker, for the publisher: any message "on the wire" / not yet at the broker could get lost; for the consumer: any message inside the broker or "on the wire" could get lost. So you need to watch the system event callbacks in the API for RECONNECTING notices.
5. Egress discards at the broker: if your publisher is WAY too fast, and the consumer is too slow, the broker will temporarily buffer messages to a certain point. When those egress buffers are full, it will discard messages from there to fit the newer ones on. Your consumer app can watch for this by examining each messages "discard indication flag", and if == true, you know the consumer is slow and the broker had to throw away some messages.
6. Network topology changes: if using a multi-broker setup, I.e. a multi-site event mesh, then if there is a temporary network drop between the brokers, some Direct messages could be lost, even if both publisher and consumer stay connected to their respective brokers.

If all of those conditions are met, then you can very safely assume that you haven't lost any data. That'd be enough for me. But if you want to know 100.00%, then the publisher should add some metadata... usually inside either the "application message ID" or "sequence number", a continuously increasing number or something (AtomicInteger in Java?) that the consumer can use to verify no gaps.

Since you have an infinite stream, you never have to worry about signaling the end of the stream. But if you did, a common pattern is to publish an "end flag" or something similar message on a topic the consumer is also listening to (and at the same Class Of Service, so it doesn't jump ahead), to let the consumer know it is done.

Hope that helps!

aa

Aaron

Hey @radekm . How's it going?

TLDR: have the publisher insert a sequence number.

Ok, so a couple things / other things to consider:

1. Same "priority" doesn't matter for Direct messaging. Messages can only be re-ordered based on priority inside Solace Queues and Topic Endpoints, meaning: Guaranteed messaging. So priority doesn't/shouldn't matter.
   a) What does/could matter is sending different Class Of Service (1-3), as those can be re-ordered by the broker for delivery to a Direct consumer. The default is all 1, so just leave it that way.
2. Def turn off eliding. Or: it's off by default... leave it that way.
3. Topic dispatch wouldn't hurt much (are you using C API?). That is all local inside the consumer application, which callback gets run.
4. Other places you can lose messages are: TCP / network disconnect... if either app disconnects from the broker, for the publisher: any message "on the wire" / not yet at the broker could get lost; for the consumer: any message inside the broker or "on the wire" could get lost. So you need to watch the system event callbacks in the API for RECONNECTING notices.
5. Egress discards at the broker: if your publisher is WAY too fast, and the consumer is too slow, the broker will temporarily buffer messages to a certain point. When those egress buffers are full, it will discard messages from there to fit the newer ones on. Your consumer app can watch for this by examining each messages "discard indication flag", and if == true, you know the consumer is slow and the broker had to throw away some messages.
6. Network topology changes: if using a multi-broker setup, I.e. a multi-site event mesh, then if there is a temporary network drop between the brokers, some Direct messages could be lost, even if both publisher and consumer stay connected to their respective brokers.

If all of those conditions are met, then you can very safely assume that you haven't lost any data. That'd be enough for me. But if you want to know 100.00%, then the publisher should add some metadata... usually inside either the "application message ID" or "sequence number", a continuously increasing number or something (AtomicInteger in Java?) that the consumer can use to verify no gaps.

Since you have an infinite stream, you never have to worry about signaling the end of the stream. But if you did, a common pattern is to publish an "end flag" or something similar message on a topic the consumer is also listening to (and at the same Class Of Service, so it doesn't jump ahead), to let the consumer know it is done.

Hope that helps!

aa

radekm

That's absolutely exhaustive answer. Thanks!

Topic dispatch wouldn't hurt much (are you using C API?)

We are using C# API and old JCSMP Java API and Python API in production and experimenting with C API.