Is it possible that publisher get notified when consumer consumes message from queue?
We are looking for scenario where publisher application needs to get notified when the consumer application consumes the message so we can start some process at publisher application. Both consumer and publisher are nodejs application.
Answers
-
Hi Prashant,
In the publisher/consumer pattern, by design there is a decoupling between the publishing and consumption of the events/messages. This allows for reduced coupling and enables adding or removing consumers without any impact on the publisher side of the flow.
In case you do want to have some coupling between the publisher and consumer, you could look into implementing the request-reply pattern as described over here :
In this pattern, every message that is published is considered as a request which qualifies for a response within a specific timeout period.
As you can see in the feature summary : https://docs.solace.com/API/Messaging-APIs/Solace-APIs-Overview.htm, the Nodejs API supports the Request/Reply messaging pattern.
We have Nodejs code samples available as well for reference.
Requestor : https://github.com/SolaceSamples/solace-samples-nodejs/blob/master/src/basic-samples/BasicRequestor.js
Replier : https://github.com/SolaceSamples/solace-samples-nodejs/blob/master/src/basic-samples/BasicReplier.js
Please do let me know in case you need any more information.
0 -
Hello @prashantk2000,
Just in case request-reply mentioned by @HariRangarajan is not good enough, for example the reply cannot take place reasonably soon, you can agree on a destination between consumer and producer on a place for storing such events, for example queue endpoint, and have publisher subscribe to it and consumer to send your "reply" events there. In addition you can implement consumer to send "reply" and acknowledge producer message under transaction if you need this level of atomicity. If you do not need persistence/guaranteed delivery, just agree on topic names and sub/pub there.
Do not hesitate to ask any questions or if you need clarifications for advice I gave here.
1 -
Thanks @HariRangarajan and @Aleksei,
I went through both approaches however as per our design, we are not allowing the client application (Consuming clients) to publish. The consuming client will have only consume access only, so in that case we cannot go for request reply pattern or publishing on any other topic/queue. :(
0 -
@prashantk2000, what does the publisher do with this information? Is it so that the publisher can detect the event is not being consumed (or consumed in time) or some other purpose?
0 -
Hi @Emerson Chen ,
Publisher need to start some process (related to particular message) on his side once he is sure that message is successfully consumed by the consumer.
I got to know Request/Reply is best for it but we are not allowing consumers to publish anything on broker :(
0 -
I don't think there is anyway to accomplish this if the consumer is not able to respond.
If we model this as a typical email exchange:
- Publisher sends an email (message)
- Email appears to be delivered successfully a Recipient (consumer) -- it was not bounced back
We can't say, for sure, that the email was:
- Delivered to the intended Recipient
- Discarded by the email filter system
- Message was discarded after being received (e.g., moved to the spam folder)
- Recipient open the message but could not read it
- Recipient consume the message (i.e., not just open it and went onto something else)
Unless there is an acknowledgment from the Recipient that the message was received and consumed successfully, there is no way that the Publisher can be sure that it is time to move to the next step. Therefore, I don't see how this use case can be satisfied without the Consumer's response.
0 -
@prashantk2000, you could use a queue browsing feature on SMF, the producer will check if consumers got their messages. Producer won't consume messages during browsing, but you may need to write the logic for tracking produced vs consumed messages. I don't know the reason why consumers are not allowed to publish, considering there is a way to control resources and permissions for destinations/topics in a reasonably "dynamic" way. But if architecture forbids any publishing by consumers for any other non-permission and resource-control reasons, then browsing might be your only option.
0 -
Great! Thanks @Aleksei,
It sounds good, this will not impact our design requirement.
0 -
I would strongly vote against that particular solution..! Having the producer browse the destination queue to know if a message has been consumed seems like a violation of decoupled architecture, introduces complexity and tight coupling. And browsing is not a trivial lightweight activity, it must receive all messages in the queue, which would constantly be changing, requiring browsing over and over again.
I would seriously reconsider the approach of having the consumer being able to publish a notification message (even a Direct message?) to indicate when it has done something... that seems like the much more straightforward, better architected approach.
1 -
Hi @Aaron, not going to disagree with you on your points. However, OP already said they went with all the other conventional approaches (including the one you suggested) and didn't fit into the limitations imposed by the requirements and it doesn't sound like the team can bend them. If there is no need to browse too frequently and consumers not doing their job is a corner case (or transient) then this might be their only option for completing the project using Solace as a broker. Competitor products may not have been able to pull that off at all.
Hi @prashantk2000, is there any chance that the consumer does something through some other APIs that you have control over, like REST call as a result of receiving message? If yes, could that API call perhaps drop a message to the producer? There could also be a database involved which that API and producer can cross reference. But if not...
0 -
Hey @Aleksei, yeah fair enough! Thanks for trying to think outside the box. Ok, then I'll try to come up with some other creative ways too...
Using a REST call: if the consumer can do a REST POST when it has processed a message, then that could be the same as publishing a message. Using Solace's REST API, a POST to
http://broker:9000/TOPIC/notif/prod1/msg/ackwill become a message on Solace, and the publisher could then subscribe tonotif/prod1/>or something to receive any of those messages.Using Solace's brand new OpenTelemetry support via Distributed Tracing: we're in the middle of building this awesome new feature which allows telemetry and message lifetime events to be published as messages and processed by an OpenTelemetry receiver. As of current (SolOS v10.3) there is support for message enqueueing, not but dequeuing yet (e.g. when the message left a queue, or when the ACK was received for a message), so can't quite use this right now. Ideally, you could listen for the ACK on a particular message to know that the consumer is done processing it.
I'm not sure how detailed/granular the info needs to be for the publisher (e.g. a notification for each and every message, or just when finished processing all messages) but perhaps using SEMP to monitor the queue depth could also done? Periodically check if the queue is empty, and then start some process when it is. This is similar to the queue browsing approach, and probably not any better. Probably worse actually.
1 -
Thanks @Aleksei and @Aaron for your ideas and suggestions.
I am agreeing with @Aaron that it will be overload on to the producer with frequent queue browsing and introduce complexity as producer end.
The other approach of rest api is good and we are thinking of exposing the rest api for message consumptions. We do have already exposed few rest apis for consumers as need of the business functionalities. For our requirement, the messages publishing rate would be a message per min or more than minute, so rest call would help to get ack from consumer.
And as Aron mentioned, solace is going to have OpenTelemetry in coming future, which looks very much helpful in such scenarios where message lifetime plays important role.
I am very much excited for OpenTelemetry feature and hope it will be available soon.
Thanks,
Prashant
0
