Pub-sub with multiple consumers having offline capabilities

subhoatg

Hi,

We have an application running with multiple instances and consuming from a solace queue. We want all the instances to receive messages. Currently, only one of them receives messages. The instances can be scaled up and down dynamically and if one of them is offline, they should be able to receive messages after coming back online.

We looked into one more approach, having multiple queues for each instances, and then subscribing to one topic. If the publisher publishes to this topic, it is received in all the queues and each instance can consume from that. But having this approach comes with a challenge of each application instances creating, managing and maintaining their own queue and topic subscription.

Is there a better approach where the consumers do not have to explicitly create the queues and can still support this usecase ?

TomF

Hi @subhoatg,

Our pattern for this use case is as you've described: publish to a topic, consume from a queue that subscribes to the topic. Remember that if the application is using the Solace native APIs, JMS or MQTT there are features in the API or protocol that allow you to create (often called provision) the persistent endpoint programmatically and/or dynamically.

One thing you need to bear in mind is the lifecycle of persistence. You're not asking for something completely dynamic here (as is the case with a subscription), so when do you want the persistence to start and when should it stop? So, for instance, should the persistence start when the application connects? If the application disconnects, should it be stopped after a time period, or can we assume it will reconnect some time later?

This question can't be answered automatically and that's why no broker can do this without some administration. PubSub+ has a lot to help, though: I have customers whose application manages the queue itself completely automatically. Other customers make use of replay to create a queue and populate it with past events before starting consumption. Many customers make use of CI/CD and SEMP to automate the provisioning and management of queues.

subhoatg

Hi @TomF

Thanks for the reply. I understand then that the applications will have to manage the lifecycle of the queues. I tried golang MQTT libraries and atleast I was able to do it seamlessly without having to deal with the queues. If I use Solace Native APIs, could you please point me to the JS APIs or some code samples which can help in creating the queues ?

Tamimi

Hey @subhoatg if you are using golang we have released a Solace Native Go API which you can find more information about in this community post. You can also check the NodeJS and the JavaScript samples

A question about what you mentioned here

I tried golang MQTT libraries and atleast I was able to do it seamlessly without having to deal with the queues

Are you referring to replaying the messages in the past to new consumers that connect to the broker? if that's the case, have you looked into the replay feature of PubSub+ that Tom referred to?

subhoatg

Hi @Tamimi ,

I was basically looking for samples in nodejs for creating queues so that my application instances can create their own queues and then subscribe to the same topic, so that all of them can receive messages from the topic and can receive even when they are offline and comes back online.