One message multiple consumer

adamsaghy

Hi

I am trying to use Solace for cache invalidation and the use case is the following:

I have 3 consumers (3 instances of the application). They are all listeners for the same queue.

What i am trying to achieve is when one of the instance publish a message to the topic (has subsciption to the queue) all the consumers to be notified.

What happens now: Only one of the consumers are getting the message.

Is it possible?

Thanks and regards,
Adam

ChristianHoltfurth

Hi @adamsaghy ,
You probably want to create 3 queues (one for each consumer) each with the same subscription for the topic you are interested in.
That way each of your consumers will receive the messages. The fanout happens on the subscriptions to the topic while the queue acts as a persistence endpoint in this case.

adamsaghy

@ChristianHoltfurth said:
Hi @adamsaghy ,
You probably want to create 3 queues (one for each consumer) each with the same subscription for the topic you are interested in.
That way each of your consumers will receive the messages. The fanout happens on the subscriptions to the topic while the queue acts as a persistence endpoint in this case.

Hi

Thanks for the quick response.

I forgot to mention the instances are dynamically scale up and down, so having a dedicated queue is not applicable

I a topic subscription would be the best, but i am unsure whether Solace support that.

TomF

Hi @adamsaghy.

You have 2 choices here. What really matters is the quality of service you need. If you need to persist messages while the consumers are offline, you'll need queues. To do the dynamic scale up/down, you could use our SEMP API to create/delete the queues automatically and programmatically.

Given that you're doing auto scaling, I would say a topic subscription is exactly what you need. Solace definitely supports this. Hopefully the way your message flow is designed is that your publisher is publishing to a topic that the queue then subscribes to. It's then a simple matter of having your consumers subscribe to the topic.

ChristianHoltfurth

Hi @adamsaghy ,
I assumed you needed persistence since you already had a queue.
You can of course subscribe directly to topics and use non-persistent messaging (aka direct messaging) - no queues required.
Maybe that answers your question already...?

But just in case:
You can still use endpoints (queue or topic endpoint) even in dynamic scale up and down use cases.
Endpoints can be created dynamically via the API on startup and you can choose to create temporary endpoints, if you want to automatically remove endpoints for any consumers that are going offline.

Can you describe your use case in more detail, so I can advise better on what would be best in your use case?

So far I gathered that you want to fan-out messages to all consumers (everyone gets every message) and that your consumers are quite dynamic.
If you can add some more detail on the questions around persistence, then that will help determine whether to use direct subscriptions on the topic, a temporary endpoint or a permanent endpoint.

Some questions that might help answer this:
Do you want to make sure you don't miss any messages even during high bursts of publishers sending a lot of messages at once?
Do you only want to receive messages that are published while your consumers are online?
Or do you need to store messages in persistent endpoints even when your consumers are offline for some time, so they can come back online later and retrieve those messages?

adamsaghy

@TomF said:
Hi @adamsaghy.

You have 2 choices here. What really matters is the quality of service you need. If you need to persist messages while the consumers are offline, you'll need queues. To do the dynamic scale up/down, you could use our SEMP API to create/delete the queues automatically and programmatically.

Given that you're doing auto scaling, I would say a topic subscription is exactly what you need. Solace definitely supports this. Hopefully the way your message flow is designed is that your publisher is publishing to a topic that the queue then subscribes to. It's then a simple matter of having your consumers subscribe to the topic.

I am trying to have a POC with the 2nd option - you have mentioned above -, but i am getting "unknown endpoint (503: unknown queue)" when trying to subscribe to the topic.

My topic:T_test
My queue: Q_test

Qtest has subscription to T_test.

Message published to T_test
Trying to listen on T_test as the destination too

adamsaghy

@ChristianHoltfurth said:
Hi @adamsaghy ,
I assumed you needed persistence since you already had a queue.
You can of course subscribe directly to topics and use non-persistent messaging (aka direct messaging).
Maybe that answers your question already...?

But just in case:
You can still use endpoints (queue or topic endpoint) even in dynamic scale up and down use cases.
Endpoints can be created dynamically via the API on startup and you can choose to create temporary endpoints, if you want to automatically remove endpoints for any consumers that are going offline.

Can you describe your use case in more detail, so I can advise better on what would be best in your use case?

So far I gathered that you want to fan-out messages to all consumers (everyone gets every message) and that your consumers are quite dynamic.
If you can add some more detail on the questions around persistence, then that will help determine whether to use direct subscriptions on the topic, a temporary endpoint or a permanent endpoint.

Some questions that might help answer this:
Do you want to make sure you don't miss any messages even during high bursts of publishers sending a lot of messages at once?
Do you only want to receive messages that are published while your consumers are online?
Or do you need to store messages in persistent endpoints even when your consumers are offline for some time, so they can come back online later and retrieve those messages?

Fair points.

Sorry to not started a detailed use case.

I would like to use it as a way to send messages to all the available consumers. In this use case it is a message to invalidate local cache.

Guarantees are not that important in this case, the only matter is all the available consumer get the message not just one of them.

The consumers can scale up and down

TomF

@adamsaghy perfect, this should work well. You need to remember that queues and topics are counted as separate destination types in the Solace APIs. I think what you're doing is creating a queue object in the API and using the topic name when you create it. That means the API is expecting to find a queue with the same name as your topic.

What you need to do is create a topic object. Have a look at the API sample for which ever API you're using called "DirectSub" or similar. That will show you how to create a subscription. It's actually easier than binding to a queue, the flow usually being:
1. Register the message receive callback/delegate when you create the session;
2.Create Topic API object;
3. Subscribe to the topic using the Topic API Object;

TomF

Just saw your update - so subscriptions are definitely the right way to go!

adamsaghy

It works.
Thank you very much the assistance!