Multiple Producer and Consumer
Is that possible to publish from multiple Producer and multiple consumer consumes from a single queue ?
Comments
-
I am not @marc (I can only dream to be as smart as that guy afterall) but looking at your question, I think if we broke it down into Producer and Consumer, that might be useful.
Producer
- Multiple producers can publish to the same queue
- There would not be any guarantee of publishing order in this scenario as each publisher would go at its own rate and therefore you could have, for example, Publisher1 sending 3 messages in the time it takes Publisher2 to send 1.
Consumer
- Multiple consumers on a single queue is "possible" but you need to be aware of the characteristics of what they are consuming.
- Exclusive queue:
- only 1 consumer will be able to bind to the Queue flow and be active at a time. The other consumers can be "connected" but will not be active and therefore will NOT be receiving messages.
- Non-exclusive queue:
- all connected consumers will receive messages but will each receive different/unique messages in a round-robin from the Queue. E.g. Consumer1 -> msg1, Consumer2 - msg2, Consumer1 -> msg3, Consumer2 -> msg4, ...
The basic rule is that a Queue is meant to persist messages for 1 and only 1 application. If you have multiple applications needing the same messages, you create a queue for each one and each queue subscribes to the same Topic to "attract" the messages to the application queue(s).
This is why publishing to a Topic (instead of a Queue) is so flexible. Have a new application come online that needs the same messages as previous applications? Easy, create a new Queue, add a Topic Subscription to the needed messages and it immediately starts getting messages without ANY impact on your publishers or other applications.
1 -
I agree with what @amackenzie said. Maybe he's pretty smart after all 😝
I'll second that you should consider publishing to topics and consuming from queues.
0 -
Cool
I just want to publish to a topic and consume the data based on a condition need to publish another two topic. Can you please explain this type of communication with an example.
Thank you
0 -
Hi @Shaz,
I'm not entirely sure what you mean by "publish to a topic and consume the data based on a condition need to publish another two topic." I think what you might mean is publish some data from one publisher, and have two consumers consume different parts of this stream, based on some condition.
If so, the easiest way to do this is to have a field in the topic that contains the data on which your condition needs to match. Let's say you have consumer 1 that looks for messages with condition == true and consumer 2 with condition == false. Have your publisher publish to a topic with a condition field, like this:
some/topic/{true|false} (e.g. some/topic/false)
Consumer 1 can subscribe to some/topic/true (or even * / * /true)
Consumer 2 can subscribe to some/topic/false.
1 -
And if what you are referring to is based on a condition after processing your first message you want to publish on another topic, you can follow a processor pattern to choreograph your architecture. So for example:
Publisher A publishes a message on
topic1/level1/level2Processor B subscribes to
topic1/level1/level2and processes the message. Based on a condition, the processor can publish on eithertopic2/level1/trueortopic3/level1/false. This way your publisher subscribes to topics and publishes messages on topics.(assuming this is what you meant by "publish to a topic and consume the data based on a condition need to publish another two topic.")
0 -
You can definitely add multiple subscriptions to one app, by simply subscribing to another topic :)
Note that you can leverage Solace Wildcard if you want to subscribe to a range of topics. And if you know that the condition is in the topic hierarchy (like what Tom has mentioned) you can just subscribe to one topic
some/topic/*which will attract messages published onsome/topic/trueandsome/topic/false0 -
Hi @Shaz,
OK, that's clear. There's a few ways of doing this, which I'll put in order of preference.
- Make the publisher publish to two different topics. This is simplest and offers the best performance
- Have a "splitter" microservice perform the condition for you and re-publish
- Add a message header with your condition variable in and use selectors. Please don't do this - it's an anti-pattern.
2 -
@shaz,
In one you'll need to edit your publisher to publish to 2 different topics. Taking the HelloWorld sample from github.com/solacesamples, line 97 on, we create two topic strings:
String topicStringTrue = "topic/true";
String topicStringFalse = "topic/false";
if (condition == true) {
publisher.publish(message, Topic.of(topicStringTrue));
} else {
publisher.publish(message, Topic.of(topicStringFalse));
}
This is the best approach so try this one first.
1
