Topic replication on same vpn
Hi All,
Is it possible to replicate the messages received on a topic to another topic.
The use case is as follows
producer application puts messages on a topic a/b/c/d with DeliverToOne set.
Whenever a message is put, a copy of that message should go into a topic e/f/g/h ( with DeliverToOne set) on the same vpn
The objective is to split the consuming application into two ( move functionality out of consumer 1 to 2) without producers knowing about it.
Is this possible with some configuration at the vpn ( looking at a low overhead solution)
Many thanks.
Best Answers
-
@vt100, I think we're a bit confused about what you're trying to do. It sounds like what you want is 2 deliver to one groups receiving the same messages, so imagine two sets of consumers A and B with 3 consumers each. You want messages to be distributed like this:
Msg 1 goes to A1 and B1
Msg 2 goes to A2 and B2
Msg 3 goes to A3 and B3
Msg 4 goes to A1 and B1
etc.Is that what you're aiming for?
0 -
2 groups... A and B. Each group subs to the same messages. Shared sub would deliver it to 1 client in each subscribing groups.
2
Answers
-
Have you looked at Shared Subsciptions? https://docs.solace.com/Basics/Direct-Messages.htm#Shared
Shared Subscriptions is the recommended way of splitting direct messages across consumers and is recommended over DTO as DTO requires the publishers to send a message with that flag set, whereas Shared Subscriptions let's you handle it completely at the consumer side.0 -
And for your use case: Why do you don't want to have multiple applications listening to the same topic ? As long as the publisher does NOT set DTO (and I fully agree with @amackenzie and do not recommend setting it), the message is consumed by both applications.
0 -
He did say he wants to split processing. So both apps are not supposed to receive the same message. This seems like a perfect situation for shared subs. You'll need to remove the DTO from your publishers though, I believe.
0 -
@vt100, I think we're a bit confused about what you're trying to do. It sounds like what you want is 2 deliver to one groups receiving the same messages, so imagine two sets of consumers A and B with 3 consumers each. You want messages to be distributed like this:
Msg 1 goes to A1 and B1
Msg 2 goes to A2 and B2
Msg 3 goes to A3 and B3
Msg 4 goes to A1 and B1
etc.Is that what you're aiming for?
0 -
the consumer application is being split to two consuming applications each doing a specific functionality.
Each consuming application should receive all messages because till the message is parsed it is not known which functionality is to be used ( hence after split a consuming application will ignore a message if it is not meant for its functionality)When i said application, the application consists of many processes .
The DTO is set so that only one application instance ( 1 process) gets the message and responds. Each application consists of many processes which are listening to the topic. I do not disagree that shared subscriptions can be used instead of DTO ( altough i havent tried it yet)At the moment there is Consumer1, Consumer2, Consumer3 etc each is a process listening to the topic and responding to the DTO message it gets.
going forward there will be ConsumerA1, ConsumerA2, ConsumerA3 and ConsumerB1, ConsumerB2, ConsumerB3 and so on for the A and B functionality
The publishers cant be changed ( its a big organization)
The reason for using topics is latency , the message has a low time to live, so no point in persistance
0 -
@TomF said:
@vt100, I think we're a bit confused about what you're trying to do. It sounds like what you want is 2 deliver to one groups receiving the same messages, so imagine two sets of consumers A and B with 3 consumers each. You want messages to be distributed like this:
Msg 1 goes to A1 and B1
Msg 2 goes to A2 and B2
Msg 3 goes to A3 and B3
Msg 4 goes to A1 and B1
etc.Is that what you're aiming for?
Sorry i pressed some wrong button it looks like.
Yes right now, there is A1, A2 A3 each a process and one of them should get the DTO message.
going forward one process in A1,A2,A3
and one process in B1, B2, B3 should get the message0 -
so the objective and constraints are
publishers cannot be changed and hence DTO flag continues to be set.
There are two sets of consumers A1...A10 and B1...B10.
Each message should be delivered to one consumer in A1...A10 and one consumer in B1..B10 only
publishers continue to publish to a single topic
How can this be achieved with the lowest overhead
0 -
2 groups... A and B. Each group subs to the same messages. Shared sub would deliver it to 1 client in each subscribing groups.
2 -
@amackenzie said:
2 groups... A and B. Each group subs to the same messages. Shared sub would deliver it to 1 client in each subscribing groups.Ok. I see.
I assume if the sender sets the DTO, this will not work ?
The publishers cannot be changed ( not because they dont change, but because they are different entities with thier own priorities etc etc)
0
