Topic replication on same vpn

vt100
vt100 Member Posts: 13

Hi All,
Is it possible to replicate the messages received on a topic to another topic.
The use case is as follows

producer application puts messages on a topic a/b/c/d with DeliverToOne set.
Whenever a message is put, a copy of that message should go into a topic e/f/g/h ( with DeliverToOne set) on the same vpn

The objective is to split the consuming application into two ( move functionality out of consumer 1 to 2) without producers knowing about it.

Is this possible with some configuration at the vpn ( looking at a low overhead solution)

Many thanks.

Best Answers

  • TomF
    TomF Member, Employee Posts: 412 Solace Employee
    #2 Answer ✓

    @vt100, I think we're a bit confused about what you're trying to do. It sounds like what you want is 2 deliver to one groups receiving the same messages, so imagine two sets of consumers A and B with 3 consumers each. You want messages to be distributed like this:
    Msg 1 goes to A1 and B1
    Msg 2 goes to A2 and B2
    Msg 3 goes to A3 and B3
    Msg 4 goes to A1 and B1
    etc.

    Is that what you're aiming for?

  • amackenzie
    amackenzie Member, Employee Posts: 265 Solace Employee
    edited January 2022 #3 Answer ✓

    2 groups... A and B. Each group subs to the same messages. Shared sub would deliver it to 1 client in each subscribing groups.

Answers

  • amackenzie
    amackenzie Member, Employee Posts: 265 Solace Employee

    Have you looked at Shared Subsciptions? https://docs.solace.com/Basics/Direct-Messages.htm#Shared
    Shared Subscriptions is the recommended way of splitting direct messages across consumers and is recommended over DTO as DTO requires the publishers to send a message with that flag set, whereas Shared Subscriptions let's you handle it completely at the consumer side.

  • uherbst
    uherbst Member, Employee Posts: 128 Solace Employee

    And for your use case: Why do you don't want to have multiple applications listening to the same topic ? As long as the publisher does NOT set DTO (and I fully agree with @amackenzie and do not recommend setting it), the message is consumed by both applications.

  • amackenzie
    amackenzie Member, Employee Posts: 265 Solace Employee

    He did say he wants to split processing. So both apps are not supposed to receive the same message. This seems like a perfect situation for shared subs. You'll need to remove the DTO from your publishers though, I believe.

  • TomF
    TomF Member, Employee Posts: 412 Solace Employee
    #7 Answer ✓

    @vt100, I think we're a bit confused about what you're trying to do. It sounds like what you want is 2 deliver to one groups receiving the same messages, so imagine two sets of consumers A and B with 3 consumers each. You want messages to be distributed like this:
    Msg 1 goes to A1 and B1
    Msg 2 goes to A2 and B2
    Msg 3 goes to A3 and B3
    Msg 4 goes to A1 and B1
    etc.

    Is that what you're aiming for?

  • vt100
    vt100 Member Posts: 13

    the consumer application is being split to two consuming applications each doing a specific functionality.
    Each consuming application should receive all messages because till the message is parsed it is not known which functionality is to be used ( hence after split a consuming application will ignore a message if it is not meant for its functionality)

    When i said application, the application consists of many processes .
    The DTO is set so that only one application instance ( 1 process) gets the message and responds. Each application consists of many processes which are listening to the topic. I do not disagree that shared subscriptions can be used instead of DTO ( altough i havent tried it yet)

    At the moment there is Consumer1, Consumer2, Consumer3 etc each is a process listening to the topic and responding to the DTO message it gets.

    going forward there will be ConsumerA1, ConsumerA2, ConsumerA3 and ConsumerB1, ConsumerB2, ConsumerB3 and so on for the A and B functionality

    The publishers cant be changed ( its a big organization)

    The reason for using topics is latency , the message has a low time to live, so no point in persistance

  • vt100
    vt100 Member Posts: 13
    edited January 2022 #9

    @TomF said:
    @vt100, I think we're a bit confused about what you're trying to do. It sounds like what you want is 2 deliver to one groups receiving the same messages, so imagine two sets of consumers A and B with 3 consumers each. You want messages to be distributed like this:
    Msg 1 goes to A1 and B1
    Msg 2 goes to A2 and B2
    Msg 3 goes to A3 and B3
    Msg 4 goes to A1 and B1
    etc.

    Is that what you're aiming for?

    Sorry i pressed some wrong button it looks like.

    Yes right now, there is A1, A2 A3 each a process and one of them should get the DTO message.
    going forward one process in A1,A2,A3
    and one process in B1, B2, B3 should get the message

  • TomF
    TomF Member, Employee Posts: 412 Solace Employee

    Ah... technical debt. Looks like shared susbscriptions might be the way forward.

  • vt100
    vt100 Member Posts: 13
    edited January 2022 #11

    How can shared subscription help for this case ?
    Lets Assume that A1.. A10 and B1..B10 are joining the shared subscription. if a message goes to A1, how can one process in B1..B10 get the same message ?

  • vt100
    vt100 Member Posts: 13

    so the objective and constraints are

    1. publishers cannot be changed and hence DTO flag continues to be set.

    2. There are two sets of consumers A1...A10 and B1...B10.

    3. Each message should be delivered to one consumer in A1...A10 and one consumer in B1..B10 only

    4. publishers continue to publish to a single topic

    How can this be achieved with the lowest overhead

  • amackenzie
    amackenzie Member, Employee Posts: 265 Solace Employee
    edited January 2022 #13 Answer ✓

    2 groups... A and B. Each group subs to the same messages. Shared sub would deliver it to 1 client in each subscribing groups.

  • vt100
    vt100 Member Posts: 13

    @amackenzie said:
    2 groups... A and B. Each group subs to the same messages. Shared sub would deliver it to 1 client in each subscribing groups.

    Ok. I see.

    I assume if the sender sets the DTO, this will not work ?

    The publishers cannot be changed ( not because they dont change, but because they are different entities with thier own priorities etc etc)