Try PubSub+

Round robin Active-Active (non exclusive queue) for bridges

adriadri Member Posts: 2
edited July 1 in PubSub+ Event Broker

Hi everyone,
I´ve been trying out Solace and its great use of topics. I´ve been wondering however if there is a way to achieve some kind of round robin between remote brokers.

The idea would be to have a "region" which produces the messages and then have 2 other active regions that receive these messages on a round robin base (to avoid sending/processing the same message twice). I would like to be able to set it up at broker level so that the configuration of the consumers is exactly the same everywhere. I tried using a bridge with a non-exclusive queue but either I don´t know how to set it up correctly or it doesn´t work as I would expect.
Thanks in advance for your help!

Tagged:

Comments

  • kovkov Member, Employee Posts: 16 Solace Employee
    edited July 1

    Hi Adri, sounds like an interesting project; can you tell us a bit more about how you see it working end-to-end and your non-functional requirements as well? It all factors into better design.
    Here are my initial thoughts/questions:

    • Are you using static bridges or DMR for the links from the source-broker to the destination brokers?
    • Does each 'site' of the 3 use HA? We recommend that for the best guarantee of once-and-only-once delivery to consumers, when the architecture becomes distributed across regions you face many tradeoffs.
    • Do you plan to use Replay (and have adequate storage to allow that)?

    There is a version of your solution that can work with replay + bridging (either static or dynamic). I'm envisioning persistent subscriptions at the destination sites to attract all desired traffic to the replay log on both sides. Consumers provision a queue w/interested topic subscriptions, and replay from the last processed timestamp.

    Note that a destination must attract the messages even if your consumer is not connected to it, in the event they failover to it.
    How closely does this resemble the solution you have in mind?

  • adriadri Member Posts: 2

    Thanks for your reply!

    For now I am not really using it, I´m trying it (as I really like it) in order to understand if we can replace kafka for this use case.
    The full picture would be the following:

    • One central producer region (it could be private cloud or public cloud)
    • 2 or more consumer regions (public cloud)

    The idea I had was to have the central region produce messages with different topics (i.e. app/customer1, app/customer2) and then have at least 2 regions per customer topic. Doing so we would be resilient to the loss of a Region and at the same time we would take advantage on processing the messages in several regions. Processing takes some time, so processing the same message twice is a waste of resources. One important thing for operability concerns would be to have the real end consumers exactly the same and "consume" all of the topics (i.e. app/>). Hopefully only the concerned customer topics would be available for it.

    My first (testing) approach was to create bridges between the different Message VPNs and it worked great for one of the requirements. Consumers can stay the same and we can assure that only specific topics get to that region thanks to bridge subscription. However, messages are duplicated in both regions and therefore would be processed twice. Concerning your solution, if I understood it correctly, I think that you need to configure end consumers differently in each region.

    Now lets answer your questions:

    • Are you using static bridges or DMR for the links from the source-broker to the destination brokers?
      For now only static bridges. I´m not sure I fully understand DMRs, but I am preparing the test platform in order to test it.

    • Does each 'site' of the 3 use HA? We recommend that for the best guarantee of once-and-only-once delivery to consumers, when the architecture becomes distributed across regions you face many tradeoffs.
      Not now, but of course if we finally plan to go to production with Solace we would go for HA.

    • Do you plan to use Replay (and have adequate storage to allow that)?
      We could use replay. We could have storage. I suppose it could be nice to have, but to be honest I haven´t yet thought about it.

    Hopefully everything is clear... Sorry for the long post and sorry for my english.
    Thanks for your help!

  • kovkov Member, Employee Posts: 16 Solace Employee

    There is another option, and that would be to configure Destination1 and Destination2 as DR-mates with 2 messaging-VPNs across them. The DR mechanism is different than bridging/DMR in that it adds an extra layer to to keep the two sites synchronized in lockstep at all times, rather than as distinct sites with their own configurations. Let's see if Markdown can capture it :)

    +---+------++------+--+
    |D1 |VPN1  ||VPN2  |  |
    |   |(Live)||(Stby)|  |
    +---|------||------|--+
        |  |   ||  ^   |
        |  |   ||  |   |
        |  V   ||  |   |
    +---|------||------|--+
    |   |VPN1  ||VPN2  |  |
    |D2 |(Stby)||(Live)|  |
    +---+------++------+--+

    What that means is that VPN1 and VPN2 are virtualized across the two sites, and you can designate which destination is active for each VPN. That way, you can accept connections to VPN1 on Destination1 and sync it to Destination2. And accept connections to VPN2 on Destination2 while synchronizing it to Destination1.

    That way, if either side fails, you can flip its peer from Standby to Live and it will begin accepting connections from all clients, and the VPNs will be in sync with all their queues, and at the last synchronized point of consumption.

    I realize this gets pretty complicated. Can have an offline conversation with you about it if you'd like.

  • AaronAaron Member, Moderator, Employee Posts: 133 Solace Employee

    Hey all... a bit late to the game on this one, I was just browsing around in the Community and stumbled on this. Just wanted to say that if you wanted to consider using non-persistent messaging for your usecase, then you could use 2 (or more) VPN bridges between your different sites, and configure each with a non-persistent shared subscription. Format: #share/<groupname>/<topic>. That way the message flow can round-robin between the two bridges. Just a thought. But would also mean that there is a chance for message loss if the broker restarted due to using non-persistent (aka Direct) messaging.

Sign In or Register to comment.