Round robin Active-Active (non exclusive queue) for bridges

adri

Hi everyone,
I´ve been trying out Solace and its great use of topics. I´ve been wondering however if there is a way to achieve some kind of round robin between remote brokers.

The idea would be to have a "region" which produces the messages and then have 2 other active regions that receive these messages on a round robin base (to avoid sending/processing the same message twice). I would like to be able to set it up at broker level so that the configuration of the consumers is exactly the same everywhere. I tried using a bridge with a non-exclusive queue but either I don´t know how to set it up correctly or it doesn´t work as I would expect.
Thanks in advance for your help!

[Deleted User]

Hi Adri, sounds like an interesting project; can you tell us a bit more about how you see it working end-to-end and your non-functional requirements as well? It all factors into better design.
Here are my initial thoughts/questions:

• Are you using static bridges or DMR for the links from the source-broker to the destination brokers?
• Does each 'site' of the 3 use HA? We recommend that for the best guarantee of once-and-only-once delivery to consumers, when the architecture becomes distributed across regions you face many tradeoffs.
• Do you plan to use Replay (and have adequate storage to allow that)?

There is a version of your solution that can work with replay + bridging (either static or dynamic). I'm envisioning persistent subscriptions at the destination sites to attract all desired traffic to the replay log on both sides. Consumers provision a queue w/interested topic subscriptions, and replay from the last processed timestamp.

Note that a destination must attract the messages even if your consumer is not connected to it, in the event they failover to it.
How closely does this resemble the solution you have in mind?

adri

Thanks for your reply!

For now I am not really using it, I´m trying it (as I really like it) in order to understand if we can replace kafka for this use case.
The full picture would be the following:

• One central producer region (it could be private cloud or public cloud)
• 2 or more consumer regions (public cloud)

The idea I had was to have the central region produce messages with different topics (i.e. app/customer1, app/customer2) and then have at least 2 regions per customer topic. Doing so we would be resilient to the loss of a Region and at the same time we would take advantage on processing the messages in several regions. Processing takes some time, so processing the same message twice is a waste of resources. One important thing for operability concerns would be to have the real end consumers exactly the same and "consume" all of the topics (i.e. app/>). Hopefully only the concerned customer topics would be available for it.

My first (testing) approach was to create bridges between the different Message VPNs and it worked great for one of the requirements. Consumers can stay the same and we can assure that only specific topics get to that region thanks to bridge subscription. However, messages are duplicated in both regions and therefore would be processed twice. Concerning your solution, if I understood it correctly, I think that you need to configure end consumers differently in each region.

Now lets answer your questions:

• Are you using static bridges or DMR for the links from the source-broker to the destination brokers?
  For now only static bridges. I´m not sure I fully understand DMRs, but I am preparing the test platform in order to test it.

• Does each 'site' of the 3 use HA? We recommend that for the best guarantee of once-and-only-once delivery to consumers, when the architecture becomes distributed across regions you face many tradeoffs.
  Not now, but of course if we finally plan to go to production with Solace we would go for HA.

• Do you plan to use Replay (and have adequate storage to allow that)?
  We could use replay. We could have storage. I suppose it could be nice to have, but to be honest I haven´t yet thought about it.

Hopefully everything is clear... Sorry for the long post and sorry for my english.
Thanks for your help!

[Deleted User]

There is another option, and that would be to configure Destination1 and Destination2 as DR-mates with 2 messaging-VPNs across them. The DR mechanism is different than bridging/DMR in that it adds an extra layer to to keep the two sites synchronized in lockstep at all times, rather than as distinct sites with their own configurations. Let's see if Markdown can capture it

+---+------++------+--+
|D1 |VPN1  ||VPN2  |  |
|   |(Live)||(Stby)|  |
+---|------||------|--+
    |  |   ||  ^   |
    |  |   ||  |   |
    |  V   ||  |   |
+---|------||------|--+
|   |VPN1  ||VPN2  |  |
|D2 |(Stby)||(Live)|  |
+---+------++------+--+

What that means is that VPN1 and VPN2 are virtualized across the two sites, and you can designate which destination is active for each VPN. That way, you can accept connections to VPN1 on Destination1 and sync it to Destination2. And accept connections to VPN2 on Destination2 while synchronizing it to Destination1.

That way, if either side fails, you can flip its peer from Standby to Live and it will begin accepting connections from all clients, and the VPNs will be in sync with all their queues, and at the last synchronized point of consumption.

I realize this gets pretty complicated. Can have an offline conversation with you about it if you'd like.

Aaron

Hey all... a bit late to the game on this one, I was just browsing around in the Community and stumbled on this. Just wanted to say that if you wanted to consider using non-persistent messaging for your usecase, then you could use 2 (or more) VPN bridges between your different sites, and configure each with a non-persistent shared subscription. Format: #share/<groupname>/<topic>. That way the message flow can round-robin between the two bridges. Just a thought. But would also mean that there is a chance for message loss if the broker restarted due to using non-persistent (aka Direct) messaging.

adri

Hello Aaron, even later on my side! I´ve just been able to come back to this subject today and I´ve seen your last message (for some reason I didn´t see it before). I´ve done a simple test on my side, and, as you said it works as expected. Persistency shouldn´t be an issue for us as we can recover messages differently.

However, while, testing it, i´ve tried to create a queue or a topic endpoint on the "distant" VPN in order to take advantage of non-exclusive setup and have also within the VPN round robin messages to final clients but without success. I am new to solace and it´s advanced configuration so I must have misunderstood something. The idea was to create a queue, subscribe to > for example and then have the clients consume from the queue.

VPN 1 ---- Bridge with #share topic ---> VPN 2 ---> Queue ----> Client 1/Client2

Is this possible? I´m testing it fully from the UI with the samble kubernetes deployment.
Thanks so much for all your feedback.

adri

Hello again. It was actually a miss-configuration on the VPN settings side.
Thanks to Aaron I managed to configure it properly.
Thanks a lot for your help!

Aaron

Hi @adri !! Glad to hear you got it working. Although, I do want to clarify something because it sounds like you might be mixing Direct and Guaranteed messages. Remember, as soon as you have a queue or topic endpoint that receive a Direct message via a subscription, the message will get promoted to Guaranteed. https://solace.com/blog/message-promotion-demotion/

From my understanding, you want something like this:

So using VPN bridges between 1&2 and 1&3, and using Direct shared subscriptions on them, it will round-robin between VPN2 and VPN3. Then, you also would like to have Client 1 and Client 2 receive the message in a round-robin fashion. (I drew a Client 3 and Client 4 in there as well for completeness. Each of the 4 clients would each receive approx. 25% of all the messages). Since you are ok using Direct messages, why not just used the shared subscription there again? Client 1 and 2 have another shared subscription, same topic. Same for clients 3 & 4. The group name for the share will probably be different than the group name for the bridges on VPN1... but I don't think it matters since the clients are in VPN2. But I'd probably make different group names just in case.

Anyhow, yes what you suggested/proposed using a non-exclusive queue with a topic subscription would totally work... but it just means that the message is being stored on the message spool of VPN2. If no clients were there to receive the message, it would just sit on the queue. Is that what you want? If using Direct messages all the way through (i.e. Client 1&2 use another Shared sub), then the message will only be delivered if one of the clients is online to receive it.

Hope that clarifies! Let me know if there's something we can help with. 👍🏼

adri

Hi @Aaron! Thanks for your continuous help.

Indeed we could have Clients use shared subscriptions but having the queues we make sure that messages once they get to VPN 2 or 3 are persisted and not lost. Ideally I would love to have guaranteed messages between VPN1 and VPN2/3 so that I can be sure that messages are not lost anywhere (this was actually the idea of the original post), but as this cannot be done, we can manage the loss of messages between VPN1 and VPN2/3 internally and rely on Solace for the persistance of the message once it gets to the "client" VPNs. In case one of the clients fail to treat a message thanks to the queue this message can be then consumed by another client.We would have to be sure that there are always clients in the VPNs so that messages are consumed.

Just to be clear, the intention is to have different cloud regions. VPN1 would be some kind of master region and VPN2/3 and others would be client regions with their own Solace instance.

Does it make sense?

Aaron

Makes total sense, yup! We're looking internally to see if there's a way to do this with Guaranteed delivery, but maybe might have to be an enhancement request.