Horizontal scaling of event brokers?
I am investigating creating a HA PubSub+ broker setup, but a single primary broker will not have enough capacity for my needs. Do I need to create multiple triplet clusters and connect them? If so, how is load distributed between the clusters?
Comments
-
Hi @petegehrman,
You're definitely on the right track there. The default answer here would be to standup multiple brokers and setup an Event Mesh which will leverage DMR (Dynamic Message Routing) to route messages between brokers only when a subscriber wants them. A bit more info in docs here. If you want to discuss your use case in more detail to make sure you're on the right path just let me know and I can connect you with someone that has the conversations with users all the time
-Marc0 -
Thanks @marc. If I have multiple brokers connected via DMR, can I abstract that away from clients and provide a single address, or do publishers/consumers need to explicitly connect to one of the brokers?
0 -
Hi @petegehrman, it all depends on the message quality of service (DIRECT vs PERSISTENT). For DIRECT messages a client can connect to any point in the mesh: there is complete abstraction.
For PERSISTENT messages, a client needs to connect to a queue. It's therefore difficult to completely automatically abstract persistent message consumption from location in the mesh. With a bit of administrative cleverness, though, this can be done. We're working on a feature to completely automate this.
2 -
Thanks @TomF. Hope that helps @petegehrman!
0 -
Yes, thanks both of you for the help.
1
