Load Balancing on One Topic Multiple Consumer

BrianChan
BrianChan Member Posts: 18
edited April 2022 in PubSub+ Event Broker #1

Hi all, I am new to Solace.

Recently I am working on a project that assume the publisher only work on one topic and there is multiple consumer listen to the same topic.

The project will be deployed to a k8s and I expect there would be a scaling on the consumer side.

My design at this point is one publisher publishing a topic to multiple queue so the message will spread through the queues. However I would like to have unique message in each queues which means if MessageA is assign to Queue1, MessageA won't appear in another queue.

I would like to know the numbers of outstanding message in order to formulate my scaling rules for k8s.

For this architecture, how should I configure in order to achieve the goal? For further information, I am using python to compile the program.

I would be grateful if someone can provide me with suggestions or guidance. Thank you very much.

Best Answer

  • uherbst
    uherbst Member, Employee Posts: 121 Solace Employee
    #2 Answer ✓

    Hi @BrianChan,

    the typical Solace solution for this is ONE queue with multiple consumers listening.

    Each message in this queue is forwarded round-robin to one of the listening consumers.

    You can add and remove consumers any time you want.

    Just set the queue to "non-exclusive".

    Feel free to ask, if anything is unclear here.

Answers

  • uherbst
    uherbst Member, Employee Posts: 121 Solace Employee
    #3 Answer ✓

    Hi @BrianChan,

    the typical Solace solution for this is ONE queue with multiple consumers listening.

    Each message in this queue is forwarded round-robin to one of the listening consumers.

    You can add and remove consumers any time you want.

    Just set the queue to "non-exclusive".

    Feel free to ask, if anything is unclear here.

  • BrianChan
    BrianChan Member Posts: 18

    @uherbst

    Thank you so much for your reply. Actually I have tried non-exclusive queue but in my first trial, it seems all messages go to all consumer. However, I tried again and the message seems forward to either one consumer only. My bad.

    Thanks again.

  • arih
    arih Member, Employee Posts: 125 Solace Employee

    Hi @BrianChan

    Just to add for scaling, you can leverage KEDA (https://keda.sh/docs/2.6/scalers/solace-pub-sub/) for Solace, which monitors either message count or spool size.

  • BrianChan
    BrianChan Member Posts: 18

    @arih

    Thank you for your suggestion. It helps a lot!!