Load Balancing on One Topic Multiple Consumer

BrianChan

Hi all, I am new to Solace.

Recently I am working on a project that assume the publisher only work on one topic and there is multiple consumer listen to the same topic.

The project will be deployed to a k8s and I expect there would be a scaling on the consumer side.

My design at this point is one publisher publishing a topic to multiple queue so the message will spread through the queues. However I would like to have unique message in each queues which means if MessageA is assign to Queue1, MessageA won't appear in another queue.

I would like to know the numbers of outstanding message in order to formulate my scaling rules for k8s.

For this architecture, how should I configure in order to achieve the goal? For further information, I am using python to compile the program.

I would be grateful if someone can provide me with suggestions or guidance. Thank you very much.

uherbst

Hi @BrianChan,

the typical Solace solution for this is ONE queue with multiple consumers listening.

Each message in this queue is forwarded round-robin to one of the listening consumers.

You can add and remove consumers any time you want.

Just set the queue to "non-exclusive".

Feel free to ask, if anything is unclear here.

uherbst

Hi @BrianChan,

the typical Solace solution for this is ONE queue with multiple consumers listening.

Each message in this queue is forwarded round-robin to one of the listening consumers.

You can add and remove consumers any time you want.

Just set the queue to "non-exclusive".

Feel free to ask, if anything is unclear here.

BrianChan

@uherbst

Thank you so much for your reply. Actually I have tried non-exclusive queue but in my first trial, it seems all messages go to all consumer. However, I tried again and the message seems forward to either one consumer only. My bad.

Thanks again.

arih

Hi @BrianChan

Just to add for scaling, you can leverage KEDA (https://keda.sh/docs/2.6/scalers/solace-pub-sub/) for Solace, which monitors either message count or spool size.

BrianChan

@arih

Thank you for your suggestion. It helps a lot!!