Clarification on Partition Queue and how it interacts with the consumer.
I am sending messages from the Publisher to the Partition Queues with the partition key defined .
Now i have 2 consumers. Is it possible that Consumer 1 receives messages that has partition key1 and consumer2 receives messages that has partition key 2?
Best Answer
-
Selectors for this simple/basic use case are overkill, and not that performant. We always recommed using topics + subscription filtering to perform basic routing like this. The easiest way to do this IMHO is for the publisher to publish messages with a well defined topic that includes some specific information, and then each consumer has their own queue, and add the topic subscriptions as appropriate to attract messages to those queues.
E.g. queue1, with subs
payments/new/>andpayments/cancel/>queue 2, with subs
payments/*/custABC/>andpayments/*/custXYZ/>Or something. I'm just making this up. But then whenever the publisher publishes, the well-defined and descriptive topic
payments/new/custABC/USD/txn8a317fwill allow it to get routed to the right consumers based on their required info.Could even be as simple as queue1 with subscription
consumer1/>and queue2 with subscriptionconsumer2/>but this is not our best practice, the topics should describe the data contained within the message, not hardcode the desired routing behaviour.Hope that makes sense? I don't see the need for the 2 consumers to share a common queue?
1
Answers
-
HI @escjames07 - That is the exact functionality partitioned queue offers. Based on the set key, internally broker applies a hashing function to determine the partition on which the event will be delivered to.
As with the terminology - what you would populate on the published messages is a KEY, which is what is used for hashing to determine the PARTITION. At any point in time, a PARTITION will be bound to a SINGLE consumer (note, a consumer can be bound to multiple PARTITIONs) - ensuring that the messages are delivered in ORDER of arrival at the partitioned queue.
I highly recommend you to checkout the following references:
1 -
Note that, due to hashing, both key 1 and key 2 could possibly end on on the same partition. Ideally, you have a very large keyspace (e.g. Order ID, Driver's License #, IP address, etc.) to ensure you have a nice even distribution of keys vs. partitions. The more keys you have, the better.
1 -
My use case is as follows:
We have a single publisher where we can define a key to send data to a queue. I have two different clients, and based on the key or property, messages need to be routed to the appropriate consumer.
System A : Publisher 1
System B : Consumer 1
System C: Consumer 2
I explored partitioned queues and queue selectors, and I understand that selectors are not compatible with partitioned queues.
For Partitioned Queues:
I set the key for the message in the publisher using Solace-User-Property-JMSXGroupID. Since each partitioned queue can accommodate multiple keys, it balances messages across partitions evenly. While this preserves the message order, I noticed that the consumer binds to the partitions automatically. In this case, I am unable to route specific messages to a designated consumer as intended. I now understand that partitioned queues are ideal when there is a single client with multiple consumers, allowing autoscaling while maintaining message order for specific keys.
For Queue Selectors:
When using queue selectors, I can segregate messages to different consumers by setting the property in the publisher:
.with_property("SystemName","B")
In the consumer, I configured the selector as:
.with_message_selector("SystemName= 'B'")
This allows the correct consumer to receive the messages. However, since the queue is non-exclusive, message order is not preserved.
Kindly correct me if there is anything wrong in my understanding.
Would be really helpful to know if this flow is possible or not ?
0 -
Selectors for this simple/basic use case are overkill, and not that performant. We always recommed using topics + subscription filtering to perform basic routing like this. The easiest way to do this IMHO is for the publisher to publish messages with a well defined topic that includes some specific information, and then each consumer has their own queue, and add the topic subscriptions as appropriate to attract messages to those queues.
E.g. queue1, with subs
payments/new/>andpayments/cancel/>queue 2, with subs
payments/*/custABC/>andpayments/*/custXYZ/>Or something. I'm just making this up. But then whenever the publisher publishes, the well-defined and descriptive topic
payments/new/custABC/USD/txn8a317fwill allow it to get routed to the right consumers based on their required info.Could even be as simple as queue1 with subscription
consumer1/>and queue2 with subscriptionconsumer2/>but this is not our best practice, the topics should describe the data contained within the message, not hardcode the desired routing behaviour.Hope that makes sense? I don't see the need for the 2 consumers to share a common queue?
1
