Partitioned Queue and Topic Subscription
We have a setup with the following
A producer posting events to topic "T/o-all/event-type/{eventType}" where {eventType} is dynamic value. When posting the event to the topic we also set
jmsTemplate.convertAndSend(topic, event, message -> {
message.setStringProperty(SupportedProperty.JMSX_GROUP_ID,partitionKey);
return message;
});
A partitioned queue "o-all-events" with a Topic Subscription "T/o-all/event-type/* "
When I post events to the topic "T/o-all/event-type/{eventType}" , I dont see any events routed to the partitioned queue "o-all-events"
Is this configuration of a Partitioned Queue with a topic subscription not supported ?
Comments
-
Hey @vn01 - may I ask how you configured your
o-all-eventsqueue? I partition queue is nothing but a non-exclusive along with a child queue for each partition. The parent queue holds the configuration state of the partitioned queue, including the subscription. So if you add a subscriptionT/o-all/event-type/*on the parent queue, you should see the events routed to the consumer bound to the queue.And how are you checking that events are routed to the queue?
0 -
Sorry @vn01 I think I wasn't clear enough in my question.
I'm actually looking for an example of the topic your queue is subscribing to.
If the full topic you're subscribing to is something like "
T/o-all/event-type/conference" then your setup should work.However, if your topic is longer, say something like "
T/o-all/event-type/conference/attended" then your subscription needs to include a>at the end, so something like "T/o-all/event-type/*/>".Let me know if this helps.
0 -
@rey Here is an example of the topics
T/o-all/event-type/OrderCreatedT/o-all/event-type/OrderConfirmedMy topic subscription that I added to the queue
Here is the log of the event posted to the topic
c.solacesystems.jms.SolMessageProducer : Client-1:Flow-165: Entering sendMessage. Destination: T/o-all/event-type/OrderCreated delivery mode: 2 priority: 4 timeToLive: 0
0 -
Hi @vn01 I just tried your config using your topics and it evenly balanced the messages between the two partitions as I didn't provide a key. Are you sure you don't have another consumer bound to the queue? Have a look at the Consumers tab to see if you have a consumer attached. If not have a look at your cumulative stats on your Stats tab to see the historical messages queued and the Consumer Bind stats.
0 -
ok, looks like all the events were ignored with the below reason
I was aware XA is not supported consuming/receiving from a partitioned queue. I was checking if we can still consume a Partitioned Queue using a NonXA transaction and orchestrate a db + topic as a XA transaction, which would have worked for our use case, but looks like its an issue even when producing to a topic that has a partitioned queue as a subscriber.
Summary of my evaluation
If we need to leverage XA - then we have to go with Exclusive Queues and Non Partitioned Queues. With Exclusive Queues trade-off is on parallel processing and with Non Partitioned Non Exclusive queues trade-off is out-of-order events
1 -
A few thoughts:
- If you only have one db in your "db + topic" scenario does it need to be an XA transaction. Will a regular transaction not suffice?
- Some Transaction Managers (not all) support the concept of "last participant". If yours does, could this be an option?
- Before we had Partitioned Queues we used to use a pattern for Sticky Load Balancing between queues when message order is important that is explained in this blog: https://solace.com/blog/sticky-load-balancing-in-solace-pubsub-event-broker/
Hope this helps.
0 -
@mstobo It is still not fool proof. In a situation where we apply the "topic" as the last participant, and the topic transaction fails, we roll back (db) and the transaction fails, this works. In lot of other scenarios it might be lot more complex to handle compensating actions on the last participant that does not participate in an atomic transaction completely.
Now consider below situation
We transaction on the topic completed successfully, but then the db commit fails, this leads to a scenario where we send a message but the db transaction has failed. Even we switch the commit order db and then topic, we will have a situation where we will end up with a scenario where message is not sent when it was supposed to be sent. There could be many other such scenarios to be handled.
I did check the sticky load balancing too much of setup to be maintained, ideally XA support for a Partitioned Queue would have been a nice, though I can understand it might not be an easy one to solve.
0 -
For reference, here is @vn01 's other thread about PQs and XA transactions from August, where I already tried to talk them out of using XA.
0
