Partitioned Queue and Topic Subscription

vn01 · October 24

We have a setup with the following

A producer posting events to topic "T/o-all/event-type/{eventType}" where {eventType} is dynamic value. When posting the event to the topic we also set

jmsTemplate.convertAndSend(topic, event, message -> {
  message.setStringProperty(SupportedProperty.JMSX_GROUP_ID,partitionKey);
  return message;
});

A partitioned queue "o-all-events" with a Topic Subscription "T/o-all/event-type/* "

When I post events to the topic "T/o-all/event-type/{eventType}" , I dont see any events routed to the partitioned queue "o-all-events"

Is this configuration of a Partitioned Queue with a topic subscription not supported ?

Tamimi · October 24

Hey @vn01 - may I ask how you configured your o-all-events queue? I partition queue is nothing but a non-exclusive along with a child queue for each partition. The parent queue holds the configuration state of the partitioned queue, including the subscription. So if you add a subscription T/o-all/event-type/* on the parent queue, you should see the events routed to the consumer bound to the queue.

And how are you checking that events are routed to the queue?

rey · October 24

Hey there @vn01,

Could you give me an example of a topic you're sending?

vn01 · October 24

This is how I created the queue

rey · October 24

Sorry @vn01 I think I wasn't clear enough in my question.

I'm actually looking for an example of the topic your queue is subscribing to.

If the full topic you're subscribing to is something like "T/o-all/event-type/conference " then your setup should work.

However, if your topic is longer, say something like "T/o-all/event-type/conference/attended" then your subscription needs to include a > at the end, so something like "T/o-all/event-type/*/>".

Let me know if this helps.

vn01 · October 24

@rey Here is an example of the topics

T/o-all/event-type/OrderCreated

T/o-all/event-type/OrderConfirmed

My topic subscription that I added to the queue

Here is the log of the event posted to the topic

c.solacesystems.jms.SolMessageProducer : Client-1:Flow-165: Entering sendMessage. Destination: T/o-all/event-type/OrderCreated delivery mode: 2 priority: 4 timeToLive: 0

mstobo · October 24

Hi @vn01 I just tried your config using your topics and it evenly balanced the messages between the two partitions as I didn't provide a key. Are you sure you don't have another consumer bound to the queue? Have a look at the Consumers tab to see if you have a consumer attached. If not have a look at your cumulative stats on your Stats tab to see the historical messages queued and the Consumer Bind stats.

vn01 · October 25

ok, looks like all the events were ignored with the below reason

@Tamimi @rey @mstobo

I was aware XA is not supported consuming/receiving from a partitioned queue. I was checking if we can still consume a Partitioned Queue using a NonXA transaction and orchestrate a db + topic as a XA transaction, which would have worked for our use case, but looks like its an issue even when producing to a topic that has a partitioned queue as a subscriber.

Summary of my evaluation

If we need to leverage XA - then we have to go with Exclusive Queues and Non Partitioned Queues. With Exclusive Queues trade-off is on parallel processing and with Non Partitioned Non Exclusive queues trade-off is out-of-order events

rey · October 25

Thanks so much @vn01 for the feedback, it's always amazing to have a user reply with information so others can learn!

mstobo · October 25

A few thoughts:

If you only have one db in your "db + topic" scenario does it need to be an XA transaction. Will a regular transaction not suffice?
Some Transaction Managers (not all) support the concept of "last participant". If yours does, could this be an option?
Before we had Partitioned Queues we used to use a pattern for Sticky Load Balancing between queues when message order is important that is explained in this blog: https://solace.com/blog/sticky-load-balancing-in-solace-pubsub-event-broker/

Hope this helps.

vn01 · October 27

@mstobo It is still not fool proof. In a situation where we apply the "topic" as the last participant, and the topic transaction fails, we roll back (db) and the transaction fails, this works. In lot of other scenarios it might be lot more complex to handle compensating actions on the last participant that does not participate in an atomic transaction completely.

Now consider below situation

We transaction on the topic completed successfully, but then the db commit fails, this leads to a scenario where we send a message but the db transaction has failed. Even we switch the commit order db and then topic, we will have a situation where we will end up with a scenario where message is not sent when it was supposed to be sent. There could be many other such scenarios to be handled.

I did check the sticky load balancing too much of setup to be maintained, ideally XA support for a Partitioned Queue would have been a nice, though I can understand it might not be an easy one to solve.

Aaron · October 28

For reference, here is @vn01 's other thread about PQs and XA transactions from August, where I already tried to talk them out of using XA.

vn01 · November 4

@Aaron : I was ok to move away from consuming a Solace Queue using a non XA transaction, but removing XA from sending to a topic, that is routed to a partitioned queue would be a limitation from using Partitioned Queues even with out an XA to consume the queue.

vn01 · November 25

@Aaron : With your reference around moving away from XA, are you suggesting we use

https://microservices.io/patterns/data/transactional-outbox.html when a service does a DB transaction along with producing a message to solace topic in a single transaction.

🎄 Happy Holidays! 🥳

Happy Holidays!

Partitioned Queue and Topic Subscription

Comments

Categories

This Month's Leaders

This Week's Leaders