Partitioned Queue and Topic Subscription

vn01
vn01 Member Posts: 17 ✭✭✭

We have a setup with the following

A producer posting events to topic "T/o-all/event-type/{eventType}" where {eventType} is dynamic value. When posting the event to the topic we also set

jmsTemplate.convertAndSend(topic, event, message -> {
  message.setStringProperty(SupportedProperty.JMSX_GROUP_ID,partitionKey);
  return message;
});

A partitioned queue "o-all-events" with a Topic Subscription "T/o-all/event-type/* "

When I post events to the topic "T/o-all/event-type/{eventType}" , I dont see any events routed to the partitioned queue "o-all-events"

Is this configuration of a Partitioned Queue with a topic subscription not supported ?

Tagged:

Comments

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 541 admin

    Hey @vn01 - may I ask how you configured your o-all-events queue? I partition queue is nothing but a non-exclusive along with a child queue for each partition. The parent queue holds the configuration state of the partitioned queue, including the subscription. So if you add a subscription T/o-all/event-type/* on the parent queue, you should see the events routed to the consumer bound to the queue.

    And how are you checking that events are routed to the queue?

  • rey
    rey Member, Administrator, Employee Posts: 11 admin

    Hey there @vn01,

    Could you give me an example of a topic you're sending?

  • vn01
    vn01 Member Posts: 17 ✭✭✭
    edited October 24 #4

    This is how I created the queue

  • rey
    rey Member, Administrator, Employee Posts: 11 admin

    Sorry @vn01 I think I wasn't clear enough in my question.

    I'm actually looking for an example of the topic your queue is subscribing to.

    If the full topic you're subscribing to is something like "T/o-all/event-type/conference " then your setup should work.

    However, if your topic is longer, say something like "T/o-all/event-type/conference/attended" then your subscription needs to include a > at the end, so something like "T/o-all/event-type/*/>".

    Let me know if this helps.

  • vn01
    vn01 Member Posts: 17 ✭✭✭

    @rey Here is an example of the topics

    T/o-all/event-type/OrderCreated

    T/o-all/event-type/OrderConfirmed

    My topic subscription that I added to the queue

    Here is the log of the event posted to the topic

    c.solacesystems.jms.SolMessageProducer : Client-1:Flow-165: Entering sendMessage. Destination: T/o-all/event-type/OrderCreated delivery mode: 2 priority: 4 timeToLive: 0

  • mstobo
    mstobo Member, Employee Posts: 26 Solace Employee
    edited October 24 #7

    Hi @vn01 I just tried your config using your topics and it evenly balanced the messages between the two partitions as I didn't provide a key. Are you sure you don't have another consumer bound to the queue? Have a look at the Consumers tab to see if you have a consumer attached. If not have a look at your cumulative stats on your Stats tab to see the historical messages queued and the Consumer Bind stats.

  • vn01
    vn01 Member Posts: 17 ✭✭✭

    ok, looks like all the events were ignored with the below reason

    @Tamimi @rey @mstobo

    I was aware XA is not supported consuming/receiving from a partitioned queue. I was checking if we can still consume a Partitioned Queue using a NonXA transaction and orchestrate a db + topic as a XA transaction, which would have worked for our use case, but looks like its an issue even when producing to a topic that has a partitioned queue as a subscriber.

    Summary of my evaluation

    If we need to leverage XA - then we have to go with Exclusive Queues and Non Partitioned Queues. With Exclusive Queues trade-off is on parallel processing and with Non Partitioned Non Exclusive queues trade-off is out-of-order events

  • rey
    rey Member, Administrator, Employee Posts: 11 admin

    Thanks so much @vn01 for the feedback, it's always amazing to have a user reply with information so others can learn!

  • mstobo
    mstobo Member, Employee Posts: 26 Solace Employee

    A few thoughts:

    1. If you only have one db in your "db + topic" scenario does it need to be an XA transaction. Will a regular transaction not suffice?
    2. Some Transaction Managers (not all) support the concept of "last participant". If yours does, could this be an option?
    3. Before we had Partitioned Queues we used to use a pattern for Sticky Load Balancing between queues when message order is important that is explained in this blog: https://solace.com/blog/sticky-load-balancing-in-solace-pubsub-event-broker/

    Hope this helps.

  • vn01
    vn01 Member Posts: 17 ✭✭✭

    @mstobo It is still not fool proof. In a situation where we apply the "topic" as the last participant, and the topic transaction fails, we roll back (db) and the transaction fails, this works. In lot of other scenarios it might be lot more complex to handle compensating actions on the last participant that does not participate in an atomic transaction completely.

    Now consider below situation

    We transaction on the topic completed successfully, but then the db commit fails, this leads to a scenario where we send a message but the db transaction has failed. Even we switch the commit order db and then topic, we will have a situation where we will end up with a scenario where message is not sent when it was supposed to be sent. There could be many other such scenarios to be handled.

    I did check the sticky load balancing too much of setup to be maintained, ideally XA support for a Partitioned Queue would have been a nice, though I can understand it might not be an easy one to solve.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 644 admin

    For reference, here is @vn01 's other thread about PQs and XA transactions from August, where I already tried to talk them out of using XA.

  • vn01
    vn01 Member Posts: 17 ✭✭✭

    @Aaron : I was ok to move away from consuming a Solace Queue using a non XA transaction, but removing XA from sending to a topic, that is routed to a partitioned queue would be a limitation from using Partitioned Queues even with out an XA to consume the queue.

  • vn01
    vn01 Member Posts: 17 ✭✭✭
    edited November 25 #14

    @Aaron : With your reference around moving away from XA, are you suggesting we use

    https://microservices.io/patterns/data/transactional-outbox.html when a service does a DB transaction along with producing a message to solace topic in a single transaction.