Solace Topic Destination Header

sdeveloper
sdeveloper Member Posts: 8 ✭
edited October 2022 in General Discussions #1

Hi Team - this is regarding the scenario of publisher using topic string as destination header (using topic hierarchy) and consumer using wild card (> or *) to consume required message. This is in order to facilitate filtering at event broker level.

Does the above scenario ensure guarantee message delivery when none of the consumer is online?

Example: Publisher sends below messages (there is no end point created in broker)

Msg1 Destination Header: OMS/EQ/AK/1

Msg2 Destination Header: OMS/EQ/AK/2

Msg3 Destination Header: OMS/EQ/AS

Consumer1 has 2 instances subscribing to 'OMS/EQ/>'. Consumer2 has 2 instances subscribing to 'OMS/EQ/AK/>'. Assume that both instances of Consumer2 are down.

Consumer1 receives all 3 messages. But since Consumer2 has both instances down, does Solace event broker store those 2 messages (with header OMS/EQ/AK/1 and OMS/EQ/AK/2) and deliver them to Consumer2 once it comes online?

Comments

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    edited October 2022 #2

    Hi there @sdeveloper. Thanks for the detailed question! So, first thing: use of topics for message filtering == great, but this is only for addressing... if you want Guaranteed delivery / persistent messaging, then you'll also need to be using Queues to store the messages in the broker. That is: your messages should always be published to topics, but if you want Guaranteed delivery, then the publisher will have to set DeliveryMode to PERSISTENT (or use a PersistentMessagePublisher if using our new APIs), and the consumers will need to use Queues that are subscribed to the topic(s) that they want. Which API are you using, so I can point you to some relevant samples? If your consumer apps subscribe directly to the topics, then you aren't using Guaranteed delivery.

    In your example, you have two consumers apps, with two instances each? Are these expected to be round-robin, or active-standby?

    So if your Consumer1 apps had a queue (q1) subscribed to OMS/EQ/> , and Consumer2 apps had another queue q2 subscribed to OMS/EQ/AK/> and they were both down, then the broker would hold any messages for Consumer2 apps in q2 until at least one of them connected to the broker to receive it. Each queue would receive a copy of the message, and if Consumer1 apps were online they'd receive it right away from q1.

    Hope that helps! Let us know if you have more questions.

  • sdeveloper
    sdeveloper Member Posts: 8 ✭
    #3

    Thanks a lot @Aaron for your insightful feedback. In order for this to work, it is my understanding that I need to do following. Please let me know if this makes sense.

    1. Create Topic Endpoints: 'OMS/EQ/AK' and 'OMS/EQ/AS'

    2. Create Queue: 'OMS/EQ' with mapped topics 'OMS/EQ/AK' and 'OMS/EQ/AS'

    3. The publisher will send following messages:

    a. Msg1 Destination Header: OMS/EQ/AK (I think I can no longer send OMS/EQ/AK/1)

    b. Msg2 Destination Header: OMS/EQ/AK (I think I can no longer send OMS/EQ/AK/2)

    c. Msg3 Destination Header: OMS/EQ/AS

    4. Consumer1 having 2 instances subscribes to queue 'OMS/EQ' and uses wild card of 'OMS/EQ/>' to get all 3 messages

    5. Consumer2 having 2 instances subscribes to queue 'OMS/EQ' and uses wild card of 'OMS/EQ/AK' to get only first 2 messages. I think I cannot make this more granular.

    I am using Spring Cloud Stream 3.0.7 and Stream Listener based annotation for consumer (not yet on functional approach). Please do share any code sample if you have, especially the configuration for filtering.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    #4

    One big thing to remember here is that you do not configure Topics in Solace... topics are ephemeral metadata on each published message. Watch this: https://www.youtube.com/watch?v=PP1nNlgERQI

    Topic Endpoints and Queues have quite similar functions, they are both persistent endpoints to store messages. Typically, Topic Endpoints are only used in JMS applications as an analog for durable subscriptions. We recommend Queues with topic subscriptions for most use-cases. See this blog by Leah: https://solace.com/blog/queues-vs-topic-endpoints/

    So skip step #1, and everything else should work fine. 👍🏼

    The Spring Cloud Stream binder helps create queues and adds topic subscriptions to them automatically. I believe there is a way to tell it to use premade/predefined queues. I'm not too familiar with it. Check the docs here: https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter

  • sdeveloper
    sdeveloper Member Posts: 8 ✭
    #5

    Thanks again @Aaron! I will try this out and update.

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 538 admin
    edited November 2022 #6

    And to add to what @Aaron mentioned, a couple of notes on terminologies:

    • You create queues on solace and add topic subscriptions to the queue. This basically means that any msg published to the broker that matches the topic subscription will be attracted to the queue --> Guaranteed messaging
    • You can still publishes messages on OMS/EQ/AK/1 from your publisher and adding OMS/EQ/AK/* or OMS/EQ/AK/> as topic subscriptions to your queue
    • Consumers dont really "subscribe" to queues they bind to a queue and consume messages from it
  • sdeveloper
    sdeveloper Member Posts: 8 ✭
    #7

    Thanks @Tamimi for the clarification!

  • sdeveloper
    sdeveloper Member Posts: 8 ✭
    #8

    Hi @marc, @Aaron, @Tamimi

    1_Publisher_Properties.JPG
    2_Publishing_Code.JPG
    3_Subscriber_TryMe.JPG
    4_Consumer_Code.JPG
    5_Consumer_Properties_Solace2.0.0.JPG
    6_Consumer_Properties_Solace3.2.0.JPG
    7_Consumer_Error_Solace3.2.0.txt
    8_Consumer_Error_Solace3.3.0.txt

    @TamimiI tried to test above setup. I think it worked on publisher side but running into issues on consumer side. Please find below details. Can you please kindly review and let me know your thoughts?

    1. The NBS.POC.EQ is a pre-created queue with pre-created topics, OMS/EQ/AK and OMS/EQ/AS

    2. Publishing side properties (1_Publisher_Properties.JPG) and code (2_Publishing_Code.JPG) is attached. I tested it with Solace 'Try Me' and it worked. Attached is the screenshot using the subscriber of 'Try Me', 3_Subscriber_TryMe.JPG. It shows the expected topic for different message (messages sent from REST end point to sendMessage).

    3. I then tried to programatically consume the same set of messages. I tried with multiple version of spring-cloud-starter-stream-solace. Same consumer properties were set across different versions. Attached is the 4_Consumer_Code.JPG

    a. spring-cloud-starter-stream-solace version of 2.0.0-string-serialization

    Attached is 5_Consumer_Properties_Solace2.0.0.JPG. The consumer gets all messages and the property 'queueAdditionalSubscriptions: OMS/EQ/AS' is not applied.

    b. spring-cloud-starter-stream-solace version of 3.2.0

    Attached is 6_Consumer_Properties_Solace3.2.0.JPG. For this, I get error on start up of consumer application, 403: Permission Not Allowed. Attached is the stack trace, 7_Consumer_Error_Solace3.2.0.txt.

    c. spring-cloud-starter-stream-solace version of 3.3.0

    Same properties, as listed in 6_Consumer_Properties_Solace3.2.0.JPG was used. For this, I get error on start up of consumer application, EL1008E: Property or field 'isAnonymous' cannot be found on object of type 'com.solace.spring.cloud.stream.binder.provisioning.ExpressionContextRoot' - maybe not public or not valid?. Attached is the stack trace, 8_Consumer_Error_Solace3.3.0.txt.

    Thanks!

Categories

This Month's Leaders

This Week's Leaders