How can i gurantee sequence of events to be consistent.
Lets assume i have 4 events
Event A producing message to Queue AQ
Event B producing message to Queue BQ
Event C producing message to Queue CQ
Event D producing message to Queue DQ
Lets assume consumer named TestConsumer consumer consumes messages from all 4 queues.
I want to make sure when solace emit these events it will be in order A->B->C->D
and consumer also consumes in same order.
How i can implement this, I am implementing using SpringBoot.
Answers
-
Hi @gauravvats, the simplest and recommended way to do this would be to combine the 4 queues in to one. You can then always be sure your events are delivered in order.
How might you do this? Ensure all events are produced on a topic, and have all your queues subscribe to the topic(s). You can then manipulate the subscriptions on a given queue to ensure it has all the events your application needs, and only those events.
This naturally leads you to an architecture where the producers own the event, and the consumers own the queue. The consumer then controls what events it receives, the parallelism, ordering etc without having any effect on other consumers or the producer.
3 -
Actually the reason behind different queue creation is that each queue is associated with a unique operation and so it is not possible to combine queues.
and we have 4 topics as well which are connected to the respective queues based on the operation.
what i need to implement is that we got the events in a specific order on different queues and how we can control the ordering between different queues.
0 -
Are there any other consumers on these queues? Are there any other consumers of these events?
If not, then it doesn't matter that different events are on the same queue. If you're concerned that the consumer needs to understand which event it is about to process (for instance, if the payload is different for each event), get the topic destination on the event when you read it from the queue and dispatch it to the right processing code in your application.
0 -
Hi @gauravvats,
I agree with @TomF that the simplest solution here is likely to publish your events to topics and have one queue which subscribes to all of the topics. If you're not familiar with how this works @Aaron made a great video which will hopefully help: https://www.youtube.com/watch?v=PP1nNlgERQI&list=PLY1Ks8JEfJR57vCkrQK0Y9bn8DKNMMJI_&ab_channel=Solace
As Tom mentioned, on the consumer side if you need helping understanding what event you're receiving then you could either parse that out of the topic or another option would be to have the publisher add a user property (key/value pair) to the event which identifies what it is. The consumer can then look at that to identify which operation to invoke.
Hope that helps
0 -
@TomF Yes there are a lot of consumers of the same queues.
I just need to control the order of events means
Ex. 4 events are :
From source a publisher is publishing these 4 events and we have no control over it means order:
New Event
Pending Event
Create Event
Placed Event
i.e. sometime Pending comes first and sometime create comes first
Each one have associated queue with it.
We are processing them at our end (Consumer) in order New->Pending->Created->Placed so if they are not in this order we are just NACK the request.
What i am expecting some configuration in queues that it will always emit the events in the above order.
If we have any configuration like that please share.
Thanks
Gaurav
0 -
Hi Gaurav,
We've had a confusion in terminology here, so forgive us. When we in Solace hear "in order," to us it means "in the order that the events were received."
What you seem to be looking for is "in the order I've defined," so you'd like the broker to re-order the events based on your preferred sequence. We don't do this in the broker, you'd need to have an application do this. I think it's an anti-pattern to do things this way because:
1) You're putting business logic in the broker. This confuses architectural layer and will bite you later on: supposing you wish to add a new state in the ordering? What if you wish to change the ordering? Where do you make this change, in the broker or in the application? What if there's a bug in your business logic? Your broker will become an agility choke point.
2) You're putting business logic in the broker. This means the broker has to do more work (computationally), which makes it a performance choke-point.
These reasons are why people are moving away from centralised ESB type architectures.
0
