Message Ordering

Naga

I have a requirement where Message ordering is the key.
When 10 messages are published to solace with order numbers OD1,OD2,OD3…OD10
While consuming if the first message OD1 fails the consumer should be receiving OD1 and then OD2 so on. The consumer should not be receiving OD2 before OD1.

This way the order should be maintained. Is there a way we can achieve this ?

TomF

Hi @Naga, message ordering sounds simple but the conversation always ends up being involved.
Solace by default enforces message ordering, with the unit of order being reception time at the broker. You should use a queue to ensure that ordering is maintained even if the consumer is offline.
Things get more complicated when we start to think about consumer failures. What if the consumer gets OD1 and it causes the consumer to crash? Well, you have some knobs to twiddle. If OD1 must be before OD2 even at the expense of system failure, leave the queue defaults as they are. If, however, you'd like to set a maximum number of re-tries before the broker considers OD1 to be "poisonous" and optionally discard or move it to a dead message queue, then change the "Maximum Redelivery Count" setting on the queue.

Aaron

Just to be explicit, you can't use non-exclusive queues if strong message ordering is required... only Exclusive queues will ensure that message order is properly maintained, even in the event of an application crash / unbind.

TomF

Good point @Aaron. Non-exclusive queues give you application load balancing at the expense of breaking message ordering as messages are round-robin-ed between applications.

Robert

I honestly see a problem with that limitations.

• If message ordering is connected to Exclusive Queues then it means that you will have problems to scale ? Right ?

That is exactly why kafka and some other order concepts (oracle web logic) base there ordering on groups or partitions (kafka).

Example:

Sales Order 1 - create

Sales Order 1 - update 1

Sales Order 1 - update 2

Sales Order 2 - create

Sales Order 2 - update 1

Sales Order 2 - update 2

Sales Order 2 - update 3

So you need all Sales Order 1 events in order and all Sales Order 2 events.

Like this you can scale to handle things in parallel per group/partition.

So my key question is how to handle in Solace ordering when i have to handle millions or messages per hour which require more then one consumer ?

TomF

Hi @Robert , this is one way of solving a scaling problem. It only works for a very specific subset of scaling problems, caused by the need to pick a stream of related events from a coarse, static topic address scheme that does not enable fine grained topic filtering and where the operational overhead of running your message broker is so high you have no alternative but to use it as your sole persistence layer. Event sourcing is great, but not when the persistence layer forces you to adopt one particular application style like this.

Solace has many use cases handling many more that millions of messages per hour - try 100B a day.

Some other approaches:

1. Sharding with topics. Have topic taxonomy salesOrder/{verb}/{id}. Now you can filter for order "1" and allocate that to a particular application/application instance. You can filter on just about any criteria, actually. What's great is that if, later on, you need to further decompose your microservices so that you have separate create and update services, you can do that two. No need to change your topic. No need to change the number of partitions as your scale. No topic or partition rebalancing.
2. Use polyglot persistence. Your example assumes each instance of the service has to own everything that happens to a particular order id, because the microservice's persistence is local to that instance of the microservice. People tend to assume this is what ployglot persistence means, but it doesn't: it means "use the persistence layer most suited to that particular microservice." So, don't throw away caches and traditional DBs, amongst other options. This is especially true of situations where updates are not easy to separate in to isolated entities like this (where in this case the entity is the order number).
3. Client based routing/filtering. Instead of basing your unit of scaling on the business object (sales order) in this case, why not base it on the real unit of work here - the customer? That way your event stream has all the events relating to the customer - a much more powerful concept that makes building omni-channel interfaces and single customer views much simpler, and makes dynamic allocation of applications instances much simpler.

There are many, many different ways of scaling. At Solace, we enable you to pick whichever suits your use case, rather than forcing you to architect everything to overcome limitations in your messaging layer.

Robert

@TomF I can not really relate your reply to what i needed clarification on and see some problem.

When message ordering is related to exclusive queues then i wonder how you scale to have multiple parallel consumers to handle workload on subscriber/consumer side ?

So key question is how does Solace support ordering with high volumes ?

So please tell me how you scale this example:

• Events 1 mio per hour
• Distinct Object: 100000
• Order is needed

Processing of message takes 50 ms

Messages/sec = 20

Throughput: 3600 seconds * 20 = 72000 msg/hour

So you need to scale with concurrent consumer to handle this volumes in parallel.

So 10 parallel consumers (to have some buffer for peaks) would solve the problem but that is not possible with exclusive queues. Right ?

The only options i can see to solve that problem are:

• Switch to Push Subscription RDP - that decouples the publish from processing (the slow part is the processing on consumer) - You can scale up to 50 calls to endpoint in parallel.
• Sure you can add some partition field in topic and filter on that. But that is not an out of the box solution which is nicely solvable by providing a feature of partitions to split up the big work into smaller chunks. Order Id you can not add just as filter as you get orders randomly. So partition could be on region order belongs to or even or uneven order or MODULO to get any split you want. MOD(orderId, 10) provides a split of 10 parallel partitions.

TomF

@Robert, approach 1 works like this:

• Events 1 mio per hour
• Distinct Object: 100000
• Order is needed

Topic hierarchy is salesOrder/{salesOrderID}

Exclusive queues, with subscriptions:

Queue0 -> salesOrder/0*

Queue1 -> salesOrder/1*

...

Queue9 -> salesOrder/9*

That gives you your 10 parrallel consumers with ordering strictly maintained.

Robert

@TomF

Correct. But when not using modulo to get a dedicated: orderPartitionKey (which can be between 0 and 9) i would go with:

Queue0 -> salesOrder/*0

Queue1 -> salesOrder/*1

..

Queue1 -> salesOrder/*9

As the last digit would change more often then first.

So i can use wildcards event in parts of the elements on path. Correct ?

(i was not aware of that)

TomF

@Robert, ah ha! I didn't realise it was wildcards that you were missing.

Yes, you can add wildcards to any subscription. There are two: "*" for "match at this level" and ">" for "match all levels below this." You can combine them:

sales*/01234*/> will match salesOrder/012345/something/else

Note, though, that you can't place any characters other than the hierarchy delimiter after "*". So

solace*/... will work, but *Order/... won't. In your example, you can do salesOrder/0* but unfortunately not salesOrder/*0

Have a look at the documentation for topic wildcards here.

Robert

@TomF then Modulu function and add partitionkey would be only reasonable option.

Otherwise if you reach a order range of 1000 you would have 1000 messages only be handled by same consumer until you reach 2000. So no distribution. :-)

I look forward for queue partitioning which i heard solace works on.

I hope you can do some distribution without any need that publisher adds a partition key.

Maybe you can just configure that and then assign consumers to each partition. (similar to Kafka)