How does the COS setting of Message and the priority of Message work in Direct Message?

sunnybaek

HI. All

COS and Message Priority are very confusing to me.

How does the COS setting of Message and the priority of Message work in Direct Message?

For example, if I set the COS of Message high(COS-3, D-3) and the Message Priority low(Message Priority 1 ), how would it work? Is Direct Message not applicable to Message Priority?

I tried to perform a test, but the message is resolved immediately or delayed too much because it is Direct Message, making it difficult to test.

And what is the purpose of the Control Priority of the Client Profile settings?

Does anyone know?

Aaron

Hi @sunnybaek, welcome to the Community!

RIght ok, so I can help out here. But I would hope that the Solace docs would've been able to answer some of these questions? Any feedback there regarding what you checked or confusion or anything that we can maybe improve our docs with? Please let us know, constructive criticism..! 🙏🏼

These two terms / settings / features are very different and not related… and I'll walk through everything I know about them:

So first off, Priority is only for Guaranteed / persistent messages. I mean, you can set the priority if you want on a Direct message, but it won't do anything. I think the broker or APIs will assign a default priority of 4 for any message that doesn't have one set. The only use for priority (that I know?) is it if you enable Respect Message Priority on a queue, then higher priority messages will jump ahead of lower priority messages. See here and here. And here.

Message-priority-respecting queues is fairly a recent feature, like within the last 2 years? I don't remember the exact Solace release. But for the longest time it was in our APIs (mostly due to JMS compatibility), but didn't do anything at all in the broker.

Now, conversely, Class of Service (CoS) is a very old and not-very-used feature anymore. It has two very different uses, depending on whether you are publishing Direct or Guaranteed messages. It's been around since before I joined Solace (in 2011) so there is some "legacy" terminology coming up!

When publishing messages as Direct, the CoS (1, 2, 3, default=1) will determine which "egress queue buffer" the message will go into for any consumer/subscriber that would receive the message. When a client application connects to a Solace broker, the broker will keep a chunk of RAM-per-client to store outbound messages. There are 5 of these buffers/queues per client, called G-1, D-1, D-2, D-3, C-1. See here. These buffers multiplex via a weighted round-robin into a single queue that then gets written out to the TCP socket eventually. G-1 holds Guaranteed messages, and has the lowest priority since it is backed by a persistent store. C-1 holds control messages (like keepalives, subscription ACKs, etc.), used by the broker for API communication. The D-1, D-2, D-3 queues correspond to the Direct messaging CoS.

As an example, let's say 99% of your data is published as CoS1. But sometimes you have a very "high value" message that you want to make sure gets through to the subscriber no matter what. Like a chat message, or a super alert. Even if you subscriber is slow and their CoS1 queue (D-1) is very backed up, by publishing a message as CoS2 it will get put into the subscriber's D-2 queue/buffer (which is most likely empty) and will cause it to jump ahead of all the older messages backed up in the D-1 queue.

Since most of newer Solace use cases use persistent/Guaranteed messaging, and/or the message rates are lower and networks are faster and Direct performance is not as concerning, unless you are building a Financial Services trading platform where microseconds count and Direct messaging rates are in the 10s or 100s of thousands of messages per-second, then CoS for Direct messages is probably not super important. And since it means that order of messages changes (i.e. CoS2 messages jump ahead of CoS1, or 2 ahead of 3) then only use it for unrelated messages (e.g. chats, alerts, shutdowns) vs. your usual data stream.

OK! Whew. Now, when publishing messages as Guaranteed / persistent, CoS has a completely different meaning. There was a feature Solace came up with about 8 (?) years ago where you could configure a queue to reject any new messages if it was somewhat full. Unfortunately, this feature was named "low priority discard" on queues. But it has nothing to do with the Priority integer value you set on a message! It works like: configure a queue to automatically reject new messages if it is over 10% full; unless the Guaranteed message was published as CoS3… then it would allow it. The typical use case cited for this is: the queue is sitting in front of an order entry gateway, and tracks orders by OrderIDs. It accepts "new" orders, "amend"s to orders, and "cancel" orders. I want to configure my queue such that if it starts to fill up / fall behind processing, then it will reject any "new" orders (sent as "low priority" QoS1) (and force the publisher to send to a different queue), but allow "amend"s and "cancel"s (sent as "high priority" QoS3) to enter the queue because the order entry gateway would have the current state for that ID.

It's a cool feature, but very niche. So yeah, in summary: you probably don't need to worry about CoS unless you're building Direct messaging apps and want to ensure that a particular message (sent with CoS2 or CoS3) will always get through. (but message order will be affected since it jumps ahead). Otherwise, if doing Guaranteed messaging, Priority is probably what you'd want to use.

Also, of course, you can "roll your own" priority mechanism… e.g. have one queue for high, one queue for low, and they each subscribe to different topics, and then your consumer can determine the schedule for how to service these queues.

Hope that helps! Let me know if you have any further questions.

Aaron

Hi @sunnybaek, welcome to the Community!

RIght ok, so I can help out here. But I would hope that the Solace docs would've been able to answer some of these questions? Any feedback there regarding what you checked or confusion or anything that we can maybe improve our docs with? Please let us know, constructive criticism..! 🙏🏼

These two terms / settings / features are very different and not related… and I'll walk through everything I know about them:

So first off, Priority is only for Guaranteed / persistent messages. I mean, you can set the priority if you want on a Direct message, but it won't do anything. I think the broker or APIs will assign a default priority of 4 for any message that doesn't have one set. The only use for priority (that I know?) is it if you enable Respect Message Priority on a queue, then higher priority messages will jump ahead of lower priority messages. See here and here. And here.

Message-priority-respecting queues is fairly a recent feature, like within the last 2 years? I don't remember the exact Solace release. But for the longest time it was in our APIs (mostly due to JMS compatibility), but didn't do anything at all in the broker.

Now, conversely, Class of Service (CoS) is a very old and not-very-used feature anymore. It has two very different uses, depending on whether you are publishing Direct or Guaranteed messages. It's been around since before I joined Solace (in 2011) so there is some "legacy" terminology coming up!

When publishing messages as Direct, the CoS (1, 2, 3, default=1) will determine which "egress queue buffer" the message will go into for any consumer/subscriber that would receive the message. When a client application connects to a Solace broker, the broker will keep a chunk of RAM-per-client to store outbound messages. There are 5 of these buffers/queues per client, called G-1, D-1, D-2, D-3, C-1. See here. These buffers multiplex via a weighted round-robin into a single queue that then gets written out to the TCP socket eventually. G-1 holds Guaranteed messages, and has the lowest priority since it is backed by a persistent store. C-1 holds control messages (like keepalives, subscription ACKs, etc.), used by the broker for API communication. The D-1, D-2, D-3 queues correspond to the Direct messaging CoS.

As an example, let's say 99% of your data is published as CoS1. But sometimes you have a very "high value" message that you want to make sure gets through to the subscriber no matter what. Like a chat message, or a super alert. Even if you subscriber is slow and their CoS1 queue (D-1) is very backed up, by publishing a message as CoS2 it will get put into the subscriber's D-2 queue/buffer (which is most likely empty) and will cause it to jump ahead of all the older messages backed up in the D-1 queue.

Since most of newer Solace use cases use persistent/Guaranteed messaging, and/or the message rates are lower and networks are faster and Direct performance is not as concerning, unless you are building a Financial Services trading platform where microseconds count and Direct messaging rates are in the 10s or 100s of thousands of messages per-second, then CoS for Direct messages is probably not super important. And since it means that order of messages changes (i.e. CoS2 messages jump ahead of CoS1, or 2 ahead of 3) then only use it for unrelated messages (e.g. chats, alerts, shutdowns) vs. your usual data stream.

OK! Whew. Now, when publishing messages as Guaranteed / persistent, CoS has a completely different meaning. There was a feature Solace came up with about 8 (?) years ago where you could configure a queue to reject any new messages if it was somewhat full. Unfortunately, this feature was named "low priority discard" on queues. But it has nothing to do with the Priority integer value you set on a message! It works like: configure a queue to automatically reject new messages if it is over 10% full; unless the Guaranteed message was published as CoS3… then it would allow it. The typical use case cited for this is: the queue is sitting in front of an order entry gateway, and tracks orders by OrderIDs. It accepts "new" orders, "amend"s to orders, and "cancel" orders. I want to configure my queue such that if it starts to fill up / fall behind processing, then it will reject any "new" orders (sent as "low priority" QoS1) (and force the publisher to send to a different queue), but allow "amend"s and "cancel"s (sent as "high priority" QoS3) to enter the queue because the order entry gateway would have the current state for that ID.

It's a cool feature, but very niche. So yeah, in summary: you probably don't need to worry about CoS unless you're building Direct messaging apps and want to ensure that a particular message (sent with CoS2 or CoS3) will always get through. (but message order will be affected since it jumps ahead). Otherwise, if doing Guaranteed messaging, Priority is probably what you'd want to use.

Also, of course, you can "roll your own" priority mechanism… e.g. have one queue for high, one queue for low, and they each subscribe to different topics, and then your consumer can determine the schedule for how to service these queues.

Hope that helps! Let me know if you have any further questions.

sunnybaek

Hi Aaron

I was very curious about that. Thank you for your excellent response.

And I was testing the Rejection of Low-Priority Messages feature in Guaranteed messaging.

Even when the Reject Low Priority Messages Limit value was set to 100 and the Message Priority was set to 9 (very high), the queue remained at 100.

When I used COS as you suggested, I confirmed that COS-3 was loaded into the queue and COS-1 was discarded.

Does the Priority of "Rejection of Low-Priority Messages" refer to COS and not Message Priority?

If this is correct, it would be better to clearly write COS in the document.

Thank you.

Aaron

Yup, exactly. The "priority" in "low priority discard" has nothing to do with message priority but instead controlled by CoS. It's legacy reasons for the naming weirdness. Changing it would be hard (SEMP schemas, CLI command tree, etc…). I'll raise it again internally and see if they'd be willing to change things.

Glad I could help..!