503 Spool Over Quota. Queue or Topic endpoint limit exceeded

barisbt

Hi Community,

I have q1 and q2 which subscribed to topic t/test/baris

When I publish a message to the topic t/test/baris
if q1 quota is exceeded (even if q2 can receive messages in terms of space), the message is rejected by the broker with the below message and none of the queues receive the message as a result.

I understand that this is by design but what is the reasoning for the broker to reject it as a whole?(all or nothing) or is there a way to allow the "other-no problem queues" to receive the messages without any problem?

Thanks

<solace-error-response>
<code>503</code>
<reason>
<![CDATA[Spool Over Quota. Queue or Topic endpoint limit exceeded]]>
</reason>
<detail>
<![CDATA[
Topic "t/test/baris";
SMF AD control client ack response error
]]>
</detail>
<internal-use>2:14136</internal-use>
</solace-error-response>

nicholasdgoodman

Yes - by default when any subscription-matching queue is full, the entire message is rejected back to the sender and the message is not enqueued at all.

This behavior can be changed on a per-queue basis, specifically via the Queue Settings > Advanced Settings > Congestion Control > Reject Messages to Sender on Discard dropdown, which has three options: When Queue Enabled (default), Always, and Never.

To allow the broker and other queues to still receive a message when a particular queue is full, you can change this setting to Never.

barisbt

It looks like my options are different for congestion control. (attached file)

It worked with "silent" option selected. Thank you for the help / answer Nicholas.

BTW I am using using Event Broker Service Version10.8.1.140-5) it is in fact SAP Advanced Event Mesh.

Queue-Cong.jpg

nicholasdgoodman

Yes - by default when any subscription-matching queue is full, the entire message is rejected back to the sender and the message is not enqueued at all.

This behavior can be changed on a per-queue basis, specifically via the Queue Settings > Advanced Settings > Congestion Control > Reject Messages to Sender on Discard dropdown, which has three options: When Queue Enabled (default), Always, and Never.

To allow the broker and other queues to still receive a message when a particular queue is full, you can change this setting to Never.

barisbt

It looks like my options are different for congestion control. (attached file)

It worked with "silent" option selected. Thank you for the help / answer Nicholas.

BTW I am using using Event Broker Service Version10.8.1.140-5) it is in fact SAP Advanced Event Mesh.

Queue-Cong.jpg

Aaron

Interesting… I didn't know that even some of the terminology in Manager is remapped when deployed as SAP AEM. EDIT: it looks like the term "Silent" was added to more recent Solace brokers as well… I guess Nick and I are a bit out of date!

Yeah, "Silent" will make your q1 queue into essentially a "best effort" queue, and that it will be fully Guaranteed with ACKs and everything until it is full, and then it'll just lose new messages while its spool is over quota.

barisbt

Thanks for the explanation Aaron. Yes it should be broker version that makes this difference.

Baris

Gopal4506

I have something similar options as @barisbt. But the silent option is not working for me. Am still getting the spool over quota error as some other queue is consuming to the same subscription.

Any help would be grateful.

Thanks
Gopal

Aaron

Hi @Gopal4506, welcome to the Community..! Yeah, if some other queue is full, and not configured to discard silently, then the publisher will get a NACk.

Just to be clear to all reading this thread: 99.99% of the time, you do NOT want the broker throwing away your Guaranteed messages. That's essentially what can happen if you configure your queue to be Silent for the "Reject Messages to Sender on Discard"… even though my queue is full, doesn't matter, don't tell the publisher.

Why is your queue full?? This should only happen if your consumers are crashed out for a LONG time.

If this is development, why not configure the queue's Max TTL to automatically clean it out after a day or a week or whatever?

barisbt

In addition to what @Aaron mentioned, @Gopal4506 What was the expected behaviour from your side when you select "silent"? Is it possible that you have other queues that is also not "silent" with the same subscription, as the broker works as expected for me.

Baris

Gopal4506

Sorry to get back here after many days. and many thanks for your replies
@Aaron I have configured only my queue to "silent", so as @barisbt mentioned all the queues should be set to Silent

So now I get it that- all the queues which has matching subscription must be set to "silent", so that the publisher will get a successfull ACK.
My follow query would be: Instead of setting all the queues to "Silent", Only the queue which got consumed fully set to "Silent"- Will this work?

also @Aaron, TTL works in my case, but I just wanted to test this setting of "Reject Messages to Sender on Discard".

Aaron

Only the queue that is full needs to be set to Silent to have your publisher receive an ACK. But again, this essentially means your queue is LOSING DATA. Publisher receives ACK even though queue is full and discards message.

What is your use case?

Gopal4506

Yes, Only the queue that is full needs to be set to Silent. So that the other queue can consume the message.
This is actually a production issue, where one queue was full with no consumers and we don't have the poc who was using that queue( and so we cannot empty the queue & dont care if this queue is losing data). We just put this queue to "Silent" and the new incoming msgs were acknowledged successfully.

Thanks @Aaron :)

Aaron

Hey @Gopal4506, sounds good. Another option would be to shutdown the ingress of that queue. Shutting down the ingress side should block any new messages coming into it, but the egress side of the queue is still enabled which means the app could still connect to it and drain its messages.