Max transacted sessions exceeded (503) error - PubSub+ Event Broker and Camel Solace component issue

JDudgeon

Our team is currently working on a proof of concept with Apache Camel to split a file and publish the events to a Solace queue (PubSub+ Event Broker). Unfortunately, we are experiencing a persistent issue from the consuming route where, after a few minutes of consuming messages using the Solace Camel component, we start to receive "Max transacted sessions exceeded (503)" errors. This error continues to get worse as processing continues.

Here is what we've done so far in an attempt to fix the error:

1. Ensured that the authenticated user is associated with a client profile that sets the maximum transacted session limit to 1000.

2. Set maxConcurrentConsumers to 200 and tried numerous values for concurrentConsumers. We've even gone as low as maxConcurrentConsumers = 20.

3. Set the Solace component query parameter, cacheLevelName, to CACHE_CONSUMER. We've also tried CACHE_CONNECTION and CACHE_NONE all of which resulted in similar behaviour. Our complete component URI is:

// Camel route for consuming events from Solace
from("solace:queue:{{solace.queue.name}}?transacted=true&concurrentConsumers=20&maxConcurrentConsumers=200&lazyCreateTransactionManager=false&cacheLevel=CACHE_CONSUMER")

One thing we've noticed is that, regardless of using CACHE_CONNECTION/CONSUMER, the smfclient ids continue to increase because, according to the logs, the connections to Solace are continually closed and re-opened. The consumer count, as viewed from the Solace admin dashboard, never exceed the maxConcurrentConsumers value. This seems to point to some sort of session/connection garbage collection as connections are open/closed so quickly; i.e. the session associated with the channel needs to be cleaned up by the PubSub+ Event Broker and lags behind. I don't know why this is the case.

Any suggestions on how to proceed?

Thank you!

Edit: we are using solace-spring-boot version 1.1.0 (which pulls in solace-jms-spring-boot-starter-4.1.0.jar / sol-jms-10.8.1.jar)

mmoreno

Hi @JDudgeon

So, Configured Max Transacted Sessions will show you the value you have set for that Client Profile.
On the other hand, Max Transacted Sessions will show you the actual limit for your Client Profile, showing that:

1 - Configured Max Transacted Sessions equals Max Transacted Sessions - Because the configured value is lower than the broker system's limit, like in @Abhikesh example where both were equal to 10

2 - Max Transacted Sessions is lower than Configured Max Transacted Sessions - Because the configured value is greater than what the system can support, like in the case you shared: 1,000 configured vs 100 supported by the broker.

You are probably using a 100 max-connection broker, hence the 100 Transacted Sessions system limit. If you need the maximum transacted session limit to reach 1,000, you should upgrade your broker to a 1,000 max-connection tier.

tkunnumpurath

@JDudgeon - putting the root cause/solution to this problem based on our discussion for anyone else who might run into this issue.

The root cause was that you were not using a CachingConnectionFactory which caused your connections to constantly tear down and spin back up on every message. This caused the broker to exceed the transaction limits as the connections to the broker were not being torn down fast enough.

Adding the following line helped resolve the problem:

CachingConnectionFactory ccf = new CachingConnectionFactory(solConnectionFactory);

Wrapping the solConnectionFactory with a CachingConnectionFactory maintained a cached connection that could be reused with all your consumers.

Abhikesh

Hi @JDudgeon Seems you are hitting max assigned limit at Client profile for Transacted Sessions Per Client.
Can you check what value has been set for max Transacted Sessions Per Client on Client profile ! I think default is 10.

primary-solace# show client-profile default message-vpn default detail
Profile Name : default
Message VPN : default

(G-Guaranteed, D-Direct, C-Control)
Queue Max Depths
G-1 : 20000 work units
D-1 : 20000 work units
D-2 : 20000 work units
D-3 : 20000 work units
C-1 : 20000 work units
Priority Queue Min Burst
G-1 : 255 messages
D-1 : 4 messages
D-2 : 4 messages
D-3 : 4 messages
C-1 : 4 messages
Message Spool
Guaranteed Message Send : allow
Guaranteed Message Receive : allow
Guaranteed Endpoint Create : allow
Guaranteed Endpoint Durability : all
Transacted Sessions : allow
Cut-Through Forwarding : deny
Max Transacted Sessions : 10
Configured Max Transacted Sessions : 10
Max Transactions : 500
Configured Max Transactions : 5000
Max Messages Per Transaction : 256

Abhikesh

Hi @JDudgeon
It seems you are still hitting max assigned transacted sessions limit on client profile, Let me do some research on it, will let you know if find anything meaningful.

JDudgeon

Hi @Abhikesh, thanks for the reply. Yes, we configured the client profiles that we use to allow more than 10 max transacted sessions:

Max Transacted Sessions             : 100
Configured Max Transacted Sessions  : 1000
Max Transactions                    : 500
Configured Max Transactions         : 5000
Max Messages Per Transaction        : 256
Max Endpoints per client-username   : 100
Configured Max Endpoints per c-u    : 1000

I'm curious though, what is the difference between Max Transacted Sessions and Configured Max Transacted Sessions?

mmoreno

Hi @JDudgeon

So, Configured Max Transacted Sessions will show you the value you have set for that Client Profile.
On the other hand, Max Transacted Sessions will show you the actual limit for your Client Profile, showing that:

1 - Configured Max Transacted Sessions equals Max Transacted Sessions - Because the configured value is lower than the broker system's limit, like in @Abhikesh example where both were equal to 10

2 - Max Transacted Sessions is lower than Configured Max Transacted Sessions - Because the configured value is greater than what the system can support, like in the case you shared: 1,000 configured vs 100 supported by the broker.

You are probably using a 100 max-connection broker, hence the 100 Transacted Sessions system limit. If you need the maximum transacted session limit to reach 1,000, you should upgrade your broker to a 1,000 max-connection tier.

JDudgeon

@mmoreno Thank you for the explanation. I was starting to wonder if that might be the case. I believe we have an Enterprise license for Solace PubSub+ Event Broker, but I'm not sure that is what was installed in our dev OpenShift environment; I'll need to confirm. Do you know off hand how many max transacted sessions the Enterprise license supports?

Now, with that said, I believe we are still encountering this error even when we set the Camel inbound route to a maximum of 50 concurrent consumers. I'll run some more tests today and report back.

tkunnumpurath

@JDudgeon - putting the root cause/solution to this problem based on our discussion for anyone else who might run into this issue.

The root cause was that you were not using a CachingConnectionFactory which caused your connections to constantly tear down and spin back up on every message. This caused the broker to exceed the transaction limits as the connections to the broker were not being torn down fast enough.

Adding the following line helped resolve the problem:

CachingConnectionFactory ccf = new CachingConnectionFactory(solConnectionFactory);

Wrapping the solConnectionFactory with a CachingConnectionFactory maintained a cached connection that could be reused with all your consumers.