Max transacted sessions exceeded (503) error - PubSub+ Event Broker and Camel Solace component issue

JDudgeon
JDudgeon Member Posts: 3
edited September 2021 in PubSub+ Event Broker #1

Our team is currently working on a proof of concept with Apache Camel to split a file and publish the events to a Solace queue (PubSub+ Event Broker). Unfortunately, we are experiencing a persistent issue from the consuming route where, after a few minutes of consuming messages using the Solace Camel component, we start to receive "Max transacted sessions exceeded (503)" errors. This error continues to get worse as processing continues.

Here is what we've done so far in an attempt to fix the error:

  1. Ensured that the authenticated user is associated with a client profile that sets the maximum transacted session limit to 1000.

  2. Set maxConcurrentConsumers to 200 and tried numerous values for concurrentConsumers. We've even gone as low as maxConcurrentConsumers = 20.

  3. Set the Solace component query parameter, cacheLevelName, to CACHE_CONSUMER. We've also tried CACHE_CONNECTION and CACHE_NONE all of which resulted in similar behaviour. Our complete component URI is:

// Camel route for consuming events from Solace
from("solace:queue:{{solace.queue.name}}?transacted=true&concurrentConsumers=20&maxConcurrentConsumers=200&lazyCreateTransactionManager=false&cacheLevel=CACHE_CONSUMER")

One thing we've noticed is that, regardless of using CACHE_CONNECTION/CONSUMER, the smfclient ids continue to increase because, according to the logs, the connections to Solace are continually closed and re-opened. The consumer count, as viewed from the Solace admin dashboard, never exceed the maxConcurrentConsumers value. This seems to point to some sort of session/connection garbage collection as connections are open/closed so quickly; i.e. the session associated with the channel needs to be cleaned up by the PubSub+ Event Broker and lags behind. I don't know why this is the case.

Any suggestions on how to proceed?

Thank you!

Edit: we are using solace-spring-boot version 1.1.0 (which pulls in solace-jms-spring-boot-starter-4.1.0.jar / sol-jms-10.8.1.jar)

Tagged:

Best Answers

  • mmoreno
    mmoreno Member, Employee Posts: 1 Solace Employee
    #2 Answer ✓

    Hi @JDudgeon

    So, Configured Max Transacted Sessions will show you the value you have set for that Client Profile.
    On the other hand, Max Transacted Sessions will show you the actual limit for your Client Profile, showing that:

    1 - Configured Max Transacted Sessions equals Max Transacted Sessions - Because the configured value is lower than the broker system's limit, like in @Abhikesh example where both were equal to 10

    2 - Max Transacted Sessions is lower than Configured Max Transacted Sessions - Because the configured value is greater than what the system can support, like in the case you shared: 1,000 configured vs 100 supported by the broker.

    You are probably using a 100 max-connection broker, hence the 100 Transacted Sessions system limit. If you need the maximum transacted session limit to reach 1,000, you should upgrade your broker to a 1,000 max-connection tier.

  • tkunnumpurath
    tkunnumpurath Member, Employee Posts: 11 Solace Employee
    #3 Answer ✓

    @JDudgeon - putting the root cause/solution to this problem based on our discussion for anyone else who might run into this issue.

    The root cause was that you were not using a CachingConnectionFactory which caused your connections to constantly tear down and spin back up on every message. This caused the broker to exceed the transaction limits as the connections to the broker were not being torn down fast enough.

    Adding the following line helped resolve the problem:

    CachingConnectionFactory ccf = new CachingConnectionFactory(solConnectionFactory);

    Wrapping the solConnectionFactory with a CachingConnectionFactory maintained a cached connection that could be reused with all your consumers.

Answers

  • Abhikesh
    Abhikesh Member Posts: 34 ✭✭

    Hi @JDudgeon Seems you are hitting max assigned limit at Client profile for Transacted Sessions Per Client.
    Can you check what value has been set for max Transacted Sessions Per Client on Client profile ! I think default is 10.

    primary-solace# show client-profile default message-vpn default detail
    Profile Name : default
    Message VPN : default

    (G-Guaranteed, D-Direct, C-Control)
    Queue Max Depths
    G-1 : 20000 work units
    D-1 : 20000 work units
    D-2 : 20000 work units
    D-3 : 20000 work units
    C-1 : 20000 work units
    Priority Queue Min Burst
    G-1 : 255 messages
    D-1 : 4 messages
    D-2 : 4 messages
    D-3 : 4 messages
    C-1 : 4 messages
    Message Spool
    Guaranteed Message Send : allow
    Guaranteed Message Receive : allow
    Guaranteed Endpoint Create : allow
    Guaranteed Endpoint Durability : all
    Transacted Sessions : allow
    Cut-Through Forwarding : deny
    Max Transacted Sessions : 10
    Configured Max Transacted Sessions : 10
    Max Transactions : 500
    Configured Max Transactions : 5000
    Max Messages Per Transaction : 256

  • Abhikesh
    Abhikesh Member Posts: 34 ✭✭
    edited September 2021 #5

    Hi @JDudgeon
    It seems you are still hitting max assigned transacted sessions limit on client profile, Let me do some research on it, will let you know if find anything meaningful.

  • JDudgeon
    JDudgeon Member Posts: 3
    edited October 2021 #6

    Hi @Abhikesh, thanks for the reply. Yes, we configured the client profiles that we use to allow more than 10 max transacted sessions:

    Max Transacted Sessions             : 100
    Configured Max Transacted Sessions  : 1000
    Max Transactions                    : 500
    Configured Max Transactions         : 5000
    Max Messages Per Transaction        : 256
    Max Endpoints per client-username   : 100
    Configured Max Endpoints per c-u    : 1000
    

    I'm curious though, what is the difference between Max Transacted Sessions and Configured Max Transacted Sessions?

  • mmoreno
    mmoreno Member, Employee Posts: 1 Solace Employee
    #7 Answer ✓

    Hi @JDudgeon

    So, Configured Max Transacted Sessions will show you the value you have set for that Client Profile.
    On the other hand, Max Transacted Sessions will show you the actual limit for your Client Profile, showing that:

    1 - Configured Max Transacted Sessions equals Max Transacted Sessions - Because the configured value is lower than the broker system's limit, like in @Abhikesh example where both were equal to 10

    2 - Max Transacted Sessions is lower than Configured Max Transacted Sessions - Because the configured value is greater than what the system can support, like in the case you shared: 1,000 configured vs 100 supported by the broker.

    You are probably using a 100 max-connection broker, hence the 100 Transacted Sessions system limit. If you need the maximum transacted session limit to reach 1,000, you should upgrade your broker to a 1,000 max-connection tier.

  • JDudgeon
    JDudgeon Member Posts: 3
    edited October 2021 #8

    @mmoreno Thank you for the explanation. I was starting to wonder if that might be the case. I believe we have an Enterprise license for Solace PubSub+ Event Broker, but I'm not sure that is what was installed in our dev OpenShift environment; I'll need to confirm. Do you know off hand how many max transacted sessions the Enterprise license supports?

    Now, with that said, I believe we are still encountering this error even when we set the Camel inbound route to a maximum of 50 concurrent consumers. I'll run some more tests today and report back.

  • tkunnumpurath
    tkunnumpurath Member, Employee Posts: 11 Solace Employee
    #9 Answer ✓

    @JDudgeon - putting the root cause/solution to this problem based on our discussion for anyone else who might run into this issue.

    The root cause was that you were not using a CachingConnectionFactory which caused your connections to constantly tear down and spin back up on every message. This caused the broker to exceed the transaction limits as the connections to the broker were not being torn down fast enough.

    Adding the following line helped resolve the problem:

    CachingConnectionFactory ccf = new CachingConnectionFactory(solConnectionFactory);

    Wrapping the solConnectionFactory with a CachingConnectionFactory maintained a cached connection that could be reused with all your consumers.