StaleSessionException - Tried to call receive on a stopped message consumer
at java.base/java.lang.Thread.run(Thread.java:834), exception_class=com.solacesystems.jcsmp.StaleSessionException, exception_message=Tried to call receive on a stopped message consumer.}","threadID":"pool-6-thread-3","sourceHost":"cardholder-ia-verify-uat2-6-nql8r","logVersion":"1.5","category":"com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener"}
{"timestmp":"2022-10-14_07:26:46.822+0000","logLevel":"WARN","msg":"Received error while trying to read message from endpoint QUE_APP2_SEND","exception":"{stacktrace=at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.throwClosedException(FlowHandleImpl.java:1957)
at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:899)
at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:866)
at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:279)
at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:211)
at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.receive(InboundXMLMessageListener.java:93)
at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.run(InboundXMLMessageListener.java:73)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
I am using Spring Cloud Stream api with Solace. It was working all fine but when Solace fail over or if any small outage happens, app is trying to re-establish the solace connection. But since it is not happening, it gives tones of above log lines keep on printed in our log files and filling the server space. Need your immediate help to solve this issue. Below is my solace properties set in my yaml files.
type: solace environment: solace: java: clientUsername: username connectRetries: 3 connectRetriesPerHost: 0 reconnectRetries: 3 host: 'tcps://domain:port' msgVpn: VPN-NAME1
Comments
-
Can I have an update on this issue please ?
0 -
Hi @Kaliappans Will get back to you on this.
0 -
Hi @Kaliappans
The
reconnectRetriesbeing set to 3 is causing the issue. By default, the binder will attempt to reconnect until it connects successfully - which I believe is the behavior you want based on your message.Let us know if this change allows your application to reconnect when the fail-over or disconnect-reconnect occurs.
However, the infinite loop of exceptions with a
reconnectRetriesset is an issue. We will report to the dev team for suggestions and see if there is a bug.0 -
Hi @Kaliappans,
As @giri mentioned you'll want to tune your reconnect retries to retry for a longer period of time to successfully reconnect and process messages after a failover. -1 will retry forever ;)
That said, the logging issue you've seen is also not ideal and I believe it's captured to be worked here: https://github.com/SolaceProducts/solace-spring-cloud/issues/174
(Edit) and Giri opened another issue here to capture what you ran into: https://github.com/SolaceProducts/solace-spring-cloud/issues/179
0 -
Thanks @giri for your response. Our expectation is to retry only for 3 times. Even though we set
reconnectRetriesto 3, solace is behaving in a opposite way that is it is keep on trying until it is getting connected. But, actually, the connection is not established with Solace. Because of that, our logging infrastructure getting flooded with solace connection exception traces. So the questions would be,1. Why was solace not able to re-establish the connection
2. Why is Solace trying to reconnect even after the third time when the property "reconnectretries" set to 3.
0 -
Hi @Kaliappans - The only reason I can think of is the broker did not come up when the 3-retries were made. It might have come up later, but the application is stuck in a different thread - which is what I have escalated on the Git issue.
I hope we get a resolution on this soon - until that time, how about setting reconnectRetries set to -1? This will ensure that the application will reconnect whenever the broker becomes available and won't enter the logging loop. Will this work?
Feel free to chime in on the issue thread https://github.com/SolaceProducts/solace-spring-cloud/issues/179
0 -
Hi @Kaliappans,
It looks like you're using a HA pair and wait to ensure your app re-connects after a retry. Our recommendation for that is actually to retry for 5 minutes as it can sometimes take ~30 seconds to a minute for the failover to happen. This gives you some leeway. With your reconnect retries at 3 and the connect retries per host at 0 you are only trying to reconnect for 9 seconds. You can see the recommended minimum settings for that in the docs here: https://docs.solace.com/API/API-Developer-Guide/Configuring-Connection-T.htm. That said, these are actually the defaults in the Spring Cloud Stream binder so you shouldn't have to change them.
The logging part is a bug that we will address in the issue that Giri mentioned above (#179). After the retries expire the JCSMP session is no longer trying to actually reconnect, but the binder is still trying to continuously use the resources from that session that will never come back. At this point, like is mentioned in the github issue, the health of the binder goes to
DOWNand will not recover.Hope that clarifies things!
0 -
Hey @Kaliappans ! Following up on this and checking to see the above recommendation resolved your issue?
0 -
