StaleSessionException spam when disconnected from host

alanfish

Situation:

Using spring cloud stream with a solace binder. Connection to host dropped.

Current behavior:

After all attempts to reconnect failed, the application starts producing tons of logs with StaleSessionException (which took over all disk space on the server over night).

Expected behavior:

Application shutdown.

Tried to do:

Manipulating reconnect properties did not yield any result. Disabled reconnect with the following:

solace:
  java:
    connect-retries-per-host: 0
    reconnect-retries: 0

The application did not make any reconnect attempts (which is expected) and went straight to exception spam. Right now the only option I see to avoid spam is setting those properties to -1 so that the application keeps trying to reconnect indefinitely. But in case of disconnect, I need the whole application to go down. Am I missing something?

Stack trace of the exception log:

2022-08-19 18:33:09,854 WARN  [-11-thread-1]  c.s.s.c.s.b.i.InboundXMLMessageListener: Received error while trying to read message from endpoint #P2P/QTMP/v:ldnpsol2005n01/scst/2a0a2991-0496-48b0-bffa-3431de5a2897
com.solacesystems.jcsmp.StaleSessionException: Tried to call receive on a stopped message consumer.
	at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.throwClosedException(FlowHandleImpl.java:1849)
	at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:793)
	at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:760)
	at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:293)
	at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:225)
	at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.receive(InboundXMLMessageListener.java:114)
	at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.run(InboundXMLMessageListener.java:91)
	at com.solace.spring.cloud.stream.binder.inbound.BasicInboundXMLMessageListener.run(BasicInboundXMLMessageListener.java:21)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.solacesystems.jcsmp.JCSMPTransportException: (JCSMPTransportException) Error receiving data from underlying connection.
	at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2299)
	... 4 common frames omitted
Caused by: com.solacesystems.jcsmp.JCSMPSecurityException: Error performing login to LoginContext (intranet.barcapint.com)
	at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.getKRBToken(SimpleSmfClient.java:1244)
	at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.open(SimpleSmfClient.java:839)
	at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.doPostNoResponse(SimpleSmfClient.java:305)
	at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.doPost(SimpleSmfClient.java:222)
	at com.solacesystems.jcsmp.protocol.impl.TcpChannel.executePostOnce(TcpChannel.java:206)
	at com.solacesystems.jcsmp.protocol.impl.ChannelOpStrategyClient.performOpen(ChannelOpStrategyClient.java:90)
	at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.performOpenSingle(TcpClientChannel.java:415)
	at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.access$800(TcpClientChannel.java:105)
	at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2135)
	... 4 common frames omitted
Caused by: javax.security.auth.login.LoginException: intranet.barcapint.com
	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:788)
	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:747)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:581)
	at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.getKRBToken(SimpleSmfClient.java:1242)
	... 12 common frames omitted
Caused by: java.net.UnknownHostException: intranet.barcapint.com
	at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at java.base/java.net.InetAddress.getByName(InetAddress.java:1252)
	at java.security.jgss/sun.security.krb5.internal.UDPClient.<init>(NetClient.java:186)
	at java.security.jgss/sun.security.krb5.internal.NetClient.getInstance(NetClient.java:44)
	at java.security.jgss/sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:406)
	at java.security.jgss/sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:369)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.security.jgss/sun.security.krb5.KdcComm.send(KdcComm.java:353)
	at java.security.jgss/sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:253)
	at java.security.jgss/sun.security.krb5.KdcComm.send(KdcComm.java:229)
	at java.security.jgss/sun.security.krb5.KdcComm.send(KdcComm.java:200)
	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:345)
	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:756)
	... 20 common frames omitted

Libraries I'm using

com.solacesystems:sol-jcsmp:10.11.0

com.solace.spring.cloud:spring-cloud-starter-stream-solace:3.3.1

com.solace.spring.boot:solace-java-spring-boot-starter:4.2.2

GreenRover

+1 i created an github issue for this problem:

https://github.com/SolaceProducts/solace-spring-cloud/issues/174