Our microservices were not able to connect to clients respective Solace queues. Although it seems microservices were able to connect to Solace, the queues themselves were not available, causing a huge volume of time outs.
At the moment the configuration has a set number of retries for the connection to Solace, but no provision for what should occur when a queue is unavailable. The result in this instance was a near endless loop of timeouts that caused the log files to grow to 20+ terabytes.
Although the queue unavailability is a client side issue, we need to have a contingency in place to prevent this from happening in future.
Please could you advise how can we prevent this indefinite attempts to connect to the queue that is not available.
The example of the error is bellow:
2024-12-12 WARN com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener [pool-9-thread-4] Received error while trying to read message from endpoint HOSTNAME.LOCATES.SECURITY.PRICE
com.solacesystems.jcsmp.StaleSessionException: Tried to call receive on a stopped message consumer.
at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.throwClosedException(FlowHandleImpl.java:2042)
at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:920)
at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:887)
at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:326)
at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:258)
at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.receive(InboundXMLMessageListener.java:123)
at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.run(InboundXMLMessageListener.java:96)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.solacesystems.jcsmp.JCSMPTransportException: (JCSMPTransportException) Error communicating with the router. (KeepAlive)
at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2493)
... 4 common frames omitted
Caused by: com.solacesystems.jcsmp.JCSMPTransportException: (Client name: client-lon-name/nnn/nnn/dddd Local port: -1 Remote addr: n.n.n.n Remote port: nnn) - Timeout happened when reading response from the router.
at com.solacesystems.jcsmp.protocol.impl.TcpChannel.executePostOnce(TcpChannel.java:243)
at com.solacesystems.jcsmp.protocol.impl.ChannelOpStrategyClient.performOpen(ChannelOpStrategyClient.java:99)
at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.performOpenSingle(TcpClientChannel.java:426)
at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.access$800(TcpClientChannel.java:122)
at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2329)
... 4 common frames omitted
Caused by: java.net.SocketTimeoutException: null
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:129)
at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.open(SimpleSmfClient.java:905)
at com.solacesystems.jcsmp.protocol.smf.ZSmfClient.open(ZSmfClient.java:62)
at com.solacesystems.jcsmp.protocol.smf.SSLSmfClient.open(SSLSmfClient.java:90)
at com.solacesystems.jcsmp.protocol.smf.SSLSmfClient.doPostNoResponse(SSLSmfClient.java:241)
at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.doPost(SimpleSmfClient.java:235)
at com.solacesystems.jcsmp.protocol.impl.TcpChannel.executePostOnce(TcpChannel.java:199)
... 8 common frames omitted
My config is:
webService:
url: http://localhost:nnnn/
user: userName
password: userPassword
maxRetries: 6
retryIntervalSecs: 120
readTimeoutSecs: 120
connectTimeoutSecs: 120
poolSize: 10
server:
port: nnn
spring:
cloud:
function:
definition: myFunction
stream:
binders:
local-solace:
type: solace
environment:
solace:
java:
host: tcp://n.n.n.n:nnn
msgVpn: default
clientUsername: default
clientPassword: default
connectRetries: 3
connectRetriesPerHost: 0
reconnectRetries: 3
bindings:
#The bindings section is used to define your input and output channels.
myFunction-in-0:
destination: client.hostName.securities
#The presence of "group" tells the binder to follow the "consumer group" pattern; if not present the "publish-subscribe" pattern will be used.
group: hostName
consumer:
concurrency: 5
solace:
bindings:
myFunction-in-0:
consumer:
#Avoid the scst prefix etc
queueNameExpression: destination
autoBindErrorQueue: false
#Queue should not be created by the application
provisionDurableQueue: false
provisionErrorQueue: false
addDestinationAsSubscriptionToQueue: false
