Solace MQ Connector - Solace Connector is broken when Failover/Failback occurs in Solace HA

Hi all,

We are currently validating Solace MQ Connector in staging environment for future deployment in production environment. Solace MQ Connector instances (deployed on OCP4 platform in mode active/standby) are installed and configured With Solace HA environment. The Solace MQ connector is configured with 16 workflows whose 8 send the messages from 8 Solace Queues to a MQSeries Queues and 8 others publish from the sames MQSeries Queues to Solace topics. The sending of messages from both sides is in transactional mode.

In our test scenario we do a failover from Solace active to solace backup. This way, the active Solace Connector also fails over to Solace backup and messages continue to be sent in both directions.
The problem occurs when we fail back on the Solace Side. In this case, messages continue to be published from Solace to MQSeries but no message is received on Solace from MQseries.

When we look at the status of the active connector with the following query:
oc exec -it solacemq-connector-ffl-1 – wget -q -O - http://localhost:8090/actuator/health | jq

We see following messages:

{
“status”: “RECONNECTING”,
“components”: {
“binders”: {
“status”: “RECONNECTING”,
“components”: {
“jmsFFL”: {
“status”: “UP”,
“components”: {
“connection”: {
“status”: “UP”,
“details”: {
“provider”: “IBM MQ Jakarta Messaging Provider”
}
}
}
},
“solacePub”: {
“status”: “RECONNECTING”,
“components”: {
“connection”: {
“status”: “RECONNECTING”,
“details”: {
“error”: “com.solacesystems.jcsmp.JCSMPTransportException: Channel is closed by peer”
}
}
}
},
“solaceSub”: {
“status”: “UP”,
“components”: {
“bindings”: {
“status”: “UP”,
“components”: {
“input-0”: {
“status”: “UP”,
“components”: {
“flows”: {
“status”: “UP”,
“components”: {
“flow-0”: {
“status”: “UP”
}
}
}
}
},

The log file from Active Solace Connector indicates following message:
2025-03-14T14:48:03.278Z WARN 1 — [solace-scst-consumer-input-61] c.s.s.c.s.b.i.InboundXMLMessageListener : Exception thrown while processing batch. Batch’s messages will be requeued.
com.solace.spring.cloud.stream.binder.util.SolaceAcknowledgmentException: Failed to resolve transaction
at com.solace.spring.cloud.stream.binder.inbound.acknowledge.TransactedJCSMPAcknowledgementCallback.acknowledge(TransactedJCSMPAcknowledgementCallback.java:62)
at org.springframework.integration.acks.AckUtils.autoAck(AckUtils.java:43)
at com.solace.spring.cloud.stream.binder.inbound.BasicInboundXMLMessageListener.handleMessage(BasicInboundXMLMessageListener.java:77) at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.processBatchIfAvailable(InboundXMLMessageListener.java:201)
at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.receive(InboundXMLMessageListener.java:142) at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.run(InboundXMLMessageListener.java:93)
at com.solace.spring.cloud.stream.binder.inbound.BasicInboundXMLMessageListener.run(BasicInboundXMLMessageListener.java:24)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.solacesystems.jcsmp.JCSMPInterruptedException: Wait for session reconnect interrupted
at com.solacesystems.jcsmp.impl.JCSMPBasicSession.waitUntilSessionReconnectDone(JCSMPBasicSession.java:1510)
at com.solacesystems.jcsmp.impl.transaction.AdCtrlV4TransactedSessionImpl.commit(AdCtrlV4TransactedSessionImpl.java:875)
at com.solace.spring.cloud.stream.binder.inbound.acknowledge.TransactedJCSMPAcknowledgementCallback.acknowledge(TransactedJCSMPAcknowledgementCallback.java:40)
… 11 common frames omitted Caused by: java.lang.InterruptedException: null
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at com.solacesystems.jcsmp.impl.JCSMPBasicSession.waitUntilSessionReconnectDone(JCSMPBasicSession.java:1494)
… 13 common frames omitted

Anybody has already encountered this kind of problem ? Is there any idea about this issue ?


Please find below an extract from the application.yml file containing the configuration of the Solace MQ Connector:

spring:
cloud:
stream:
binders:
solaceSub:
type: solace
environment:
solace:
java:
host: tcp://solace-int.xxxxx:55555
msgVpn: PUBSUB
client-username: xxxx.s_cu
client-password: ‘xxxxxxxx’
connect-retries: -1
reconnect-retries: -1
connectTimeoutInMillis: 240000
reconnectRetryWaitInMillis: 15000
keepAliveLimit: 240
solacePub:
type: solace
environment:
solace:
java:
host: tcp://solace-int.xxxxx:55555
msg-vpn: EVENTMGT
client-username: xxxx.p_cu
client-password: ‘xxxxxxx’
connect-retries: -1
reconnect-retries: -1
connectTimeoutInMillis: 240000
reconnectRetryWaitInMillis: 15000
keepAliveLimit: 240
jmsFFL:
type: jms
environment:
ibm:
mq:
user: xxxxxx
password: xxxxxxxx
channel: XXXX.YYY_SVRCONN
queue-manager: XXXX
conn-name: mvsxxxx(1416)
additional-properties:
WMQ_CLIENT_RECONNECT_TIMEOUT_DEFAULT: 1000
connect-retries: -1
reconnect-retries: -1
undefined:
type: undefined
bindings:
input-0:
destination: SOLACE_MQ_0
binder: solaceSub
output-0:
destination: FL999KE05T0
binder: jmsFFL
input-1:
destination: SOLACE_MQ_1
binder: solaceSub
output-1:
destination: FL999KE05T1
binder: jmsFFL
input-2:
destination: SOLACE_MQ_2
binder: solaceSub
output-2:
destination: FL999KE05T2
binder: jmsFFL
input-3:
destination: SOLACE_MQ_3
binder: solaceSub
output-3:
destination: FL999KE05T3
binder: jmsFFL
input-4:
destination: SOLACE_MQ_4
binder: solaceSub
output-4:
destination: FL999KE05T4
binder: jmsFFL
input-5:
destination: SOLACE_MQ_5
binder: solaceSub
output-5:
destination: FL999KE05T5
binder: jmsFFL
input-6:
destination: SOLACE_MQ_6
binder: solaceSub
output-6:
destination: FL999KE05T6
binder: jmsFFL
input-7:
destination: SOLACE_MQ_7
binder: solaceSub
output-7:
destination: FL999KE05T7
binder: jmsFFL
input-8:
destination: FL999KE05T0
binder: jmsFFL
output-8:
destination: test0/t0
binder: solacePub
input-9:
destination: FL999KE05T1
binder: jmsFFL
output-9:
destination: test1/t1
binder: solacePub
input-10:
destination: FL999KE05T2
binder: jmsFFL
output-10:
destination: test2/t2
binder: solacePub
input-11:
destination: FL999KE05T3
binder: jmsFFL
output-11:
destination: test3/t3
destination: test3/t3
binder: solacePub
input-12:
destination: FL999KE05T4
binder: jmsFFL
output-12:
destination: test4/t4
binder: solacePub
input-13:
destination: FL999KE05T5
binder: jmsFFL
output-13:
destination: test5/t5
binder: solacePub
input-14:
destination: FL999KE05T6
binder: jmsFFL
output-14:
destination: test6/t6
binder: solacePub
input-15:
destination: FL999KE05T7
binder: jmsFFL
output-15:
destination: test7/t7
binder: solacePub
jms:
default:
consumer:
destination-type: queue
batch-max-size: 255
transacted: true
producer:
destination-type: queue
transacted: true

  binding-retry-interval: 30

  solace:
    default:
      consumer:
        destination-type: queue
        batch-max-size: 255
        transacted: true
      producer:
        destination-type: topic
        transacted: true

solace:
connector:
workflows:
0:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
1:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
2:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
3:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
4:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
5:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
6:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
7:
enabled: true
transform-headers:
expressions:
JMS_IBM_Format: “‘MQSTR’”
JMS_IBM_Character_Set: “‘273’”
8:
enabled: true
9:
enabled: true
10:
enabled: true
11:
enabled: true
12:
enabled: true
13:
enabled: true
14:
enabled: true
15:
enabled: true

management:
  leader-election:
    mode: active_standby
    fail-over:
      max-attempts: 3
      back-off-initial-interval: 1000
      back-off-max-interval: 10000
      back-off-multiplier: 2.0
  queue: Q-T-QMGR-SOLACEMQ-CONNECTOR-TEST
  session:
    host: tcp://solace-int.xxxxx:55555
    msg-vpn: PUBSUB
    client-username: xxxx.s_cu
    client-password: 'xxxxxxxx'

Hi, @Farzin ,
I managed to get this post out from the spam false positive prison it was in :wink:
I think you might be encountering a bug in the underlying JCSMP API that we use for the connectors.
I would suggest that you log a support ticket and the connector logs that show the reconnect, as Support will be able to coordinate a response with development.