Solace JMS Channel closing on it's own when running on a Kubernetes cluster (Spring Boot Autoconfig)

pooja
pooja Member Posts: 3

Log snippet:
{"com.solacesystems.jcsmp.protocol.impl.TcpClientChannel","label":
"Client-109: Connected to host 'orig=host:55555, scheme=tcp://, host=host, port=55555' (smfclient 109)"}

{"com.solacesystems.jcsmp.protocol.impl.TcpClientChannel","label":"Client-109: Channel Closed
(smfclient 109)"}

It basically loops like this all day long and I can't figure out why. When I run this locally on my development machine the connection remains open and and not closing on its own****.

There is no other log to give me a clue why the channel is closing on its own like this.

Any one have any ideas what the problem might be?

PS: I am already using CachingConnectionfactory

Tagged:

Comments

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 568 admin

    Hi @pooja,

    You tried my first guess - not using the CachingConnectionFactory is usually the culprit here. When you run your app do the logs say that the CachingConnectionFactory is being instantiated?

    A few questions:
    1. Are you using core spring framework or Spring boot?
    2. Is this app a consumer, sender, or both?
    3. is it disconnecting after each send/receive?
    4. can you share more of your Spring configuration?

  • pooja
    pooja Member Posts: 3

    Hey marc, thanks for the quick response :smile: . Answering your questions inline. Let me know anything else required from my side.

    @marc said:
    Hi @pooja,

    You tried my first guess - not using the CachingConnectionFactory is usually the culprit here. When you run your app do the logs say that the CachingConnectionFactory is being instantiated?

    [Pooja] No it doesnt say so

    A few questions:
    1. Are you using core spring framework or Spring boot?

    [Pooja]Spring boot.

    1. Is this app a consumer, sender, or both?

    [Pooja]It is both, but even with sender alone it loops through creating closing connection

    1. is it disconnecting after each send/receive?

    [Pooja]Not actually. Connection creation and closing are happening in every fixed interval of time. Seems like some timer is set for connection .

    1. can you share more of your Spring configuration?

    [Pooja] Connection is created using spring autoconfig(spring managed connection) through JmsTemplate.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 568 admin
    edited July 8 #4

    Interesting. I just re-read your question and noticed this which makes me think it is a configuration on the broker or network settings in K8s

    It basically loops like this all day long and I can't figure out why. When I run this locally on my development machine the connection remains open and and not closing on its own****.

    @Aaron @TomF - any ideas here?

    Just a guess from me, but can you check the client-profile and the JMS JNDI Connection Factory settings for keepalives and see if there is anything disabled or drastically different than what you have locally?

    As defaults I see this in my local docker but your admin may change some of these things based on what they expect to see or network restrictions in your k8s cluster.

    On the client-profile

    On the JMS Connection Factory

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 568 admin

    @pooja Another thing to consider would be the network path that your app is taking to connect to the broker. Is the broker also within K8s? is it external to K8s? is there a load balancer in the mix?

  • pooja
    pooja Member Posts: 3

    @marc These settings are same as yours, in local as well as in the server broker. Broker is (outside) external to the k8s cluster.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 568 admin

    @pooja...interesting. Is there a load balancer between K8s and your broker? I wonder if there is some sort of load balancer flapping going on.

    @TomF @KenBarr any other ideas?

  • KenBarr
    KenBarr Member, Employee Posts: 13 Solace Employee

    might be worth looking at the connect/disconnect events on the event broker. "show log events" and match these to your JMS logs. The broker logs have a disconnect reason.

  • bhavik20
    bhavik20 Member Posts: 1

    Hi Team,

    I'm also facing same issue. solace connections are keep dropping every 10s and reconnect again it own while running spring boot application via docker image. i don't see any issue running spring boot jar independently.