SSL/TLS connection rejected by remote host

Velu
Velu Member Posts: 6
edited February 2022 in PubSub+ Event Broker #1

Below error is triggered and fills the log file event and system log files. Please assist what could be the issue.

2021-08-25T09:46:47.432+00:00 <local3.info> **** SOLACEINFO: SYSTEM: SYSTEM_SSL_CONNECTION_REJECTED: - - SSL Connection rejected: reason (Connection closed by remote host); connection to 127.0.0.1:8091 from 127.0.0.1:56656

Above client-username is #client-username mapped to default profiles.

replaced the VPN name to ****

One of my analysis:

127.0.0.1 with port 8091 is trying to connect as per the service configuration on the broker MQTT port from 8000 is disabled for SSL. Wil this be an issue ? and why all of sudden the alert (INFO) is triggered ?

MQTT WEB N - - Mgmt Test 8000 D D Disabled at Vpn

Comments

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 508 admin
    edited August 2021 #2

    Hello @Velu ! Welcome to the Community.

    Ok, so there's a couple things. First, you can't use the #client-username username. That is for internal use (along with anything else you see starting with "#"). Use the default client-username, or make a new one.

    From your CLI output, I'll copy in the column headers as it makes it easier to explain:

    Flags Legend:
    TP - Transport
    T+U - TCP and UDP
    S - SSL (Y=Yes, N=No, -=not-applicable)
    C - Compressed (Y=Yes, N=No, -=not-applicable)
    R - Routing Ctrl (Y=Yes, N=No, -=not-applicable)
    VRF - VRF (Mgmt=management, MsgBB=msg-backbone)
    A - Admin State (U=Up, D=Down, -=not-applicable)
    O - Oper State (U=Up, D=Down, -=not-applicable)
    
                                                  Status
    Service    TP  S C R VRF   MsgVpn          Port  A O Failed Reason
    ---------- --- ----- ----- --------------- ----- --- --------------------------
    MQTT       WEB N - - Mgmt  Test             8000 D D Disabled at Vpn
    
    • the "N" means it is a non-SSL port, so you can't connect using SSL
    • the "D D" means that the port is down as well, both configured and operational status
    • the reason is because it has been disabled at the VPN level
    • and I'm not sure why you're trying to connect to port 8091 ..?

    So, if you want an SSL connection, you'll probably want to connect to port 8883 (SSL) or 8443 (WSS)... but those are just for the default VPN. If you are connecting to another Message VPN, then you'll have to figure out what ports are configured. E.g.:

    show message-vpn <BLAH> service
    

    Finally, you'll need to ensure you see "U U" for admin and operational state. You'll need to ensure you have copied a server certificate onto the broker so that it can support SSL connections. https://docs.solace.com/Configuring-and-Managing/Managing-Server-Certs.htm

    Good luck!

  • Velu
    Velu Member Posts: 6

    @Aaron,
    Thank you for the welcome and the brief.

    Am not sure why it picks 8091 port? Is there any configuration explicit to get this system based service ?

    Is that by enabling MQTT for SSL address the issue ?

    Flags Legend:
    TP - Transport
    S - SSL (Y=Yes, N=No, -=not-applicable)
    C - Compressed (Y=Yes, N=No, -=not-applicable)
    VRF - VRF (Mgmt=management, MsgBB=msg-backbone)
    A - Admin State (U=Up, D=Down, -=not-applicable)
    O - Oper State (U=Up, D=Down, -=not-applicable)

    ---Press any key to continue, or `q' to quit---
    Service TP S C VRF MsgVpn Port A O Failed Reason


    SMF TCP N N MsgBB 55555 D D Disabled at vpn
    SMF TCP N Y MsgBB 55003 D D Disabled at vpn
    SMF TCP Y N MsgBB 55443 U U
    REST WEB N - MsgBB Test 9000 D D Disabled at Vpn
    REST WEB Y - MsgBB Test 9443 U U
    SMF WEB N - MsgBB 8008 D D Disabled at vpn
    SMF WEB Y - MsgBB 1443 U U
    MQTT TCP N - MsgBB Test 1883 D D Disabled at Vpn
    MQTT TCP Y - MsgBB Test 8883 U U
    MQTT WEB N - MsgBB Test 8000 D D Disabled at Vpn
    MQTT WEB Y - MsgBB Test 8443 U U
    AMQP TCP Y - MsgBB U D Not Permitted
    AMQP TCP N - MsgBB Test 5672 D D Disabled at Vpn
    AMQP TCP Y - MsgBB Test 5671 U U

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 508 admin

    Hi again @Velu . Ok, from your whole output, I see a lot of "U U", which means "admin up" and "operational up". So I guess you have a server certificate successfully installed!

    From this line:

    Service    TP  S C VRF   MsgVpn          Port  A O Failed Reason
    ---------- --- --- ----- --------------- ----- --- --------------------------
    MQTT       WEB Y - MsgBB Test             8443 U U
    

    So if you're looking to use WebSockets to connect (e.g. from JavaScript or something), then you'll want port 8443, and that will connect you to your "Test" Message VPN.

    And I can tell that you're on a hardware appliance, rather than a software broker, so make sure you are using the message bus IP address, not the management one.

    Finally, just an FYI for formatting: you can post blocks of code/text inside three backticks ```. Makes it a lot easier to read CLI output! 👍

  • Velu
    Velu Member Posts: 6
    edited August 2021 #5

    Hi @Aaron,

    The broker is running on Kubernetes. We have disabled SMF ports. Clients are using only REST API's and SSL cert auth is mandatory.

    How does 8091 is picked by the internal process.
    Any config changes ? like any web ports to be enabled.

    ( I would like to modify the port (8091) to the REST SSL port (9443). is it possible ?).

    Please suggest.

    (Surely will adopt the usage/formatting :smile: ).

  • Abhikesh
    Abhikesh Member Posts: 34 ✭✭

    Hi @Velu Yes you can change port number, For REST you can use below to specify port.

    solace(configure)# message-vpn
    solace(configure/message-vpn)# service rest
    solace(configure/message-vpn/service/rest)# incoming
    solace(...ure/message-vpn/service/rest/incoming)# listen-port [ssl]

  • Velu
    Velu Member Posts: 6

    Thank you for the cli @Abhishek
    @Aaron

    Trying to understand, that the port 8091 was not configured for the service on the broker, but the #client-username picks it ?

    client details : #client-username

    Client:                       #client
    Type:                         Internal
    Client Profile:               #client-profile
    ACL Profile:                  #acl-profile
    Authorization Group:          
    Originating IP Address:       127.0.0.1
    Subscriptions:                6
    No Local Delivery:            No
    Eliding Enabled:              No
    Eliding Topics:                 
      Current:                    0
      High Water Mark:            0
    Deliver To One:                 
      Local Priority:             1
      Network Priority:           1
    Client Id:                    4
    Message VPN:                  Test
    Uptime:                       41d 13h 6m 24s
    Scheduled Disconnect Time:    N/A
    Slow Subscriber:              No
    Client Username:              #client-username
    Original Client Username:     #client-username
    User:                         SolOS
    Description:                  Internal Message Bus
    Software Version:             7.14.0.9
    Software Date:                Jul 20 2020 09:45:34
    Platform:                     Linux26-x86_64_opt - C SDK
    ---Press any key to continue, or `q' to quit---
    One Shot Events:                
      Large Message:              not raised
      Message Too Big:            not raised
      Parse Error:                not raised
      Max Eliding Topics:         not raised
    Total Ingress Flows:          0
    Total Egress Flows:           0
    Web Transport Session:        N/A
    SSL Version:                  N/A
    SSL Cipher:                   N/A
    SSL Downgraded to Plain Text: No
    
  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 508 admin
    edited August 2021 #8

    Hi again @Velu ... ok, so I think the confusion here is that you're looking at an "internal use only" client that the broker maintains whenever a Message VPN is enabled. #client is not your client connection. It is for the broker's own internal communication. That's why you're seeing a weird port. Note the uptime is 41 days! 😅

    As mentioned in my first response, ignore anything that starts with # as they are all for internal stuff. Check this post: https://solace.community/discussion/356/what-are-these-objects-that-start-with

    So, what you need to do to get connected to your broker:

    • in your Message VPN, create a new client-username to use. Configure however you want for authentication. Maybe "basic none" is the best way to start.
    • If you've made a new client-profile for your client-username, and you want to use Guaranteed messaging, please make sure you enable a bunch of message spool settings in that profile
    • If you've made a new acl-profile for your client-username, you'll need to change the default behaviour of everything to "allow". It is set to disallow by default.

      • Easier to just use the default acl-profile
    • Finally, as a test, make sure you can connect to the broker with just a regular Solace (SMF) client to make sure your username/password/whatever is working. Either SdkPerf, or the PubSub+ Manager "TryMe!" test applications.

    Hope that helps!

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 508 admin

    Hey, also maybe just double-check / verify that Kubernetes isn't doing something weird with port 8091? I don't know where that's coming from.

  • Velu
    Velu Member Posts: 6

    Thanks @Aaron
    Will check in details on the setup and revert if any clarifications required.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 508 admin

    @Velu Any update?

    (more characters to make post long enough)

  • Velu
    Velu Member Posts: 6

    @Aaron,
    Trying to deploy a test broker on Kube, to find out the configurations that are related. I will revert with the findings/analysis from the new deployment. Also with the current environment, couldn't prolong the manipulations since clients are active. mystery as of now :)

  • arih
    arih Member, Employee Posts: 125 Solace Employee

    hi all,

    I remember seeing this log entry repeatedly in a customer environment (GKE Anthos) last year, and it was something about load balancer heartbeat. Though it was not port 8091 that time.

    For K8s based deployment, the port exposed to outside world could be different from the ports shown in the service, since the LB will define those external port and map it back to the Pod's ports.

    For my case last time, it was found out to be something with the GKE environment and the support from GKE helped the client out, I didn't get what was the root cause etc.