SSL/TLS connection rejected by remote host

Velu

Below error is triggered and fills the log file event and system log files. Please assist what could be the issue.

2021-08-25T09:46:47.432+00:00 <local3.info> **** SOLACEINFO: SYSTEM: SYSTEM_SSL_CONNECTION_REJECTED: - - SSL Connection rejected: reason (Connection closed by remote host); connection to 127.0.0.1:8091 from 127.0.0.1:56656

Above client-username is #client-username mapped to default profiles.

replaced the VPN name to ****

One of my analysis:

127.0.0.1 with port 8091 is trying to connect as per the service configuration on the broker MQTT port from 8000 is disabled for SSL. Wil this be an issue ? and why all of sudden the alert (INFO) is triggered ?

MQTT WEB N - - Mgmt Test 8000 D D Disabled at Vpn

Aaron

Hello @Velu ! Welcome to the Community.

Ok, so there's a couple things. First, you can't use the #client-username username. That is for internal use (along with anything else you see starting with "#"). Use the default client-username, or make a new one.

From your CLI output, I'll copy in the column headers as it makes it easier to explain:

Flags Legend:
TP - Transport
T+U - TCP and UDP
S - SSL (Y=Yes, N=No, -=not-applicable)
C - Compressed (Y=Yes, N=No, -=not-applicable)
R - Routing Ctrl (Y=Yes, N=No, -=not-applicable)
VRF - VRF (Mgmt=management, MsgBB=msg-backbone)
A - Admin State (U=Up, D=Down, -=not-applicable)
O - Oper State (U=Up, D=Down, -=not-applicable)

                                              Status
Service    TP  S C R VRF   MsgVpn          Port  A O Failed Reason
---------- --- ----- ----- --------------- ----- --- --------------------------
MQTT       WEB N - - Mgmt  Test             8000 D D Disabled at Vpn

• the "N" means it is a non-SSL port, so you can't connect using SSL
• the "D D" means that the port is down as well, both configured and operational status
• the reason is because it has been disabled at the VPN level
• and I'm not sure why you're trying to connect to port 8091 ..?

So, if you want an SSL connection, you'll probably want to connect to port 8883 (SSL) or 8443 (WSS)... but those are just for the default VPN. If you are connecting to another Message VPN, then you'll have to figure out what ports are configured. E.g.:

show message-vpn <BLAH> service

Finally, you'll need to ensure you see "U U" for admin and operational state. You'll need to ensure you have copied a server certificate onto the broker so that it can support SSL connections. https://docs.solace.com/Configuring-and-Managing/Managing-Server-Certs.htm

Good luck!

Velu

@Aaron,
Thank you for the welcome and the brief.

Am not sure why it picks 8091 port? Is there any configuration explicit to get this system based service ?

Is that by enabling MQTT for SSL address the issue ?

Flags Legend:
TP - Transport
S - SSL (Y=Yes, N=No, -=not-applicable)
C - Compressed (Y=Yes, N=No, -=not-applicable)
VRF - VRF (Mgmt=management, MsgBB=msg-backbone)
A - Admin State (U=Up, D=Down, -=not-applicable)
O - Oper State (U=Up, D=Down, -=not-applicable)

---Press any key to continue, or `q' to quit---
Service TP S C VRF MsgVpn Port A O Failed Reason

---

SMF TCP N N MsgBB 55555 D D Disabled at vpn
SMF TCP N Y MsgBB 55003 D D Disabled at vpn
SMF TCP Y N MsgBB 55443 U U
REST WEB N - MsgBB Test 9000 D D Disabled at Vpn
REST WEB Y - MsgBB Test 9443 U U
SMF WEB N - MsgBB 8008 D D Disabled at vpn
SMF WEB Y - MsgBB 1443 U U
MQTT TCP N - MsgBB Test 1883 D D Disabled at Vpn
MQTT TCP Y - MsgBB Test 8883 U U
MQTT WEB N - MsgBB Test 8000 D D Disabled at Vpn
MQTT WEB Y - MsgBB Test 8443 U U
AMQP TCP Y - MsgBB U D Not Permitted
AMQP TCP N - MsgBB Test 5672 D D Disabled at Vpn
AMQP TCP Y - MsgBB Test 5671 U U

Aaron

Hi again @Velu . Ok, from your whole output, I see a lot of "U U", which means "admin up" and "operational up". So I guess you have a server certificate successfully installed!

From this line:

Service    TP  S C VRF   MsgVpn          Port  A O Failed Reason
---------- --- --- ----- --------------- ----- --- --------------------------
MQTT       WEB Y - MsgBB Test             8443 U U

So if you're looking to use WebSockets to connect (e.g. from JavaScript or something), then you'll want port 8443, and that will connect you to your "Test" Message VPN.

And I can tell that you're on a hardware appliance, rather than a software broker, so make sure you are using the message bus IP address, not the management one.

Finally, just an FYI for formatting: you can post blocks of code/text inside three backticks ```. Makes it a lot easier to read CLI output! 👍

Velu

Hi @Aaron,

The broker is running on Kubernetes. We have disabled SMF ports. Clients are using only REST API's and SSL cert auth is mandatory.

How does 8091 is picked by the internal process.
Any config changes ? like any web ports to be enabled.

( I would like to modify the port (8091) to the REST SSL port (9443). is it possible ?).

Please suggest.

(Surely will adopt the usage/formatting ).

Abhikesh

Hi @Velu Yes you can change port number, For REST you can use below to specify port.

solace(configure)# message-vpn
solace(configure/message-vpn)# service rest
solace(configure/message-vpn/service/rest)# incoming
solace(...ure/message-vpn/service/rest/incoming)# listen-port [ssl]

Velu

Thank you for the cli @Abhishek
@Aaron

Trying to understand, that the port 8091 was not configured for the service on the broker, but the #client-username picks it ?

client details : #client-username

Client:                       #client
Type:                         Internal
Client Profile:               #client-profile
ACL Profile:                  #acl-profile
Authorization Group:          
Originating IP Address:       127.0.0.1
Subscriptions:                6
No Local Delivery:            No
Eliding Enabled:              No
Eliding Topics:                 
  Current:                    0
  High Water Mark:            0
Deliver To One:                 
  Local Priority:             1
  Network Priority:           1
Client Id:                    4
Message VPN:                  Test
Uptime:                       41d 13h 6m 24s
Scheduled Disconnect Time:    N/A
Slow Subscriber:              No
Client Username:              #client-username
Original Client Username:     #client-username
User:                         SolOS
Description:                  Internal Message Bus
Software Version:             7.14.0.9
Software Date:                Jul 20 2020 09:45:34
Platform:                     Linux26-x86_64_opt - C SDK
---Press any key to continue, or `q' to quit---
One Shot Events:                
  Large Message:              not raised
  Message Too Big:            not raised
  Parse Error:                not raised
  Max Eliding Topics:         not raised
Total Ingress Flows:          0
Total Egress Flows:           0
Web Transport Session:        N/A
SSL Version:                  N/A
SSL Cipher:                   N/A
SSL Downgraded to Plain Text: No

Aaron

Hi again @Velu ... ok, so I think the confusion here is that you're looking at an "internal use only" client that the broker maintains whenever a Message VPN is enabled. #client is not your client connection. It is for the broker's own internal communication. That's why you're seeing a weird port. Note the uptime is 41 days! 😅

As mentioned in my first response, ignore anything that starts with # as they are all for internal stuff. Check this post: https://solace.community/discussion/356/what-are-these-objects-that-start-with

So, what you need to do to get connected to your broker:

• in your Message VPN, create a new client-username to use. Configure however you want for authentication. Maybe "basic none" is the best way to start.
• If you've made a new client-profile for your client-username, and you want to use Guaranteed messaging, please make sure you enable a bunch of message spool settings in that profile

• If you've made a new acl-profile for your client-username, you'll need to change the default behaviour of everything to "allow". It is set to disallow by default.

  • Easier to just use the default acl-profile

• Finally, as a test, make sure you can connect to the broker with just a regular Solace (SMF) client to make sure your username/password/whatever is working. Either SdkPerf, or the PubSub+ Manager "TryMe!" test applications.

Hope that helps!

Aaron

Hey, also maybe just double-check / verify that Kubernetes isn't doing something weird with port 8091? I don't know where that's coming from.

Velu

Thanks @Aaron
Will check in details on the setup and revert if any clarifications required.

Aaron

@Velu Any update?

(more characters to make post long enough)

Velu

@Aaron,
Trying to deploy a test broker on Kube, to find out the configurations that are related. I will revert with the findings/analysis from the new deployment. Also with the current environment, couldn't prolong the manipulations since clients are active. mystery as of now

arih

hi all,

I remember seeing this log entry repeatedly in a customer environment (GKE Anthos) last year, and it was something about load balancer heartbeat. Though it was not port 8091 that time.

For K8s based deployment, the port exposed to outside world could be different from the ports shown in the service, since the LB will define those external port and map it back to the Pod's ports.

For my case last time, it was found out to be something with the GKE environment and the support from GKE helped the client out, I didn't get what was the root cause etc.