OpenShift TCPS routes

andan02
andan02 Member Posts: 17 ✭
in PubSub+ Event Broker #1

I am able to connect to the ws endpoints in solace (running on openshift) with the nodejs samples. I am unable to connect to the tcp/tcps endpoints (55555 and 55443) with nodejs and Java samples. Can someone tell me what I need to do to get the Java or nodejs samples to connect to the tcp and tcps endpoints?

Comments

  • andan02
    andan02 Member Posts: 17 ✭
    #2

    Telnet to the tcp port works:
    telnet ytcp-solace-pubsub.blah-blah 80

    here is one test:
    bin/DirectSubscriber tcp://ytcp-solace-pubsub.blah-blah:80 default password
    DirectSubscriber initializing...
    --LOG-- 11:27:04.127 [main] INFO com.solac.jcsmp.proto.impl.TcpClientChannel - Connecting to host 'orig=tcp://ytcp-solace-pubsub.blah-blah:80, scheme=tcp://, host=ytcp-solace-pubsub.blah-blah, port=80' (host 1 of 1, smfclient 1, attempt 1 of 1, this_host_attempt: 1 of 6)
    --LOG-- 11:27:04.287 [main] INFO com.solac.jcsmp.proto.impl.TcpClientChannel - Connection attempt failed to host 'ytcp-solace-pubsub.blah-blah' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: (Client name: pro-desk.local/54811/#000f0001/Y6Tz6MYY_j Local addr: 192.168.4.37 Local port: 59872 Remote addr: ytcp-solace-pubsub.blah-blah Remote port: 80) - Error communicating with the router. cause: java.io.IOException: Could not read valid SMF Header from network. found smf version=0 ((Client name: pro-desk.local/54811/#000f0001/Y6Tz6MYY_j Local addr: 192.168.4.37 Local port: 59872 Remote addr: ytcp-solace-pubsub.blah-blah Remote port: 80) - )

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #3

    Hi @andan02,
    A few questions:
    1. Did you deploy the Solace Event Brokers into OpenShift yourself? Or if someone else did it do you know if they used the openshift quickstart to do so? https://github.com/SolaceProducts/pubsubplus-openshift-quickstart
    2. Are you running the samples inside of openshift or trying to connect externally?

    My guess would be that the ports/routes may need to be configured. Note that there is a Validating the Deployment section of the quickstart guide that might prove to be useful.

  • andan02
    andan02 Member Posts: 17 ✭
    #4
    1. yes - I ran the quickstart
    2. the samples are running external to the cluster. I have masked the full url in the output log above to have "blah-blah" as the host.
  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #5

    Hi @andan02 - I think the issue might be that you're not running it against port 55555 unless I'm missing where that is mapped to port 80. Did you try this bin/DirectSubscriber tcp://tcp-solace-pubsub.blah-blah:55555 default password ?

  • andan02
    andan02 Member Posts: 17 ✭
    #6

    port 80 (for this route) is mapped in openshift to port 55555. The error I get - from the above example - "Error communicating with the router. cause: java.io.IOException: Could not read valid SMF Header from network. found smf version=0"

  • andan02
    andan02 Member Posts: 17 ✭
    #7

    btw, the nodejs example (using ws) works fine using ws://ws-solace-pubsub.blah-blah:80 (tcp-web port). Rest also works over the tcp-rest port. This seems to be unique to the tcp-smf port for the service.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #8

    thanks for the additional info @andan02. I'm not sure of the solution off the top of my head but will see what I can do to help out!
    In the mean time were you able to successfully validate the deployment using this info?

    Specifically I would double check that the tcp-smf ports all line up so port 55555 to the container itself goes to the NodePort which lines up with the port you're trying to connect to on the load balancer.

  • andan02
    andan02 Member Posts: 17 ✭
    #9

    oc get statefulset,service,pods,pvc,pv --show-labels
    I0503 15:26:25.795322 86634 request.go:621] Throttling request took 1.188045213s, request: GET:https://c114-e.us-south.containers.cloud.ibm.com:30488/apis/operators.coreos.com/v1alpha1?timeout=32s
    NAME READY AGE LABELS
    statefulset.apps/my-release-pubsubplus-ha 3/3 4d5h app.kubernetes.io/instance=my-release,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=pubsubplus-ha,helm.sh/chart=pubsubplus-ha-2.4.0

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE LABELS
    service/my-release-pubsubplus-ha LoadBalancer 172.21.243.89 2222:32484/TCP,8080:30166/TCP,1943:30710/TCP,55555:32231/TCP,55003:30151/TCP,55443:30827/TCP,55556:32422/TCP,8008:30635/TCP,1443:30142/TCP,9000:31919/TCP,9443:32398/TCP,5672:32641/TCP,5671:31032/TCP,1883:31015/TCP,8883:30378/TCP,8000:31336/TCP,8443:32428/TCP 4d5h app.kubernetes.io/instance=my-release,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=pubsubplus-ha,helm.sh/chart=pubsubplus-ha-2.4.0
    service/my-release-pubsubplus-ha-discovery ClusterIP None 8080/TCP,8741/TCP,8300/TCP,8301/TCP,8302/TCP 4d5h app.kubernetes.io/instance=my-release,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=pubsubplus-ha,helm.sh/chart=pubsubplus-ha-2.4.0

    NAME READY STATUS RESTARTS AGE LABELS
    pod/my-release-pubsubplus-ha-0 1/1 Running 0 4d5h active=true,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus-ha,controller-revision-hash=my-release-pubsubplus-ha-7c5895cff6,statefulset.kubernetes.io/pod-name=my-release-pubsubplus-ha-0
    pod/my-release-pubsubplus-ha-1 1/1 Running 0 4d5h active=false,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus-ha,controller-revision-hash=my-release-pubsubplus-ha-7c5895cff6,statefulset.kubernetes.io/pod-name=my-release-pubsubplus-ha-1
    pod/my-release-pubsubplus-ha-2 1/1 Running 0 2d13h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus-ha,controller-revision-hash=my-release-pubsubplus-ha-7c5895cff6,statefulset.kubernetes.io/pod-name=my-release-pubsubplus-ha-2

    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE LABELS
    persistentvolumeclaim/data-my-release-pubsubplus-ha-0 Bound pvc-99fe7366-694b-47d7-9449-84bd1f687ccc 30Gi RWO ibmc-vpc-block-10iops-tier 4d5h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus-ha
    persistentvolumeclaim/data-my-release-pubsubplus-ha-1 Bound pvc-812bbec7-806c-441f-9a9c-d85ba4a63aca 30Gi RWO ibmc-vpc-block-10iops-tier 4d5h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus-ha
    persistentvolumeclaim/data-my-release-pubsubplus-ha-2 Bound pvc-d76f6454-0901-4ea2-8abd-0c4f0fa978f3 30Gi RWO ibmc-vpc-block-10iops-tier 4d5h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus-ha

  • andan02
    andan02 Member Posts: 17 ✭
    #10

    I checked the logs on the 3 pods. I could not see any logs of any successful or unsuccessful publish or subscription. I do not know how to monitor if a connection is taking place or not over tcp/tcps to the smf ports.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #11

    Hi @andan02,
    I got a bit of info from our support team that might help:

    What causes JCSMP to give this error: "cause: java.io.IOException: Could not read valid SMF Header from network." when trying to connect? Something is wrong with the received packet and the JCSMP API cannot decode it. What is most likely happening is the openshift route is routing to the wrong host/port or something is corrupting the network data.

    With this info I was actually able to reproduce the issue by pointing the sample at tcp://google.com:80 which makes me think that for some reason the smf traffic isn't being properly routed through to port 55555.

    MJD-MacBook-Pro.local:~/git/solace-samples-java-jcsmp/build/staged$ ./bin/DirectSubscriber tcp://google.com:80 default default
DirectSubscriber initializing...
--LOG-- 08:42:00.014 [main] INFO  com.solac.jcsmp.proto.impl.TcpClientChannel - Connecting to host 'orig=tcp://google.com:80, scheme=tcp://, host=google.com, port=80' (host 1 of 1, smfclient 1, attempt 1 of 1, this_host_attempt: 1 of 6)
--LOG-- 08:42:00.093 [main] INFO  com.solac.jcsmp.proto.impl.TcpClientChannel - Connection attempt failed to host 'google.com' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: (Client name: MJD-MacBook-Pro.local/63808/#000f0001/aoVc2QtIeh   Local addr: 192.168.1.25 Local port: 62037   Remote addr: google.com  Remote port: 80) - Error communicating with the router. cause: java.io.IOException: Could not read valid SMF Header from network. found smf version=0 ((Client name: MJD-MacBook-Pro.local/63808/#000f0001/aoVc2QtIeh   Local addr: 192.168.1.25 Local port: 62037   Remote addr: google.com  Remote port: 80) - )

    I can however help with this question:

    I do not know how to monitor if a connection is taking place or not over tcp/tcps to the smf ports.

    You can see if a client is connected a few ways...note that you likely won't see your client since it is failing to connect.
    1. In PubSub+ Manager you can see connected SMF clients by choosing your Message VPN and then choosing the "Clients" Menu on the left hand side, and choosing "Solace Clients"
    2. In the logs use the show log event command to see the event log. This log will tell you when a client connects, if you look for events that say CLIENT: CLIENT_CLIENT_CONNECT that tells you when a messaging client is connecting to the broker. It will look something like this:

    2021-05-03T18:37:37.780+00:00 <local3.info> b0e95afab69a event: CLIENT: CLIENT_CLIENT_CONNECT: default MJD-MacBook-Pro.local/16740/#000f0001/LOrqg4D3OS Client (4) MJD-MacBook-Pro.local/16740/#000
f0001/LOrqg4D3OS username default OriginalClientUsername(default) WebSessionId (N/A) connected to 172.17.0.2:55555 from 172.17.0.1:64534 version(10.10.0) platform(Mac OS X-x86_64 (Java 11.0.4+11)
 - JCSMP SDK) SslVersion() SslCipher() APIuser('marcdipasquale' Computer: 'MJD-MacBook-Pro.local' Process ID: 16740) authScheme(Basic) authorizationGroup() clientProfile(default) ACLProfile(defau
lt) SSLDowngradedToPlainText(No) SSLNegotiatedTo() SslRevocation(Not Checked), Capabilities(unbind-ack, bind-response-endpoint-error-id)

    For next steps in troubleshooting this would there by some way of doing packet capture to see what is arriving into the load balancer on the outside of OpenShift vs. what is actually getting to the pod itself? Or even seeing stats within OpenShift that maybe tells us that the request is actually traversing the Load Balancer to the NodePort to the Pod?

  • andan02
    andan02 Member Posts: 17 ✭
    #12

    I need to figure out how to monitor this ingres. I do not know how to do this yet. In your example above, of course google wont respond with a proper SMF version. Port 80 is what openshift uses for non-ssl comms for any route you create. 443 is used for any ssl route you create. The route is connected to port 55555 in the port 80 route and port 55443 in the port 443 route I have created

    Is there someone within Solace that has used the TCP or TCPs connection out of openshift? I have now seen others with this issue in other forums. It would be great to figure this out.

  • andan02
    andan02 Member Posts: 17 ✭
    #13

    I found a way to monitor the route traffic. When I curl the url for the tcp-smf on port 80 you will see a spike in the traffic graph. At 13:46 on the graph I attempted to run the java code against the same url and port number - no traffic was detected. Is there something specific about the java code that is looking for 55555 or 55443 and if it does not find it, then it fails?

  • andan02
    andan02 Member Posts: 17 ✭
    #14

    can you test the java sample with a port other than 55555 or 55443 to see if it is the code or the connection?

  • andan02
    andan02 Member Posts: 17 ✭
    #15

    when I ran the code with the tcps connection info on port 443 I get a different result. It appears that traffic is getting through, but ssl is failing.

    --LOG-- 13:54:31.952 [main] INFO com.solac.jcsmp.proto.impl.TcpClientChannel - Connection attempt failed to host 'ytcps-solace-pubsub.blah' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: (Client name: pro-desk.local/77914/#000f0001/Ri222wdFr9 Local addr: 192.168.4.37 Local port: 61030 Remote addr: ytcps-solace-pubsub.blah Remote port: 443) - Error communicating with the router. cause: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure ((Client name: pro-desk.local/77914/#000f0001/Ri222wdFr9 Local addr: 192.168.4.37 Local port: 61030 Remote addr: ytcps-solace-pubsub.blah Remote port: 443) - )

  • andan02
    andan02 Member Posts: 17 ✭
    #16

    Is there someone who can help with this issue?

  • andan02
    andan02 Member Posts: 17 ✭
    #17

    @marc is there any more information you need to recreate or debug this?

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #18

    Hi @andan02, sorry for the delay! I'm working on getting someone to help you out here :)

  • pkondrat
    pkondrat Member, Employee Posts: 27 Solace Employee
    #19

    Hi @andan02
    I think there are two things that are interacting here to produce the behavior that you are seeing. SMF runs over multiple transports; it runs over TCP/TLS and HTTP(s)/web-transport. The OpenShift routes are generally meant to work with HTTP transports. This is why your tests with Node worked but, not with JCSMP (the Node SDK runs SMF over web-sockets and JCSMP works over a TCP connection). Unfortunately, JCSMP does not have an option to run SMF over web-sockets (JS/node, CCSMP and JAVA RTO SDKs can run over web-sockets).
    Let's see what else we can do?
    Options for OpenShift are described here:
    v3.11 - https://docs.openshift.com/container-platform/3.11/dev_guide/expose_service/index.html
    v4 - https://docs.openshift.com/container-platform/4.7/networking/configuring_ingress_cluster_traffic/overview-traffic.html
    Note: The route/ingress controller option allows TLS-secured protocols other than HTTPS.
    We normally recommend using a load-balancer to provide external access to PubSub+ running inside OpenShift but, if you must use a router then it seems like the choices are using an API that supports web-socket transport or use TLS. The best option is probably to use JCSMP with TLS.
    Note that if you are going to use TLS then you have a little more configuration to do on the broker to enable it; you will need configure a server certificate. If you are using OpenShift 4, the latest version of the quickstart has support for setting the server certificate when the broker is deployed (https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#enabling-use-of-tls-to-access-broker-services). Configuring TLS on your existing broker is also possible (https://docs.solace.com/Configuring-and-Managing/Managing-TLS-SSL-Service.htm).
    Best Regards,
    Paul

  • andan02
    andan02 Member Posts: 17 ✭
    #20

    @pkondrat, thank you for the update. I thought this might be the case. Here is what I did this morning to align my setup with your guidance.

    1. helm uninstall my-release
    2. removed the old project/namespace and re-added 'solace-pubsub'
    3. I setup the openshift secret. oc create secret tls solace-tls-secret --key=controlcenter.key --cert=controlcenter.crt. (i re-used a key and crt I had from another component - don't know if I can do this or if I need to setup a new self-signed cert - please advise)
    4. I installed again with helm - helm install my-release solacecharts/pubsubplus-ha --set securityContext.enabled=false,tls.enabled=true,tls.serverCertificatesSecret=solace-tls-secret
    5. The UI is working and I added a messagevpn "onboarding"
    6. added a route for 55443
    7. used route to test - bin/DirectSubscriber tcps://tpc-tls-solace-pubsub.blah:443 onboarding default password
    8. watched the traffic on OCP - I can confirm that I did receive traffic on the route.

    received the following:

    --LOG-- 09:28:31.100 [main] INFO com.solac.jcsmp.proto.impl.TcpClientChannel - Connection attempt failed to host 'tpc-tls-solace-pubsub.blah' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: CertificateException - java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors cause: java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors ((Client name: pro-desk.local/31215/#000f0001/758P5iMYVV Local addr: 192.168.4.37 Local port: 54796 Remote addr: tpc-tls-solace-pubsub.blah Remote port: 443) - )

    Andy

  • pkondrat
    pkondrat Member, Employee Posts: 27 Solace Employee
    #21

    @andan02 It looks like the API is unable to validate the server certificate from the broker. I suspect you need to configure the trust store in the API. If the server certificate that you configured on the broker is self-signed then you can use that same certificate in the APIs trust store.
    Paul

  • andan02
    andan02 Member Posts: 17 ✭
    #22

    @pkondrat - I think its working now.
    I added:
    properties.setProperty(JCSMPProperties.SSL_TRUST_STORE, "/Users/andan02/projects/solace/truststore.jks");
    properties.setProperty(JCSMPProperties.SSL_TRUST_STORE_PASSWORD, "password");
    to both the DirectPublisher and DirectSubscriber.
    I can see traffic hitting the route, and I see messages publishing and consumed on the pubsub dashboard. I was able to subscribe to the topic and see messages in the TRY ME section also.

    Thank you very much for your help. I think I can continue my testing from here. Would be great to see some guidance online for those using openshift and java. Perhaps this thread can help others.

    Andy

  • pkondrat
    pkondrat Member, Employee Posts: 27 Solace Employee
    #23

    I think it already has. Thank you for your persistence.
    Paul

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #24

    Thank you so much @pkondrat for helping and that's great to hear it's working @andan02.
    This thread will definitely be useful for others and we'll have to take a look at how we can clarify some of this on the openshift template repo as well.

  • andan02
    andan02 Member Posts: 17 ✭
    #25

    I can add a pull request with the two lines commented out if you like. @marc

  • andan02
    andan02 Member Posts: 17 ✭
    #26

    @marc @pkondrat
    pull requests added:
    directsubscriber - https://github.com/SolaceSamples/solace-samples-java-jcsmp/pull/5
    directpublisher - https://github.com/SolaceSamples/solace-samples-java-jcsmp/pull/4

    I hope this helps,
    Andy

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    edited May 2021 #27

    @andan02 said:
    @marc @pkondrat
    pull requests added:
    directsubscriber - https://github.com/SolaceSamples/solace-samples-java-jcsmp/pull/5
    directpublisher - https://github.com/SolaceSamples/solace-samples-java-jcsmp/pull/4

    I hope this helps,
    Andy

    Awesome thanks @andan02!

Categories

This Month's Leaders

This Week's Leaders