Distributed Tracing Connecting Solace with Jaeger

kilbac
kilbac Member Posts: 6
edited November 2022 in General Discussions #1

Hey there,

I'm trying to get the solace telemetry demo to run. I've followed instructions [1] creating a telemetry-profile default with the password default.

I've configured a receiver [2] and my telemetry queue is called #telemetry-default , which is subscribed to ">"

receivers:
  solace/primary:
    broker: [localhost:5671]
    auth:
      sasl_plain:
        username: default
        password: default
    queue: queue://#telemetry-default

It seems the telemetry itself is working (when I send messages, the telemetry queue has a message too), but I can't find anything in Jaeger.

My Docker logs show the following:

debug  solacereceiver@v0.55.0/receiver.go:147 Encountered error while connecting messaging service  {"kind": "receiver", "name": "solace", "pipeline": "traces", "error": "dial tcp 127.0.0.1:5671: connect: connection refused"}

What am I missing?

What do I need to configure in order for Jaeger to display my traces?

Thanks and kind regards!

[1] https://docs.solace.com/Features/Distributed-Tracing/Distributed-Tracing-Example.htm#Distributed_Tracing_Example

[2] https://docs.solace.com/Features/Distributed-Tracing/Distributed-Tracing-Receiver.htm

Best Answer

  • kilbac
    kilbac Member Posts: 6
    #2 Answer ✓

    Hi,

    we solved our Problems. This is our otel-collector-config.yaml:

    processors:
      batch:
    
    exporters:
      logging:
        loglevel: "debug"
    
      jaeger:
        endpoint: jaeger-all-in-one:14250
        tls:
          insecure: true
    
    receivers:
      otlp:
        protocols:
          grpc:
    
      solace:
        broker: [ solbroker:5672 ]
        auth:
          sasl_plain:
            username: trace
            password: trace
        tls:
          insecure: true
          insecure_skip_verify: true
        queue: queue://#telemetry-trace
    
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [ otlp, solace ] 
          processors: [ batch ]
          exporters: [ jaeger, logging ]
    

    Also we did the following:

    • Access Control > Client Authentication > Settings > Change Type of Basic Authentication to "Internal Database"
    • Telemetry > Receiver Connect ACLs > default action allow

    Jaeger is now showing a Trace for a message that is sent to the broker.

    Thanks for your help.

    Kind regards,

    Kilian

Answers

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 914 admin

    Hi @kilbac,

    I shot your question to one of the devs internally and this was the guidance they provided. I read through it and agree that I think it will:

    The OpenTelemetry collector when following the demo runs within a docker container. Localhost points, in this case, to the localhost of the docker container, NOT to the broker container (assuming its running on the same machine). Theres a couple ways to fix this. One is to add --network=host to the OpenTelemetry container, since localhost will now point to the localhost of the machine rather than the docker container. Another is to add both the OpenTelemetry container and the broker to the same network, and then point to the broker container’s hostname (if in docker compose it might be something like solbroker). Lastly, a “depends on” can be added if the broker is in the same docker compose.

    The same thing can happen when sending from the collector to jaeger. They also need to be on the same network or depends on or host networked. Essentially they must be able to communicate with eachother. Easiest in my opinion for PoC is to put them all in the same docker-compose.yml and add them all to the same network, then refer to the other containers based on their names, e.g. jaeger:14250 or solbroker:5672

    Hope that helps!

  • kilbac
    kilbac Member Posts: 6

    Hey there,

    thanks for the fast reply.

    The error is gone, but I still don't receive anything in Jaeger. I believe I am missing something essential in my setup. I will post step-by-step what I do:

    1. Setup Docker with .yaml file
    version: "3.5"
    
    services:
      # Jaeger
      jaeger-all-in-one:
        image: jaegertracing/all-in-one:latest
        ports:
          - "16686:16686"
          - "14268"
          - "14250"
        depends_on:
          - otel-collector
    
      # Collector
      otel-collector:
        image: otel/opentelemetry-collector-contrib:latest
        command: [ "--config=/etc/otel-collector-config.yaml" ]
        volumes:
          - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
        ports:
          - "1888:1888" # pprof extension
          - "8888:8888" # Prometheus metrics exposed by the collector
          - "8889:8889" # Prometheus exporter metrics
          - "13133:13133" # health_check extension
          - "4317:4317" # OTLP gRPC receiver
          - "4318:4318" # OTLP http receiver
          - "55679:55679" # zpages extension
        depends_on:
          - solbroker
    
      solbroker:
        image: solace/solace-pubsub-standard:10.2.0.26
        hostname: $PUBSUB_HOSTNAME
        env_file:
          - ./solace_config_keys.env
        ports:
          - "2222:2222"
          - "8080:8080"
          - "55003:55003"
          - "55443:55443"
          - "55445:55445"
          - "${PUBSUB_PORT_NUMBER}:55555"
          - "55556:55556"
          - "5671:5671"
          - "5672:5672"
          - "5550:5550"
          - "1943:1943"
          - "9000:9000"
          - "8008:8008"
        shm_size: 2g
        ulimits:
          memlock: -1
          nofile:
            soft: 2448
            hard: 42192
    

    2. Config Collector with otel-collector-config.yaml

    processors:
      batch:
    
    exporters:
      logging:
        loglevel: "debug"
    
      jaeger:
        endpoint: jaeger-all-in-one:14250
        tls:
          insecure: true
    
    receivers:
      solace:
        broker: [ solbroker:5671 ]
        auth:
          sasl_plain:
            username: telemetry-receiver
            password: default
        queue: queue://#telemetry-default
    
    service:
      pipelines:
        traces:
          receivers: [ solace ]
          exporters: [ jaeger ]
    

    3. Start docker and configure

    docker compose up -d
    docker exec -it tracing-ea-solbroker-1 /bin/bash
    (...)
    solbroker> enable
    solbroker# configure
    solbroker(configure)# message-vpn default
    solbroker(configure/message-vpn)# create telemetry-profile default
    solbroker(configure/message-vpn/telemetry-profile)# end
    solbroker# configure
    solbroker(configure)# create client-username telemetry-receiver message-vpn default
    solbroker(configure/client-username)# acl-profile #telemetry-default
    solbroker(configure/client-username)# client-profile #telemetry-default
    solbroker(configure/client-username)# password default
    solbroker(configure/client-username)# no shutdown
    solbroker(configure/client-username)# exit
    solbroker(configure)# message-vpn default
    solbroker(configure/message-vpn)# telemetry-profile default
    solbroker(configure/message-vpn/telemetry-profile)# trace
    solbroker(...e/message-vpn/telemetry-profile/trace)# no shutdown
    solbroker(...e/message-vpn/telemetry-profile/trace)# create filter all
    solbroker(...ge-vpn/telemetry-profile/trace/filter)# create subscription ">"
    solbroker(...try-profile/trace/filter/subscription)# exit
    solbroker(...ge-vpn/telemetry-profile/trace/filter)# no shutdown
    solbroker(...ge-vpn/telemetry-profile/trace/filter)# exit
    solbroker(...e/message-vpn/telemetry-profile/trace)#
    

    4. Stop and restart Docker

    5. Open Solace Webinterface (localhost:8080)

    • create Queue "q"
    • create Subscription "tracing" in Queue "q"
    • Go to "Try Me!"
    • Establish Connection to localhost:8008
    • Publish to topic "tracing"

    There is now 1 message in Queue "q" and 1 message in queue "#telemetry-default"

    6. Check Jaeger

    • go to localhost:16686
    • there is no Service there except "jaeger-query"

    What am I missing? :)

    Must be something stupid obvious, right?

    Thanks again in advance!

    Kind regards,

    Kilian

  • kilbac
    kilbac Member Posts: 6
    #5 Answer ✓

    Hi,

    we solved our Problems. This is our otel-collector-config.yaml:

    processors:
      batch:
    
    exporters:
      logging:
        loglevel: "debug"
    
      jaeger:
        endpoint: jaeger-all-in-one:14250
        tls:
          insecure: true
    
    receivers:
      otlp:
        protocols:
          grpc:
    
      solace:
        broker: [ solbroker:5672 ]
        auth:
          sasl_plain:
            username: trace
            password: trace
        tls:
          insecure: true
          insecure_skip_verify: true
        queue: queue://#telemetry-trace
    
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [ otlp, solace ] 
          processors: [ batch ]
          exporters: [ jaeger, logging ]
    

    Also we did the following:

    • Access Control > Client Authentication > Settings > Change Type of Basic Authentication to "Internal Database"
    • Telemetry > Receiver Connect ACLs > default action allow

    Jaeger is now showing a Trace for a message that is sent to the broker.

    Thanks for your help.

    Kind regards,

    Kilian

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 491 admin

    Thanks for the update on this @kilbac ! I'm sure other members will find this useful :)

  • kilbac
    kilbac Member Posts: 6
    edited November 2022 #7

    Sure thing :)


    Also we did notice a strange behavior, maybe a bug, maybe a feature ;D

    When one does change the Access Control > Client Authentication > Settings to "Internal Database", we had to reset the password of default user afterwords to login.

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 491 admin

    interesting behaviour, I tried to replicate it locally and the default username has a default password even after setting the basic authentication type to "internal Database"... I'll look at this in more details and if it persists I'll raise a bug for it 👍

  • kriishan_anz
    kriishan_anz Member Posts: 14

    Hi @kilbac - Hope you are doing fine.

    I followed this link but getting an error which is like:

    2023-03-16 23:38:12 solacetracing2-otel-collector-1   | 2023-03-16T12:38:12.822Z   debug  solacereceiver@v0.73.0/receiver.go:150 Encountered error while connecting messaging service {"kind": "receiver", "name": "solace", "data_type": "traces", "error": "SASL PLAIN auth failed with code 0x2: "}


    The only difference from the given setup is that I am using latest Solace version 10.3.1.17.

    Any pointers on what could be the cause?


    Thanks in advance

    Cheers

    Krishan

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 491 admin
    edited March 2023 #10

    Hey @kriishan_anz - can you confirm you have the right username/password for your client profile? the SASL PLAIN auth failed error happens when the username/password you configured in your profile does not match what you have in your otel collector configuration. Make sure that the solace receiver configuration

    solace:
        broker: [ solbroker:5672 ]
        auth:
          sasl_plain:
            username: trace
            password: trace
        tls:
          insecure: true
          insecure_skip_verify: true
        queue: queue://#telemetry-trace
    


    matches what you configured in your client username on the broker. Hope this helps!

  • kriishan_anz
    kriishan_anz Member Posts: 14

    Thanks a lot, @Tamimi - got it working.