Distributed Tracing Connecting Solace with Jaeger

Hey there,
I’m trying to get the solace telemetry demo to run. I’ve followed instructions [1] creating a telemetry-profile default with the password default .
I’ve configured a receiver [ 2 ] and my telemetry queue is called #telemetry-default > , which is subscribed to “>”
receivers:

solace/primary:
broker: [localho st > :5671]
auth:
sasl_plain:
username: default
password: default
queue: queue://#telemetry-default
It seems the telemetry itself is working (when I send messages, the telemetry queue has a message too), but I can’t find anything in Jaeger.
My Docker logs show the following:
debug  solacereceiver@v0.55.0/receiver.go:147 Encountered error while connecting messaging service  {“kind”: “receiver”, “name”: “solace”, “pipeline”: “traces”, “error”: “dial tcp 127.0.0.1:5671: connect: connection refused”}
What am I missing?
What do I need to configure in order for Jaeger to display my traces?
Thanks and kind regards!
[1] Distributed Tracing Example
[2] Solace Open Telemetry Receiver

Hi @kilbac ,
I shot your question to one of the devs internally and this was the guidance they provided. I read through it and agree that I think it will:
The OpenTelemetry collector when following the demo runs within a docker container. Localhost points, in this case, to the localhost of the docker container, NOT to the broker container (assuming its running on the same machine). Theres a couple ways to fix this. One is to add  --network=host  to the OpenTelemetry container, since localhost will now point to the localhost of the machine rather than the docker container. Another is to add both the OpenTelemetry container and the broker to the same network, and then point to the broker container’s hostname (if in docker compose it might be something like  solbroker ). Lastly, a “depends on” can be added if the broker is in the same docker compose. The same thing can happen when sending from the collector to jaeger. They also need to be on the same network or depends on or host networked. Essentially they must be able to communicate with eachother. Easiest in my opinion for PoC is to put them all in the same docker-compose.yml and add them all to the same network, then refer to the other containers based on their names, e.g. jaeger:14250 or solbroker:5672 Hope that helps!

Hey there,
thanks for the fast reply.
The error is gone, but I still don’t receive anything in Jaeger. I believe I am missing something essential in my setup. I will post step-by-step what I do:
Setup Docker with .yaml file > version: “3.5”

services:
  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - “16686:16686”
      - “14268”
      - “14250”
depends_on:
  - otel-collector
  # Collector
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: [ “–config=/etc/otel-collector-config.yaml” ]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - “1888:1888” # pprof extension
      - “8888:8888” # Prometheus metrics exposed by the collector
      - “8889:8889” # Prometheus exporter metrics
      - “13133:13133” # health_check extension
      - “4317:4317” # OTLP gRPC receiver
      - “4318:4318” # OTLP http receiver
      - “55679:55679” # zpages extension
    depends_on:
      - solbroker
  solbroker:
    image: solace/solace-pubsub-standard:10.2.0.26
    hostname: $PUBSUB_HOSTNAME
    env_file:
      - ./solace_config_keys.env
    ports:
      - “2222:2222”
      - “8080:8080”
      - “55003:55003”
      - “55443:55443”
      - “55445:55445”
      - “${PUBSUB_PORT_NUMBER}:55555”
      - “55556:55556”
      - “5671:5671”
      - “5672:5672”
      - “5550:5550”
      - “1943:1943”
      - “9000:9000”
      - “8008:8008”
    shm_size: 2g
    ulimits:
      memlock: -1
      nofile:
        soft: 2448
        hard: 42192
2. Config Collector with otel-collector-config.yaml
processors:
  batch:
exporters:
  logging:
    loglevel: “debug”
  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true
receivers:
  solace:
    broker: [ solbroker:5671 ]
    auth:
      sasl_plain:
        username: telemetry-receiver
        password: default
    queue: queue://#telemetry-default
service:
  pipelines:
    traces:
      receivers: [ solace ]
      exporters: [ jaeger ]
3. Start docker and configure
docker compose up -d
docker exec -it tracing-ea-solbroker-1 /bin/bash
(…)
solbroker> enable
solbroker# configure
solbroker(configure)# message-vpn default
solbroker(configure/message-vpn)# create telemetry-profile default
solbroker(configure/message-vpn/telemetry-profile)# end
solbroker# configure
solbroker(configure)# create client-username telemetry-receiver message-vpn default
solbroker(configure/client-username)# acl-profile #telemetry-default
solbroker(configure/client-username)# client-profile #telemetry-default
solbroker(configure/client-username)# password default
solbroker(configure/client-username)# no shutdown
solbroker(configure/client-username)# exit
solbroker(configure)# message-vpn default
solbroker(configure/message-vpn)# telemetry-profile default
solbroker(configure/message-vpn/telemetry-profile)# trace
solbroker(…e/message-vpn/telemetry-profile/trace)# no shutdown
solbroker(…e/message-vpn/telemetry-profile/trace)# create filter all
solbroker(…ge-vpn/telemetry-profile/trace/filter)# create subscription “>”
solbroker(…try-profile/trace/filter/subscription)# exit
solbroker(…ge-vpn/telemetry-profile/trace/filter)# no shutdown
solbroker(…ge-vpn/telemetry-profile/trace/filter)# exit
solbroker(…e/message-vpn/telemetry-profile/trace)#

  1. Stop and restart Docker
  2. Open Solace Webinterface (localhost:8080)
    create Queue “q” create Subscription “tracing” in Queue “q” Go to “Try Me!” Establish Connection to localhost:8008 Publish to topic “tracing” There is now 1 message in Queue “q” and 1 message in queue “#telemetry-default
  3. Check Jaeger
    go to localhost:16686 there is no Service there except “jaeger-query” What am I missing? :slight_smile:
    Must be something stupid obvious, right?
    Thanks again in advance!
    Kind regards,
    Kilian

Hi,
we solved our Problems. This is our otel-collector-config.yaml:
processors:
  batch:
exporters:
  logging:
    loglevel: “debug”
  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true
receivers:
  otlp:
    protocols:
      grpc:
  solace:
    broker: [ solbroker:5672 ]
    auth:
      sasl_plain:
        username: trace
        password: trace
    tls:
      insecure: true
      insecure_skip_verify: true
    queue: queue://#telemetry-trace
service:
  telemetry:
    logs:
      level: “debug”
  pipelines:
    traces:
      receivers: [ otlp, solace ]
      processors: [ batch ]
      exporters: [ jaeger, logging ]
Also we did the following:
Access Control > Client Authentication > Settings > Change Type of Basic Authentication to “Internal Database” Telemetry > Receiver Connect ACLs > default action allow Jaeger is now showing a Trace for a message that is sent to the broker.
Thanks for your help.
Kind regards,
Kilian

Thanks for the update on this @kilbac ! I’m sure other members will find this useful :slight_smile:

Sure thing :slight_smile:

Also we did notice a strange behavior, maybe a bug, maybe a feature ;D
When one does change the Access Control > Client Authentication > Settings to “Internal Database”, we had to reset the password of default user afterwords to login.

interesting behaviour, I tried to replicate it locally and the default username has a default password even after setting the basic authentication type to “internal Database”… I’ll look at this in more details and if it persists I’ll raise a bug for it ?

Hi @kilbac - Hope you are doing fine.
I followed this link but getting an error which is like:
2023-03-16 23:38:12 solacetracing2-otel-collector-1   | 2023-03-16T12:38:12.822Z   debug  solacereceiver@v0.73.0/receiver.go:150 Encountered error while connecting messaging service {“kind”: “receiver”, “name”: “solace”, “data_type”: “traces”, “error”: "SASL PLAIN auth failed with code 0x2: "}

The only difference from the given setup is that I am using latest Solace version 10.3.1.17.
Any pointers on what could be the cause?

Thanks in advance
Cheers
Krishan

Hey @kriishan_anz - can you confirm you have the right username/password for your client profile? the SASL PLAIN auth failed > error happens when the username/password you configured in your profile does not match what you have in your otel collector configuration. Make sure that the solace receiver configuration
solace:

broker: [ solbroker:5672 ]
    auth:
      sasl_plain:
        username: trace
        password: trace
    tls:
      insecure: true
      insecure_skip_verify: true
    queue: queue://#telemetry-trace

matches what you configured in your client username on the broker. Hope this helps!

Thanks a lot, @Tamimi - got it working.