Distributed Tracing Connecting Solace with Jaeger

kilbac

Hey there,

I'm trying to get the solace telemetry demo to run. I've followed instructions [1] creating a telemetry-profile default with the password default.

I've configured a receiver [2] and my telemetry queue is called #telemetry-default , which is subscribed to ">"

receivers:
  solace/primary:
    broker: [localhost:5671]
    auth:
      sasl_plain:
        username: default
        password: default
    queue: queue://#telemetry-default

It seems the telemetry itself is working (when I send messages, the telemetry queue has a message too), but I can't find anything in Jaeger.

My Docker logs show the following:

debug  solacereceiver@v0.55.0/receiver.go:147 Encountered error while connecting messaging service  {"kind": "receiver", "name": "solace", "pipeline": "traces", "error": "dial tcp 127.0.0.1:5671: connect: connection refused"}

What am I missing?

What do I need to configure in order for Jaeger to display my traces?

Thanks and kind regards!

[1] https://docs.solace.com/Features/Distributed-Tracing/Distributed-Tracing-Example.htm#Distributed_Tracing_Example

[2] https://docs.solace.com/Features/Distributed-Tracing/Distributed-Tracing-Receiver.htm

kilbac

Hi,

we solved our Problems. This is our otel-collector-config.yaml:

processors:
  batch:

exporters:
  logging:
    loglevel: "debug"

  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true

receivers:
  otlp:
    protocols:
      grpc:

  solace:
    broker: [ solbroker:5672 ]
    auth:
      sasl_plain:
        username: trace
        password: trace
    tls:
      insecure: true
      insecure_skip_verify: true
    queue: queue://#telemetry-trace

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers: [ otlp, solace ] 
      processors: [ batch ]
      exporters: [ jaeger, logging ]

Also we did the following:

• Access Control > Client Authentication > Settings > Change Type of Basic Authentication to "Internal Database"
• Telemetry > Receiver Connect ACLs > default action allow

Jaeger is now showing a Trace for a message that is sent to the broker.

Thanks for your help.

Kind regards,

Kilian

marc

Hi @kilbac,

I shot your question to one of the devs internally and this was the guidance they provided. I read through it and agree that I think it will:

The OpenTelemetry collector when following the demo runs within a docker container. Localhost points, in this case, to the localhost of the docker container, NOT to the broker container (assuming its running on the same machine). Theres a couple ways to fix this. One is to add --network=host to the OpenTelemetry container, since localhost will now point to the localhost of the machine rather than the docker container. Another is to add both the OpenTelemetry container and the broker to the same network, and then point to the broker container’s hostname (if in docker compose it might be something like solbroker). Lastly, a “depends on” can be added if the broker is in the same docker compose.

The same thing can happen when sending from the collector to jaeger. They also need to be on the same network or depends on or host networked. Essentially they must be able to communicate with eachother. Easiest in my opinion for PoC is to put them all in the same docker-compose.yml and add them all to the same network, then refer to the other containers based on their names, e.g. jaeger:14250 or solbroker:5672

Hope that helps!

kilbac

Hey there,

thanks for the fast reply.

The error is gone, but I still don't receive anything in Jaeger. I believe I am missing something essential in my setup. I will post step-by-step what I do:

1. Setup Docker with .yaml file

version: "3.5"

services:
  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268"
      - "14250"
    depends_on:
      - otel-collector

  # Collector
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: [ "--config=/etc/otel-collector-config.yaml" ]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888" # pprof extension
      - "8888:8888" # Prometheus metrics exposed by the collector
      - "8889:8889" # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317" # OTLP gRPC receiver
      - "4318:4318" # OTLP http receiver
      - "55679:55679" # zpages extension
    depends_on:
      - solbroker

  solbroker:
    image: solace/solace-pubsub-standard:10.2.0.26
    hostname: $PUBSUB_HOSTNAME
    env_file:
      - ./solace_config_keys.env
    ports:
      - "2222:2222"
      - "8080:8080"
      - "55003:55003"
      - "55443:55443"
      - "55445:55445"
      - "${PUBSUB_PORT_NUMBER}:55555"
      - "55556:55556"
      - "5671:5671"
      - "5672:5672"
      - "5550:5550"
      - "1943:1943"
      - "9000:9000"
      - "8008:8008"
    shm_size: 2g
    ulimits:
      memlock: -1
      nofile:
        soft: 2448
        hard: 42192

2. Config Collector with otel-collector-config.yaml

processors:
  batch:

exporters:
  logging:
    loglevel: "debug"

  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true

receivers:
  solace:
    broker: [ solbroker:5671 ]
    auth:
      sasl_plain:
        username: telemetry-receiver
        password: default
    queue: queue://#telemetry-default

service:
  pipelines:
    traces:
      receivers: [ solace ]
      exporters: [ jaeger ]

3. Start docker and configure

docker compose up -d
docker exec -it tracing-ea-solbroker-1 /bin/bash
(...)
solbroker> enable
solbroker# configure
solbroker(configure)# message-vpn default
solbroker(configure/message-vpn)# create telemetry-profile default
solbroker(configure/message-vpn/telemetry-profile)# end
solbroker# configure
solbroker(configure)# create client-username telemetry-receiver message-vpn default
solbroker(configure/client-username)# acl-profile #telemetry-default
solbroker(configure/client-username)# client-profile #telemetry-default
solbroker(configure/client-username)# password default
solbroker(configure/client-username)# no shutdown
solbroker(configure/client-username)# exit
solbroker(configure)# message-vpn default
solbroker(configure/message-vpn)# telemetry-profile default
solbroker(configure/message-vpn/telemetry-profile)# trace
solbroker(...e/message-vpn/telemetry-profile/trace)# no shutdown
solbroker(...e/message-vpn/telemetry-profile/trace)# create filter all
solbroker(...ge-vpn/telemetry-profile/trace/filter)# create subscription ">"
solbroker(...try-profile/trace/filter/subscription)# exit
solbroker(...ge-vpn/telemetry-profile/trace/filter)# no shutdown
solbroker(...ge-vpn/telemetry-profile/trace/filter)# exit
solbroker(...e/message-vpn/telemetry-profile/trace)#

4. Stop and restart Docker

5. Open Solace Webinterface (localhost:8080)

• create Queue "q"
• create Subscription "tracing" in Queue "q"
• Go to "Try Me!"
• Establish Connection to localhost:8008
• Publish to topic "tracing"

There is now 1 message in Queue "q" and 1 message in queue "#telemetry-default"

6. Check Jaeger

• go to localhost:16686
• there is no Service there except "jaeger-query"

What am I missing? :)

Must be something stupid obvious, right?

Thanks again in advance!

Kind regards,

Kilian

kilbac

Hi,

we solved our Problems. This is our otel-collector-config.yaml:

processors:
  batch:

exporters:
  logging:
    loglevel: "debug"

  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true

receivers:
  otlp:
    protocols:
      grpc:

  solace:
    broker: [ solbroker:5672 ]
    auth:
      sasl_plain:
        username: trace
        password: trace
    tls:
      insecure: true
      insecure_skip_verify: true
    queue: queue://#telemetry-trace

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers: [ otlp, solace ] 
      processors: [ batch ]
      exporters: [ jaeger, logging ]

Also we did the following:

• Access Control > Client Authentication > Settings > Change Type of Basic Authentication to "Internal Database"
• Telemetry > Receiver Connect ACLs > default action allow

Jaeger is now showing a Trace for a message that is sent to the broker.

Thanks for your help.

Kind regards,

Kilian

Tamimi

Thanks for the update on this @kilbac ! I'm sure other members will find this useful :)

kilbac

Sure thing :)

Also we did notice a strange behavior, maybe a bug, maybe a feature ;D

When one does change the Access Control > Client Authentication > Settings to "Internal Database", we had to reset the password of default user afterwords to login.

Tamimi

interesting behaviour, I tried to replicate it locally and the default username has a default password even after setting the basic authentication type to "internal Database"... I'll look at this in more details and if it persists I'll raise a bug for it 👍

kriishan_anz

Hi @kilbac - Hope you are doing fine.

I followed this link but getting an error which is like:

2023-03-16 23:38:12 solacetracing2-otel-collector-1   | 2023-03-16T12:38:12.822Z   debug  solacereceiver@v0.73.0/receiver.go:150 Encountered error while connecting messaging service {"kind": "receiver", "name": "solace", "data_type": "traces", "error": "SASL PLAIN auth failed with code 0x2: "}

The only difference from the given setup is that I am using latest Solace version 10.3.1.17.

Any pointers on what could be the cause?

Thanks in advance

Cheers

Krishan

Tamimi

Hey @kriishan_anz - can you confirm you have the right username/password for your client profile? the SASL PLAIN auth failed error happens when the username/password you configured in your profile does not match what you have in your otel collector configuration. Make sure that the solace receiver configuration

solace:
    broker: [ solbroker:5672 ]
    auth:
      sasl_plain:
        username: trace
        password: trace
    tls:
      insecure: true
      insecure_skip_verify: true
    queue: queue://#telemetry-trace

matches what you configured in your client username on the broker. Hope this helps!

kriishan_anz

Thanks a lot, @Tamimi - got it working.