Spring cloud stream solace + Solace distributed tracing

jremeseiro · August 29

Hello!

I have a question/issue I hope I can get some help with!

I'm using Spring boot 3, with the Spring cloud stream solace integration. Everything's working fine, but I would like to have distributed tracing across applications and solace.

I've got traceability between producer < - > consumer, but no luck with adding solace traces in-between.

I can see the events have traceparent attached in the headers, that is how the consumer knows about the traces, and that is working out of the box with the applications traces, but not with Solace PubSub+ trace.

Here's an example of a trace between producer & consumer, and the second one is the solace trace, but is not attached to the original trace, it's creating a new one.

These are the properties of the message, you can see the traceparent being present there:

I've followed the docs on these pages, but I was unable to achieve it:

Summary of tech stack:

Spring boot 3.3
Spring cloud stream solace binder (5.5.0)
OpenTelemetry java agent
- otel.instrumentation.jms.enabled=true
- Tried also otel.javaagent.extensions="./build/libs/solace-opentelemetry-jcsmp-integration-1.1.0.jar", but no result
OpenTelemetry contrib collector

Is it supported? Am I missing any configuration? Any ideas?

Thank you!

BenGottstein · August 30

Hi @jremeseiro, my name is Ben, I`m a Solutions Engineer at Solace with a focus on Retail.

Great that you want to introduce Context Propagation with Distributed Tracing in your environment.

We separate context propagation in auto-instrumentation and manual-instrumentation. I have a SCSt app running locally which uses the auto-instrumentation of OpenTelemetry. The way to auto-instrument the SCSt app of yours is to use additionally the Solace JMS Binder as we need the capabilities of JMS for the auto-instrumentation: https://tutorials.solace.dev/spring/spring-boot-autoconfigure/

Once your app is built use the following command to implement auto-instrumentation for it:

java -javaagent:<absolute-path-to-otel-jar>/opentelemetry-javaagent-all-<otel-java-version>.jar 
-Dotel.javaagent.extensions=<absolute-path-to-solace-jar>/solace-opentelemetry-jms-integration-<solace-opentelementry-version>.jar 
-Dotel.propagators=solace_jms_tracecontext 
-Dotel.exporter.otlp.endpoint=<http://localhost:4317> 
-Dotel.traces.exporter=otlp 
-Dotel.metrics.exporter=none 
-Dotel.instrumentation.jms.enabled=true 
-Dotel.resource.attributes="service.name=SolaceJMSPublisher" 
-jar <your-jms-publisher-application>.jar

Of course some variables such as the exporter.otlp.endpoint should be replaced with the actual endpoint (that`s all mentioned in the documentation):

https://docs.solace.com/API/Solace-JMS-API/Context-Propagation-Distributed-Tracing.htm#Automat

In addition we have a code-lab which might help as well: https://codelabs.solace.dev/codelabs/dt-otel/?_gl=1rm5oh0_gaNzU3ODQ1ODcuMTcyNTAzMTI5NQ.._ga_XZ3NWMM83E*MTcyNTAzMTI5NC4xLjEuMTcyNTAzMjMyMy4wLjAuMA..#15

Let me know if moving to JMS will help to get auto-instrumentation working on your app.

jremeseiro · September 3

Hello BenGottstein, thanks for the quick reply!

Sorry, but I didn't fully understood your suggested solution, could you elaborate a bit more?

We are using SCSt configured like this, which under the hood is using Jcsmp if I got it right.

I already tried the configuration mentioned before, but I think I'm missing the part on how to combine JMS + SCSt

This is how we configure our app:

  cloud:
    function:
      definition: "log|routeCreateInternalReturn;"
      …
    stream:
      function:
        bindings:
          log|routeCreateInternalReturn-in-0: routeCreateInternalReturn-in-0
          …
      binders:
        solace:
          type: solace
          environment:
            solace:
              java:
                msgVpn: \${SOLACE_VPN}
                clientPassword: \${SOLACE_PASSWORD}
                clientUsername: \${SOLACE_USERNAME}
                host: \${SOLACE_HOST}

We are happy with the SCSt ways of working, meaning we're using cloud functions to receive events, and for publishing we have a generic producer:

    fun publish(message: Any, topic: String, headers: Map<String, Any> = emptyMap()) {
        val payload = mapper.writeValueAsString(message)
        val springMessage = MessageBuilder.withPayload(payload).copyHeaders(headers).build()
        logger.info("Publishing message $springMessage to topic $topic")
        streamBridge.send(topic, springMessage)

I already took a look before posting to the manual propagation in Jcsmp, but because we're using SCSt abstraction layer, I couldn't figure out where to add the propagation context, since we don't have access to the lower level (JCSMP)publisher

Is it possible to keep functions / SCSt approach using the JMS Binder? Is it only possible to have the context propagation using JMS and not SCSt?

Thank you!

BenGottstein · September 5

Thank you @jremeseiro!

You're absolutely right - the app I was referencing is Spring Boot using the Solace JMS Binder - for SCSt JMS isn`t directly possible with the Solace Binder. That was the difference here.

With JMS you get the advantage that you don`t have to implement the instrumentation as auto-instrumentation is supported. JCSMP doesn`t support this as we need the layer of the JMS API to leverage certain functions.

Manual instrumentation with JCSMP and the Solace Spring Cloud Streams Binder is possible as well - Context Propagation isn`t yet supported here as you pointed out during your testing. I`ll keep you posted once this is natively built into the JCSMP Solace Cloud Streams Binder and will inform our Product Management about your demand.

I`d be happy to join a meeting to discuss this in detail with you/your team as well.

Spring cloud stream solace + Solace distributed tracing

Answers

Categories

This Month's Leaders

This Week's Leaders