Distributed Tracing Context Propagation

Hi @ChristianHoltfurth ,

sure I can try to share as many details as possible :-)

I use the sdkperf sdkperf-jms-8.4.9.27 which I downloaded from products.solace.com under the path products/10.17GA/Tools/sdkperf_jms/Current/8.4.9.27/

The SOLACE_VM_ARGS I am using before running the sdkperf are listed above. The parameters I use are the following for

Publishing (I replaced the actual values with placeholders <>):

./sdkperf_jms.sh -cip=smfs://<Broker#1>:55443 -ptl=<mytopic> -mn=1 -msa=100 -mt=persistent -cu=<user>@<vpn> -cp=<password> -jcf=<connectionfactory>

Subscribing (Again, placeholders for actual values):

./sdkperf_jms.sh -cip=smfs://<Broker#2>:55443 -sql=<myqueue> -md -cu=<user>@<vpn> -cp=<password> -jcf=<connectionfactory>

For this usecase I setup two new Brokers in Version 10.2.1.32.

Any specific information you need about the Bridge Setup ?

I tried to create a draw.io image of the setup I have. I use queues for the subscribtions, no direct message transfer. The bridge is setup as unidirectional - so only Broker#2 has established a connection to Broker#1.

The message transfer works as expected in the setup. So

1. sdkperf publish to a topic on Broker#1
2. On Broker#1 is the bridge queue which subscribes to this topic
3. Message goes through the Bridge onto Broker #2 where a queue lives which subscribes to the topic
4. sdkperf subscribes to this queue

Greetings,

Jan-Filip.

Hello Alex,

thank you for your feedback. Okay, so now I totally assume that the error is on my end.

I checked the compability matrix you provided aswell as the libraries which are in use.

For the components I have:

• Broker#1 is version 10.2.1.32
• Broker#2 is version 10.2.1.32
• OpenTelemtry Collector is otel/opentelemetry-collector-contrib:0.68.0
• PubSub+ Messaging API for JMS - I am using the sdkperf from GA10.17
• PubSub+ OpenTelemetry Integration for Solace JMS API: I am using version 1.0
• OpenTelemetry Instrumentation for Java: I am using version 1.19.0

I also compared the freshly downloaded md5 sums from the maven with the libs provided in the sdkperf and they are the same. The first md5sum is the one downloaded from maven, the second one (with lib/) is the one coming with sdkperf.

filip@SIT-:~/solace-sdkperf-jms/send$ cat opentelemetry-instrumentation-api-1.19.0.jar | md5sum
c5c8c6c8a769f73b8eeabc7e4a16469b  -
filip@SIT-:~/solace-sdkperf-jms/send$ cat lib/opentelemetry-instrumentation-api-1.19.0.jar | md5sum
c5c8c6c8a769f73b8eeabc7e4a16469b  -
filip@SIT-:~/solace-sdkperf-jms/send$ cat solace-opentelemetry-jms-integration-1.0.0.jar | md5sum
508b616474f74d4b3469c93599cbd28f  -
filip@SIT-:~/solace-sdkperf-jms/send$ cat lib/solace-opentelemetry-jms-integration-1.0.0.jar | md5sum
508b616474f74d4b3469c93599cbd28f  -

The only - main difference - I can currently see would be that I am using sdkperf and you are using a custom java programm

com.solace.samples.dt.jms.example.TimeEventPublisher

I sadly do not have access to the distributed tracing EA version. Maybe can you check your example with sdkperf if it still works as expected ? Or you can somehow provide me your example setup so I can test it with my Broker setup to see if that makes any difference.

Greetings,

Jan-Filip

Hi Christian,

I contacted Alexander and asked him to provide me access to the special DT EA version.

But for sdkperf I am using the latest version. At least which is available to me.

This is my products Tools list available

Only in 10.17GA I can find the sdkperf_jms. Here I am using the one from "Current" - 8.4.9.27 (seems to be the latest)

In 10.18GA I only have sdkperf_cs

In 10.19GA it is the same

Or am I missing something and the tool is provided in a later version through other paths ?

Greetings,

Jan-Filip.

Hello Christian & Alex,

thank you very much for your patience and help.

Now it works! The hint from Christian with the "-tcc" flags was the push in the right direction. Instead of using the example from the codelab which uses a older version of sdkperf I needed to use the "new" tcc flags provided by sdkperf.

Now I get the expected result and have a trace which spans from start to finish :-)

If someone else has struggles like I do - here is the solution that worked in the end.

On the sender sdkperf:

./sdkperf_jms.sh \
    -cip=smfs://<brokerurl>:55443 \
    -ptl=<topicname> \
    -mn=1 \
    -msa=100 \
    -mt=persistent \
    -cu=<user>@<vpn> \
    -cp=<password> \
    -jcf=<connectionfactory> \
    -tcc \
    -tecip="http://<url to your opentelemetry collector>:4317"

On the receiver sdkperf:

./sdkperf_jms.sh  \
    -cip=smfs://<brokerurl>:55443 \
    -sql=<queue> \
    -cu=<user>@<vpn> \
    -cp=<password> \
    -jcf=<connectionfactory> \
    -tcrc \
    -tecip="http://<url to your opentelemetry collector>:4317"

Just a final hint maybe - if you are using the "-tmd" flag provided by sdkperf on the receiver

  -tmd              Trace message dump. When printing messages, the payload will be decoded
                    from the protobuf to show trace data (Also sets -md to true)

you will get an error

2023-01-13 09:12:39,997 [Context_2_jms_1_ConsumerDispatcher] ERROR [com.solacesystems.jms.impl.JCSMPMessageListener] Error on async receiving of BytesXMLMessage
java.lang.NoClassDefFoundError: com/google/protobuf/MessageOrBuilder

Not sure if this lib can not be added to sdkperf because of licensing issues but I had to manually download and add it to my lib directory from here Maven Repository: com.google.protobuf » protobuf-java (mvnrepository.com)

But even after you add it, you still receive an error

2023-01-13 09:11:43,646 [Context_2_jms_1_ConsumerDispatcher] ERROR [com.solacesystems.pubsub.sdkperf.core.AbstractClient] Failed to parse trace:
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

Not sure if I still did something wrong - but it was not important for me. I just did not use the flag and everything else worked as expected. So I am happy ;-)

Greetings,

Jan-Filip.