How do I non-destructively view a message payload of the most recent message in a queue?

AllenW

I've come across this issue a few times recently. The two tools I've tried using are JMSToolbox and SDKPerf. I've found neither really have the documentation or resources to help with this.

The problem I'm having is I have large queues and DMQs (some with 2M+ messages) and we are trying to determine the payloads of these messages to troubleshoot, and implement more data cleansing before the messages reach Solace. However, in queues so large I'm having difficulty filtering messages to the latest. Ideally I want the latest messages in the Queue to be examined, JMSToolbox shows messages from oldest to newest unfortunately..

SDKPerf:

I found this command from anther thread usefull.

sdkperf_jms.bat -cip smfs://:55443 -cu <client-username@VPN -cp -jcf -jndi -sql -qb -md >> 03122020_Latest.txt

To view all the messages in the Queue and write to a text file. However, for huge queues this is unfeasible. Is there any way to add a timestamp or filter the messages by message-id perhaps?

JMSToolbox is usefull however for queues this large it freezes. I've also tried using selectors and a unix timestamp to filter, however this also doesn't work as expected.

JMSTimestamp > 1687823291

Open to suggestions of any other tools some might recommend.

Tamimi

You could look into creating a browser in most of our APIs. For example, here is how you can do it with JMS (https://docs.solace.com/API-Developer-Online-Ref-Documentation/jms/javax/jms/QueueBrowser.html) and with JCSMP (https://docs.solace.com/API-Developer-Online-Ref-Documentation/java/com/solacesystems/jcsmp/Browser.html) by creating a Browser object.

Tamimi

Hey @AllenW - have you considered queue browsing from the API? you can pass extra configuration parameters to your queue browsing.

We also have a chrome extension queue browser project on our Solace Community that you might find interesting https://github.com/solacecommunity/solace-queue-browser-extension. Note: this is a community contributed project, so you can also contribute to it if you find it helpful to extend it!

I'll see what others also have to say about this

AllenW

Hi @Tamimi - thanks for the reply.

I've only tried using SDKPerf & JMSToolbox. Which API were you referring to? I'm keen to check it out if you have a link? Chrome extensions are blocked by our company unfortunately.

Thanks a lot for your help.

Tamimi

You could look into creating a browser in most of our APIs. For example, here is how you can do it with JMS (https://docs.solace.com/API-Developer-Online-Ref-Documentation/jms/javax/jms/QueueBrowser.html) and with JCSMP (https://docs.solace.com/API-Developer-Online-Ref-Documentation/java/com/solacesystems/jcsmp/Browser.html) by creating a Browser object.

Aaron

There is no way to "browse from latest". Unfortunately.

One option (not a great one) might be to setup another queue, configure it with the same topic subscriptions as the original main (very full) queue. Then initiate a replay request (if you have replay enabled in your VPN) for the last n minutes (or whatever) to give you at least some of the messages. Might be easier than "while hasMore next" on a Browser for millions of loops.

sjaak

The Chrome extension is what we use (we developed it ourselves), but it is not ideal. Our problem is that we, as a Solace partner, are currently managing 20-30 Solace instances. To switch from Solace VPN A to B, you must update the credentials of the Chrome extension. JMS Toolbox does not scale.

So, when does Solace add a message viewer to PubSub+? 😀

leonard

The Chrome extension has received a significant enhancement! It now supports multiple "Connections" to Message VPNs making the examination of message payloads for different environments frictionless.

Check it out on the Chrome Store or directly from the Github repo.

Aaron

Hi @AllenW, just a bump/update from me on this specific issue: I've got a tool that has been upgraded to fix the exact issues you're complaining about w/ JMSToolBox… deep queues.

It's my terminal app PrettyDump: https://github.com/SolaceLabs/solace-pretty-dump

The problem I'm having is I have large queues and DMQs (some with 2M+ messages) and we are trying to determine the payloads of these messages to troubleshoot, and implement more data cleansing before the messages reach Solace. However, in queues so large I'm having difficulty filtering messages to the latest. Ideally I want the latest messages in the Queue to be examined, JMSToolbox shows messages from oldest to newest unfortunately..

So… there's no way for a queue browser to only "browse the end"… you have to scrub through all the messages. But I've added a feature (due to customer demand for this exact situation) where the browser will still pull down the messages but not display them, keeping the last n messages, and only dump them when the program terminates (i.e. Ctrl+C when you get to the end of your queue).

EDIT: I forgot to mention the PrettyDump argument to use this feature! It's --count=-100 which will keep the last 100 messages seen, and dump them when you quit the app. Use any negative number. Check docs here: https://github.com/SolaceLabs/solace-pretty-dump?tab=readme-ov-file#count

For 2M messages on a queue on my local broker, it took about 3 minutes to scrub through the whole queue.

And another feature that might be helpful: sometimes messages payloads are too big to display, and you just want to dump message header information: you can add an "indent" parameter of special code 00 to do this. I had a customer with some 1MB XML payloads, and JMSToolBox kept crashing.

PrettyDump supports Selectors, like JMSToolBox does… so if you know the Sender ID or some other piece of metadata on the message(s) you want, it can be a lot faster than browsing every single message off a queue. It also supports client-side filtering using a regex on any part of the payload as a convenience feature.

Note, unfortunately you can't use Selectors on the message spool ID (the one you see in PubSub+ Manager), or on the spooled time. Just on the message metadata listed here: https://docs.solace.com/API/API-Developer-Guide/Using-Selectors.htm

Hope this helps! Give it a try.

sjaak

@leonard I think it would be good to add a feature to the Solace message viewer that only displays the first X characters of the payload. This would prevent the Chrome browser from becoming unresponsive.

Aaron

Ok, here's another solution for browsing the back of a very big queue..! It uses Solace admin/SEMP copy feature, and the RGMID of each message which you can get from the queue's details.

Here, I have my queue with 2.8M messages on it:

solace1081b> show queue pq-shadow

Queue Name                     Messages      Spool             Bind Status
Message VPN                     Spooled  Usage(MB)   HWM (MB) Count I E A S D P
--------------------------- ----------- ---------- ---------- ----- -----------
pq-shadow
default                         2806771     268.94     268.94     0 U U E N D N

I start up PrettyDump, and create a temporary queue to use, but don't apply any subscriptions to it:

$ prettydump tq:

PrettyDump initializing...
PrettyDump connected to VPN 'default' on broker 'solace1081b' v10.8.1.152.

Creating temporary Queue.
Queue name: #P2P/QTMP/v:solace1081b/8edd8686-931c-4475-8a10-33cf3fdb9a72
No subscriptions added, I hope you're copying messages into this queue!

Starting. Press Ctrl-C to quit.

(NOTE: you could use SdkPerf for this as well. Also, you don't need to use a temp queue, any empty queue will work).

Then run the shell script I made called copy-last-n-msgs.sh in the scripts directory of the PrettyDump distribution. It uses SEMPv1 to query the RGMIDs of the messages at the back of your source queue, and then in a loop copy them one-by-one to the dest queue.

$ ./copy-last-n-msgs.sh
┌ This utility copies the last/newest n messages from one queue to another. ┐
│ It uses SEMPv1 to retrieve message details (RGMIDs) and SEMPv1 again to   │
│ copy each message one-by-one.  Requires curl and xmllint.                 │
└───────────────────────────────────────────────────────────────────────────┘
SEMP URL (e.g. https://aaron.messaging.solace.cloud:943): http://localhost:8080
SEMP username: admin
SEMP password:
Message VPN: default
Source Queue: pq-shadow
Dest Queue: #P2P/QTMP/v:solace1081b/8edd8686-931c-4475-8a10-33cf3fdb9a72  (copied from PrettyDump)
Number of messages from back of queue: 100

SEMP call successful. Copying messages...
Last 100 messages successfully copied from Queue 'pq-shadow'.

At which point PrettyDump shows all 100 messages. NOTE: the only drawback to this approach is that these are copies of the original messages, so their Message Spool ID and RGMID will be different. But at least you can see the topics/header/payload of the messages from the very back of a big queue without having to browse through all the messages!

Oh… another approach, if not obvious: if you have replay enabled/configured for your VPN, and it's setup to listen (via replay topic filter) to the topics that you are interested in, then you can just create a temp queue, add the topic subscriptions that match your source queue, and then initiate a replay on that queue going back only a couple minutes (or whatever time the last message was published to the queue).

AllenW

@Aaron - Thanks so much for sharing.

This is very useful! Awesome work.
We will be using it for sure