Scalable solution for republish a message WITHOUT the payload
For analysis purposes, we want to consume messages and republish them on a new queue without the payload (only the topic that is was published on and the header/properties info do we need). What can be a scalable solution for this? The environment can be on-prem, AWS/Google/Azure cloud so need to run everywhere 
any experiences here?
Comments
-
Hi @PimvdNoll,
So I think you have 2 options here:
1. If these events are already published to topics just create a separate queue that subscribes to the topics of interest and do your analysis on the already published events. But I assume you don't want to do this b/c of payload security reasons or just needing to have a queue large enough to hold the larger message sizes with the payload?
2. Create a simple processor app that listens on a queue which is subscribed to the topics you care about. It will then gather the info you want from that message and publish a new message to a different topic with that info. Create a queue that subscribes to the new topic(s) and you're good to go. Note that if you care about header info like the original message id, expiration time, etc those headers would be different in the new message so you'd either have to map them to a different custom property or maybe just store the info you want in the payload.What are you looking at from a scalability perspective? Like expected msg/sec? An app like this should be relatively simple to code using any of our APIs so you could choose from Java, Python, JavaScript, Spring, etc and probably deploy anywhere using a docker container. Easiest (from my perspective) would probably do using Spring Cloud Stream but that might not meet your performance needs.
Side note: we have a cool solace topic explorer tool that might help with analyzing your topics. It's a bit tricky to find right now as we're about to release Solace Discovery Beta in ~10 days but I'll try to create a screen capture to demonstrate how you can try it out and see if it will be useful for you
1 -
Hey @PimvdNoll I created this quick video to show how you can use the Solace Topic Hierarchy Explorer (originally created by @Aaron!) to analyze your topics as well. It could be useful for what you want to do
2 -
As of today, the broker can't do this (strip out the payload). Maybe one day! But yeah, I'd also recommend building a little "processor" app that builds a new message for every one it receives, copies over the properties and everything, and then republishes to a different topic. Then your other queue for analytics can subscribe to those topics.
1 -
Just a tip @PimvdNoll if you go the Spring Cloud Stream route be sure to use the target destination header way of sending to dynamic topics. There will be less overhead than using StreamBridge. A quick example of how to do it is here. Also I don't believe Spring allows for a null payload so you'd just have to make it an empty string.
@Aaron I know you're creating some new Java + JCSMP samples. When you have a GuaranteedProcessor for each it would be cool to link that here. I'd definitely be curious to see the performance difference of JCSMP vs. New Java API vs. Cloud Stream for something like this!
1
https://youtu.be/2lBWnJ4A39E
