Subscribing to a subset of the fields on a chatty topic

djdk

Hello Solace Community,
I'm just starting out with Solace and I'm trying to figure out how topics work. Reading documentation I can see how to subscribe to topics and how to use wildcards but I don't see how to subscribe to a subset of a topic. I have a very chatty topic that sends out SDT maps and I'm only interested in a small number of very infrequently changing fields in the map. The software that connects to the topic currently needs to process each message even though 99+% of the time the fields I'm interested haven't changed. Is there a way to only subscribe to a small subset of a topic and only get pushed events when the fields I'm interested in have changed?
Thank you,
Allen

ChristianHoltfurth

Hi @djdk ,

Sorry, I misread your problem statement and thought you were looking to filter on fields that are not part of the topic. I'm updating my answer to reflect that you are looking for a way to only receive events where certain fields inside the message payload have changed.

In short, there's no easy way to do this with in-built features in the broker. Solace broker don't touch any message payload for performance reasons as stated in my original answer (below). So what you are looking for would entail that someone parses the message payload and flags the change of certain fields via the topic string.
I think there might be a path where the publisher (or a re-publisher) could do this and use some levels in the topic hierarchy to indicate the change, but I don't think this would result in any sort of elegant solution.

What I would rather recommend is that you ask the publisher to publish a slimmed down version of the full stream that would only contain the deltas (only the fields that where changed). This may require adopting another data format that allows to drop certain fields when the data hasn't changed like JSON (I don't think you can do this with SDTs).
This is actually a very common approach in financial services to distribute market data. To preserve bandwidth and processing capacity, instead of publishing a full data object every time, they have adopted the practice to publish delta messages, which will only contain the changed field values.

Hope this helps!

---

Old answer (preserving for continuity):
Solace brokers route and filter on topic strings and hierarchies by applying/matching each published topic with matching subscriptions and their wildcards. This is a highly optimised matching algorithm and the way it scales best.
There's no support for filtering on message content built into the brokers as this would require accessing and parsing each and every message, which imposes way lower scaling limits as it is a CPU intensive task.
I see three options for you to achieve the filtering you are looking for:

a) Expanding the topic by appending further topic levels with the content of the fields you are interested in filtering on.
b) Moving those fields into user properties, so you can apply selectors (I personally wouldn't recommend that approach as selectors are also very CPU intensive and will limit the scalability of the solution).
c) More of a variation to a. - If you publisher can't expand the topic and append the fields, have a microservice that listens to the original topic string and republishes to a finer grained topic.

It sounds like the topic you are publishing to is not detailed enough at the moment to support your use case, so extending that topic with additional fields is the best way to go.
Let me give an example to make it more tangible:
If your current topic is:

a/b/c

And you have three fields that you want to additionally filter on that are currently only in the body of the message, then have the publisher append the topic as follows:

a/b/c/field1/field2/field3

Any current subscribers should listen to :

a/b/c (the original topic) &
a/b/c/> (to catch any extensions, like the one described)

Your subscriber could then add tailored subscriptions to get only the specific values for field1, field2 and field3.

djdk

Hi Christian,
Thank you for your detailed and quick reply. Unfortunately I have no control over the publisher as the publisher is required to stream out all the market data it receives. Deltas would help with the capacity but really I want to write a small piece of code to monitor for changes in a subset of the market data that doesn't change all that often, so even deltas would need to receive, interpret and then either discard or use each tick. I understand that giving Solace itself the ability to identify a subset of a topic would require Solace to understand something about the format of the data inside the SDT map, whereas Solace wants to avoid any pre-suppositions (i.e. Map<String, Object>) to allow users to put anything they want in the map.

The solution you suggest to have an intermediate piece of code that takes in the full feed, caches the previously received value for these infrequently changing fields and rebroadcasts out to Solace only when a change occurs could work for me.

Most of the data I need to process is of the form Map<String, Map<String,Object>> that can be thought of as a table (like an excel sheet) with rows and columns, the first string (rows) are filterable using the topic name or wildcard but that second string (the columns) would be nice to be able to filter on as well.

Thanks again!
Allen

ChristianHoltfurth

Hi Djdk,
I think creating a new delta publisher micro service providing/republishing a specialised/filtered stream (should we call it a derivative stream? ) with only the fields you are interested in might be an interesting option to explore further.
With regards to the topic space, to maximise and make the best of Solace's filtering capabilities via topic wildcards, you are facing the challenge of trying to compress a multi dimensional map into a single dimensional topic space.
One way to approach this could be to break each row into a root topic followed by the column values (or an indicator for a changed value keeping the actual value in the payload of the message), like:

row1/columnA/columnB/columnC/...
row2/columnA/columnB/columnC/...
...

And so on. Not sure how big your two dimensional maps are, so this might get challenging, if they are very large.
Keep in mind that the max length for Solace topics is 250 bytes when designing your topic taxonomy.
Best regards,
Christian