Solace now supports Payload Compression!

Aaron · August 6

Hi team, just wanted to put a note out there about a new feature Solace is rolling out across all our APIs. And that's the option of being able to automatically compress/decompress message payloads within the API.

As of this moment, it's supported in JCSMP, C, JS/Node, and Python, with more APIs coming soon (check here). For example JMS requires a broker-side change to expose this setting as part of the JNDI Connection Factory.

It works using the deflate compression method, and when sending a message compressed, the API will also update the HTTP Content Encoding to indicate this: deflate. To enable sending messages with compressed payloads, you simply add a Session property:

properties.setProperty(JCSMPProperties.PAYLOAD_COMPRESSION_LEVEL, 9);

Valid values are between 1..9.

Note that every application receiving a compressed message will need to be using an API that supports compression/decompression, otherwise a consumer will just see a bunch of compressed bytes instead of the actual payload. So using this feature requires upgrading all applications to support it.

The biggest advantage of using this feature vs. our existing streaming compression on the TCP link is that messages stored inside the broker (i.e. Guaranteed messages on the message spool) will take up less space.

See the docs for more details: https://docs.solace.com/API/API-Developer-Guide/Message-Payload-Compression.htm

dreamoka · August 11

does it work for amqp protocol ?

Aaron · August 17

Hi there @dreamoka, sorry for the delay, I had to confirm some stuff with R&D and do some testing myself.

So the short answer is: kind of. More below.

This feature was primarily built for SMF-to-SMF applications, specifically for 3rd-party JMS apps that don't really do "coding" and more just set properties on their connections. Implementing payload compression yourself in code is actually pretty trivial… this is for apps that are very hands-off!

So: an AMQP (or MQTT) client receiving a message from a payload-compressed-enabled SMF client, it would be possible for it to simply check the Content Encoding is deflate and then inflate the payload using standard mechanisms. For example, it would look something like this in Java:

Inflater inflater = new Inflater();
ByteArrayOutputStream os = new ByteArrayOutputStream();
try {
    inflater.setInput(bytes);
    byte[] buffer = new byte[1024];
    while (!inflater.finished()) {
        int decompressedSize = inflater.inflate(buffer);
        os.write(buffer, 0, decompressedSize);
    }
    bytes = os.toByteArray();
} catch (DataFormatException e) {
    logger.warn("Had a message marked as 'deflate' but wasn't");
} finally {
    try {
        os.close();
    } catch (IOException e2) { }
}

However, the other direction is not possible… having an AMQP client sending a compressed payload (even with the correct Content Encoding set) and have it automatically inflated won't work. There are some magic bits set by the API that it needs to automatically inflate. But, the receiving app could do the exact same checks above and inflate it manually.

Again, implementing payload compression yourself isn't hard…. just ~15 lines of code. This feature was mainly introduced for those customers that don't like coding.

dreamoka · August 21

I don't mind coding. I am thinking what is the best practice to send large file (more than 200 mb) via amqp using solace.

Based on my understanding, Apache ActiveMQ Artemis supports sending and receiving of huge messages. I am curious whether solace support sending huge messages.

Aaron · August 21

Our max is 30MB for Guaranteed. We have a file splitting connector for chunking up larger files to send over Solace. Or, if you don't mind coding… 😉

dreamoka · August 22

Interesting.. Do you have any sample for the file splitting connector ??

forward100 · August 25

Nice. Approx. 30 MB message is compressed to 28 KB with API compression level 9. Using network compression with 55003 it's also approx. 28 KB.

Aaron · September 5

@dreamoka checked internally, and apparently the file splitting connector will not be open-source. It's getting close to release though, so keep an eye out here for at least a way to trial/demo it. It's part of our broader "File Mesh" strategy for helping make MFT more event-driven.

@forward100, welcome to the Community! I'm impressed you got such good compression, must be very repetitive JSON or XML or something? I'm wondering though: were you comparing the API payload compression feature with the streaming connection compression, or were you trying to use both together and not seeing any additional improvement? If the latter, it makes sense that you wouldn't see much improvement as the payload is already compressed, and only the message header (which is mostly binary) would be available to be compressed, which probably wouldn't do much of anything.

leonard · October 4

Will this feature be added to the Solace connector in Boomi? I could see a few good use cases for something like this 👍️

Aaron · October 16

Hey @leonard. I'm not a Boomi expert, so can't say for certain when this would be added… but it's literally as easy as updaing the JCSMP JAR to at least 10.24.0, and adding a single line of code to the JCSMPProperties before connection. That's it.

🎄 Happy Holidays! 🥳

Happy Holidays!

Solace now supports Payload Compression!

Comments

Categories

This Month's Leaders

This Week's Leaders