Solace now supports Payload Compression!

Aaron
Aaron Member, Administrator, Moderator, Employee Posts: 595 admin
edited August 17 in Tips and Tricks #1

Hi team, just wanted to put a note out there about a new feature Solace is rolling out across all our APIs. And that's the option of being able to automatically compress/decompress message payloads within the API.

As of this moment, it's supported in JCSMP, C, JS/Node, and Python, with more APIs coming soon (check here). For example JMS requires a broker-side change to expose this setting as part of the JNDI Connection Factory.

It works using the deflate compression method, and when sending a message compressed, the API will also update the HTTP Content Encoding to indicate this: deflate. To enable sending messages with compressed payloads, you simply add a Session property:

properties.setProperty(JCSMPProperties.PAYLOAD_COMPRESSION_LEVEL, 9);

Valid values are between 1..9.

Note that every application receiving a compressed message will need to be using an API that supports compression/decompression, otherwise a consumer will just see a bunch of compressed bytes instead of the actual payload. So using this feature requires upgrading all applications to support it.

The biggest advantage of using this feature vs. our existing streaming compression on the TCP link is that messages stored inside the broker (i.e. Guaranteed messages on the message spool) will take up less space.

See the docs for more details: https://docs.solace.com/API/API-Developer-Guide/Message-Payload-Compression.htm

Comments

  • dreamoka
    dreamoka Member Posts: 53 ✭✭✭

    does it work for amqp protocol ?

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 595 admin
    edited August 18 #3

    Hi there @dreamoka, sorry for the delay, I had to confirm some stuff with R&D and do some testing myself.

    So the short answer is: kind of. More below.

    This feature was primarily built for SMF-to-SMF applications, specifically for 3rd-party JMS apps that don't really do "coding" and more just set properties on their connections. Implementing payload compression yourself in code is actually pretty trivial… this is for apps that are very hands-off!

    So: an AMQP (or MQTT) client receiving a message from a payload-compressed-enabled SMF client, it would be possible for it to simply check the Content Encoding is deflate and then inflate the payload using standard mechanisms. For example, it would look something like this in Java:

    Inflater inflater = new Inflater();
    ByteArrayOutputStream os = new ByteArrayOutputStream();
    try {
        inflater.setInput(bytes);
        byte[] buffer = new byte[1024];
        while (!inflater.finished()) {
            int decompressedSize = inflater.inflate(buffer);
            os.write(buffer, 0, decompressedSize);
        }
        bytes = os.toByteArray();
    } catch (DataFormatException e) {
        logger.warn("Had a message marked as 'deflate' but wasn't");
    } finally {
        try {
            os.close();
        } catch (IOException e2) { }
    }
    

    However, the other direction is not possible… having an AMQP client sending a compressed payload (even with the correct Content Encoding set) and have it automatically inflated won't work. There are some magic bits set by the API that it needs to automatically inflate. But, the receiving app could do the exact same checks above and inflate it manually.

    Again, implementing payload compression yourself isn't hard…. just ~15 lines of code. This feature was mainly introduced for those customers that don't like coding.

  • dreamoka
    dreamoka Member Posts: 53 ✭✭✭

    I don't mind coding. I am thinking what is the best practice to send large file (more than 200 mb) via amqp using solace.

    Based on my understanding, Apache ActiveMQ Artemis supports sending and receiving of huge messages. I am curious whether solace support sending huge messages.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 595 admin

    Our max is 30MB for Guaranteed. We have a file splitting connector for chunking up larger files to send over Solace. Or, if you don't mind coding… 😉

  • dreamoka
    dreamoka Member Posts: 53 ✭✭✭

    Interesting.. Do you have any sample for the file splitting connector ??

  • forward100
    forward100 Member Posts: 1

    Nice. Approx. 30 MB message is compressed to 28 KB with API compression level 9. Using network compression with 55003 it's also approx. 28 KB.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 595 admin

    @dreamoka checked internally, and apparently the file splitting connector will not be open-source. It's getting close to release though, so keep an eye out here for at least a way to trial/demo it. It's part of our broader "File Mesh" strategy for helping make MFT more event-driven.

    @forward100, welcome to the Community! I'm impressed you got such good compression, must be very repetitive JSON or XML or something? I'm wondering though: were you comparing the API payload compression feature with the streaming connection compression, or were you trying to use both together and not seeing any additional improvement? If the latter, it makes sense that you wouldn't see much improvement as the payload is already compressed, and only the message header (which is mostly binary) would be available to be compressed, which probably wouldn't do much of anything.

  • leonard
    leonard Member Posts: 10 ✭✭

    Will this feature be added to the Solace connector in Boomi? I could see a few good use cases for something like this 👍️