partition and offset

is there any spring samples which deal with partition , offset and grouping message concept in solace?

Answers

  • maniv
    maniv Member Posts: 10

    Hi I am new to solace, kindly let me know, do we have partition concept when payload size exceeds compare to message spool size.?

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 508 admin

    Hi Maniv! These sound very much like Kakfa questions. What are you trying to achieve? Are you looking for a way to send very large messages? Or how to load-balance among multiple consumers? Or store a large number of messages..? Let us know, thanks!

  • maniv
    maniv Member Posts: 10
    edited February 2020 #4

    @Aaron , Yes trying to co-relate those action activities..
    case :-
    Large message which need to be chunked with sequence and then need to be regrouped while consuming the same.

    Let me know if i am not clear in above statement. Looking for your great help.

  • TomF
    TomF Member, Employee Posts: 406 Solace Employee

    Hi @maniv, the first thing I'd say is "how large is large?" If you can get away without chunking, your life will be very much simpler. If you are looking to persist messages (i.e. you can't tolerate any loss) then the largest message PubSub+ will accept is 30MB. For direct messages (you can afford to lose messages, for instance if the subscriber is off-line) then the limit is 64MB.

    That said, there might be a case where you can't guarantee what the size of the message might be. In that case the application will need to chunk the message, just as it would with Kafka if it exceeds the Kafka message.max.bytes configurations. In this case there are a couple of things to remember about PubSub+ vs. Kafka:

    PubSub+ doesn't have the concept of topic partitions. Simply send everything on a (dynamic, hierachical) topic. Because the topics are hierarchical, you could even create an id for this chunked message and use that as the topic - then there is no way chunked messages from multiple publishers will get interleaved:

    • Publisher1 sends to large/message/1234
    • Publisher2 sends to large/message/1235
    • Subscriber subscribes to large/message/>, and knows that chunks from 1234 are not part of 1235 from the topic.

    Solace guarantees the ordering of messages on a topic at ingress to the broker. With the above this buys to a lot of what you need for free.

    That said, if you still need sequence IDs, just add them in a message header.

    If you need load balancing for the subscribers, you can shard messages based on the topic.

  • maniv
    maniv Member Posts: 10
    edited February 2020 #6

    @TomF thanks, correct me if i am wrong.

    • Chunk operation to be handled in producer side with sequence..
    • where subscriber part need to take care of regrouping those messages in same sequence basis.
  • TomF
    TomF Member, Employee Posts: 406 Solace Employee

    @maniv yes. Note that the limits are per message - there are separate limits (called quota) on the maximum size of messages held on a queue.