Hi Giri,
I am adding bit more context to above question. Kirthi and I work together.
Context :
We are using processor function which consumes and publishes in batch. Method signature is as below. It publishes each message in batch as individual messages. TPS observed is around 150.
Function<Message<List>, Collection<Message<POJO>> > receiveMessage()
Issue :
Batch publishing is very slow. For a batch size of 255, it takes more than a second to publish.
Observation :
If the method signature is updated as below then the entire batch is published as a single message. TPS observed is around 1000. There is a huge performance difference.
Function<Message<List>, Collection<<POJO>> > receiveMessage()
Questions :
We are building a service to handle around 2000 TPS. The service is very simple, just consume, perform minor transformation and publish. Can you please suggest config options to optimise publish. With batch publish are there any config to avoid acknowledge or round trip for each message?
Thanks