Client Ack using Apache Beam

naveenv712

Hi - I am using Apache Beam (2.30) with Solace connector. I receive messages from Solace, process it, save it and then wish to ack to Solace once I am done with processing. I explicitly sets the status of my POJO to true once saved. So for such successfully processed messages, I wish to ack.

However, I am facing issues as Spark executors are unable to see the status of message processed as CACHE of Solace messages may be maintained on a completely different node then the message processed on a different node. Is there any way to solve this?

Also I have noticed finializeCheckpoint is called at random where I wished to ack to Solace. Therefore I made ACK as pipeline step but flowReceiver closes the connection at random and unable to ACK as pipeline step.

[Deleted User]

Hi, I would like to know more.

finializeCheckpoint is the correct method to ack processed messages in the current window and you should not need to explicitly ack the message.

Can you explain what you mean by the statement "finializeCheckpoint is called at random"? Does this mean it is called at an undetermined window or at a time undetermined after a window has been processed? Also note, The windows are not serialized, meaning window "n+1" can be started before window "n" is completely ACKed. But the finalize for window "n" will be called before the finalize for window "n+1".