Client Ack using Apache Beam
Hi - I am using Apache Beam (2.30) with Solace connector. I receive messages from Solace, process it, save it and then wish to ack to Solace once I am done with processing. I explicitly sets the status of my POJO to true once saved. So for such successfully processed messages, I wish to ack.
However, I am facing issues as Spark executors are unable to see the status of message processed as CACHE of Solace messages may be maintained on a completely different node then the message processed on a different node. Is there any way to solve this?
Also I have noticed finializeCheckpoint is called at random where I wished to ack to Solace. Therefore I made ACK as pipeline step but flowReceiver closes the connection at random and unable to ACK as pipeline step.