Consuming messages from solace queue but not writing to GCS

sachingupta

I am working on java pipeline code to consume messages from solace queue and write it to GCS. With the same code I was able to do the same but from past 3 days with the same code I am only able to consume messages but it is not writing to GCS.
Today I ran the code again so it is not consuming messages from solace and writing to GCS but after some time with the same code it is just consuming messages but not writing to the GCS.

TomF

Hi @sachingupta , could you give us some more details, please? When you say a pipeline, is this in Beam or some other framework? Are you using a Google Cloud Function? When this is happening, what does the queue screen in PubSub+ Broker Manager say: are there any unacknowledged messages?

sachingupta

SO, I am using Beam pipeline in java whose job is to consume messages from solace queue and after consuming messages we are converting it to PCollection then we are writing this file to the GCS Bucket with .json extension.

So the problem is our code was working fine earlier with 3 different queues, but from past 4 days with 2 of the queue it was only consumuing messages but was not writing it to GCS bucket.
On the console we are getting messages like messageRecevied: lets say 10, messagesCheckpointReady:10,messageCheckpointComplete:0 .............. etc
then after some time
messageRecevied: 0 messagesCheckpointReady:0,messageCheckpointComplete:0 .............. etc
that means we have consumed messages from the queue but it is suppose to write it into GCS bucket which was working fine earlier but now it is not writing.
And we are running our code using direct runner on cloud shell

TomF

Hi Sachin, I'm working for an answer for you - I'll get back to you as soon as I can.