Error occurred for Solace queue depth request message: 0 while reading from Solace using dataflow

Options
smart4_one
smart4_one Unconfirmed, Member Posts: 4

Hi,

I am trying to stream messages from Solace to BigQuery using DataFlow where I am using the pubsub+ IO connector for Apache Beam provided in the below github repo. https://github.com/SolaceProducts/solace-apache-beam

I am able to read the messages but also getting below errors which is due to failures in reading the backlogs from the queue. Based on the queue depth dataflow workers need to scale up which is NOT happening. The user account I am using has subscribe access in the queue. I checked with the Solace team and "Show SEMP Queue" is enabled, Solace team is able to make below SEMP queries too in the same queue and vpn.

"<rpc semp-version=\"%s\"><show><smrp><subscriptions><vpn-name>%s</vpn-name><client></client><topic></topic><topic-str>%s</topic-str></subscriptions></smrp></show></rpc>",

Errors:

warning: Error Response (403) - Publish ACL Denied - Topic '#SEMP/<host router>/SHOW'

 Error: Error occurred for Solace queue depth request message: 0

 Notice: Exception occurred in message producer notification handler

 Notice: Failed to execute message bus SEMP request, restarting Requestor: Timeout occurred performing request

 Error: Encountered a JCSMPException querying queue depth

 Error: getSplitBacklogBytes() unable to read bytes from: q/<the solace queue name>

Just to read the queue depth why the user ACL need publish access? There may be many consumers to the queue where for security purpose they will not be given publisher access. As per my analysis the error is occurring from MsgBusSempUtil.java. function -> queryRouter() in the below line:

return requestor.request(requestMsg, 5000, sempShowTopic);
where requestMsg is BytesXMLMessage having query string attached as : String.format("<rpc><show><queue><name>%s</name><vpn-name>%s</vpn-name></queue></show></rpc>",
      queueName, vpnName)
and sempShowTopic:
final String sempShowTopicString = String.format("#SEMP/%s/SHOW", routerName);
sempShowTopic = JCSMPFactory.onlyInstance().createTopic(sempShowTopicString);

Please help how to resolve this issue. Thanks in advance.

Answers

  • smart4_one
    smart4_one Unconfirmed, Member Posts: 4
    Options

    @KenBarr Can you please help?

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 920 admin
    edited September 2022 #3
    Options

    Hi @smart4_one,

    Ken is OOO right now but I'll ping some folks and see if we can get you an answer :)

    If you have support I would also open a support ticket and link to this post.

  • Dennis Brinley
    Dennis Brinley Member, Employee Posts: 3 Solace Employee
    Options

    Hi @smart4_one - When you instantiate MsgBusSempUtil, are you using the constructor:

    MsgBusSempUtil(JCSMPSession jcsmpSession)

    or are you setting up with MsgBusSempUtil( jcsmpSession, createProducer=false, createConsumer=true) ?

    The first form defaults both createProducer and createConsumer to true

  • smart4_one
    smart4_one Unconfirmed, Member Posts: 4
    Options

    It's working with MsgBusSempUtil(JCSMPSession jcsmpSession), However when I tried with this MsgBusSempUtil( jcsmpSession, createProducer=false, createConsumer=true)  I got different error:

    Error: Encountered a JCSMPException querying queue depth

    jsonpayload: {com.solacesystems.jcsmp.InvalidOperationException: No producer to perform operation

    at com.solacesystems.jcsmp.impl.requestreply.RequestorImpl.validateCanRequest(RequestorImpl.java:36)

    at com.solacesystems.jcsmp.impl.requestreply.RequestorImpl.request(RequestorImpl.java:62)

    at com.solace.connector.beam.MsgBusSempUtil.queryRouter(MsgBusSempUtil.java:125)

    at com.solace.connector.beam.MsgBusSempUtil.queryRouter(MsgBusSempUtil.java:133)

    at com.solace.connector.beam.UnboundedSolaceReader.queryQueueBytes(UnboundedSolaceReader.java:315)

    at com.solace.connector.beam.UnboundedSolaceReader.getSplitBacklogBytes(UnboundedSolaceReader.java:330)

    at org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.flushState(StreamingModeExecutionContext.java:441)

    at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1442)

    at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:165)

    at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1113)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

    at java.lang.Thread.run(Thread.java:748) }

    Error: getSplitBacklogBytes() unable to read bytes from: {queue}

    Exception: "Failed to execute message bus SEMP request, restarting Requestor: No producer to perform operation"

  • smart4_one
    smart4_one Unconfirmed, Member Posts: 4
    Options

    @marc Yes I do have support. Can you please open a support ticket?

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 920 admin
    Options

    Hi @smart4_one,

    In order to open a support ticket go ahead and shoot an email over to support@solace.com from your company email address. This way you own the ticket, can mark it as resolved, etc.