Getting getSplitBacklogBytes() unable to read bytes error after draining Streaming Dataflow job

abhitej
abhitej Member Posts: 2 ✭
edited April 2021 in General Discussions #1

I am trying to read messages from solace queues through solace io connector and then writing them to GCS bucket , i was able to read and write messages to GCS as it was streaming job i need to drain it so when i drain the Streaming dataflow job getting below error
error:
getSplitBacklogBytes() unable to read bytes from: queue name

Encountered a Parser Exception querying queue depth: java.lang.NullPointerException: Failed to evaluate /rpc-reply/rpc/show/queue/queues/queue/info/current-spool-usage-in-bytes in <?xml version="1.0" encoding="UTF-8" standalone="no"?>SEMP over Message Bus is disabled for commands published on #SEMP topic 'SHOW'
what might the reason for this error?

Tagged:

Comments

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #2

    Hi @abhitej,
    Can you share more info around what solace io connector you're using? Also out of curiosity why are you querying queue depth via a SEMP command? In general you usually want to register a listener in an event-driven paradigm and then your app gets messages pushed to it when available. This way you shouldn't need to poll to see if messages are available.

    If for some reason you really need to poll using that SEMP command you'll need to enable show commands. You'll find more info in this docs section: https://docs.solace.com/SEMP/Using-Legacy-SEMP.htm#Configur
    Note the sub-section "Allowing Access to Show Commands"

  • nram
    nram Member, Employee Posts: 80 Solace Employee
    #3

    Hello @abhitej , Are you using this BEAM I/O Connector ? This requires "show semp" commands to be enabled. (see below from the documentation)
    https://github.com/SolaceProducts/solace-apache-beam#allow-apache-beam-to-detect-message-backlog-to-scale-workers

    Allow Apache Beam to Detect Message Backlog to Scale Workers
Apache Beam uses the message backlog as one of its parameters to determine whether or not to scale its workers. To detect the amount of backlog that exists for a particular queue, the Beam I/O Connector sends a SEMP-over-the-message-bus request to the broker. But for it to be able to do this, show commands for SEMP-over-the-message-bus must be enabled.

    Are you running broker on your container env (like docker) or using Solace Cloud service?

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #4

    Note that you can also toggle SEMP over Msg Bus options in PubSub+ Manager (WebUI) under advanced settings on the message VPN.

  • abhitej
    abhitej Member Posts: 2 ✭
    #5

    Hi marc thanks for your reply will try the suggested approach.. I was using java connector and draining job as i want to only run it for some time though its a streaming job

  • nram
    nram Member, Employee Posts: 80 Solace Employee
    #6

    Going by the principle of least privilege, you may want to enable only "Show Commands" in the screen shot provided by @marc :-)
    If you are on Solace Cloud, you need to enable SEMP under Cluster Manager/Manage/Advanced Options.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #7

    Yes, great point @nram. Only enable "show" commands initially and enable others as needed :)

Categories

This Month's Leaders

This Week's Leaders