Solace on Apache Beam with Java 2.25+ (SDF issue)

alexkoay
alexkoay Member Posts: 3 ✭
in General Discussions #1

Is there anybody using Solace with Apache Beam (non-Dataflow) with SDK 2.25 or later? It seems that there are some issues with how the new SDF Read interacts with Solace.

Comments

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #2

    Hi @alexkoay,

    Thanks for the heads up, I'm checking to see if there are any known issues. Can you share any more details around the issue that you're seeing? What is the error/exception?

  • alexkoay
    alexkoay Member Posts: 3 ✭
    edited June 2021 #3

    Sure thing, so the root of the problem is the new SplittableDoFn which sort of screws up the consumers.
    The pipeline will run, but there will be a few things that happen:

    1. The queue will never get exhausted. I've queued about 20k messages only for it to go down to 5k, and then a large amount of consumers will start getting connected and then eventually fail (due to #2).

    2. The large number of consumers actually cause a separate problem because of the SEMP query. Once it reaches about 130, SEMP replies with a limit-error (which means you're no longer getting any watermarking information). As an aside, what happens to very large pipelines with 130+ simultaneous workers in this situation? It seems untenable to not be able to getSplitBacklogBytes().

    3. There's also an error on a null msgSempBusUtil which seemingly occurs because start() was not called at the right time -- I've seen another thread on this (and I can confirm SEMP is enabled for me.)

    These all come from the new SDF Read that's been implemented.

    My main problem is actually that I'm trying to use this with Python via an external transform provider that I've created (it works in batching scenarios and all that) but these issues above are blocking it for a streaming pipeline. Dataflow has a native transform substituted when I use the Java transform directly that sidesteps this whole issue, but as I'm using it in an external transform, it doesn't perform the substitution.

    I've been talking to the Beam user mailing list as well here where you might find some additional details.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    edited June 2021 #4

    Thanks for that great info @alexkoay. Just a quick update that a member of our support team has reproduced similar behavior when using Solace Apache Bream connect running against Apache Beam Java SDK v2.25+ and are investigating

    I'll provide and update when I get one, but let us know if you figure it out as well :)

  • alexkoay
    alexkoay Member Posts: 3 ✭
    #5

    Definitely, I've made some progress today (which I wrote about on the Beam thread). It feels like I'm almost at the answer already. Let me know if anybody should be CC-ed in that thread as well.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #6

    Hey @alexkoay, thanks again for bringing this to our attention. It sounds like you already got more info via support, but in case others come across this thread I wanted to share that the Solace Apache Beam connector is currently not interoperable with the new Splittable DoFn wrapper introduced in Apache Beam SDK 2.25+. It is now on our teams' radar and we plan on resolving the issue in a future release.

    Also for anyone that comes across this in the future, just a note that our Apache Beam connector is open source and we encourage contributions. Let us know what you find/want via an issue (or in this community ;) ) or fork the project and make a PR with suggested changes :)

    https://github.com/SolaceProducts/solace-apache-beam

Categories

This Month's Leaders

This Week's Leaders