How to use streamset connectors ?

ProsenjitDas

Hi @marc

The following connectors are not available under Solace pub sub cloud ( trial version). Can you please suggest how to configure the below 2 connectors ? Is there any documentation for connecting below 2 streamsets connector from solace ?

Google Pub/Sub Subscriber
Google Pub/Sub Producer

I have checked the doc and found the above 2 are vendor licensed with partner integrated. Do you have any documentation like to how to configure the same from SOLACE broker ?

https://docs.streamsets.com/portal/#datacollector/latest/help/datacollector/UserGuide/Destinations/PubSubPublisher.html

ProsenjitDas

Hi @marc

Can you please check my above note and suggest ?

hong

Hi @ProsenjitDas, we just published a new blog post about Solace and StreamSets by @himanshu. You may want to take a look: https://solace.com/blog/change-data-capture-solace-streamsets-dr/. @himanshu, can you answer the above question?

ProsenjitDas

@himanshu
Can you please guide if Solace REST Consumer capable of publishing message to Google pub Sub topic over the API? I have tried few options from message broker using REST but still not succeeded. How to pass the below URL from RDP configuration of the message broker ? Any help will be highly appreciated.

The below is my google Pubsub API url:

"https://pubsub.googleapis.com/v1/projects/{project-name}/topics/{topic-name}:publish"

Message:
{
"id": "1",
"price": "$1.99",
"product": "shirt"

    }

himanshu

@ProsenjitDas

You should be able to get it to work with REST. You need to ensure you have set up an RDP and REST Consumer. Can you please explain what steps you took and what your configuration looks like and finally, what error are you getting so we can help debug?

I wrote this post earlier which explains how to use Solace's REST feature to integrate with AWS Lambda. You can have a look at it for reference. Ignore the lambda part but the REST stuff should be applicable.

ProsenjitDas

Hi @himanshu

I have added the detailed configuration in the attachment. I have not seen any REST consumer error but messages are not delivered from Solace Queue to Google pubsub Topic.

Initially I have started by adding google apis oauth token/secrets to REST Consumer under REST connectors but it's failing to connect pubsub endpoint but same is working now without using any authentication scheme. Both cases messages are not delivered yet .

Note : I have already added google trust CA certificate in solace from https://pki.goog/repository/

himanshu

@ProsenjitDas

Hmm...according to RDP's stats, the REST consumer is not up. Can you share screenshot of REST Consumer's stats/details? Trying to see if there is any error message there which can help us troubleshoot.

@Aaron - any ideas what might be happening here?

ProsenjitDas

@himanshu @Aaron
attached screenshot of REST Consumer's stats/details. Please have a look. Kindly note that Rest Consumer is working fine when I am using Authentication Scheme as Http Header by sending Bearer token in the header. The oauth 2.0 client credentials /oauth JWT is not working while using google service account for calling google Pub sub API.

Can you please share if there is any specific treatment required after getting the key files from GCP service account ?

I have added roles/policy to the service account for using google pub sub as producer and consumer with admin role.