Publish / Subscribe - REST Way

[jaikrattariyal]

I have been following article Publish / Subscribe for long time but not really able to get it 100% working.

Below are some of the issue I am confused with

1. Operational state of my RDP, Rest Consumer and Q is down but they are enabled.
2. Node Server is up and running and below both the commands are working fine.

Not able to differentiate whats the difference

curl -X POST http:localhost:8089/T/rest/pubsub \
-d "Hello World REST" \
-H "content-type: text" \
-H "Solace-delivery-mode: direct" \
--user solace-cloud-client:mypassword

and

curl -X POST http:localhost:8089/T/rest/pubsub \
-d "Hello World REST" \
-H "content-type: text" \
-H "Solace-delivery-mode: direct"  

3. Third point is, please correct my understanding. In this setup, messages can be posted to broker in REST way and once received at broker end they will be passed to registered rest customers like in the example rc1 is the registered customer
4. What is POST Request Target i.e. /rest/tutorials
5. Can that node server be replaced with one Spring Boot Application. If yes, do I need to expose any end point in it?

Thanks

[jaikrattariyal]

Sorry for the typo – its http:**//**localhost:8089/T/rest/pubsub

[jaikrattariyal]

Also below command

curl -X POST https://mr8bk5ki6t2uk.messaging.solace.cloud:9443/T/rest/pubsub \
-d "Hello World REST" \
-H "content-type: text" \
-H "Solace-delivery-mode: direct" \
--user solace-cloud-client:mypassword

is showing nothing on console.

[himanshu]

Hi @jaikrattariyal ,

To give you some background, Solace PubSub+ broker has native support for REST. From a publisher’s perspective, you can run a CURL command (like the one you have in your original post) and it will publish message to a topic. This topic can be mapped to a queue so that message can be enqueued.

From a consumer’s perspective, PubSub+ supports webhooks via RDP (rest delivery endpoints). By creating RDPs, you basically tell the broker which REST endpoint the message should be pushed to when it is enqueued in a queue.

What is POST Request Target i.e. /rest/tutorials

This is the endpoint which will be invoked every time a message is enqueued in your queue. This should be the endpoint your webservice exposes.

Have a look at this blog post I wrote recently which shows you how to achieve this:

Demo: hybrid architecture with RESTful and streaming APIs – a bit deployed

[jaikrattariyal]

Thanks Himanshu… Let me try that.

[marc]

Hi @jaikrattariyal,

To troubleshoot why the initial tutorial is working you should be able to see a last failure reason on the “stats” page of the connector in PubSub+ Manager. I’d be curious what that says after you walked through the tutorial.

Note that I have opened an issue to update that tutorial to use PubSub+ Manager instead of CLI.

Hope that helps
-Marc

[jaikrattariyal]

Thanks @Marc… All my components (RDP, BQs, Qs, RCs) are UP and enabled and I am able to hit some of postman-echo.com endpoints as well. I am following _Event-Enable your REST Architecture with Solace PubSub+, Part 1. But the same way when I deployed my API in heroku, I am not able to see the response when I am hitting my pubsub localhost URL. I am able to hit the same heroku endpoint via my postman or browser.

Curl command is giving me blank on console - _curl -u default:default 'http://localhost:9000/rest/tutorials_
On browser its saying - Not Found
Heroku endpoint - https://spring-solace-consumer.herokuapp.com/rest/tutorials

Under Stats, I am seeing “No REST Queue Bindings Up”…but its up as of now.

I have one doubt, as following above mentioned URL, I am putting PORT as 80 for postman-echo.com but not sure what to mention when I am hitting Heroku endpoint. Since its mandatory so cant leave it blank.
For time being I mentioned port which is visible in console of heroku console when application is deployed (i.e. Tomcat started on port(s): 32411 (http) with context path)

[jaikrattariyal]

Hello @Marc, I was following _Event-Enable your REST Architecture with Solace PubSub+, Part 1 steps and done with my basic setup for pubsub broker as REST way but still when I am trying to setup my own endpoint, that is not working properly. I setup two APIs in Heroku and created two RCs. They are up and running but when I am trying to hit them through broker way like below, its not working. Response is nothing on console.

curl -u default:default ‘http://localhost:9000/rest/tutorials’

Endpoint1 - https://spring-solace-consumer.herokuapp.com/rest/tutorials
Emdpoint2 - https://spring-solace-consumer1.herokuapp.com/rest/tutorials

I have one doubt, While following up the link given above, I tried hitting postman-echo.com endpoints and there I had mentioned port as 80 but not sure what to mention when I am hitting Heroku exposed URL. Though I did mention port which is visible in console of heroku.

As per your your screenshot, I am seeing

Operational State. Up
Last Failure Reason. No REST Queue Bindings Up
Last Failure Time 2021-07-27 09:57:15
Client Name. #rdp/rdp_get
Time Connections Blocked (%) 0

Thanks

[jaikrattariyal]

Sometimes console hit gives me nothing and sometimes its timeout with message “Timed out waiting for reply after 30415ms; Solace-Reply-Wait-Time=30000ms”

[jaikrattariyal]

@himanshu said:
Hi @jaikrattariyal ,

To give you some background, Solace PubSub+ broker has native support for REST. From a publisher’s perspective, you can run a CURL command (like the one you have in your original post) and it will publish message to a topic. This topic can be mapped to a queue so that message can be enqueued.

From a consumer’s perspective, PubSub+ supports webhooks via RDP (rest delivery endpoints). By creating RDPs, you basically tell the broker which REST endpoint the message should be pushed to when it is enqueued in a queue.

What is POST Request Target i.e. /rest/tutorials

This is the endpoint which will be invoked every time a message is enqueued in your queue. This should be the endpoint your webservice exposes.

Have a look at this blog post I wrote recently which shows you how to achieve this:
Demo: hybrid architecture with RESTful and streaming APIs – a bit deployed

Hello @Himanshu,
Not sure why but I am not able to open link given by you due to some security reason.

Anyways, I am able to bring my RD, BQ, RCs and Qs up successfully. But when I am adding any new RC with my own API… Its not working… Please check my comments above in reply of @marc

FYI, I am running this POC as Gateway Service Mode.

[Aaron]

hi @jaikrattariyal . How’s it going? I wrote that REST Event-Enable CodeLab… hope I can help!

If you’re still having error messages about “no REST queue bindings up”, that means your RDP is not connecting to a Solace queue correctly, the queue is needed for buffering messages before being sent out the RDP. You need to create the queue, makes sure you enable it, then in your RDP Queue Binding config, specify that queue name. Also, make sure you add your subscriptions to the queue to attract the MicroGateway topics, e.g. GET/>.

One thought: if running the Solace broker locally as Docker container, you can’t use “localhost” as the RDP Consumer remote host… you need to use your machine IP address. Even though localhost works on your browser (hopefully), the Docker container has different networking setup.

Also, did you see this? Handy Node mini server for testing: Quick-and-dirty REST server for RDP testing — Solace Community

The POST Request Target you don’t need to worry about in Gateway mode, that’s only for Messaging mode. In the PubSub+Manager GUI, you can see on the right-hand side of the screen a “Tip”:

Post Request Target
The request-target string to use when sending requests. It identifies the target resource on the far-end REST Consumer upon which to apply the request. There are generally two common forms for the request-target. The origin-form is most often used in practice and contains the path and query components of the target URI. If the path component is empty then the client must generally send a “/” as the path. When making a request to a proxy, most often the absolute-form is required. This configuration is only applicable when the Message VPN is in REST messaging mode.

So on current local setup, I have Ubuntu in WSL2, I start the Node REST test server above, binding to “172.28…”, and then use the same IP address in my RDP REST Consumer configuration. The REST test server is listening on port 9090, and my PubSub+ broker is using the default REST port 9000. So if I go into my Chrome browser and hit http://localhost:9000/test then it gets routed to the test server on port 9090, and I see this on the console:

Received message: GET /test at 16:55:43
  HEAD:host = 172.28.123.115:9090
  HEAD:content-length = 0
  HEAD:solace-correlation-id = ID:Solace-f814b095ab26879d
  HEAD:solace-delivery-mode = Non-Persistent
  HEAD:solace-message-id = ID:Solace-f814b095ab26879d
  HEAD:solace-reply-wait-time-in-ms = FOREVER
  HEAD:solace-time-to-live-in-ms = 30000
  HEAD:sec-ch-ua = " Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"
  HEAD:sec-ch-ua-mobile = ?0
  HEAD:upgrade-insecure-requests = 1
  HEAD:user-agent = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
  HEAD:accept = text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
  HEAD:sec-fetch-site = none
  HEAD:sec-fetch-mode = navigate
  HEAD:sec-fetch-user = ?1
  HEAD:sec-fetch-dest = document
  HEAD:accept-encoding = gzip, deflate, br
  HEAD:accept-language = en-GB,en-US;q=0.9,en;q=0.8
  HEAD:cookie = _ga=GA1.1.567890820.1615794484; fs_uid=rs.fullstory.com#13AXQ4#6077823464579072:4684159190376448#0ef8131a#/1656476679; ajs_user_id=d071d4c2-e717-41ac-b344-3ccd9f8d10cd; ajs_anonymous_id=632237e8-ab12-4e22-9036-37b493ca8bc6; TSID=d4b280e5f5850a48
  BODY:

Hope that helps!!

[jaikrattariyal]

Thanks @Aaron… Thanks for commenting …Article is good to use… I raised one PR to fix one minor typo there… Please approve here _https://github.com/SolaceDev/solace-dev-codelabs/pull/149_

1. All my RDP, RCs, BDs and Qs are showing green and UP,
2. Yes, I am clear on POST Request Target… Missed to read the TIP earlier,
3. Yes, I do have GET/rest/tutorials subscription in my Queue. I notice that if I remove this subscription then error I get error No subscription matching topic “GET/rest/tutorials”
4. And once I add it back, its timeout.
5. localhost:9000 hit is reaching to service broker… I can see my messages in Try Me section if I add subscription on topic GET/rest/tutorials

Below the text I notice there.

Delivery Mode: Direct
Time To Live (sec): 30
Reply To: [Topic #P2P/v:9c11a2f55b5d/_rest-95a7b3d477d4abdb/GET/rest/tutorials]
Correlation Id: ID:Solace-5bb05e5ff567a017

Not getting reply of what I asked earlier…While following up the link given above, I tried hitting postman-echo.com endpoints and there I had mentioned port as 80 but not sure what to mention when I am hitting Heroku exposed URL. Though I did mention port which is visible in console of heroku.

Thanks

[jaikrattariyal]

Hello @Aaron @marc , My locally installed, docker way, pubsub broker is not able to talk to Heroku API but when I deployed my same spring boot API locally (in docker) then its connecting with them and I am able to hit them via Solace broker’s REST endpoint. i.e. localhost:9000/endpointname

But the things that I am confused with are following

1. I dont get response always. Sometimes its timeout and sometime its blank response.
2. When I am starting multiple containers, from same spring boot image, on different-2 ports… they all are not getting requests from broker. Only one of them is getting the req. What config I need to make changes in to make all the containers eligible to receive req from broker.

Thanks

[Aaron]

Hi @jaikrattariyal . Sounds like you’re making some progress! I’ve never tried Heroku. Is the API just a standard REST endpoint, with basic user auth? Or is there some special magic HTTP headers that need to be in the POST? When connecting to various cloud services, some of them require specific things in the HTTP message, and we are slowly adding all sorts of various customization to allow RDPs to talk with GCP, AWS, Azure, etc.

If you want the RDP to send to multiple REST endpoints (multiple containers), then just configure multiple REST Consumers inside your RDP. Here is mine with two consumers, pointing to different IP addresses:

Good that you can always see the REST message coming into the broker on port 9000, when you subscribe to the special topic using the TryMe! Good to know. So the inbound side is definitely working.

So from what I can tell, if you’ve properly configured your REST Consumer(s) inside your RDP, and they are turning “UP” and are green, that should mean you have HTTP connectivity to your REST API / web server. I did some testing with my little Node server I posted above, and that’s what it looked like.

And you managed to get that PostMan Echo service working, right? I mean, you should be able to use any API that accepts basic authentication. I pointed the RDP back to my own Solace broker, the management port 8080, and I used it to control SEMP. What URL, API, port are you trying to reach in Heroku? I’ll try to find someone who knows more about that.

[jaikrattariyal]

@Aaron

Hi @jaikrattariyal . Sounds like you’re making some progress! I’ve never tried Heroku. Is the API just a standard REST endpoint, with basic user auth? Or is there some special magic HTTP headers that need to be in the POST? When connecting to various cloud services, some of them require specific things in the HTTP message, and we are slowly adding all sorts of various customization to allow RDPs to talk with GCP, AWS, Azure, etc.
If you want the RDP to send to multiple REST endpoints (multiple containers), then just configure multiple REST Consumers inside your RDP. Here is mine with two consumers, pointing to different IP addresses:

Good that you can always see the REST message coming into the broker on port 9000, when you subscribe to the special topic using the TryMe! Good to know. So the inbound side is definitely working.
So from what I can tell, if you’ve properly configured your REST Consumer(s) inside your RDP, and they are turning “UP” and are green, that should mean you have HTTP connectivity to your REST API / web server. I did some testing with my little Node server I posted above, and that’s what it looked like.
And you managed to get that PostMan Echo service working, right? I mean, you should be able to use any API that accepts basic authentication. I pointed the RDP back to my own Solace broker, the management port 8080, and I used it to control SEMP. What URL, API, port are you trying to reach in Heroku? I’ll try to find someone who knows more about that.

Thanks @Aaron for reply. There is no authentication in my heroku deployed API. Even if you hit URL _https://spring-solace-consumer.herokuapp.com/rest/tutorials_ you will see something on browser. But looks like something else is needed.

But there I am no more concerned about. Thats looks more infra issue than code or application issue. Where I am not clear now is, how can my multiple consumers receive message in one push from producer?

[MartinL]

@jaikrattariyal Seems we have the same problem. Are you sitting behind a firewall? I also have the same situation - everything seems to run just the operational State is down. When i changed the url in my brokers REST consumer to “localhost”, it was suddenly up. Looks like at the time of configuration, solace tries to connect to the remote rest service.