Detect which services are alive

radekm
radekm Member Posts: 23

Hi,

we have several services which use Solace to communicate.
We also want to notify users if some service is not running.
To do that every service publishes a heartbeat
to a topic service-name/heartbeat and when the heartbeat stops
we assume that the service is not running.

This is working, but I'm wondering if there is a better/easier/more natural solution?
Because Solace broker itself must monitor which sessions are connected
so if we could somehow get this information or notification
when a new session is connected or some existing session disconnected
we could simplify our implementation.

Thanks

Best Answer

  • TomF
    TomF Member, Employee Posts: 405 Solace Employee
    #2 Answer ✓

    Hi @radekm, an interesting question. I think the answer is that you need to keep your heartbeat solution.

    The long answer is: yes the broker will tell you when a client stops responding. The broker and client share heartbeats (keepalives) depending on the protocol you're using. When there's a keepalive failure, the broker will disconnect the client, and send a notification via Syslog (CLIENT_CLIENT_DISCONNECT, if I remember correctly). So you could easily set up syslog rules to deal with this.

    BUT... and it's a big but: this all operates at the TCP and API layer. Provided your process/JVM etc is still running, the API processing will continue and API keepalives will still flow. What this won't tell you is if your application code is stuck somewhere: say it's processing a loop, got stuck and can't allocate any more threads. In this case, you need your application code to send a keepalive message - which is what you're already doing.

    Now, if you hadn't already created a heartbeat mechanism I would say use the client disconnection notice as a good approximation for application level heartbeats. However, you've already implemented the most thorough solution: my advice is to stick with it now you've already done the hard work.

Answers

  • TomF
    TomF Member, Employee Posts: 405 Solace Employee
    #3 Answer ✓

    Hi @radekm, an interesting question. I think the answer is that you need to keep your heartbeat solution.

    The long answer is: yes the broker will tell you when a client stops responding. The broker and client share heartbeats (keepalives) depending on the protocol you're using. When there's a keepalive failure, the broker will disconnect the client, and send a notification via Syslog (CLIENT_CLIENT_DISCONNECT, if I remember correctly). So you could easily set up syslog rules to deal with this.

    BUT... and it's a big but: this all operates at the TCP and API layer. Provided your process/JVM etc is still running, the API processing will continue and API keepalives will still flow. What this won't tell you is if your application code is stuck somewhere: say it's processing a loop, got stuck and can't allocate any more threads. In this case, you need your application code to send a keepalive message - which is what you're already doing.

    Now, if you hadn't already created a heartbeat mechanism I would say use the client disconnection notice as a good approximation for application level heartbeats. However, you've already implemented the most thorough solution: my advice is to stick with it now you've already done the hard work.