🎄 Happy Holidays! 🥳

Most of Solace is closed December 24–January 1 so our employees can spend time with their families. We will re-open Thursday, January 2, 2024. Please expect slower response times during this period and open a support ticket for anything needing immediate assistance.

Happy Holidays!

Please note: most of Solace is closed December 25–January 2, and will re-open Tuesday, January 3, 2023.

How to send application log to a Solace queue

suhelrizvi
suhelrizvi Member Posts: 3 ✭
in General Discussions #1

We need to send log file contents on a linux server to a remote queue in Solace. The log files are from a perl application.

Is there a standard way to do this for Solace?

Or are there any known output plugins for log forwarders like logstash/filebeat/fluentd etc which could send to a Solace queue?

Comments

  • arih
    arih Member, Employee Posts: 125 Solace Employee
    #2

    Hi @suhelrizvi ,
    Few of us here uses community plugin from Logstash or Fluentd to publish the logs via MQTT into Solace PubSub+ using topics. From there, we use topic to queue mapping to persistently store the logs and also most importantly do some routing/filtering as needed.
    One sample plugin is here: https://github.com/toyokazu/fluent-plugin-mqtt-io. Look at the output part.

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 963 admin
    #3

    Also if you don't have much luck with MQTT you can also use JMS. I believe Logstash has a plugin for either.

  • David Fung
    David Fung Member Posts: 7 ✭
    edited December 2020 #4

    Hi,
    I am evaluating fluentd to collect the event logs from Solace by MQTT. The sample plugin (https://github.com/toyokazu/fluent-plugin-mqtt-io) is used to collect the events. Below please find my plugin conf. Nothing is received. Other MQTT topic is working fine except the "$SYS/LOG/#". Is there some trick in defining a special character "$" in MQTT topic?

    <source>
  @type mqtt
  host x.x.x.x
  port 1883
  topic $SYS/LOG/#
  tag mytag
  <security>
     username 'user'
     password 'password'
   </security>
  <parse>
    @type json
  </parse>
</source>
  • arih
    arih Member, Employee Posts: 125 Solace Employee
    #5

    Hi David,
    I don't have the env to test right now, but can you try "\$SYS/LOG/#" and see?
    And you can also go check in the broker itself like the screenshot below to see what is the subscription that your fluentd client has, so you can check if the string is correctly captured or it was mistaken as a env var.

  • David Fung
    David Fung Member Posts: 7 ✭
    #6

    Thank you for your input. Both "\$SYS/LOG/#" and "/$SYS/LOG/#" options were tried. No event message was received. Below please find the screen captures.

  • arih
    arih Member, Employee Posts: 125 Solace Employee
    #7

    Hi @David Fung
    In that case, the simpler "$SYS/INFO/#" string should do the trick.
    But, just to make sure, that format is used for MQTT, and my previous screenshot was done lazily that I just use the try-me that uses SMF. The more correct sample should be like this where I actually use an MQTT client:

    And, if that is all already correct, you can also double-check if events are actually published there. You can check that using any MQTT client you have, SDKPerf MQTT or MQTTfx for example. And just in case, the publishing of events must be enabled first in the management message-vpn such as documented here: https://docs.solace.com/System-and-Software-Maintenance/Receiving-Message-Bus-Events.htm#subscribing_to_message_bus_events_1651767527_301887

    Below is my test with SDKPerf MQTT:

  • David Fung
    David Fung Member Posts: 7 ✭
    #8

    Thank you for your support.. I can receive the message by MQTT. Topic is $SYS/LOG/#

  • arih
    arih Member, Employee Posts: 125 Solace Employee
    edited December 2020 #9

    Ah cool, so it's working now? Great then :)
    Anyway, just sharing what I missed earlier when I was testing this locally. I realised the broker by default does not publish these events. And even if it does, the default is gonna be SMF format (#LOG/>) but it needs to be changed to $SYS because # is reserved for MQTT for wildcard. Hope that helps others later too :)

  • David Fung
    David Fung Member Posts: 7 ✭
    #10

    Because MQTT generates a lot of traffic for log collection purpose, we tend to use Syslog as an input and Elasticsearch as an output by Fluentd. Many thanks for your support!

  • suhelrizvi
    suhelrizvi Member Posts: 3 ✭
    #11

    The posts provide some useful info if we're using MQTT

    Thought I would share my findings to help others who want to send app logs to a Solace queue.

    There are a number of possible ways to send logs to Solace :

    1) some people are using a community plugin from Logstash or Fluentd to publish the logs via MQTT into Solace PubSub+ using topics. From there, use topic to queue mapping to persistently store the logs and also most importantly do some routing/filtering as needed.

    One sample plugin is here: https://github.com/toyokazu/fluent-plugin-mqtt-io. Look at the output part.

    See the previous discussion on this thread.

    2) logstash can send logs to Solace via a JMS output plugin.

    https://www.elastic.co/guide/en/logstash/5.5/plugins-outputs-jms.html

    3) its possible to use the curl command to post log messages to Solace queues directly

    4) or write a custom Java daemon to read logs and forward to Solace via JMS.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 644 admin
    #12

    Looks like we have 2 different threads in here..!

    @David Fung out of curiosity, why are you choosing to receive Solace broker log events via MQTT, rather than using the (management plane) Syslog integration? Are you using Solace Cloud? If not, perhaps consider using the Syslog export facility..?

    @suhelrizvi thanks for the summary! Something I recently discovered (although it's not that new) is that you can build custom Logstash plugins using Java... someone built a Gradle project to compile Java into a Ruby Gem to use with Logstash. So I made a custom Solace Java (JCSMP API) Logstash Output plugin to publish logs into Solace, with custom topics and everything. I'd could show you the code and stuff if you'd be interested? Or are you looking for more "off-the-shelf" solutions?

  • David Fung
    David Fung Member Posts: 7 ✭
    #13

    @Aaron Yes, we use Syslog + Fluentd to receive the log and display it in Graylog.

  • suhelrizvi
    suhelrizvi Member Posts: 3 ✭
    #14

    @Aaron said:
    Looks like we have 2 different threads in here..!

    @David Fung out of curiosity, why are you choosing to receive Solace broker log events via MQTT, rather than using the (management plane) Syslog integration? Are you using Solace Cloud? If not, perhaps consider using the Syslog export facility..?

    @suhelrizvi thanks for the summary! Something I recently discovered (although it's not that new) is that you can build custom Logstash plugins using Java... someone built a Gradle project to compile Java into a Ruby Gem to use with Logstash. So I made a custom Solace Java (JCSMP API) Logstash Output plugin to publish logs into Solace, with custom topics and everything. I'd could show you the code and stuff if you'd be interested? Or are you looking for more "off-the-shelf" solutions?

    @Aaron we are trying to build an enterprise solution so looking for something off the shelf preferably for easier support but I'd like to take a look at what you have done in case we decide to do that instead to meet our requirements :smile:

Categories

This Month's Leaders

This Week's Leaders