HOWTO connect lumberjack output of logstash and lumberjack spout of storm

Why do that¶

Your client has a configured lumberjack output of logstash and you want to connect the logstash pipeline with your own storm pipeline.

Prerequisites¶

You have a certificate and a key (der and pksc8 format)

openssl req -x509 -newkey rsa:2048 -keyout /home/user/keys/logstash/logstash.key -out /home/user/keys/logstash/logstash.crt -nodes -days 3650

You have a logstash with plugin
Clone or download the logstash-output-lumberjack repository
Download logstash (our tests were done with 5.5.1)
Edit the GemFile of your logstash to integrate the plugin

gem "logstash-output-lumberjack", :path => "./logstash-output-lumberjack"

Add a pipeline with a lumberjack output

output {
    lumberjack {
        hosts => "localhost"
        port => 6782
        ssl_certificate => "/home/user/keys/logstash/logstash.crt"
    }
}

What to do¶

Certificate¶

You will need the certificate and the key in pkcs8. You already have the certificate (see previous command of logstash) but we still need to convert key in der format to pkcs8.

openssl pkcs8 -topk8 -nocrypt -in /home/user/keys/logstash/logstash.key -out /home/user/keys/logstash/punchplatform.key8

Storm topology¶

Logstash sends the log in the map : key= 'line ', value= 'my log ... '

The PunchPlatform provide a Lumberjack spout. The configuration of the lumberjack must contain :

ssl activated and configured
compression deactivated
the same port between logstash output and storm input
the published field 'line '

Example :

"spouts" : [
      {
          "type" : lumberjack_input",
          "settings" : { 
            "listen" : {
              "host" : "0.0.0.0",  
              "port" : 6782,
              "compression" : false,
              "ssl" : true,
              "ssl_private_key" : "/home/user/keys/logstash/punchplatform.key8",
              "ssl_certificate" : "/home/user/keys/logstash/logstash.crt"
            },              
            "self_monitoring.activation" : true
          },
          "storm_settings" : {
            "executors": 1,
            "component" : "syslog_spout_lumberjack",
            "publish" : [ 
              { 
                "stream" : "logs", 
                "fields" : ["line"]
              }
            ] 
          }
      }
  ],

!!! warning "Don 't forget the publish configuration