Skip to content

HOWTO connect lumberjack output of logstash and lumberjack spout of storm

Why do that

Your client has a configured lumberjack output of logstash and you want to connect the logstash pipeline with your own storm pipeline.

Prerequisites

  • You have a certificate and a key (der and pksc8 format)
1
openssl req -x509 -newkey rsa:2048 -keyout /home/user/keys/logstash/logstash.key -out /home/user/keys/logstash/logstash.crt -nodes -days 3650
  • You have a logstash with plugin

  • Clone or download the logstash-output-lumberjack repository

  • Download logstash (our tests were done with 5.5.1)
  • Edit the GemFile of your logstash to integrate the plugin

sh gem "logstash-output-lumberjack", :path => "./logstash-output-lumberjack"

  • Add a pipeline with a lumberjack output
1
2
3
4
5
6
7
output {
    lumberjack {
        hosts => "localhost"
        port => 6782
        ssl_certificate => "/home/user/keys/logstash/logstash.crt"
    }
}

What to do

Certificate

You will need the certificate and the key in pkcs8. You already have the certificate (see previous command of logstash) but we still need to convert key in der format to pkcs8.

1
openssl pkcs8 -topk8 -nocrypt -in /home/user/keys/logstash/logstash.key -out /home/user/keys/logstash/punchplatform.key8

Storm topology

Logstash sends the log in the map : key= 'line ', value= 'my log ... '

The PunchPlatform provide a Lumberjack spout. The configuration of the lumberjack must contain :

  • ssl activated and configured
  • compression deactivated
  • the same port between logstash output and storm input
  • the published field 'line '

Example :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
"spouts" : [
      {
          "type" : "lumberjack_spout",
          "spout_settings" : { 
            "listen" : {
              "host" : "0.0.0.0",  
              "port" : 6782,
              "compression" : false,
              "ssl" : true,
              "ssl_private_key" : "/home/user/keys/logstash/punchplatform.key8",
              "ssl_certificate" : "/home/user/keys/logstash/logstash.crt"
            },              
            "self_monitoring.activation" : true
          },
          "storm_settings" : {
            "executors": 1,
            "component" : "syslog_spout_lumberjack",
            "publish" : [ 
              { 
                "stream" : "logs", 
                "fields" : ["line"]
              }
            ] 
          }
      }
  ],

!!! warning "Don 't forget the publish configuration