Lumberjack Output¶

The lumberjack output is very similar to the syslog output node, excepts it encodes the data using the Lumberjack protocol. That protocol has two characteristics:

It is acknowledged : the server will acknowledge each received log once that log has been fully processed.
it supports efficiently a key-value format. Lumberjack encodes key-value pairs using a binary format, efficiently decoded by the server.
the punchplatform lumberjack protocol supports an additional keep alive mechanism

Refer to the Syslog output explanation, the Lumberjack output supports destinations groups the same way (Destination Groups<destinationGroups/). Per Stream Destinations are also supported.

Here is a simple example with all the defaults, no compression and no ssl:

  {
    "type": "lumberjack_output",
    "settings": {
      "destination" : [
        {
          "host" : "localhost",
          "port" : 9903,
          "compression" : true,
          "ssl" : false
        }
      ]
    },
    "subscribe": [
      {
            "component": "your_input_node",
            "stream": "logs"
      }
    ]
  }

Here is an complete example configuration:

    {
      "type": "lumberjack_output",
      "component": "lumberjack_output",
      "settings": {
        "destination": [
          {
            "compression": false,
            "host": "target.ip.address",
            "port": 9999,
            "drop_if_queue_full": false,
            "send_queue_size": 1000,
            "send_queue_flush_size": 1000,
            "send_queue_flush_interval_ms": 3000,
            "connect_retry_interval_ms": 3000,
            "connect_timeout_ms": 3000,

            // Use a keep alive applicative message exchange to make sure 
            // the server is alive
            // Here we send such keep alive message every 30 seconds
            "keep_alive_interval": 30,

            // and we give 20 seconds to the server to send us back the 
            // corresponding acknowledgement.
            // If not received in that time interval the socket will be closed 
            "keep_alive_timeout": 20,

            "ssl": true,
            "ssl_provider": "JDK",
            "ssl_private_key": "/opt/keys/punchplatform.key.pkcs8",
            "ssl_certificate": "/opt/keys/ca.pem"
          }
        ]
      },
      "subscribe": [
        {
          "component": "previous_node",
          "stream": "logs"
        }
      ]
    }

Info

note the keep alive options. It let the node check for the connection aliveness and closes inactive sockets.

SSL/TLS¶

To learn more about encryption possibilities, refer to the SSL/TLS configurations dedicated chapter.

Compression¶

The Lumberjack supports two compression mode. If you use the compression property, compression will be performed at the socket level using the Netty ZLib compression. If instead you use the lumberjack_compression parameter, compression is performed as part of Lumberjack frame.

Note

Netty compression is most efficient, but will work only if the peer is a punch Lumberjack inpout node. If you send your data to a standard Lumberjack server such as a Logstash daemon, use the lumberjack compression instead.

Streams And Fields¶

The Lumberjack output works nicely using storm streams and fields. It encode the received fields in a lumberjack frame. This is illustrated next:

/

Info

make sure you understand input nodes and output nodes stream and field fundamental concepts.

Refer to the Lumberjack Output Node javadoc documentation.

Metrics¶

Refer to the lumberjack output metrics