Skip to content

Lumberjack Output

The lumberjack output is very similar to the syslog output node, excepts it encodes the data using the Lumberjack protocol. That protocol has two characteristics:

  1. It is acknowledged : the server will acknowledge each received log once that log has been fully processed.
  2. it supports efficiently a key-value format. Lumberjack encodes key-value pairs using a binary format, efficiently decoded by the server.
  3. the punchplatform lumberjack protocol supports an additional keep alive mechanism

You configure the destination using a per stream destination logic. Here is a simple example with all the defaults, no compression and no ssl:

- type: lumberjack_output
  settings:
    destinations:
    - stream: logs
      destination:
      - host: 127.0.0.1
        port: 5052
  subscribe:
  - component: input
    stream: logs

Here is an complete example configuration:

- type: lumberjack_output
  settings:
    destinations:
    - stream: logs
      destination:
      - host: 127.0.0.1
        port: 5052
        compression: false
        drop_if_queue_full: false
        queue_size: 1000
        queue_flush_size: 1000
        queue_flush_interval_ms: 3000
        connect_retry_interval_ms: 3000
        connect_timeout_ms: 3000

        # Use a keep alive applicative message exchange to make sure 
        # the server is alive
        # Here we send such keep alive message every 30 seconds
        keep_alive_interval: 30000

        # and we give 20 seconds to the server to send us back the 
        # corresponding acknowledgement.
        # If not received in that time interval the socket will be closed 
        keep_alive_timeout: 2000

        ssl: true,
        ssl_provider: JDK
        ssl_private_key": /opt/keys/punchplatform.key.pkcs8
        ssl_certificate": /opt/keys/ca.pem

Destination groups are supported as well, as explained in the syslog ouput node. Here is an example:

type: lumberjack_output
settings:
  destination:
  - group: primary
    weight: 100
    host: 125.67.67.10
    port: 9999
  - group: primary
    weight: 100
    host: 125.67.67.11
    port: 9999
  - group: primary
    weight: 100
    host: 125.67.67.12
    port: 9999
  - group: secondary
    weight: 60
    host: 125.67.67.20
    port: 9999
  - group: secondary
    weight: 60
    host: 125.67.67.21
    port: 9999
  - group: secondary
    weight: 60
    host: 125.67.67.22
    port: 9999

Info

Note the keep alive options. It lets the node check for the connection aliveness and closes inactive sockets.

TLS

To learn more about encryption possibilities, refer to the TLS configurations dedicated chapter.

Compression

The Lumberjack supports two compression modes. If you use the compression property, compression will be performed at the socket level using the Netty ZLib compression. If instead you use the lumberjack_compression parameter, compression is performed as part of Lumberjack frame.

Note

Netty compression is most efficient, but will work only if the peer is a punch Lumberjack inpout node. If you send your data to a standard lumberjack server such as a Logstash daemon, use the lumberjack compression instead.

Streams And Fields

The Lumberjack output works nicely using storm streams and fields. It encode the received fields in a lumberjack frame. This is illustrated next:

image

Metrics

Refer to the lumberjack output metrics