The lumberjack output is very similar to the syslog output node, excepts it encodes the data using the Lumberjack protocol. That protocol has two characteristics:
- It is acknowledged : the server will acknowledge each received log once that log has been fully processed.
- it supports efficiently a key-value format. Lumberjack encodes key-value pairs using a binary format, efficiently decoded by the server.
- the punchplatform lumberjack protocol supports an additional keep alive mechanism
You configure the destination using a per stream destination logic. Here is a simple example with all the defaults, no compression and no ssl:
- type: lumberjack_output settings: destinations: - stream: logs destination: - host: 127.0.0.1 port: 5052 subscribe: - component: input stream: logs
Here is an complete example configuration:
- type: lumberjack_output settings: destinations: - stream: logs destination: - host: 127.0.0.1 port: 5052 compression: false drop_if_queue_full: false queue_size: 1000 queue_flush_size: 1000 queue_flush_interval_ms: 3000 connect_retry_interval_ms: 3000 connect_timeout_ms: 3000 # Use a keep alive applicative message exchange to make sure # the server is alive # Here we send such keep alive message every 30 seconds keep_alive_interval: 30000 # and we give 20 seconds to the server to send us back the # corresponding acknowledgement. # If not received in that time interval the socket will be closed keep_alive_timeout: 2000 ssl: true, ssl_provider: JDK ssl_private_key": /opt/keys/punchplatform.key.pkcs8 ssl_certificate": /opt/keys/ca.pem
Destination groups are supported as well, as explained in the syslog ouput node. Here is an example:
type: lumberjack_output settings: destination: - group: primary weight: 100 host: 18.104.22.168 port: 9999 - group: primary weight: 100 host: 22.214.171.124 port: 9999 - group: primary weight: 100 host: 126.96.36.199 port: 9999 - group: secondary weight: 60 host: 188.8.131.52 port: 9999 - group: secondary weight: 60 host: 184.108.40.206 port: 9999 - group: secondary weight: 60 host: 220.127.116.11 port: 9999
Note the keep alive options. It lets the node check for the connection aliveness and closes inactive sockets.
To learn more about encryption possibilities, refer to the TLS configurations dedicated chapter.
The Lumberjack supports two compression modes. If you use the
property, compression will be performed at the socket level using
the Netty ZLib compression. If instead you use the
lumberjack_compression parameter, compression is performed as part
of Lumberjack frame.
Netty compression is most efficient, but will work only if the peer is a punch Lumberjack inpout node. If you send your data to a standard lumberjack server such as a Logstash daemon, use the lumberjack compression instead.
Streams And Fields¶
The Lumberjack output works nicely using storm streams and fields. It encode the received fields in a lumberjack frame. This is illustrated next:
Refer to the lumberjack output metrics