org.thales.punch.libraries.storm.bolt

Class SyslogOutput

java.lang.Object

org.thales.punch.libraries.storm.api.BaseProcessingNode

org.thales.punch.libraries.storm.bolt.SyslogOutput

All Implemented Interfaces:

Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt

public class SyslogOutput
extends org.thales.punch.libraries.storm.api.BaseProcessingNode

The Syslog bolt forwards messages to a next hop UDP server. It is a terminating bolt, i.e. it does not forwards the tuple further the topology.

Overview

The syslog bolt works the other way the Syslog spout works : it takes one or several destination address(es) and forwards the lines it receives from the Storm previous bolt(s) to one or several destination using a load-balancing algorithm.

The SyslogBolt supports SSL/TLS and compression. For efficiency the SyslogBolt relies on an internal send queue, in order to perform bulk writes towards the server, as well as supporting a good level of compression. Check the real time reported metrics to estimate the compression level or your bolt.

*

Destination Settings

Because the Syslog bolt does not rely on an acknowledged protocol, sent data can be lost in case the remote peer fails or restart. The SyslogBolt will acknowledge the Storm tuples without ensuring the corresponding data has been received and processed by the Server. The configuration parameters of each destination are the following: Each destination takes the following properties

property	mandatory	type	default	comment
host	yes	string	-	The server host address. Accepted values are dotted number IP addresses, or hostname. You can set several destination addresses using the destination property.
port	yes	int	-	The server listening port.
proto	no	string	"tcp"	The socket protocol, "tcp" or "lumberjack"
compression	no	boolean	false	activate netty level compression
connect_timeout_ms	no	long	5000	the socket connection timeout.
connect_retry_interval_ms	no	long	5000	If the connection to the destination(s) fails, a new connection attempt is performed regularly. Set this to the retry interval.
send_queue_size	no	int	1000	The size of the blocking send queue. If the remote server is too slow, this queue will eventually fills up. Once full, you have two options: you can decide to give up sending data or make your bolt wait to catch up. You can also set this property for each destination.
drop_if_queue_full	no	boolean	true	Shutdown the connection if the input queue is full. This will make the bolt throw away the queued data and reconnect to the server. This option has no effect with the lumberjack protocol.
ssl	no	string	false	Set to true to listen on SSL.
secret	no	string	-	The secret to load a TLS PKCS12 SSL context

Load Balancing

Fromatting the data

Often what you need is to forward the data to another format, and/or to filter out some unnecessary fields. To do that you must insert a PunchBolt before this SyslogBolt. For example here is a punchlet that transform the (Json) log into a plain key-value string:

 
  {
         // get the Json under the stream and field id. 
         Tuple input = root:[logs][log];
         // transform it into a key value string
         String kv = toKv().on(input);
         // replace the input document by that String.
         root:[logs][log] = kv;
  }
  

With such a punchlet, the bolt will send something like

 
  "message"="original log", "vendor"="apache" ...
 

Main Settings

Besides destination the bolt has the following settings:

property	mandatory	type	default	comment
acked	no	boolean	false	if the syslog bolt is connected to none of its configured destination, you can make it eventually block the processing, or instead considered it harmless. You control that with this parameter, if set to true, Storm tuples will be acknowledged (in the Storm sense) only if it was sucessfully written to the output socket. In turn if the bolt cannot connect to any destination it will make the tuple fail, which eventually will make the topology repeat or block tuple processing. If false, tuples will always be acked. Note that even if tuples are acked, it does not mean yet that the remote server has received the data. It only means the data has been written to the socket. It you need real end-to-end acnowledgement with the remote server, you must in addition use the lumberjack acknowledged protocol. Note also that even is set to false, your bolt can slow down your topology if the remote server is too slow. Check the destination drop_if_queue_full property below.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.thales.punch.libraries.storm.api.BaseProcessingNode

ackRate, collector, componentId, every, failRate, metricContext, nodeSettings, topologyContext, traversalTime

Constructor Summary

Constructors

Constructor and Description
SyslogOutput(org.thales.punch.libraries.storm.api.NodeSettings config) Public constructor.

Method Summary

Modifier and Type	Method and Description
void	declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
void	prepare(Map map, org.apache.storm.task.TopologyContext topologyContext, org.apache.storm.task.OutputCollector collector) Setup this spout before starting emitting Tuples into the topology.
void	process(org.apache.storm.tuple.Tuple tuple) The Syslog bolt expects a single field, and forwards that field to the destination.

Methods inherited from class org.thales.punch.libraries.storm.api.BaseProcessingNode

ack, cleanup, enrichAndForwardMonitoringTuple, execute, fail, getComponentConfiguration, getNewPoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SyslogOutput

public SyslogOutput(org.thales.punch.libraries.storm.api.NodeSettings config)

Public constructor. A Spout ctor is in charge of setting the fields that will be required later on when the bolt will be effectively started and run somewhere in the cluster. Each field set here must be serializable.

Parameters:

config - the config

Method Detail

prepare

public void prepare(Map map,
                    org.apache.storm.task.TopologyContext topologyContext,
                    org.apache.storm.task.OutputCollector collector)

Setup this spout before starting emitting Tuples into the topology. This is called by the Storm runtime at activation.

Specified by:

prepare in interface org.apache.storm.task.IBolt

Overrides:

prepare in class org.thales.punch.libraries.storm.api.BaseProcessingNode

Parameters:

map - the storm settings

topologyContext - the topology settingss

collector - the output collector for sending out tuples.

process

public void process(org.apache.storm.tuple.Tuple tuple)

The Syslog bolt expects a single field, and forwards that field to the destination. If several fields are received, it sends a key-value format field1=value1 field2=value2.

Specified by:

process in class org.thales.punch.libraries.storm.api.BaseProcessingNode

Parameters:

tuple - the tuple to send

declareOutputFields

public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)

Specified by:

declareOutputFields in interface org.apache.storm.topology.IComponent

Overrides:

declareOutputFields in class org.thales.punch.libraries.storm.api.BaseProcessingNode