org.thales.punch.libraries.storm.bolt

Class LumberjackOutput

java.lang.Object

org.thales.punch.libraries.storm.api.BaseProcessingNode

org.thales.punch.libraries.storm.bolt.LumberjackOutput

All Implemented Interfaces:

Serializable, org.apache.storm.task.IBolt, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichBolt

public class LumberjackOutput
extends org.thales.punch.libraries.storm.api.BaseProcessingNode

The PunchPlatformBolt forwards messages to a next hop PunchPlatformSpout. It is a terminating bolt, i.e. it does not forwards the tuple further the topology.

Overview

This bolt uses a single socket per configured destination(s). If it fails to connect to the peer, it will periodically attempt to reconnect.

The PunchPlatformBolt forwards the Storm Tuple it receives, using the lumberjack protocol encoding. For example on a log management PunchPlatform you have something like :

 
 { "logs" : { "log" : { "message" : "original log" , "vendor" : "apache" ...}, "es_id" = "37643" }
 

What will be sent is a lumberjack frame with

 
         "log" : { "message" : "original log" , "vendor" : "apache" ...}
         "es_id" = "37643" 
 

Bolt level settings

Here is an example of a typical configuration :

 
   {
           "type" : "lumberjack_bolt",
           "bolt_settings" : {
              "destination" : [{
                "host" : "localhost",
                "port" : 9999,
                "compression" : false,
                "ssl" : true,
                "ssl_private_key" : "/opt/keys/punchplatform.key.pkcs8",
                "ssl_certificate" : "/opt/keys/punchplatform.crt" 
              }]
           },
           "storm_settings" : {
                "executors": 1
           }
      }
 

Main Settings

property	mandatory	type	default	comment

Destination Settings

Each destination takes the following properties

property	mandatory	type	default	comment
host	yes	string	-	The server host address. Accepted values are dotted number IP addresses, or hostname. You can set several destination addresses using the destination property.
port	yes	int	-	The server listening port.
send_queue_flush_size	no	int	0	If that amount of data is received, flush the queue right away. Use this only if you have low traffic and?or to benefit from a better compression level
send_queue_size	no	integer	100000	Set this property to avoid having your client send more frames to the server than it reads the corresponding acks. This property is advised to avoid your client library to enqueue too many frames in the send queue.
send_queue_flush_period	no	int	100	Only effective if queue_flush_size is greater than 0. If little data is sent, flush the size of the blocking queue. This period is expressed in milliseconds.
ssl	no	string	false	Set to true to listen on SSL.
secret	no	string	-	The secret to load a TLS PKCS12 SSL context
connect_timeout_ms	no	long	5000	the socket connection timeout.
connect_retry_interval_ms	no	long	5000	If the connection to the destination(s) fails, a new connection attempt is performed regularly. Set this to the retry interval.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.thales.punch.libraries.storm.api.BaseProcessingNode

ackRate, collector, componentId, every, failRate, metricContext, nodeSettings, topologyContext, traversalTime

Constructor Summary

Constructors

Constructor and Description
LumberjackOutput(org.thales.punch.libraries.storm.api.NodeSettings config) Constructor.

Method Summary

Modifier and Type	Method and Description
void	cleanup()
void	declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)
void	prepare(Map map, org.apache.storm.task.TopologyContext topologyContext, org.apache.storm.task.OutputCollector collector) Setup this spout before starting emitting Tuples into the topology.
void	process(org.apache.storm.tuple.Tuple tuple) Storm gives us tuple one by one.

Methods inherited from class org.thales.punch.libraries.storm.api.BaseProcessingNode

ack, enrichAndForwardMonitoringTuple, execute, fail, getComponentConfiguration, getNewPoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LumberjackOutput

public LumberjackOutput(org.thales.punch.libraries.storm.api.NodeSettings config)

Constructor.

Parameters:

config - the bolt configuration

Method Detail

prepare

public void prepare(Map map,
                    org.apache.storm.task.TopologyContext topologyContext,
                    org.apache.storm.task.OutputCollector collector)

Setup this spout before starting emitting Tuples into the topology. This is called by the Storm runtime at activation.

Specified by:

prepare in interface org.apache.storm.task.IBolt

Overrides:

prepare in class org.thales.punch.libraries.storm.api.BaseProcessingNode

Parameters:

map - the storm settings

topologyContext - the topology settingss

collector - the output collector for sending out tuples.

process

public void process(org.apache.storm.tuple.Tuple tuple)

Storm gives us tuple one by one. It will be the strategy of the underlying NioTcpClient to do aggregation. What we do here is a simple forward.

Specified by:

process in class org.thales.punch.libraries.storm.api.BaseProcessingNode

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.storm.task.IBolt

Overrides:

cleanup in class org.thales.punch.libraries.storm.api.BaseProcessingNode

declareOutputFields

public void declareOutputFields(org.apache.storm.topology.OutputFieldsDeclarer declarer)

Specified by:

declareOutputFields in interface org.apache.storm.topology.IComponent

Overrides:

declareOutputFields in class org.thales.punch.libraries.storm.api.BaseProcessingNode