org.thales.punch.libraries.storm.spout

Class GeneratorInput

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichSpout

org.thales.punch.libraries.storm.api.BaseInputNode

org.thales.punch.libraries.storm.spout.GeneratorInput

All Implemented Interfaces:

Serializable, org.apache.storm.spout.ISpout, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichSpout

public class GeneratorInput
extends org.thales.punch.libraries.storm.api.BaseInputNode

The generator publishes fake data in configurable streams. It is extra simple and can be used to design topology unit test files, or simply used to play with. Here is an example

 
 {
   type : generator_input
   component : generator
   settings : {
     interval : 1000
     messages : [
       hello I am message 1
       followed by message 2
       .. and finally message 3
     ]
     messages_count : 20
   }
   publish : [
     {
       stream : logs
       fields : [
         log
       ]
     }
   ]
 }
 
 

To generate content with a limited rate, set the 'interval' parameter to the expected value in milliseconds. Its default value is set to 1 second.

If you want to generate a big number of messages, you can provide "messages_count" setting, and the generator will send the messages multiple times until the required messages count is reached.

You cannot make it simpler ! If you need several fields, specify your messages as maps using a "stream" : { "field" : value } format. Here is an example to send one message on the "logs" stream (with two fields "log" and "id") and another one on a different "error" stream:

 {
   type: generator_input
   component: spout
   settings: {
     messages : [
       {
         logs : {
           log : hello I am a valid message
           id  : id1
         }
       }
       {
         error : {
           log : hello I am a error message
           id  : id2
         }
       }
     ]
   }
 }
   

For test automation purpose, it is possible to request an automatic exit of the JVM depending on 'expectation' setting of the spout and the ack/fail of the tuples (See GeneratorInput.Expectation for possible values). One mode even allows the failed tuples to be automatically reemitted, so that in the end, we can check the expected number of tuples and/or idempotence of the topology in disturbed environment.

Author:

dimi

See Also:

Serialized Form

Field Summary

Fields inherited from class org.thales.punch.libraries.storm.api.BaseInputNode

collector, exitCondition, latencyRecordSender, loadController, metricContext, myself, nodeSettings

Constructor Summary

Constructors

Constructor and Description
GeneratorInput(org.thales.punch.libraries.storm.api.NodeSettings spoutSettings) Create a new Generator spout

Method Summary

Modifier and Type	Method and Description
void	ack(Object msgId)
void	fail(Object msgId)
void	nextTuple()
void	open(Map conf, org.apache.storm.task.TopologyContext topologyContext, org.apache.storm.spout.SpoutOutputCollector collector)

Methods inherited from class org.thales.punch.libraries.storm.api.BaseInputNode

close, deactivate, declareOutputFields, getPublishedStreams, regulate, sendLatencyRecord

Methods inherited from class org.apache.storm.topology.base.BaseRichSpout

activate

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.thales.punch.libraries.storm.api.ISpout

registerNextTupleCallback

Methods inherited from interface org.apache.storm.spout.ISpout

activate

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Constructor Detail

GeneratorInput

public GeneratorInput(org.thales.punch.libraries.storm.api.NodeSettings spoutSettings)

Create a new Generator spout

Parameters:

spoutSettings - the spout configuration.

Method Detail

open

public void open(Map conf,
                 org.apache.storm.task.TopologyContext topologyContext,
                 org.apache.storm.spout.SpoutOutputCollector collector)

Specified by:

open in interface org.apache.storm.spout.ISpout

Overrides:

open in class org.thales.punch.libraries.storm.api.BaseInputNode

nextTuple

public void nextTuple()

ack

public void ack(Object msgId)

Specified by:

ack in interface org.apache.storm.spout.ISpout

Overrides:

ack in class org.thales.punch.libraries.storm.api.BaseInputNode

fail

public void fail(Object msgId)

Specified by:

fail in interface org.apache.storm.spout.ISpout

Overrides:

fail in class org.thales.punch.libraries.storm.api.BaseInputNode