org.thales.punch.libraries.storm.spout

Class ExtractionInput

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichSpout

org.thales.punch.libraries.storm.api.BaseInputNode

org.thales.punch.libraries.storm.spout.ExtractionInput

All Implemented Interfaces:

Serializable, org.apache.storm.spout.ISpout, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichSpout

public class ExtractionInput
extends org.thales.punch.libraries.storm.api.BaseInputNode

Light topology elastic input node designed for extraction

This node is designed for single worker only !

Designed

Elasticsearch scroll API is used for fetching index data containing a relatively huge number of documents.

The client being used for retrieving data enable you to configure it's behavior through it's elastic_settings key.

Custom metrics are implemented for users to be able to follow their extraction progression... Those metrics are published in an elasticsearch index.

 
 {
 "runtime_id": "generated_by_kibana",
 "tenant": "mytenant",
 "version": "6.0",
 "platform_id": "myplatformid",
 "runtime": "storm",
 "dag": [
 {
 "type": "extraction_input",
 "component": "input",
 "settings": {
 "index": "mytenant-events-*",
 "id_column": "myid",
 "mode": "hits_hits",
 "nodes": [
 "localhost"
 ],
 "elastic_settings": {
 "es.scroll.size": 10000
 },
 "output_columns": [
 {
 "field": "channel"
 }
 ]
 },
 "publish": [
 {
 "stream": "logs",
 "fields": ["log"]
 }
 ]
 },
 {
 "type": "punchlet_node",
 "component": "processor",
 "settings": {
 "punchlet_code": "{ print(root); }"
 },
 "subscribe": [
 {
 "component": "input",
 "stream": "logs"
 }
 ],
 "publish": [
 {
 "stream": "logs",
 "fields": ["log"]
 }
 ]
 },
 {
 "type": "file_output",
 "component": "todisk",
 "settings": {
 "destination": "file:///tmp/test_elastic_wrapper",
 "streaming": true,
 "strategy": "at_least_once",
 "fields": ["log"],
 "batch_size": 10000
 },
 "subscribe": [
 {
 "component": "processor",
 "stream": "logs",
 "fields": ["log"]
 }
 ]
 }
 ],
 "metrics": {
 "reporters": [
 {
 "type": "console"
 }
 ]
 }
 }
 
 

Author:

jonathan

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_HOST
static int	DEFAULT_PORT
static String	UNDERSCORE_SEARCH

Fields inherited from class org.thales.punch.libraries.storm.api.BaseInputNode

collector, exitCondition, latencyRecordSender, loadController, metricContext, myself, nodeSettings

Constructor Summary

Constructors

Constructor and Description
ExtractionInput(org.thales.punch.libraries.storm.api.NodeSettings nodeSettings, List<org.thales.punch.settings.api.ISettingsMap> esMetas, List<org.thales.punch.settings.api.ISettingsMap> outputColumns, org.thales.punch.settings.api.ISettingsMap punchlineMeta, org.thales.punch.settings.api.ISettingsMap componentSettings, String mode)

Method Summary

Modifier and Type	Method and Description
void	ack(Object msgId)
void	close()
void	fail(Object msgId)
void	nextTuple()
void	open(Map conf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)

Methods inherited from class org.thales.punch.libraries.storm.api.BaseInputNode

deactivate, declareOutputFields, getPublishedStreams, regulate, sendLatencyRecord

Methods inherited from class org.apache.storm.topology.base.BaseRichSpout

activate

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.thales.punch.libraries.storm.api.ISpout

registerNextTupleCallback

Methods inherited from interface org.apache.storm.spout.ISpout

activate

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Detail

DEFAULT_HOST

public static final String DEFAULT_HOST

See Also:

Constant Field Values

DEFAULT_PORT

public static final int DEFAULT_PORT

See Also:

Constant Field Values

UNDERSCORE_SEARCH

public static final String UNDERSCORE_SEARCH

See Also:

Constant Field Values

Constructor Detail

ExtractionInput

public ExtractionInput(org.thales.punch.libraries.storm.api.NodeSettings nodeSettings,
                       List<org.thales.punch.settings.api.ISettingsMap> esMetas,
                       List<org.thales.punch.settings.api.ISettingsMap> outputColumns,
                       org.thales.punch.settings.api.ISettingsMap punchlineMeta,
                       org.thales.punch.settings.api.ISettingsMap componentSettings,
                       String mode)

Method Detail

open

public void open(Map conf,
                 org.apache.storm.task.TopologyContext context,
                 org.apache.storm.spout.SpoutOutputCollector collector)

Specified by:

open in interface org.apache.storm.spout.ISpout

Overrides:

open in class org.thales.punch.libraries.storm.api.BaseInputNode

nextTuple

public void nextTuple()

ack

public void ack(Object msgId)

Specified by:

ack in interface org.apache.storm.spout.ISpout

Overrides:

ack in class org.thales.punch.libraries.storm.api.BaseInputNode

fail

public void fail(Object msgId)

Specified by:

fail in interface org.apache.storm.spout.ISpout

Overrides:

fail in class org.thales.punch.libraries.storm.api.BaseInputNode

close

public void close()

Specified by:

close in interface org.apache.storm.spout.ISpout

Overrides:

close in class org.thales.punch.libraries.storm.api.BaseInputNode