org.thales.punch.libraries.storm.spout

Class SFTPSpout

java.lang.Object

org.apache.storm.topology.base.BaseComponent

org.apache.storm.topology.base.BaseRichSpout

org.thales.punch.libraries.storm.api.BaseInputNode

org.thales.punch.libraries.storm.spout.SFTPSpout

All Implemented Interfaces:

Serializable, org.apache.storm.spout.ISpout, org.apache.storm.topology.IComponent, org.apache.storm.topology.IRichSpout

public class SFTPSpout
extends org.thales.punch.libraries.storm.api.BaseInputNode

SFTPSpout downloads files that matches a certain regex expression.

Downloaded files are stored in a temporary directory where the topology has been executed.

Regex should be the same as used in unix system when searching for files.

It is expected to use another node to download the desired files...

Note by design, we consider this node to be used in a batch-like topology.

Although the client by design makes it easy to have a 'streaming behavior'

 
{
    version: "6.0"
    type: punchline
    runtime: storm
    dag: [
        {
            type: sftp_input
            component: input
            settings: {
                cleanup: false
                download_path: /tmp/toto
                download_ignore_suffix: ".complete"
                download_add_suffix: ".nm4"
                consumer_mode: last_committed
                sftp_settings: {
                    sftp.ssh.host: server3.com
                    sftp.ssh.auth.user: user
                    sftp.ssh.auth.pass: pass
                    sftp.ssh.file.name_regex: "*.complete"
                    sftp.ssh.scan_directories: [
                        SAISData
                    ]
                }
                checkpoint_settings: {
                    checkpoint.application_runtime_id: sftp_application_id_test
                    checkpoint.es_index_prefix: jonathan_tenant-test-
                    checkpoint.es_nodes: [
                        {
                            host: localhost
                        }
                    ]
                }
            }
            publish: [
                {
                    stream: files
                    fields: [
                        meta
                    ]
                }
            ]
            subscribe: [

            ]
        }
        {
            type: punchlet_node
            component: stdout
            settings: {
                punchlet_code: "{ print(root); }"
            }
            subscribe: [
                {
                    stream: files
                    component: input
                }
            ]
            publish: [
                {
                    stream: files
                    fields: [
                        meta
                    ]
                }
            ]   
        }
    ]
}
 
 

Author:

jonathan

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKPOINT_APPLICATION_RUNTIME_ID
static String	CHECKPOINT_ES_INDEX_PREFIX
static String	CHECKPOINT_ES_NODES

Fields inherited from class org.thales.punch.libraries.storm.api.BaseInputNode

collector, exitCondition, latencyRecordSender, loadController, metricContext, myself, nodeSettings

Constructor Summary

Constructors

Constructor and Description
SFTPSpout(org.thales.punch.libraries.storm.api.NodeSettings nodeSettings, org.thales.punch.settings.api.ISettingsMap sftpSettingsMap, boolean withCleanUp, String downloadFilePath, String downloadIgnoreSuffix, String downloadAddSuffix, org.thales.punch.settings.api.ISettingsMap checkpointSettings, String consumeMode)

Method Summary

Modifier and Type	Method and Description
void	ack(Object msgId) commit will be done only if all emitted data have been ack If at least one emitted file is not ack, this topology will fail
void	close()
void	fail(Object msgId)
void	nextTuple()
void	open(Map conf, org.apache.storm.task.TopologyContext context, org.apache.storm.spout.SpoutOutputCollector collector)

Methods inherited from class org.thales.punch.libraries.storm.api.BaseInputNode

deactivate, declareOutputFields, getPublishedStreams, regulate, sendLatencyRecord

Methods inherited from class org.apache.storm.topology.base.BaseRichSpout

activate

Methods inherited from class org.apache.storm.topology.base.BaseComponent

getComponentConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.thales.punch.libraries.storm.api.ISpout

registerNextTupleCallback

Methods inherited from interface org.apache.storm.spout.ISpout

activate

Methods inherited from interface org.apache.storm.topology.IComponent

getComponentConfiguration

Field Detail

CHECKPOINT_APPLICATION_RUNTIME_ID

public static final String CHECKPOINT_APPLICATION_RUNTIME_ID

See Also:

Constant Field Values

CHECKPOINT_ES_INDEX_PREFIX

public static final String CHECKPOINT_ES_INDEX_PREFIX

See Also:

Constant Field Values

CHECKPOINT_ES_NODES

public static final String CHECKPOINT_ES_NODES

See Also:

Constant Field Values

Constructor Detail

SFTPSpout

public SFTPSpout(org.thales.punch.libraries.storm.api.NodeSettings nodeSettings,
                 org.thales.punch.settings.api.ISettingsMap sftpSettingsMap,
                 boolean withCleanUp,
                 String downloadFilePath,
                 String downloadIgnoreSuffix,
                 String downloadAddSuffix,
                 org.thales.punch.settings.api.ISettingsMap checkpointSettings,
                 String consumeMode)

Method Detail

nextTuple

public void nextTuple()

open

public void open(Map conf,
                 org.apache.storm.task.TopologyContext context,
                 org.apache.storm.spout.SpoutOutputCollector collector)

Specified by:

open in interface org.apache.storm.spout.ISpout

Overrides:

open in class org.thales.punch.libraries.storm.api.BaseInputNode

ack

public void ack(Object msgId)

commit will be done only if all emitted data have been ack If at least one emitted file is not ack, this topology will fail

Specified by:

ack in interface org.apache.storm.spout.ISpout

Overrides:

ack in class org.thales.punch.libraries.storm.api.BaseInputNode

fail

public void fail(Object msgId)

Specified by:

fail in interface org.apache.storm.spout.ISpout

Overrides:

fail in class org.thales.punch.libraries.storm.api.BaseInputNode

close

public void close()

Specified by:

close in interface org.apache.storm.spout.ISpout

Overrides:

close in class org.thales.punch.libraries.storm.api.BaseInputNode