Skip to content

SFTP Input Node

Introduction

This node can be use to fetch files through ssh protocol. You can download files to a specific path and persist them or download them temporarily to a temporary directory.

Most of this node parameters are similar to what you are used on an sshd server by using the key:

  • sftp.session or
  • sftp.channel

Example(s)

{
    version: "6.0"
    type: punchline
    runtime: storm
    dag: [
        {
            type: sftp_input
            component: input
            settings: {
                cleanup: false
                download_path: /tmp/toto
                sftp_settings: {
                    sftp.ssh.host: mysftphost.com
                    sftp.ssh.auth.user: auser
                    sftp.ssh.auth.pass: apass
                    sftp.ssh.file.name_regex: "*.txt"
                    sftp.ssh.scan_directories: [
                        dir1,
                        dir2
                    ]
                }
            }
            publish: [
                {
                    stream: files
                    fields: [
                        meta
                    ]
                }
            ]
            subscribe: [

            ]
        }
        {
            type: punchlet_node
            component: stdout
            settings: {
                punchlet_code: "{ print(root); }"
            }
            subscribe: [
                {
                    stream: files
                    component: input
                }
            ]
            publish: [
                {
                    stream: files
                    fields: [
                        meta
                    ]
                }
            ]   
        }
    ]
}

Settings

Main Settings

Name Type Default Value Mandatory Description
sftp_settings Map (k-v) None true A list of key(string)-value(string) parameters. See SFTP_SETTINGS section below for more information
cleanup boolean true false Wipe all downloaded path after punchline finished or failed
download_path String System tmp dir false Specify an absolute path in which this node will use as storage
download_ignore_suffix String None false When files will be retrieved, you can ignore a suffix string of the real path of the file, normally used in together with download_add_suffix
download_add_suffix String None false When files will be retrieved and download_ignore_suffix is set, you can append a suffix string to the real path to be used for retrieving
consumer_mode String earliest false earliest (consume all files) or last_committed (consume files not consumed before)
checkpoint_settings Map (K-V) None false Is required if consume_mode is last_committed

SFTP_SETTINGS

Name Type Default Value Mandatory Description
sftp.ssh.auth.user String None true ssh user name for sftp server
sftp.ssh.auth.pass String None true ssh user password for sftp server
sftp.ssh.port String 22 false port of your sftp server
sftp.ssh.host String None true host name of your sftp server
sftp.ssh.file.name_regex String * false valid unix regex expression for searching files
sftp.ssh.timeout String 0 false timeout for sftp channel and session due to inactivity
sftp.ssh.scan_directories List of Str None false only top level files of a given folder are scanned. in case you wish to scan files within a folder, use this parameter where each element of the list is the name of a folder you want to scan
sftp.session Map k-v None false valid sftp configuration for a session (unix-like) - see /etc/ssh/ssh_config
sftp.channel Map k-v None false valid sftp configuration for a channel (unix-like)

CHECKPOINT_SETTINGS

Name Type Default Value Mandatory Description
checkpoint.application_runtime_id String None True A unique ID to the running pipeline
checkpoint.es_index_prefix String None True Elasticsearch index name where checkpoint will be stored: should end with '-', a date suffix will be added to the name: year.month.day
checkpoint.es_nodes List(Map K-V) None True Each map should have 2 fields: host -> String and port -> int

Note: Some settings for sftp.channel and sftp.session requires the sftp server to have the right configuration for it to work.