Skip to content

Python File Input

Before you start...

Before using...

This node is intended to be used when spark features are not required. For instance, in your pipeline, you are not manipulating Spark's dataframe at all...

The resulting output of this node is a list of string where each element is a line of your file.

Pyspark ->

Spark ->

Examples

Use-cases

Our "hello world" punchline configuration.

Beware

Each line of your file should be a valid elasticsearch query

beginner_use_case.punchline

{
    type: punchline
    version: "6.0"
    runtime: spark
    tenant: default
    dag: [
        {
            type: python_file_input
            component: queries
            publish: [
                {
                    stream: data
                }
            ]
            settings: {
                file_path: /full/path/to/file/query
            }
        }
        {
            type: python_elastic_input
            component: python_elastic_input
            settings: {
                index: mydata
                nodes: [
                    localhost
                ]
            }
            subscribe: [
                {
                    stream: data
                    component: queries
                }
            ]
            publish: [
                {
                    stream: data
                }
            ]
        }
        {
            type: python_elastic_output
            component: python_elastic_output
            settings: {
                nodes: [
                    localhost
                ]
                index: multiquerytest
            }
            subscribe: [
                {
                    stream: data
                    component: python_elastic_input
                }
            ]
        }
    ]
}

run beginner_use_case.punchline by using the command below:

CONF=beginner_use_case.punchline
punchlinectl start -p $CONF

Comming soon

Comming soon

Parameters

Common Settings

Name Type mandatory Default value Description
file_path String true NONE full path the file you want to ingest.

Advanced Settings

No advanced settings