Skip to content

Python Elastic Output

Overview

Compatible Pyspark only

This node is intended to be used when spark features are not required. For instance, in your pipeline, you are not manipulating Spark's dataframe at all...

With this node, you can save your an incoming stream of list (where each element is a python dictionary) to an elasticsearch index.

Example(s)

1: Multiple queries

Below is an example where our Python Elastic Ouput Node is subscribed to a node publishing a list of python dictionary. The whole list is bulked inside an elasticsearch index.

Note

Each line of your file should be a valid elasticsearch query

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
job: [
    {
        type: python_file_input
        component: queries
        publish: [
            {
                stream: data
            }
        ]
        settings: {
            file_path: /full/path/to/file/query
        }
    }
    {
        type: python_elastic_input
        component: python_elastic_input
        settings: {
            index: mydata
            nodes: [
                localhost
            ]
        }
        subscribe: [
            {
                stream: data
                component: queries
            }
        ]
        publish: [
            {
                stream: data
            }
        ]
    }
    {
        type: python_elastic_output
        component: python_elastic_output
        settings: {
            nodes: [
                localhost
            ]
            index: multiquerytest
        }
        subscribe: [
            {
                stream: data
                component: python_elastic_input
            }
        ]
    }
]

Configuration(s)

  • index: String

    Description: [Required] The name of your elasticsearch index where data will be fetched.

  • port: Integer

    Description: [Optional] Your Elasticsearch server Port.

  • type: String

    Description: [Optional] Document type that will be retrieved from your elasticsearch index.

  • nodes: List

    Description: [Required] Hostnames of your elasticsearch nodes. In general, only one hostname is needed.