Skip to content

Python Elastic Output

Warning

This node is beta. In particular it does not support TLS encryption. Prefer using the java variants.

Overview

This node can be used for simple use cases where you do not need to manipulate dataframe APIs. With this node, you can save an incoming stream of list where each element is a python dictionary to an elasticsearch index.

Runtime Compatibility

  • PySpark :
  • Spark :

Example

Here is a simple "hello world" example:

{
  type: punchline
  version: "6.0"
  runtime: pyspark
  tenant: default
  dag:
  [
    {
      type: python_elastic_input
      component: python_elastic_input
      settings:
      {
        index: mydata
        nodes:
        [
          localhost
        ]
      }
      subscribe: []
      publish:
      [
        {
          stream: data
        }
      ]
    }
    {
      type: python_elastic_output
      component: python_elastic_output
      settings:
      {
        nodes:
        [
          localhost
        ]
        index: multiquerytest
      }
      subscribe:
      [
        {
          stream: data
          component: python_elastic_input
        }
      ]
    }
  ]
}
You can run this punchline using the following command
punchlinectl start -p punchline.hjson

Parameters

Name Type mandatory Default value Description
index String true NONE The name of your elasticsearch index where data will be fetched.
port Integer false 9200 Your Elasticsearch server Port.
nodes List of String true NONE Hostnames of your elasticsearch nodes. In general, only one hostname is needed.
type String false NONE document type that will be retrieved from elasticsearch index
login String false NONE the login required if your elasticsearch server requests authentication
password String false NONE the password required if the elasticsearch server requests authentication