Python Elastic Output

Warning

This node is beta. In particular it does not support TLS encryption. Prefer using the java variants.

Overview

This node can be used for simple use cases where you do not need to manipulate dataframe APIs. With this node, you can save an incoming stream of list where each element is a python dictionary to an elasticsearch index.

Runtime Compatibility

• PySpark : ✅
• Spark : ❌

Example

Here is a simple "hello world" example:

---
type: punchline
version: '6.0'
runtime: pyspark
dag:
- type: python_elastic_input
  component: python_elastic_input
  settings:
    index: mydata
    nodes:
    - localhost
  subscribe: []
  publish:
  - stream: data
- type: python_elastic_output
  component: python_elastic_output
  settings:
    nodes:
    - localhost
    index: multiquerytest
  subscribe:
  - stream: data
    component: python_elastic_input

You can run this punchline using the following command

punchlinectl start -p punchline.yaml

Parameters

Name	Type	mandatory	Default value	Description
index	String	true	NONE	The name of your elasticsearch index where data will be fetched.
port	Integer	false	9200	Your Elasticsearch server Port.
nodes	List of String	true	NONE	Hostnames of your elasticsearch nodes. In general, only one hostname is needed.
type	String	false	NONE	document type that will be retrieved from elasticsearch index
login	String	false	NONE	the login required if your elasticsearch server requests authentication
password	String	false	NONE	the password required if the elasticsearch server requests authentication