Elastic Output

Overview

The ElasticSearchOutput receives a Dataset, convert each Row in a Json document and write that document to ElasticSearch. It supports both SSL and/or basic authentication.

To get the best performance , this node uses a bulked strategy, and allow parallelism. Messages are batched by partition and sent once to your elasticsearch nodes.

Example

type: elastic_output
settings:
  # Select elasticsearch nodes
  http_hosts:
    - host: localhost
      port: 9200
  # Send the document to a daily index with a given prefix
  index:
    type: daily
    prefix: example-index-
  # Select fields to index (default all)
  fields:
    - name
    - age
  # Use another field as unique id for inserting that document to elasticsearch
  document_id_field: example_id
subscribe:
  - component: punchlet_node
    stream: logs

Global Parameters

The elasticsearch output accepts the following settings:

• http_hosts (array)

  Elasticsearch nodes addresses (host and port).

• index (dictionary)

  Target index settings. See next section.

• request_timeout (string: "20s")

  Bulk indexation request timeout.

• credentials

  If you need basic auth, use a credentials dictionary to provide the user password to use. For example : "credentials" : { "user" : bob, "password" : "bob's password" }

  These settings can be combined with ssl. token parameter can be specified like that: "credentials": { "token": "mytoken", "token_type": "ApiKey" }. Note, if user and password are specified, they will be ignored in favor of token parameter. Token are the base64 encoded string "user:password" if set to type: Basic

Target Index

You can set different types of index.

# insert into a daily index. You must provide the prefix.
# Note that your also can use 'monthly' or 'yearly' if your index content is intended to be small, to avoid Elasticsearch cluster cluttering with too many indices to manage.
index:
  type: daily
  prefix: some_prefix_

# insert into an index provided by one of the subscribed field.
index:
  type: field
  field: index_name

# insert into a fixed named index
index:
  type: constant
  value: constant_index_name

Target Type

# insert into an index with the type mytype, the default value is _doc.
document_type: mytype

!!! Warning Type mapping are deprecated since ElasticSearch 7. Therefore, this parameter is optional and should be used only when dealing with an Elasticsearch version prior to the 7th.

Security

Additional parameters are available to configure the Elasticsearch output node with security settings for :

• TLS : using keystores or key files
• Authentication : using credentials with a basic user and password or a token

Example :

type: elastic_output
settings:
  http_hosts:
    - host: localhost
      port: 9200
  credentials:
    user: bob
    password: bob_secret
  ssl: true
  ssl_keystore_location: /data/certs/keystore.jks
  ssl_truststore_location: data/certs/truststore.jks
  ssl_keystore_pass: keystore_secret
  ssl_truststore_pass: truststore_secret

• credentials.user: (string)

  Username used by th Elasticsearch output node to authenticate to the Elasticsearch cluster. If provided, credentials.password MUST be configured. Cannot work with credentials.token and credentials.token_type.

• credentials.password: (string)

  Username used by th Elasticsearch output node to authenticate to the Elasticsearch cluster. If provided, credentials.user MUST be configured. Cannot work with credentials.token and credentials.token_type.

• credentials.token: (string)

  Token string used by th Elasticsearch output node to authenticate to the Elasticsearch cluster. If provided, credentials.token_type MUST be configured. Cannot work with credentials.user and credentials.password.

• credentials.token_type: (string)

  Token type used by th Elasticsearch output node to authenticate to the Elasticsearch cluster. If provided, credentials.token MUST be configured. Cannot work with credentials.user and credentials.password. Use Basic for a base64 encoded token string for authentication, Bearer for tokens based on the OAuth2 specs, or ApiKey for custom api key generation. Check Token-based authentication services Elasticsearch's documentation for more information.

• ssl: (boolean: false)

  Enable TLS encryption over the Elasticsearch output node's connexion to the Elasticsearch cluster. If false, all the following configurations are ignored.

Warning

For SSL files, use key files OR keystores settings, but not both

• ssl_private_key: (string)

  Path to the PKCS8 private key of the Elasticsearch output node.

• ssl_certificate: (string)

  Path to the x509 public key of the Elasticsearch output node. This certificate should always be generated with at least the usr_cert extension as its x509v3 extended usage for client purpose.

• ssl_trusted_certificate: (string)

  Path to the CA file containing the trusted certificates by the Elasticsearch output node. It should also contain its own CA root and all the intermediate certificates if this one is not self-signed.

• ssl_keystore_location: (string)

  Path to the keystore containing the Elasticsearch output node's public and private keys. jks, pkcs12 and p12 keystore types are supported.

• ssl_keystore_pass: (string)

  Password of the keystore provided with ssl_keystore_location. Do not provide this configuration if no password protects the keystore.

• ssl_truststore_location: (string)

  Path to the truststore containing the Elasticsearch output node's CA file and all the certificates trusted by this node. jks, pkcs12 and p12 truststore types are supported.

• ssl_truststore_pass: (string)

  Password of the truststore provided with ssl_truststore_location. Do not provide this configuration if no password protects the truststore.

• ssl_hostname_verification (boolean: true)

  Whether the node client should resolve the nodes hostnames to IP addresses or not.