Skip to content

Channels

Abstract

This chapter explains how you assemble various applications into a channel and why this concept is both powerful and necessary.

A channel describes where one or several application must be executed. The channel abstraction provides you with a concise and clear declarative approach to organise your applications in ways easy to manage.

Configuration

Each channel is defined using a yaml channel_structure.yaml file. Refer to the configuration file to understand the tenant/channel/application tree organisation.

The channel_structure.yaml file has the following structure:

version: "6.0"
start_by_tenant: true
stop_by_tenant: false
applications:
  - name: mypunchline
    runtime: shiva
    cluster: common
    reload: none
    command: punchlinectl
    quartzcron_schedule : 0/30 * * * * ? *
    args:
      - "start"
      - "--punchline"
      - "mypunchline.yaml"
      - "--runtime"
      - "punch"
      - "--childopts"
      - "-Xms256m -Xmx256m"
    apply_resolver_on:
    - afile.json
    - anotherfile.hjson

resources:
 - type: kafka_topic
   topic: syslog-to-kafka
   cluster: kafka
   partitions: 1
   replication_factor: 1
   delete_retention_time: "1d"
   retention_time: "-1"
   retention_bytes: "-1"
   cleanup_policy: compact
Parameter Description Default
version Correspond to the major punch release. nil
start_by_tenant set to true to make that channel startable by a tenant-level command true
stop_by_tenant set to true to make that channel stoppable by a tenant-level command false
application the list of applications nil
resources an optional list of resources associated to this channel nil

Each applications has the following properties:

Parameter Description Default
name The application name. nil
runtime storm or shiva nil
cluster the name of the target (kubernetes|shiva) cluster nil
reload control the reload policy of this application. none or kill_then_start none
command one of the punch supported commands nil
args the command args as you would type them in a plain terminal nil
apply_resolver_on an optional list of additional files on which to apply the resolver nil
quartzcron_schedule an optional cron expression to periodically execute the application nil

Shiva runtime specific settings:

Parameter Description Default
shiva_runner_tags A list of required tags. A shiva runner can run this application only if it posseses (at least) all these tags (refer to shiva node configuration un deployment setting).
Common error is to think that ["host1", "host2"] tags list, will allow the runner host1 or the runner host2 to launch the application. Instead, the application will never launch because host1 does not have tag host2 and vice versa. To target a group of runners, you need to define a common tags for all the runners.
nil

The resources is an array of resources. In 6.x releases only kafka topics are supported with following settings :

Parameter Description
type Resource type : only "kafka_topic" is supported
topic Topic name
cluster Cluster name
partitions Partitions number
replication_factor Number of replicas
delete_retention_time Delete retention time. Format number : s, m, h or d
retention_time Retention time. Format number : s, m, h, or d. "-1" for infinite
retention_bytes Retention bytes. Format number : m, g, k or t. "-1" for infinite
cleanup_policy Cleanup policy.

Applications

The punch ships in with several applications ready to be used and listed hereafter. Note that each application you define is uniquely reffered to as <tenant_name>/<channel_name>/<cluster>/<application_name>.

Punchlinectl

To start a punchline use the following command and argumens:

Parameter Values
--punchline The punchline configuration name.
--runtime One of flink spark pyspark or punch for (resp.) the flink, spark, pyspark or punch execution engine.
--childopts The optional jvm arguments

Tip

The arguments correspond to the punchlinectl command on line arguments.

Planctl

To start a plan use the planctl tool on line arguments:

Parameter Values
--template The plan template configuration file. This is typically a punchline template file.
--plan The plan configuration file.
--runtime One of the supported punch execution engine: shiva, storm.

Here is an example:

stop_by_tenant: true
version: "6.0"
start_by_tenant: true

applications:
- args:
  - start
  - --plan
  - plan.yaml
  - --template
  - punchline.yaml
  - --runtime
  - spark
  - --spark-cluster
  - common
  - --deploy-mode
  - client
  - --last-committed  # persistence
  cluster: common
  shiva_runner_tags:
  - common
  name: plan-aggregation
  runtime: shiva
  command: planctl

Logstash

Logstash is fully integrated into the punch. Here is the example shipped with the standalone:

version: '6.0'
start_by_tenant: true
stop_by_tenant: true
applications:

- name: input
  runtime: shiva
  cluster: common
  shiva_runner_tags: []
  command: logstash
  args:
  - -f
  - logstash.conf
  -  --path.data
  - /tmp/myChannel_input

- name: print
  runtime: shiva
  cluster: common
  command: punchlinectl
  args:
  - start
  - --punchline
  - punchline.yaml
  - --childopts
  - -Xms256m -Xmx256m

resources:
- type: kafka_topic
  name: mytenant-logstash
  cluster: common
  partitions: 1
  replication_factor: 1

Tip

In order to deploy several instance of logstash on the same processing server each logstash instance must use its own directory referenced by the setting --path.data.

Elastalert

Elastalert is fully integrated into the punch. The corresponding command is elastalert. You must provide an Elastalert configuration and a rules folder or a single rule using the --rule Elastalert option. Here is the example shipped with the standalone:

version: '6.0'
start_by_tenant: false
stop_by_tenant: true
applications:
- name: elastalert
  runtime: shiva
  command: elastalert
  args:
  - --config
  - config.yaml
  - --verbose
  cluster: common
  shiva_runner_tags:
  - common

Take a look at Elastalert documentation.

Housekeeping

The punch provides seveveral housekeeping applications. Here is the standalone mytenant channel example that monitors the other mytenant channels.

version: '6.0'
start_by_tenant: true
stop_by_tenant: true
applications:

  - name: elasticsearch-housekeeping
    runtime: shiva
    cluster: common
    command: elasticsearch-housekeeping
    args:
      - --tenant-configuration-path
      - elasticsearch-housekeeping.json
    apply_resolver_on:
      - elasticsearch-housekeeping.json
    quartzcron_schedule: 0 * * ? * * *

  - name: archives-housekeeping
    runtime: shiva
    cluster: common
    command: archives-housekeeping
    args:
      - archives-housekeeping.yaml
      - --childopts
      - -Xms100m -Xmx500m
    quartzcron_schedule: 0 * * ? * * *
resources: [ ]
Application Description
elasticsearch-housekeeping elasticsearch data housekeeper in charge of cleaning old elasticsearch indexes
archives-housekeeping long term storage data housekeeper

Channel Monitoring

The punch provides a dedicated channel monitoring application channels-monitoring that computes health status metrics for channels. Refer to the channel monitoring guide.

Tip

this application is deployed in each tenant. This ensure each tenant is monitored using a dedicated application and not affected by other tenants operations.

Platform Monitoring

The punch provides a dedicated platform monitoring application platform-monitoring that computes health status metrics for an entire platform. Refer to the platform monitoring guide.

Tip

this application is deployed in the platform tenant.

Java Applications

Shiva can execute an arbitrary application as long as its jar is installed on each shiva node.

version: '6.0'
start_by_tenant: true
stop_by_tenant: true
applications:
- name: my-java-app 
  runtime: shiva
  cluster: common
  shiva_runner_tags:
  - common
  command: java
  args:
  - -jar
  - myjar.java
}