Channels

Abstract

This chapter explains how you assemble various applications into a channel and why this concept is both powerful and necessary.

A channel describes where one or several application must be executed. The channel abstraction provides you with a concise and clear declarative approach to organise your applications in ways easy to manage.

Configuration¶

Each channel is defined using a yaml channel_structure.yaml file. Refer to the configuration file to understand the tenant/channel/application tree organisation.

The channel_structure.yaml file has the following structure:

version: "6.0"
start_by_tenant: true
stop_by_tenant: false
applications:
  - name: mypunchline
    runtime: shiva
    cluster: common
    reload: none
    command: punchlinectl
    quartzcron_schedule : 0/30 * * * * ? *
    args:
      - "start"
      - "--punchline"
      - "mypunchline.yaml"
      - "--runtime"
      - "punch"
      - "--childopts"
      - "-Xms256m -Xmx256m"
    apply_resolver_on:
    - afile.json
    - anotherfile.hjson

resources:
 - type: kafka_topic
   topic: syslog-to-kafka
   cluster: kafka
   partitions: 1
   replication_factor: 1
   delete_retention_time: "1d"
   retention_time: "-1"
   retention_bytes: "-1"
   cleanup_policy: compact

Parameter	Description	Default
`version`	Correspond to the major punch release.	`nil`
`start_by_tenant`	set to true to make that channel startable by a tenant-level command	`true`
`stop_by_tenant`	set to true to make that channel stoppable by a tenant-level command	`false`
`application`	the list of applications	`nil`
`resources`	an optional list of resources associated to this channel	`nil`

Each applications has the following properties:

Parameter	Description	Default
`name`	The application name.	`nil`
`runtime`	`storm` or `shiva`	`nil`
`cluster`	the name of the target (kubernetes\|shiva) cluster	`nil`
`reload`	control the reload policy of this application. `none` or `kill_then_start`	`none`
`command`	one of the punch supported commands	`nil`
`args`	the command args as you would type them in a plain terminal	`nil`
`apply_resolver_on`	an optional list of additional files on which to apply the resolver	`nil`
`quartzcron_schedule`	an optional cron expression to periodically execute the application	`nil`

Shiva runtime specific settings:

Parameter	Description	Default
`shiva_runner_tags`	A list of required tags. A shiva runner can run this application only if it posseses (at least) all these tags (refer to shiva node configuration un deployment setting). Common error is to think that `["host1", "host2"]` tags list, will allow the runner host1 or the runner host2 to launch the application. Instead, the application will never launch because host1 does not have tag host2 and vice versa. To target a group of runners, you need to define a common tags for all the runners.	`nil`

The resources is an array of resources. In 6.x releases only kafka topics are supported with following settings :

Parameter	Description
`type`	Resource type : only "kafka_topic" is supported
`topic`	Topic name
`cluster`	Cluster name
`partitions`	Partitions number
`replication_factor`	Number of replicas
`delete_retention_time`	Delete retention time. Format number : s, m, h or d
`retention_time`	Retention time. Format number : s, m, h, or d. "-1" for infinite
`retention_bytes`	Retention bytes. Format number : m, g, k or t. "-1" for infinite
`cleanup_policy`	Cleanup policy.

Applications¶

The punch ships in with several applications ready to be used and listed hereafter. Note that each application you define is uniquely reffered to as <tenant_name>/<channel_name>/<cluster>/<application_name>.

Punchlinectl¶

To start a punchline use the following command and argumens:

Parameter	Values
`--punchline`	The punchline configuration name.
`--runtime`	One of `flink` `spark` `pyspark` or `punch` for (resp.) the flink, spark, pyspark or punch execution engine.
`--childopts`	The optional jvm arguments

Tip

The arguments correspond to the punchlinectl command on line arguments.

Planctl¶

To start a plan use the planctl tool on line arguments:

Parameter	Values
`--template`	The plan template configuration file. This is typically a punchline template file.
`--plan`	The plan configuration file.
`--runtime`	One of the supported punch execution engine: shiva, storm.

Here is an example:

stop_by_tenant: true
version: "6.0"
start_by_tenant: true

applications:
- args:
  - start
  - --plan
  - plan.yaml
  - --template
  - punchline.yaml
  - --runtime
  - spark
  - --spark-cluster
  - common
  - --deploy-mode
  - client
  - --last-committed  # persistence
  cluster: common
  shiva_runner_tags:
  - common
  name: plan-aggregation
  runtime: shiva
  command: planctl

Logstash¶

Logstash is fully integrated into the punch. Here is the example shipped with the standalone:

version: '6.0'
start_by_tenant: true
stop_by_tenant: true
applications:

- name: input
  runtime: shiva
  cluster: common
  shiva_runner_tags: []
  command: logstash
  args:
  - -f
  - logstash.conf
  -  --path.data
  - /tmp/myChannel_input

- name: print
  runtime: shiva
  cluster: common
  command: punchlinectl
  args:
  - start
  - --punchline
  - punchline.yaml
  - --childopts
  - -Xms256m -Xmx256m

resources:
- type: kafka_topic
  name: mytenant-logstash
  cluster: common
  partitions: 1
  replication_factor: 1

Tip

In order to deploy several instance of logstash on the same processing server each logstash instance must use its own directory referenced by the setting --path.data.

Elastalert¶

Elastalert is fully integrated into the punch. The corresponding command is elastalert. You must provide an Elastalert configuration and a rules folder or a single rule using the --rule Elastalert option. Here is the example shipped with the standalone:

version: '6.0'
start_by_tenant: false
stop_by_tenant: true
applications:
- name: elastalert
  runtime: shiva
  command: elastalert
  args:
  - --config
  - config.yaml
  - --verbose
  cluster: common
  shiva_runner_tags:
  - common

Take a look at Elastalert documentation.

Housekeeping¶

The punch provides seveveral housekeeping applications. Here is the standalone mytenant channel example that monitors the other mytenant channels.

version: '6.0'
start_by_tenant: true
stop_by_tenant: true
applications:

  - name: elasticsearch-housekeeping
    runtime: shiva
    cluster: common
    command: elasticsearch-housekeeping
    args:
      - --tenant-configuration-path
      - elasticsearch-housekeeping.json
    apply_resolver_on:
      - elasticsearch-housekeeping.json
    quartzcron_schedule: 0 * * ? * * *

  - name: archives-housekeeping
    runtime: shiva
    cluster: common
    command: archives-housekeeping
    args:
      - archives-housekeeping.yaml
      - --childopts
      - -Xms100m -Xmx500m
    quartzcron_schedule: 0 * * ? * * *
resources: [ ]

Application	Description
`elasticsearch-housekeeping`	elasticsearch data housekeeper in charge of cleaning old elasticsearch indexes
`archives-housekeeping`	long term storage data housekeeper

Channel Monitoring¶

The punch provides a dedicated channel monitoring application channels-monitoring that computes health status metrics for channels. Refer to the channel monitoring guide.

Tip

this application is deployed in each tenant. This ensure each tenant is monitored using a dedicated application and not affected by other tenants operations.

Platform Monitoring¶

The punch provides a dedicated platform monitoring application platform-monitoring that computes health status metrics for an entire platform. Refer to the platform monitoring guide.

Tip

this application is deployed in the platform tenant.

Java Applications¶

Shiva can execute an arbitrary application as long as its jar is installed on each shiva node.

version: '6.0'
start_by_tenant: true
stop_by_tenant: true
applications:
- name: my-java-app 
  runtime: shiva
  cluster: common
  shiva_runner_tags:
  - common
  command: java
  args:
  - -jar
  - myjar.java
}