Skip to content

Developer and Testing Commands

When designing punchlines, you need a way to quickly run them without the burden to pass through a channel.

The punch provides you with an additional punchlinectl command to launch the many variant of punchlines. A punchline can indeed be a stream or batch application you run in various runtime environments.

By default, the punchlinectl tool detects the required runtime. Storm compatible punchlines are executed in the punch lightweight storm compatible runtime. Spark punchline are executed as a client or foreground mode spark application.

You can control which runtime and which deployment mode you need using the punchlinectl arguments.

Once satisfied, simply include your new punchline in a channel, you will then be able to start it using the regular channelctl command.

Designing Channels with Templates

Writing channel and punchline configuration files is rendered simpler using templating. The punch provides you with a tool (channelctl configure command) to generate you complete channel configuration files to greatly simplify the settings of the various parameters and options.

Important

Using channels templates to generate all your variable channels is very much advised when you have similarily-structured channels.

This is key to efficient updates of the platform or of its configuration, enabling to apply the same change to multiple channels while reducing the configuration management effort.

Templating is based on two files:

  • a template directory folder is supposed to be stored in tenants//etc/templates or in one of its sub-folders.
  • a channel configuration hjson file providing variables used to configure the template for a specific channel. It is usually stored in tenants/<mytenant>/etc/channels_config folder.

Configuration files for a channel can be instantiated from the templates by issuing channelctl -t <mytenant> configure /path/to/the/channel/config/file.

The applicable template directory is indicated in the channel configuration file through the special channel_structure_profile setting, that is the relative path to the template directory folder from the tenants/<mytenant>/etc/templates folder.

All *.j2 files from the template directory folder will be submitted to jinja2 templating engine, for rendering actual configuration files in the channel folder (tenants/<mytenant>/channels/<channelName>/).

The jinja2 tags, in channel configuration template files, can refer to variables provided in the channel file through the channel name scope. E.g. if a 'technology: apache_httpd' variable in the channel configuration file, then {{channel.technology}} tag in the template will be replaced by apache_httpd when generating the channel configuration/.

Tip

By quickly switching from one template set to another, using the same channel configuration file (through --profile setting that overrides the used template folder), you can easily test your channels in different variations (e.g. with or without Kafka retention for instance).

This is handy and often used to generate simpler channels to focus on parsing issues in development/integration platforms. Then actual production template is used to deploy the parsers in a real production applications chain.

Punchline Troubleshooting

When a punchline does not behave correctly when started inside a Shiva cluster or Storm cluster, you can try to run it in foreground on the operator node itself, which will directly display the events/error log of the punchline on the operator console:

  • Stop it in the cluster, by using channectl -t <myTenant> stop --application <channel>/<cluster>/<applicationName>. (issue a channelctl [...] describe or channelctl [...] status command on the channel to find out the exact name for your application.)

  • Then start your processing in foreground mode : 

      punchlinectl -t <myTenant> --punchline <myPunchlineFile>

You can stop the execution by pressing <ctrl-c>.

Warning

Some punchlines are designed to run with some resources that maybe available only on the runtime cluster (shiva or storm) worker nodes, not necesserally the local operator node.

This can cause other errors than the one that you are actually investigating. MakeBe sure to know what resources your punchline is requiring locally, such as:

  • Access to (archiving) filesystems mount points (like /mnt/archive_filesystem)

  • Custom resources (e.g. geographic data, e.g. /data/geo/mydb.dat)

  • Security credentials (e.g. /data/opt/secrets/my-client-certificate.key)

Note

Most mis-configured punchlines cannot be started as they are checked for consistency before being submitted to a runtime storm or shiva cluster. Use the --audit channelctl option to have details about the most common misconfiguration issues.