Using Templates

To create new channels you have two options. First you can refer to the punch documentation (in particular the punchline nodes documentation), invent some punchlines and applications, then group them in a channel, all by hand.

A second option is to work with templates to ease the channel configuration files generations. This second option is useful if you have many similar channels. It is also useful to quickly change your channels structure as illustrated hereafter.

To understand the logic the best is to first go through the online documentation:

channelctl help configure

In a nutshell:

1. a channel high level configuration yaml or json file : in there you define only the most important properties of your channel. A typical example is the listening (tcp) port, the punchlets and the output elasticsearch cluster.
2. template file(s) to generate the detailed configuration files : these are .j2 jinja2 files, one for each required channel configuration file.

Example

Have a look at the tenants/mytenant/etc/channel_config folder. There you will find the channel high-level configuration yaml files.

Next have a look next at the tenants/mytenant/etc/templates folders. Some (storm_single or shiva_single) can be used to generate the example channels you just executed. Others (storm_input_kafka_processing or shiva_input_kafka_processing) are variants to generate channels made of two punchlines with a Kafka topic in between.

The sourcefire channel shipped with the standalone is a single punchline channel. Let us transform it into a dual punchline channel with two punchlines communcating through a kafka topic.

Stop all your channels. Then go to the channel configuration folder:

cd $PUNCHPLATFORM_CONF_DIR/tenants/mytenant/etc/channel_config

next execute

channelctl configure --profile=storm_input_kafka_processing sourcefire_channel.yaml --override

Have a look at your channel it now contains two punchlines. Restart it:

channelctl start --channel sourcefire

and inject some data:

punchplatform-log-injector.sh -c resources/injector/mytenant/asourcefire_injector.json

Your channel is now composed of two punchlines, the first one pushes the logs to a Kafka topic, the second one consumes that topic to parse the logs and inserts them into elasticsearch. An easy way to visualise this new setup is to visit the Storm UI on http://localhost:8080. You should see your two punchlines.

Note

in the tenants/mytenant/channels/sourcefire folder, have a quick look at the channel_structure.yaml file. This is the one that defines the overall structure of your channel. Compare it to the sourcefire original channel.

This concludes our 10 minutes tour, in order to come back to the original single channel layout, simply type in :

channelctl configure --profile=single sourcefire_channel.yaml --remove

Here the remove option make all files removed before regenerating the channel. This is to get rid of the additional punchline.