Skip to content

Channels

A channel groups several punchlines and/or plans into a consistent and useful unit. Channels can also can also contain Logstash, administrative tasks and much more. Using channel you can model complete applications, mixing stream, batch and arbitrary tasks.

Once you defined a channel, you can start or stop it. All its punchlines plan (or other parts) will be started or stopped accordingly. It is that simple and that powerful.

The standalone punch comes with a number of demo channels defined as part of a demo tenant called mytenant.

ls $PUNCHPLATFORM_CONF_DIR/tenants/mytenant/channels
admin  aggregation  apache_httpd  sourcefire  stormshield_networksecurity  universal  websense_web_security

Here is a quick tour of these demo channels:

  • sourcefire stormshield_networksecurity websense_web_security are examples of single punchline channels. They process logs and index them into Elasticsearch.
  • apache_httpd : that channel processes apache logs. In addition it also illustrates how you can archive your logs to the filesystem. This illustrates a two-punchlines channel and the use of a kafka topic in between.
  • universal : illustrates the use of a punch language goodie to implement a universal log parser.
  • logstash : a simple channel launching a logstash instance

To start any of these, use the channelctl command:

channelctl

Info

This command works per tenant. By default on a single tenant platform, it knows which tenant to consider. Should you have more than one, use the --tenant option. channelctl --tenant mytenant or channelctl -t mytenant

The channelctl command line provides lots of help and auto-completion facilities using the tab key. In the rest of this chapter we provide some explanations for you to get it completely.

Stream processing

Let us consider the simplest channels you can think of, composed of simple streaming punchlines to parse continuously parse logs, received on a TCP socket, and indexed into Elasticsearch once transformed in a normalised and enriched json data. Start the sourcefire channel:

channelctl:mytenant> start --channel sourcefire
application:storm:sourcefire/main/input  (mytenant_sourcefire_input-10-1579413670) .............................................. ACTIVE

You can now inject some sourcefire logs using the punch injector tool. It will generate sourcefire logs and send them to your punchlines.

punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant/sourcefire_injector.json

Play similarly with the other channels. For example the apache_httpd channel.

channelctl:mytenant> start --channel apache_httpd
application:storm:apache_httpd/main/input  (mytenant_apache_httpd_input-8-1579413470) ........................................... ACTIVE
application:storm:apache_httpd/main/archiving  (mytenant_apache_httpd_archiving-9-1579413471) ................................... ACTIVE

As you can see this channel started two distinct punchlines : input and archiving. Check their running status using the status command

channelctl:mytenant> status
channel:logstash ........................................................................................................ STOPPED
channel:stormshield_networksecurity ..................................................................................... STOPPED
channel:admin ........................................................................................................... STOPPED
application:storm:sourcefire/main/input  (mytenant_sourcefire_input-10-1579413670) .............................................. ACTIVE
channel:sourcefire ...................................................................................................... ACTIVE
channel:aggregation ..................................................................................................... STOPPED
channel:websense_web_security ........................................................................................... STOPPED
channel:universal ....................................................................................................... STOPPED
application:storm:apache_httpd/main/archiving  (mytenant_apache_httpd_archiving-9-1579413471) ................................... ACTIVE
application:storm:apache_httpd/main/input  (mytenant_apache_httpd_input-8-1579413470) ........................................... ACTIVE
channel:apache_httpd .................................................................................................... ACTIVE
To inject apache logs :

punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant/apache_httpd_injector.json

Your (sourcefire and apache) logs are now parsed and indexed into Elasticsearch. In addition apache logs are also archived as compressed csv files under the /tmp directory.

Feel free to explore the various channectl commands, auto-completion and inline documentation are your friends. In particular it is important to understand:

channelctl:mytenant> status --help

Good to know you can also use non interactive variants. Hit Ctrl-C or Ctrl-D to exit. Then simply type:

channelctl status

If you want to generate more types of logs simply type in the following command to start all injector file found at once:

punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant

This will generate all the logs as defined in the $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant folder. When you are done, stop injection with Ctrl-C and stop your channel. To do that you can use again the channelctl interactive tool. You can also simply directly type in:

channelctl stop
You might wonder where exactly are running your punchlines ? Are they started as plain java applications ? submitted to a storm, spark or punch shiva cluster ? That is an excellent question. Check out the storm ui at (http://localhost:8080)[http://localhost:8080]. You will see there that some of the punchline indeed run as storm applications.

What about the stormshield_networksecurity punchline ? This one runs in the punch runtime called shiva. Shiva is a lightweight job scheduler somehow equivalent (but simpler) to Storm or Spark. If you check the status using channelctl you will notice that it is indeed running in shiva:

application:shiva:stormshield_networksecurity/common/input  (punchplatform/mytenant/channels/stormshield_networksecurity/input) . ACTIVE

Batch processing

Now that you are comfortable with streaming, let's move on to batch processing. We will run a continuous aggregation channel based on a PML Plan (Spark). This aggregation is executed each minute and fetch all the logs stored in the mytenant-events-* Elasticsearch index since the last minute. Here, by minute, we want to compute:

  1. how many bytes have been written to this index
  2. what was the size (in bytes) of the biggest log

Before running the aggregation, we need to provide some data. To do so, let's start two channels with the channelctl and inject some logs.

channelctl start --channel sourcefire
channetctl start --channel websense_web_security

Now that channels are running, let's inject some logs:

punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injector/mytenant

It is important to keep injecting logs in real time because the aggregation will only fetch the last minute's logs. Keep the log injector running and start a new terminal. From the new terminal, type in this command to start the aggregation:

# From another terminal
channelctl start --channel aggregation
application:shiva:aggregation/common/plan-example-1  (punchplatform/mytenant/channels/aggregation/plan-example-1) ............... ACTIVE

Wait about a 1 minute, the time for the first aggregation to be completed. Then, a new Elasticsearch index should shows up with this name mytenant-aggregations-YYYY.MM.DD. Add this new index pattern to Kibana and see the results. The documents have the following fields:

{
  "_index": "mytenant-aggregations-2019.05.15",
  "_type": "_doc",
  "_id": "QmHvu2oBm9lH_e9QjytC",
  "_version": 1,
  "_score": null,
  "_source": {
    "total_size_value": 1013339,
    "max_size_value": 298,
    "key": "sourcefire",
    "timestamp": "2019-05-15T16:40:00.148+02:00"
  },
  "fields": {
    "timestamp": [
      "2019-05-15T14:40:00.148Z"
    ]
  },
  "sort": [
    1557931200148
  ]
}

As you can see, we get an overview of the total log size and the larger log size over the last minute sorted by technology vendor. Note that one event is generated by vendor each minute. The vendor can be found in the field "key". In this example, the vendor is "sourcefire".

To stop everything, run the following commands:

# first, stop the aggregation channel
channelctl stop --channel aggregation

# then, stop any existing channel
channelctl stop

Congratulation ! Now, you are ready for high performance stream or batch processing applications !