Skip to content

Punchlets

Now that you have a sense of what Elasticsearch, Kibana and Beats can do, let us move on to punch features. First we will explore punchlets. A punchlet is a small function in charge of transforming your data. A typical example is log parsing. If you are familiar with logstash, think of punchlet as the filter part of a logstash configuration.

Basics

The standalone ships in with simple examples. Go to the sample folder:

cd $PUNCHPLATFORM_CONF_DIR/samples/punchlets
Then run one as follows:
punchplatform-puncher.sh basic/hello.punch
You will get:
{
  "logs": {
    "age": 22,
    "log": "a sample log",
    "user": "bob"
  }
}
TCheck the hello.punch file. the code of that punchlet is the following:
// @test(encoding=json) {"logs":{"log": {"a sample log":}
{
    print(root);
    [logs][user] = "bob";
    print(root);
    [logs][age] = 22;
    print(root);
}
Punchlets can be packaged in vartious ways. Here is an example punchlet that is contained in a yaml file. That makes it easier to document it, and to define the test input data in a more readable way.

Execute this second example:

punchplatform-puncher.sh advanced/operators_ipmatch.yaml
You will get
{
  "check": true,
  "logs": {
    "log": "172.16.0.2"
  }
}
{
  "check": false,
  "logs": {
    "log": "5.36.18.2"
  }
}
Here is the complete yaml file.
description: >
  Here is a simple example of the ipmatch operator.
  You want to know if an IP is in a list of ranges.
tests:
- logs:
    log: 172.16.0.2
- logs:
    log: 5.36.18.2
resources:
  ranges:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16
  - 127.0.0.1/32
punchlet: >
  {
    // Retrieve our IP domains from a punch resource.
    // Resources comes from outside the punchlet. In this
    // example it is defined above but in production
    // it typically come from a S3 store, a remote filesystem
    // etc.. 
    Tuple ranges = getResourceTuple("ranges");

    // You can use that resource tuple like any other.
    [check] = ipmatch(ranges).contains([logs][log]);
  }
In there, the code of that particular punchlet is quite simple. It checks if an IP address belongs to some defined range.
{
  Tuple ipRange = getResourceTuple("ranges");
  [check] = ipmatch(ipRange).contains([logs][log]);
}
The resources tupe ipRange fetches and contains the data you provide outside your punchlet. In this all-in-one example, that data is actually defined inside the same yaml file:

resources:
  ranges:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16
  - 127.0.0.1/32

On production platform such referential or configuration data can come from various places, typically csv yaml or json files.

Have a look at that example file as well as other examples, they are self-explanatory. The Punch language is powerful and comes with a complete online documentation. You will later on see how to invoke it from various stream or batch applications.