punchplatform-puncher.sh

Synopsis

punchplatform-puncher.sh [options]

Description

Develop, test and run punchlets and parsers.

Command

• -c,--code <arg>

  • Use an inline punchlet code

• -g,--grok <arg>

  • Change default grok-pattern resources directory (resource/punch/pattern)

• -h,--help

  • Print help and usage

• -i,--input-fields <arg>

  • Use another input fields key (e.g. [logs][others])

• -I,--input-file <arg>

  • Read log samples from a file

• -it,--interactive

  • Interactive mode, the punchlet will wait for stdin inputs

• -j,--json-input

  • Give a json formatted string as root input to the punchlet

• -o,--output-fields <arg>

  • Use another output fields key (e.g. [a][b])

• -O,--output-file <arg>

  • Write your output to this file

• -p,--punchlet <arg>

  • Punchlet file (.punch)

• -q,--quiet

  • Quiet mode, root tuple will not be printed out

• -r,--resource <arg>

  • Punchlet resource file, .json file or directory

• -t,--run-unit-test <arg>

  • Run a unit test

• -tk,--tick <arg>

  • Processing tick frequency in milliseconds

• -u,--create-unit-test <arg>

  • Generate unit test files

Testing Punchlets

Using the puncher you can define unit test for your punchlets. A unit test is a json file that contains three sections. One defines the punchlet(s) you want to test. If you put several ones, the chain of punchlet will be considered.

The input section lets you define an input log. The output section defines what the output parsed log shoud look like.

Here is a simple example:

{
  "punchlets" : [
    "parser.punch"
  ],
  "input": {
    "logs": { "data" : "color=red city=Rome uri=https://punchplatform.com" }
  },
  "output": {
    "includes": {
        "logs": {
            "data" : "color=red city=Rome uri=https://punchplatform.com",
            "log" : {
                "color": "red",
          "city": "rome",
          "proto": "https",
          "host": "punchplatform.com",
          "uri": "https://punchplatform.com",
          "color_code": 1
            }
        }
    }
  }
}

To define and package such unit tests, refer to the (parser development guide)[../Contribution_Guide/Parser_Developper/Testing_parsers.md].

Examples

Interactive mode, the puncher will wait for user input and output the result for each new line:

punchplatform-puncher.sh -it -c '{[a][b] = "test";}'

Compile multiple punchlets and use [logs][data] as input field:

punchplatform-puncher.sh -i [logs][data] -p p1.punch -p p2.punch

Use a json as punchlet(s) input. It allows more complex inputs based on multiple fields:

punchplatform-puncher.sh -j '{"logs":{"data":"My log"}}' -p p1.punch

Quiet mode, do not output the final root tuple:

punchplatform-puncher.sh -q -c '{[test] = "test"}'

The Puncher can read input from stdin. Feel free to pipe it your sample inputs:

echo "my log" | punchplatform-puncher.sh -c '{;}'
punchplatform-puncher.sh -c '{;}' <<< "my log"
cat log_sample.txt | punchplatform-puncher.sh -p p1.punch
punchplatform-puncher.sh -p p1.punch < log_sample.txt

Set input and output files:

punchplatform-puncher.sh -I log_sample.txt -O dir/output.txt -p p1.txt

Limit the speed processing using a frequency tick. Here, input logs will be processed every 2 seconds (value in milliseconds):

punchplatform-puncher.sh -I log_sample.txt -p p1.punch -tk 2000

Test grok or dissect pattern

   punchplatform-puncher.sh -gi "AAA000DDDD" -gp "(?<queue_id>[0-9A-F]{10,11})"

   punchplatform-puncher.sh -di 'hello world' -dp '%{a} %{b}'

See also

environment