Skip to content

HOWTO test a parser before going to production

Why do that

In production context, the most value is in the running code (parser, aggregation, anomaly job). To improve the quality of the service, the platform owner has to frequently update this code. This HOWTO procedure is only a method example to go to production.

The PunchPlatform Professional Services provides a list of standard parser. They are best practice examples!

From the Professional Services experiences:

: - Most of the parser are not standard - Before each update, the platform owner has to test the update to check the performance impact and the result of the change. He doesn 't need a big device. It 's better if he can test on local. Benefits are: work every where, use the PunchPlatform Sublime Text plugin, just custom his environment. - Do not update code on Fridays ...

Prerequisites

  • A recent PunchPlatform standalone

What to do

Install the Standalone

1
2
$ ./install.sh -s
$ source ~/.bashrc

No need to start anything

Identify the parser (punchlets)

For example say your topology chain the following punchlets:

  • standard/common/input.punch
  • standard/common/parsing_syslog_header.punch
  • standard/apache_httpd/parsing.punch
  • standard/apache_httpd/enrichment.punch
  • standard/apache_httpd/normalization.punch

You possibly need external resources for enrichment

  • standard/apache_httpd/http_codes.json
  • standard/apache_httpd/taxonomy.json

Identify your raw logs:

  • Take it from production
  • Use a default log-injector (for instance $PUNCHPLATFORM _CONF _DIR/resources/injector/lmr/apache _injector.sh)
  • Construct a complexe injector configuration file : punchplatform-log-injector.sh

You have unit tests:

If you have followed the parser development procedure, or if you got the parser update from the PunchPlatform Professional Services, you should have unit tests associated with your punchlet.

To test your unit tests against your new parser, simply type in:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ punchplatform-puncher.sh -t path/to/my/unittest/unit_1.json
# to test all unit tests at once:
$ punchplatform-puncher.sh -t path/to/my/unittest
# To test it agains a "staging" parser chain:
$ punchplatform-puncher.sh -t path/to/my/unittest   
  -p tmp_dir_punchlets/common/input.punch  
  -p tmp_dir_punchlets/common/parsing_syslog_header.punch  
  -p tmp_dir_punchlets/apache_httpd/parsing.punch  
  -p tmp_dir_punchlets/apache_httpd/enrichment.punch  
  -p tmp_dir_punchlets/apache_httpd/normalization.punch

It will give you if the test(s) passed and the differentials.

Unit test your punchlets

The folowing command let you check everything is allright on a given log.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
$ punchplatform-log-injector.sh -c apache_httpd_injector.json --punchlets standard/common/input.punch,standard/common/parsing_syslog_header.punch,standard/apache_httpd/parsing.punch,standard/apache_httpd/enrichment.punch,standard/apache_httpd/normalization.punch --resources standard/apache_httpd/http_codes.json,standard/apache_httpd/taxonomy.json -n 1 -v

registering punchlet: standard/common/input.punch
...
19:52:07 c.t.s.c.p.p.resources [INFO] message="registered regular tuple" size=57 resource_name="http_codes"
...
punchlets compiled
[Wed Nov 22 19:52:11 CET 2017] client.apache_httpd_injector.json0 starts ....
input string ===========================
Nov 22 19:52:11 host0 128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] "GET /images/KSC-94EC-412-small.gif HTTP/1.0" 200 23279 "http://www.example.com/start.html" "Mozilla/5.0 (iPad; U; CPU OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5"
input tuple ===========================
{
  "logs": {
    "raw_log": "Nov 22 19:52:11 host0 128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] "
  }
}
19:52:11 c.t.s.c.p.u.PunchEnvironment [INFO] message="detected host ip" host_ip=127.0.0.1
19:52:11 c.t.s.c.p.u.PunchEnvironment [INFO] message="detected host name" host_name=MacBook-Pro-de-loic.local
19:52:11 c.t.s.c.p.p.r.o.Contains [INFO] built index for 189 entries for key set [code] in 8.033596ms
output tuple ===========================
{
  "logs": {
    "data": "128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] ",
    "log": {
      "app": {
        "method": "GET",
        "return": {
          "code": "200"
        }
      },
      "col": {
        "host": {
          "name": "MacBook-Pro-de-loic.local"
        }
      },
      "obs": {
        "host": {
          "name": "host0"
        },
        "ts": "2012-12-31T01:00:00.000+01:00"
      },
      "init": {
        "process": {
          "name": "Mozilla/5.0 (iPad; U; CPU OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5"
        },
        "host": {
          "ip": "128.216.77.224"
        }
      },
      "lmc": {
        "parse": {
          "host": {
            "ip": "127.0.0.1",
            "name": "MacBook-Pro-de-loic.local"
          },
          "ts": "2017-11-22T19:52:11.435+01:00"
        }
      },
      "session": {
        "out": {
          "byte": 23279
        }
      },
      "channel": "unknown",
      "type": "web",
      "target": {
        "host": {
          "name": "host0"
        },
        "uri": {
          "urn": "/images/KSC-94EC-412-small.gif"
        }
      },
      "taxo": {
        "nf": {
          "sev": "2",
          "alarm": "160018"
        }
      },
      "size": 307,
      "web": {
        "header": {
          "referer": "http://www.example.com/start.html"
        }
      },
      "vendor": "unknown",
      "action": "OK",
      "rep": {
        "host": {
          "name": "host0"
        },
        "ts": "2017-11-22T19:52:11.000+01:00"
      },
      "tenant": "unknown"
    },
    "raw_log": "Nov 22 19:52:11 host0 128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] ",
    "es_index": "unknown-events-2017.11.22"
  }
}
output string ===========================
{"logs":{"data":"128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] ","es_index":"unknown-events-2017.11.22"}}
[Wed Nov 22 19:52:11 CET 2017] client.apache_httpd_injector.json0 duration (s): 0     sent-msg : 1          rate (1/s): 17.2
[Wed Nov 22 19:52:11 CET 2017] client.apache_httpd_injector.json0 stopped.

Make sure this is the expected output.

Run a performance test:

Last chack your punchlets have good performance:

1
$ punchplatform-log-injector.sh -c apache_httpd_injector.json --punchlets standard/common/input.punch,standard/common/parsing_syslog_header.punch,standard/apache_httpd/parsing.punch,standard/apache_httpd/enrichment.punch,standard/apache_httpd/normalization.punch --resources standard/apache_httpd/http_codes.json,standard/apache_httpd/taxonomy.json -t 50000