HOWTO test a parser before going to production
Why do that¶
In production context, the most value is in the running code (parser, aggregation, anomaly job). To improve the quality of the service, the platform owner has to frequently update this code. This HOWTO procedure is only a method example to go to production.
The PunchPlatform Professional Services provides a list of standard parser. They are best practice examples!
From the Professional Services experiences:
: - Most of the parser are not standard - Before each update, the platform owner has to test the update to check the performance impact and the result of the change. He doesn 't need a big device. It 's better if he can test on local. Benefits are: work every where, use the PunchPlatform Sublime Text plugin, just custom his environment. - Do not update code on Fridays ...
Prerequisites¶
- A recent PunchPlatform standalone
What to do¶
Install the Standalone¶
1 2 | $ ./install.sh -s
$ source ~/.bashrc
|
No need to start anything
Identify the parser (punchlets)¶
For example say your topology chain the following punchlets:
- standard/common/input.punch
- standard/common/parsing_syslog_header.punch
- standard/apache_httpd/parsing.punch
- standard/apache_httpd/enrichment.punch
- standard/apache_httpd/normalization.punch
You possibly need external resources for enrichment
- standard/apache_httpd/http_codes.json
- standard/apache_httpd/taxonomy.json
Identify your raw logs:¶
- Take it from production
- Use a default log-injector (for instance $PUNCHPLATFORM _CONF _DIR/resources/injector/lmr/apache _injector.sh)
- Construct a complexe injector configuration file : punchplatform-log-injector.sh
You have unit tests:¶
If you have followed the parser development procedure, or if you got the parser update from the PunchPlatform Professional Services, you should have unit tests associated with your punchlet.
To test your unit tests against your new parser, simply type in:
1 2 3 4 5 6 7 8 9 10 | $ punchplatform-puncher.sh -t path/to/my/unittest/unit_1.json # to test all unit tests at once: $ punchplatform-puncher.sh -t path/to/my/unittest # To test it agains a "staging" parser chain: $ punchplatform-puncher.sh -t path/to/my/unittest -p tmp_dir_punchlets/common/input.punch -p tmp_dir_punchlets/common/parsing_syslog_header.punch -p tmp_dir_punchlets/apache_httpd/parsing.punch -p tmp_dir_punchlets/apache_httpd/enrichment.punch -p tmp_dir_punchlets/apache_httpd/normalization.punch |
It will give you if the test(s) passed and the differentials.
Unit test your punchlets¶
The folowing command let you check everything is allright on a given log.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | $ punchplatform-log-injector.sh -c apache_httpd_injector.json --punchlets standard/common/input.punch,standard/common/parsing_syslog_header.punch,standard/apache_httpd/parsing.punch,standard/apache_httpd/enrichment.punch,standard/apache_httpd/normalization.punch --resources standard/apache_httpd/http_codes.json,standard/apache_httpd/taxonomy.json -n 1 -v registering punchlet: standard/common/input.punch ... 19:52:07 c.t.s.c.p.p.resources [INFO] message="registered regular tuple" size=57 resource_name="http_codes" ... punchlets compiled [Wed Nov 22 19:52:11 CET 2017] client.apache_httpd_injector.json0 starts .... input string =========================== Nov 22 19:52:11 host0 128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] "GET /images/KSC-94EC-412-small.gif HTTP/1.0" 200 23279 "http://www.example.com/start.html" "Mozilla/5.0 (iPad; U; CPU OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5" input tuple =========================== { "logs": { "raw_log": "Nov 22 19:52:11 host0 128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] " } } 19:52:11 c.t.s.c.p.u.PunchEnvironment [INFO] message="detected host ip" host_ip=127.0.0.1 19:52:11 c.t.s.c.p.u.PunchEnvironment [INFO] message="detected host name" host_name=MacBook-Pro-de-loic.local 19:52:11 c.t.s.c.p.p.r.o.Contains [INFO] built index for 189 entries for key set [code] in 8.033596ms output tuple =========================== { "logs": { "data": "128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] ", "log": { "app": { "method": "GET", "return": { "code": "200" } }, "col": { "host": { "name": "MacBook-Pro-de-loic.local" } }, "obs": { "host": { "name": "host0" }, "ts": "2012-12-31T01:00:00.000+01:00" }, "init": { "process": { "name": "Mozilla/5.0 (iPad; U; CPU OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5" }, "host": { "ip": "128.216.77.224" } }, "lmc": { "parse": { "host": { "ip": "127.0.0.1", "name": "MacBook-Pro-de-loic.local" }, "ts": "2017-11-22T19:52:11.435+01:00" } }, "session": { "out": { "byte": 23279 } }, "channel": "unknown", "type": "web", "target": { "host": { "name": "host0" }, "uri": { "urn": "/images/KSC-94EC-412-small.gif" } }, "taxo": { "nf": { "sev": "2", "alarm": "160018" } }, "size": 307, "web": { "header": { "referer": "http://www.example.com/start.html" } }, "vendor": "unknown", "action": "OK", "rep": { "host": { "name": "host0" }, "ts": "2017-11-22T19:52:11.000+01:00" }, "tenant": "unknown" }, "raw_log": "Nov 22 19:52:11 host0 128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] ", "es_index": "unknown-events-2017.11.22" } } output string =========================== {"logs":{"data":"128.216.77.224 - frank [31/Dec/2012:01:00:00 +0100] ","es_index":"unknown-events-2017.11.22"}} [Wed Nov 22 19:52:11 CET 2017] client.apache_httpd_injector.json0 duration (s): 0 sent-msg : 1 rate (1/s): 17.2 [Wed Nov 22 19:52:11 CET 2017] client.apache_httpd_injector.json0 stopped. |
Make sure this is the expected output.
Run a performance test:¶
Last chack your punchlets have good performance:
1 | $ punchplatform-log-injector.sh -c apache_httpd_injector.json --punchlets standard/common/input.punch,standard/common/parsing_syslog_header.punch,standard/apache_httpd/parsing.punch,standard/apache_httpd/enrichment.punch,standard/apache_httpd/normalization.punch --resources standard/apache_httpd/http_codes.json,standard/apache_httpd/taxonomy.json -t 50000
|