In this section, you will learn how to test and validate a parser using two methods. Feel free to choose the one you are more comfortable with.
Parser Package Layout¶
Below is what looks like the structure of a parser package currently delivered by the Punchplatform Team. Let's take the example of the Apache parser throughout this tutorial.
apache_httpd/ ├── README.md ├── resources │ ├── http_codes.json │ └── taxonomy.json ├── src │ ├── enrichment.punch │ ├── normalization.punch │ └── parsing.punch ├── structure.json └── test ├── sample.txt ├── unit_1.json └── unit_2.json
This package contains the documentation relating to the parser itself, the associated tests with a sample of data and,
of course, the parser itself, named
parsing.punch by convention. You can also find eventual resource files needed for potential
normalization or enrichment of the data.
Punch Test Tools¶
In order to test a parser package in this first method, we will use a tool shipped with the
If you already have it set up, you can skip the next point.
Set up your Punchplatform environment
- Download the last stable version of the Punchplatform Standalone by visiting this link.
- Once you have got it, follow the quick installation steps available here.
- If everything is running properly, you are good to go.
Test your parser with
- First of all, unzip your parser package to the desired location, you should obtain a repository named after the parser you want to test.
- Then use the following command to go through the unit tests and check if your parser is correct:
punchplatform-puncher.sh -T apache_httpd/
For example, with the
Apache parser, here is the output we get after using this tool:
[INFO] Adding punchlet: parsing.punch [INFO] Adding punchlet: enrichment.punch [INFO] Adding punchlet: normalization.punch [INFO] Adding resource: http_codes.json [INFO] Adding resource: taxonomy.json Playing 3 tests cases. Playing "unit_1.json": ✓ 1/1 Succeed ✓ PASSED Playing "sample.txt": ✓ 1/2 Succeed ✓ 2/2 Succeed ✓ PASSED Playing "unit_2.json": ✓ 1/1 Succeed ✓ PASSED ✓ 3/3 playbooks passed.
Testing Without a Punch Environment¶
The other way to test your parser can be performed without downloading a standalone and installing the
punch related environment.
However, you will need to have
OpenJDK 8 installed in order to proceed further. We will indeed use a
JAR to run our tests on the parser.
- First, download the JAR from this download area and
Puncher Jarartifact. You should get a JAR named
- Once you are done, unzip your parser package to the desired location, you should obtain a repository named after the parser you want to test.
- Then run the following command:
java -cp punchplatform-puncher-jar-with-dependencies.jar org.thales.punch.libraries.punch.core.Puncher -T ./apache_httpd -g grok_patterns/
Some parsers are using GROK patterns to process the data, therefore the option
-g grok_patterns/ will be mandatory in order to make those parsers work properly.
You NEED to ensure that this
grok_patterns/ directory has been delivered to you independently of the parser package as it is not contained within it.
To run performance tests on your parsers before using them into production, we will need another tool called
punchplatform.log-injector.sh. A punchplatform standalone is required in order
to have access to it and proceed further so make sure to have one installed.
Please visit this page to get full information about how to test a parser before going to production. You will go through a tutorial to test the performance of our standard Apache parser.