Configuration Layout

This chapter explains where the parsers and their related files are located.

Note

Before going further, ensure you have a standalone punchplatform available. Refer to the Getting Started and make sure you completed the Punchlets section.

We start with the configuration directory layout of a PunchPlatform. It is located under the $PUNCHPLATFORM_CONF_DIR folder. Remember this directory holds all the configuration files of your platform. We focus here on those files related to log management:

  • parsers : a set of punchlets parsers, some of them generic to all log types, some of them specific to a particular log type.
  • elasticsearch templates : a set of template to define Elasticsearch mappings. In a nutshel, a mapping is a schema definition to be applied when indexing a document into a given Elasticsearch index. These mappings defines types like integer, String, date or even IPV4 ip addresses format.
  • grok patterns : a set of standard grok patterns (compatible with Logstash), made available to the punchlet Grok operator. A grok pattern is a predefined regex, it takes a String as input and convert it into a well organised Json document.
  • punchlets resources : punchlets can be packaged with their own resource files.
  • kibana dashboards : at the end of their journey, logs are stored into Elasticsearch, and rendered to you through a Kibana dashboard.

All these files must be designed carefully and consistently. Depending on the parsing result, you will probably make dedicated Elasticsearch templates and Kibana dashboards. Changing your parser could make it non indexable anymore into Elasticsearch. In that sense, all these files must be created with caution and shipped as one consistent software package.

These files can be defined for all tenants, per tenants or per channel. The directory layout to define resources at various level is illustrated next.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|   # The resource folder contains global resources, available
|   #to all tenants/channels
├── resources
│   ├── elasticsearch
│   ├── kibana
│   └── punch
│       ├── patterns
│       │   ├── grok-patterns.grok
│       │   ├── httpd.grok
│       │   └── ... 
│       └── standard
│           ├── apache_httpd
│           ├── common
│           ├── sourcefire
│           └── ... 
│      
│ # Directory layout for per tenant/channel configuration files.# These are the ones use when starting/stopping a channel. # Do not update these files by hand, they are generated from
| # samples and templates.
└── tenants
   └── <your_tenant>
       └── channels
           ├── <your_channel>
           │   ├── channel_structure.json
           │   ├── <channel_topology_file(s)>.json

You are free to organise the punchlets the way you want, as long as you end up refering to the right ones in your channels. Let us take an example. Assume you have an [input.punch] punchlet that is common to all your log channels, because it (say) only adds a timestamp in your parsed log. A good practice is to put it under a common per tenant subdirectory:

1
conf/mytenant/resources/punch/standard/common/input.punch

Now let's say you need to change that timestamp format only for a given log channel. Put an updated input.punch in the per channel resources directory.

1
conf/mytenant/resources/punch/standard/<your_log_type>/input.punch

To refer to that punchlet in a topology configuration file, use standard/common/input.punch or standard/<your_log_type>/input.punch. Here is an example of using that later punchlet in a syslog bolt :

1
2
3
4
5
6
7
8
9
"spouts" : [
  {
    "type" : "punch_bolt",
    "spout_settings" : {
      ...
      "punchlet" : "standard/<your_log_type>/input.punch"
    }
  }
]

Going back to the global directory layout, here is the one shipped with the standalone PunchPlatform. It shows you how you can have standard log parsers defined globally, then refered by various log channels. The example hereafter details how the apache channel chain several punchlets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
tenants
  └── mytenant
      ├── channels
      │   ├── apache_httpd
      │   │   ├── channel_structure.json
      │   │   └── single_topology.json
      │   │         uses   standard/common/input.punch
      │   │                standard/common/parsing_syslog_header.punch
      │   │                standard/apache_httpd/parsing.punch
      │   │                standard/apache_httpd/enrichment.punch (with standard/apache_httpd/http_codes.json)
      │   │                standard/apache_httpd/normalization.punch (with standard/apache_httpd/taxonomy.json)
      │   │                standard/common/geoip.punch
      │   └── ...
      ├── etc
      └── ...