Skip to content

Parser Deployment

Abstract

Parser deployment makes your parser usable in a log management punchline.

There are two main modes for parser deployment :

  1. Foreground mode : practical for testing parser within a punchline. Not recommended on production platforms.
  2. Production mode : more steps to manage the entire parser lifecycle. Recommended on production platforms.

Foreground Mode

The goal of the foreground mode is to test and fix a parser within a punchline.

The typical lifecycle of the foreground mode is :

  • package parser.
  • punchlinectl -t mytenant start -p /path/to/punchline.yaml.
  • fix punchlets, resources, groks...
  • punchlinectl -t mytenant start -p /path/to/punchline.yaml
  • fix punchlets, resources, groks...
  • ...
  • package parser for Production Mode.

Currently, you have 2 possible ways to make quick fixes on your parsers :

  1. Extract on Filesystem
  2. Use Maven plugin

It is better to check both and select the one that best fits your needs.

Extract on Filesystem

The first and simplest method is to transfer your punchlets, patterns and resource files to your local tenant configuration tree. This topic is explained in detail in the next chapter.

As a preview what you must do is to copy your files to a special punchplatform configuration folder:

└── conf
    └── tenants
        └─-─ sampletenant
            └── channels
            │   └── samplechannel
            │       ├── channel_structure.yml
            │       └── samplepunchline.yml
            └── resources
                └── punch
                    └── com
                       └── mycompany
                           └── sample
                               ├── MANIFEST.yml
                               ├── groks
                               │   └── pattern.grok
                               ├── enrich.punch
                               ├── parser.punch
                               └── resources
                                   └── color_codes.json

From there you can use your punchlets directly inside your punchlet_code :

version: "6.3.4"
name: dhcp-parser
runtime: storm
dag:
  - type: syslog_input
    settings:
      listen:
        proto: tcp
        host: 0.0.0.0
        port: 9902
    publish:
      - stream: logs
        fields:
          - log
  - type: punchlet_node
    settings:
      punchlet_grok_pattern_dirs:
        - com/mycompany/sample/groks
      punchlet_json_resources:
        - com/mycompany/sample/resources/color_codes.json
      punchlets:
        - com/mycompany/sample/parser.punch
    subscribe:
      - component: syslog_input
        stream: logs

Note that using this method, Maven is not used anymore. You loose the version information. If you use that method, ensure you use git to protect your configuration folder and keep track of your change history.

Important

Always maintain the same tree structure (here com/mycompany/sample) in your layout. That tree structure is an universal and recommended pattern used in Maven and Java. If you conform to that, your punchlines will run fine on other deployed Punch platforms.

Use Maven plugin

The other foreground method is to use Maven. Each time you run mvn clean install within your parser repository, Maven updates the Maven repository in ~/.m2/repository/.

The punchlet_code will fetch the latest parser version in the Maven repository.

This requires to add an extra configuration key .resources in your punchline :

version: "6.3.4"
name: dhcp-parser
runtime: storm
resources:
  - punch-parser:com.mycompany:firewall-parsers:1.0.0
dag:
  - type: syslog_input
    settings:
      listen:
        proto: tcp
        host: 0.0.0.0
        port: 9902
    publish:
      - stream: logs
        fields:
          - log
  - type: punchlet_node
    settings:
      punchlet_grok_pattern_dirs:
        - com/mycompany/sample/groks
      punchlet_json_resources:
        - com/mycompany/sample/resources/color_codes.json
      punchlets:
        - com/mycompany/sample/parser.punch
    subscribe:
      - component: syslog_input
        stream: logs

This method is closer to what is done on Production Mode.

Production Mode (6.3.4 version)

Warning

This method is supported only from release 6.3.4 and higher.

You can alternatively ship the Maven generated archive directly to the repository folder located in your configuration folder. Here it is illustrated:

└── conf
     ├─── repository
     │       └─── firewall-parsers-1.0.0.zip
     └── tenants
        └─-─ sampletenant
            └─── channels
                └── samplechannel
                    ├── channel_structure.yml
                    └── samplepunchline.yml

This requires to add an extra configuration key .resources in your punchline :

version: "6.3.4"
name: dhcp-parser
runtime: storm
resources:
  - punch-parser:com.mycompany:firewall-parsers:1.0.0
dag:
  - type: syslog_input
    settings:
      listen:
        proto: tcp
        host: 0.0.0.0
        port: 9902
    publish:
      - stream: logs
        fields:
          - log
  - type: punchlet_node
    settings:
      json_resources:
        - com/mycompany/sample/resources/color_codes.json
      punchlets:
        - com/mycompany/sample/parser.punch
    subscribe:
      - component: syslog_input
        stream: logs

This method is robust and encouraged on production platforms.

Production Mode (7.0.1 version)

The Production Deployment Mode is designed to manage the entire lifecycle of the parser. In 7.0.1 version, the Resource Manager API is the backend to keep track of the changes within your parser.

Once your parser is pushed to the Resource Manager API, it is also displayed within the Resource Manager UI. This makes it easier to find what parsers are available on your platform.

Currently, there are two ways to push your parsers to the Resource Manager API :

  1. Use Punch Maven Deploy Plugin
  2. Use resourcectl

Punch Maven Deploy Plugin

Once you have cloned the Punch Maven Deploy Plugin, just run maven clean install.

This will add the plugin to your Maven installation.

In each pom.xml generated by the Punch Maven Archetype Generator, there is a section that uses the punch-maven-deploy-plugin.

You have to fill the following fields related to your Resource Manager API :

  • URI
  • username
  • password

Then, just run maven clean deploy inside your parser project folder. This will :

  1. Generate a metadata.yml based on Maven information of your parser project folder.
  2. POST your .zip parser with the metadata.yml to the Resource Manager API.

Resourcectl

It is also possible to push your .zip parser manually with resourcectl.

This requires to write a metadata.yml with type: punch-parsers. Check resourcectl for more information about how to push resources manually.

Punch Operator Format

In Punch 7.0.1 version, the Punchlines are CRDs launched by Punch Operator. The Punch Operator requires the .spec.dependencies key to retrieve the resources in the Resource Manager API.

---
# Kube section
apiVersion: punchline.gitlab.thalesdigital.io/v1
kind: Stormline
metadata:
  name: sourcefire-input
  labels:
    app: sourcefire-input
    tenant: mytenant
    channel: sourcefire
    technology: cisco
  annotations:
    prometheus.io/path: "/metrics"
    prometheus.io/port: "9101"
    prometheus.io/scrape: "true"
    platform.gitlab.thalesdigital.io/platform: platform
spec:
  image: ghcr.io/punchplatform/stormline:7.0.1-SNAPSHOT

  # Prometheus metrics endpoint
  metrics:
    port: 9101


  # Dependencies section
  dependencies:
    - punch-parsers:org.thales.punch:punch-cisco-parsers:1.0.0
    - punch-parsers:org.thales.punch:punch-common-punchlets:1.0.2
    - file:org.thales.punch:punch-geoipv4-resources:1.0.0
    - file:org.thales.punch:punch-geoipv6-resources:1.0.0

  # Punchline section
  punchline:
    platform: dev4
    settings:
      topology.worker.childopts: "-server -Xms1g -Xmx1g"
      topology.max.spout.pending: "2000"
      topology.enable.message.timeouts: "true"
      topology.message.timeout.secs: "60"

    dag:

      # Syslog input
      - type: syslog_input
        component: syslog_input
        settings:
          listen:
            proto: tcp
            host: 0.0.0.0
            port: 9904
        publish:
          - stream: logs
            fields:
              - log
              - _ppf_local_host
              - _ppf_local_port
              - _ppf_remote_host
              - _ppf_remote_port
              - _ppf_timestamp
              - _ppf_id

      # Punchlet Parser
      - type: punchlet_node
        component: punchlet_simple
        settings:
          punchlet_json_resources: []
          punchlet_grok_pattern_dirs:
            - org/thales/punch/common/groks
          punchlet:
            - org/thales/punch/common/input.punch
            - org/thales/punch/common/parsing_syslog_header.punch
            - org/thales/punch/cisco/sourcefire/parser_sourcefire.punch
            - org/thales/punch/common/geoip.punch
        subscribe:
          - component: syslog_input
            stream: logs
        publish:
          - stream: logs
            fields:
              - log
              - _ppf_id
              - _ppf_timestamp

      # Elastic Output
      - type: elasticsearch_output
        component: es_output
        settings:
          cluster_id: es_search
          credentials:
            user: admin
            password: password
          http_hosts:
            - host: punchplatform-es-http.doc-store
              port: 9200
          credentials:
            user: elastic
            password: elastic

          # Production settings for errors
          reindex_failed_documents: true
          error_index:
            type: daily
            prefix: mytenant-events-indexation-errors-

          per_stream_settings:
            - stream: logs
              index:
                type: daily
                prefix: mytenant-events-
              document_json_field: log
              document_id_field: _ppf_id
              additional_document_value_fields:
                - type: date
                  document_field: "@timestamp"
                  format: iso
        subscribe:
          - component: punchlet_simple
            stream: logs


      - type: kafka_output
        component: to_kafka
        settings:
          brokers: common
          bootstrap.servers: kafka-kafka-bootstrap.processing:9092
          topic: sourcefire-events
          encoding: lumberjack
          producer.acks: all
          producer.batch.size: 16384
          producer.linger.ms: 5
        subscribe:
          - component: punchlet_simple
            stream: logs