Skip to content

Common Configuration

Each Spout is defined using two sections. The spout_settings section defines the spout specific parameters, while the storm_settings defines its related storm properties, in particular the stream and fields it emits in the topology.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "type" : "syslog_spout",
    "spout_settings" : {
      "listen": {
        ...
      },
      "self_monitoring.activation": true,
      "self_monitoring.period": 10
  },
  "storm_settings" : { ... }
}

Latency metrics

To enable latency metrics, you must first add the self_monitoring properties to your spouts, and second configure them to emit the corresponding latency messages onto the reserved stream and field used by the punchplatform to propagate these records.

Let us take an example with the Syslog Spout:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
  "type" : "syslog_spout",
    "spout_settings" : {
      ...
      # activate the emitting of self monitoring messages
      "self_monitoring.activation": true,
      # one every 10 seconds
      "self_monitoring.period": 10
  },
  "storm_settings": {
    "component": "tcp_spout_apache_httpd",
    "publish": [
      # your business data
      {
        "stream": "logs",
        "fields": [ ... ]
      },
      # this makes the spout publish the latency records to the reserved
      # stream and field dedicated to latency record propagation 
      {
        "stream": "_ppf_metrics",
        "fields": [ "_ppf_latency" ]
      }
    ]
  }
}

With this configuration, you should see some latency metrics in your Elasticsearch. These metrics are stored using this index pattern [tenant_name]-metrics-YYYY.MM.DD.

These metrics content will looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
  "@timestamp": "2018-07-04T12:45:24.800Z",
  "storm": {
    "latency": {
      "start": {
        "component": "syslog_spout_tcp",
        "@timestamp": 1530708316699,
        "topology": "input",
        "channel": "sourcefire",
        "platform": "punchplatform-primary",
        "tenant": "mytenant"
      },
      "diff": 7
    }
  },
  "name": "storm.latency",
  "rep": {
    "host": {
      "name": "PunchPlatform.local"
    }
  },
  "type": "storm",
  "platform": {
    "storm": {
      "component": {
        "name": "kafka",
        "task_id": 2,
        "type": "kafka_bolt"
      },
      "topology": "input",
      "container_id": "main"
    },
    "channel": "sourcefire",
    "id": "punchplatform-primary",
    "tenant": "mytenant"
  }
}

SSL/TLS encryption

Overview

Several client and server punch modules can be configured as SSL/TLS endpoints. The punch leverages the Netty library.

All these modules accepts the same configuration properties.

Server Side Authentication

The most common configuration is to have only the server configured with certificates, and to configure the client only with the public keys. The client is able to ensure the server is the one he claims to nbe, but the server does not check for clients identities.

Assuming you have your private-key.pem public-key.pem and cachain.pem:

  • private-key.pem : is the server private key. It is known only to the server. It is signed using a trusted certification chain.
  • public-key.pem : is the server public key.
  • cachain.pem : is the server certification chain.

Here is how you typically use these files on a server (resp. client) side:

1
openssl s_server -key private-key.pem -cert public-key.pem
1
openssl s_client -CAfile cachain.pem

Say now you want a punch server configured the same way. Here is the equivalent server configuration (assuming here a syslog punch module):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
     {
      "type": "syslog_spout",
      "spout_settings": {
        "listen": {
          "proto": "tcp" ,
          "host": "0.0.0.0",
          "port": 4433,
          "ssl" : true,
          "ssl_private_key": "<path to your>/private-key.pem",
          "ssl_certificate": "<path to your>/public-key.pem"
        }
      }

And here is the equivalent client :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    {
      "type": "syslog_bolt",
      "bolt_settings": {
        "destination": [
          {
            "host": "localhost",
            "port": 4433,
            "ssl" : true,   
            "ssl_certificate": "<path to your>/cachain.pem"
          }
        ]
      }

Configuration example

Here is an example of the syslog_spout configuration using SSL options:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  "type": "syslog_spout",
  "spout_settings": {
    "listen": {
      ...
      "ssl": true,
      "ssl_private_key": "/absolute/path/to/my.key.pcks8",
      "ssl_certificate": "/absolute/path/to/my.crt",
      "ssl_trusted_certificate": "/absolute/path/to/cacert.pem",
      "ssl_provider": "JDK",
      "ssl_protocol" "TLSv1.2",
      "ssl_ciphers": ["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"],
      "ssl_session_resumption": true
    }
  },
  "storm_settings": { ... }
}

The "ssl_private_key" and "ssl_certificate" fields must be absolute paths

If ssl is set to true, you can provide the following configurations :

  • ssl

    if true, TLS encryption is used and all ssl configurations are enabled

  • ssl_private_key : Mandatory

    Absolute path to the private key used for TLS encryption

  • ssl_certificate : Mandatory

    Absolute path to the certificate file used for TLS encryption. Can be self-signed.

  • ssl_trusted_certificate : Optional

    Absolute path to the CA file containing the certificates chain This configuration is optional if the certificate is self signed, or you can provide the same self-signed certificate

  • ssl_provider : Optional

    [JDK (default)] [OPENSSL] [OPENSSL_REFCNT]
    Implementation used for encryption

  • ssl_protocol : Optional

    [TLSv1] [TLSv1.1] [TLSv1.2 (default)]
    Set the TLS version used for encryption. Version 1.0 and 1.1 are not recommended due to their weakness. Depending on the version set, the ciphers are able to change.

  • ssl_ciphers : Optional

    Overrides the cipher suites used by the spout or the bolt concerned by this configuration

  • ssl_session_resumption : Optional

    Default is true
    If set to true, the TLS handshake between the concerned spout or bolt will be shortened and faster. It results in an improvement for the TLS communications.
    If set to false, the TLS handshake will start from scratch for each connection, even with the same client.

TLS Ciphers

You can set 2 different SSL provider : JDK (default) or OPENSSL

The default JDK provider uses the native encryption library provided by the Java JDK installed on the host. It is very useful for testing purpose or for a setup where a high level of security is not required. No extra installation step needed, it should work in any case. The ciphers used with this provider are listed below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384"
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
"TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
"TLS_DHE_RSA_WITH_AES_256_CBC_SHA256"
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA"
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
"TLS_RSA_WITH_AES_128_CBC_SHA256"
"TLS_RSA_WITH_AES_128_CBC_SHA"

Now, if you want to go to production with a higher level of security, you may want to rely on OpenSSL. In that case, use the OPENSSL provider, the associated ciphers are the following ones. In that case, be sure to run the topology on a compatible host, the underlying OpenSSL library and the Apache Portable Runtime (APR) must be installed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
"ECDHE-ECDSA-AES256-GCM-SHA384"
"ECDHE-ECDSA-AES256-SHA384"
"ECDHE-RSA-AES256-GCM-SHA384"
"ECDHE-RSA-AES256-SHA384"
"DHE-RSA-AES256-GCM-SHA384"
"DHE-RSA-AES256-SHA256"
"ECDHE-ECDSA-AES128-GCM-SHA256"
"ECDHE-RSA-AES128-GCM-SHA256"
"ECDHE-ECDSA-AES128-SHA256"
"ECDHE-RSA-AES128-SHA256"
"ECDHE-ECDSA-AES256-GCM-SHA384"
"ECDHE-RSA-AES256-GCM-SHA384"
"ECDHE-ECDSA-AES256-SHA384"
"ECDHE-RSA-AES256-SHA384"
"AES128-GCM-SHA256"
"AES128-SHA256"
"AES256-GCM-SHA384"
"AES256-SHA256"

Regarding security issues, these providers are tested using the "testssl" tool. With OPENSSL, no vulnerability were found. With JDK, only one vulnerability is found, the "Secure Client-Initiated Renegotiation" which is a DoS threat (see this CVE for in-depth review).

Testing

For testing, you can generate a certificate and keys using :

1
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout punchplatform.key -out punchplatform.crt

If you want to use the Lumberjack protocol, it expects private key in PKCS8 format. Use the following command to convert a non PKCS8 to PKCS8 key.

1
openssl pkcs8 -topk8 -nocrypt -in punchplatform.key -out punchplatform.key.pkcs8