Skip to content

Common Configuration

Each Spout is defined using two sections. The spout_settings section defines the spout specific parameters, while the storm_settings defines its related storm properties, in particular the stream and fields it emits in the topology.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "type" : "syslog_spout",
    "spout_settings" : {
      "listen": {
        ...
      },
      "self_monitoring.activation": true,
      "self_monitoring.period": 10
  },
  "storm_settings" : { ... }
}

Latency metrics

To enable latency metrics, youmust first add the self_monitoring properties to your spouts, and second configure them to emit the corresponding latency messages onto the reserved stream and field used by the punchplatform to propagate these records.

Let us take an example with the Syslog Spout:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
  "type" : "syslog_spout",
    "spout_settings" : {
      ...
      # activate the emitting of self monitoring messages
      "self_monitoring.activation": true,
      # one every 10 seconds
      "self_monitoring.period": 10
  },
  "storm_settings": {
    "component": "tcp_spout_apache_httpd",
    "publish": [
      # your business data
      {
        "stream": "logs",
        "fields": [ ... ]
      },
      # this makes the spout publish the latency records to the reserved
      # stream and field dedicated to latency record propagation 
      {
        "stream": "_ppf_metrics",
        "fields": [ "_ppf_latency" ]
      }
    ]
  }
}

With this configuration, you should see some latency metrics in your Elasticsearch. These metrics are stored using this index pattern [tenant_name]-metrics-YYYY.MM.DD.

These metrics content will looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
  "@timestamp": "2018-07-04T12:45:24.800Z",
  "storm": {
    "latency": {
      "start": {
        "component": "syslog_spout_tcp",
        "@timestamp": 1530708316699,
        "topology": "input",
        "channel": "sourcefire",
        "platform": "punchplatform-primary",
        "tenant": "mytenant"
      },
      "diff": 7
    }
  },
  "name": "storm.latency",
  "rep": {
    "host": {
      "name": "PunchPlatform.local"
    }
  },
  "type": "storm",
  "platform": {
    "storm": {
      "component": {
        "name": "kafka",
        "task_id": 2,
        "type": "kafka_bolt"
      },
      "topology": "input",
      "container_id": "main"
    },
    "channel": "sourcefire",
    "id": "punchplatform-primary",
    "tenant": "mytenant"
  }
}

SSL/TLS encryption

Spouts & bolts

When you use any Spout or Bolt that uses SSL/TLS encryption you rely on the same software implementation based on the well-known Netty library.

All spouts or bolts that use SSL are listed below:

Configuration example

Here is an example of the syslog_spout configuration using SSL options:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
  "type": "syslog_spout",
  "spout_settings": {
    "listen": {
      ...
      "ssl": true,
      "ssl_provider": "JDK",
      "ssl_private_key": "/absolute/path/to/my.key.pcks8",
      "ssl_certificate": "/absolute/path/to/my.crt"
    }
  },
  "storm_settings": { ... }
}

The "ssl_private_key" and "ssl_certificate" fields must be absolute paths

SSL provider

You can set 2 differents SSL provider : JDK (default) or OPENSSL

The default JDK provider uses the native encryption library provided by the Java JDK installed on the host. It is very useful for testing purpose or for a setup where a high level of security is not required. No extra installation step needed, it should work in any case. The ciphers used with this provider are listed below:

1
2
3
4
5
6
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA

Now, if you want to go to production with a higher level of security, you may want to rely on OpenSSL. In that case, use the OPENSSL provider, the associated ciphers are the following ones. In that case, be sure to run the topology on a compatible host, the underlying OpenSSL library and the Apache Portable Runtime (APR) must be installed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
AES128-GCM-SHA256
AES128-SHA256
AES256-GCM-SHA384
AES256-SHA256

About security issues, we tested these providers using this "testssl" tool. With OPENSSL, no vulnerability were found. With JDK, only one vulnerability is found, the "Secure Client-Initiated Renegotiation" which is a DoS threat (see this CVE for in-depth review).

Testing

For testing, you can generate a certificate and keys using :

1
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout punchplatform.key -out punchplatform.crt

If you want to use the Lumberjack protocol, it expects private key in PKCS8 format. Use the following command to convert a non PKCS8 to PKCS8 key.

1
openssl pkcs8 -topk8 -nocrypt -in punchplatform.key -out punchplatform.key.pkcs8