Common Configuration
All punch nodes share the same format as defined in the punchline. Nodes have a settings sections where they properties can be set. Other properties are common to all nodes. Here is an input node example:
{
"type" : "syslog_input",
"component": "syslog_input",
"executors": 1,
"settings" : {
"listen": {
...
},
"self_monitoring.activation": true,
"self_monitoring.period": 10
},
"publish": {...}
}
executorscorresponds to the number of instances you want. Think of it as the number of thread. Each instance runs in its own thread.publishandsubscribedefine the nodes dag. Input nodes only publish, output nodes only subscribe, and inner nodes do both.
Input Nodes¶
Input nodes share some common characteristics. We describe these here.
Latency metrics¶
To enable latency metrics, you must first add the self_monitoring
properties to your input node, and second configure them to emit the
corresponding latency messages onto the reserved stream and field used
by the punchplatform to propagate these records.
Let us take an example with the Syslog input node:
{
"type" : "syslog_input",
"settings" : {
...
# activate the emitting of self monitoring messages
"self_monitoring.activation": true,
# one every 10 seconds
"self_monitoring.period": 10
},
"publish": [
# your business data
{
"stream": "logs",
"fields": [ ... ]
},
# this makes the spout publish the latency records to the reserved
# stream and field dedicated to latency record propagation
{
"stream": "_ppf_metrics",
"fields": [ "_ppf_latency" ]
}
]
}
}
With this configuration, you should see some latency metrics in your
Elasticsearch. These metrics are stored using this index pattern
[tenant_name]-metrics-YYYY.MM.DD.
These metrics content will looks like:
{
"@timestamp": "2018-07-04T12:45:24.800Z",
"storm": {
"latency": {
"start": {
"component": "syslog_input",
"@timestamp": 1530708316699,
"topology": "input",
"channel": "sourcefire",
"platform": "punchplatform-primary",
"tenant": "mytenant"
},
"diff": 7
}
},
"name": "storm.latency",
"rep": {
"host": {
"name": "PunchPlatform.local"
}
},
"type": "storm",
"platform": {
"storm": {
"component": {
"name": "kafka",
"task_id": 2,
"type": "kafka_output"
},
"topology": "input",
"container_id": "main"
},
"channel": "sourcefire",
"id": "punchplatform-primary",
"tenant": "mytenant"
}
}
TLS encryption¶
Overview¶
Several input and output nodes can be configured as SSL/TLS endpoints. The punch leverages the Netty library.
All these modules accepts the same configuration properties.
Server Side Authentication¶
The server side authentication is the most common configuration for punch nodes. This configuration is used by clients to trust and verify the server identity. The server however will not authenticate the clients. This configuration requires :
- A
private keyand acertificatefor the server - The server certificate is expected to be a standard X.509 certificate chain file in PEM format.
- The server private key must be provided as a PKCS8 private key file in PEM format.
- A
CA filefor the client, to trust the server certificate
The server side configuration is as follows:
{
"type": "syslog_input",
"settings": {
"listen": {
"proto": "tcp" ,
"host": "0.0.0.0",
"port": 4433,
"ssl" : true,
"ssl_private_key": "<path to your>/private-key.pem",
"ssl_certificate": "<path to your>/public-key.pem"
}
}
the client side configuration is as follows:
{
"type": "syslog_output",
"settings": {
"destination": [
{
"host": "localhost",
"port": 4433,
"ssl" : true,
"ssl_trusted_certificate": "<path to your>/cachain.pem"
}
]
}
Mutual authentication¶
The mutual authentication involves a symmetric behavior for both clients and server. Each side needs a private key, a
certificate to identify itself and a CA file to identify the other side.
The server side configuration is as follows:
{
"type": "syslog_input",
"spout_settings": {
"listen": {
"proto": "tcp" ,
"host": "0.0.0.0",
"port": 4433,
"ssl" : true,
"ssl_private_key": "<path to your>/private-key.pem",
"ssl_certificate": "<path to your>/public-key.pem",
"ssl_trusted_certificate": "<path to your>/cachain.pem"
}
}
}
The client side configuration is as follows:
{
"type": "syslog_output",
"bolt_settings": {
"destination": [
{
"host": "localhost",
"port": 4433,
"ssl" : true,
"ssl_private_key": "<path to your>/private-key.pem",
"ssl_certificate": "<path to your>/public-key.pem",
"ssl_trusted_certificate": "<path to your>/cachain.pem"
}
]
}
}
Detailed example¶
Here is an example of the syslog_input (server side) configuration using SSL options:
{
"type": "syslog_input",
"settings": {
"listen": {
...
"ssl": true,
"ssl_private_key": "/absolute/path/to/my.key.pcks8",
"ssl_certificate": "/absolute/path/to/my.crt",
"ssl_trusted_certificate": "/absolute/path/to/cacert.pem",
"ssl_provider": "JDK",
"ssl_protocol": "TLSv1.2",
"ssl_ciphers": ["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"],
"ssl_session_resumption": true
}
}
}
The "ssl_private_key" and "ssl_certificate" fields must be absolute paths
If ssl is set to true, you can provide the following configurations :
-
sslif true, TLS encryption is used and all ssl configurations are enabled
-
ssl_private_key: with ssl, mandatory for input nodes, optional for output nodesAbsolute path to the private key used for TLS encryption
-
ssl_certificate: with ssl mandatory for input nodes, optional for output nodesAbsolute path to the certificate file used for TLS encryption. Can be self-signed.
-
ssl_trusted_certificate: OptionalAbsolute path to the CA file containing the peer certificates chain This configuration is optional if the certificate is self signed, or you can provide the same self-signed certificate
-
ssl_provider: Optional[JDK (default)] [OPENSSL] [OPENSSL_REFCNT]
Implementation used for encryption -
ssl_protocol: Optional[TLSv1] [TLSv1.1] [TLSv1.2 (default)]
Set the TLS version used for encryption. Version 1.0 and 1.1 are not recommended due to their weakness. Depending on the version set, the ciphers are able to change. -
ssl_ciphers: OptionalOverrides the cipher suites used by the spout or the bolt concerned by this configuration
-
ssl_session_resumption: OptionalDefault is true
If set to true, the TLS handshake between the concerned spout or bolt will be shortened and faster. It results in an improvement for the TLS communications.
If set to false, the TLS handshake will start from scratch for each connection, even with the same client.
TLS Ciphers¶
You can set 2 different SSL provider : JDK (default) or OPENSSL
The default JDK provider uses the native encryption library provided by the Java JDK installed on the host.
It is very useful for testing purpose or for a setup where a high level of security is not required.
No extra installation step needed, it should work in any case. The ciphers used with this provider are listed below:
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384"
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
"TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
"TLS_DHE_RSA_WITH_AES_256_CBC_SHA256"
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA"
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
"TLS_RSA_WITH_AES_128_CBC_SHA256"
"TLS_RSA_WITH_AES_128_CBC_SHA"
If you want to go to production with a higher level of security, you may want to rely on OpenSSL.
In that case, use the OPENSSL provider, the associated ciphers are the following ones.
"ECDHE-ECDSA-AES256-GCM-SHA384"
"ECDHE-ECDSA-AES256-SHA384"
"ECDHE-RSA-AES256-GCM-SHA384"
"ECDHE-RSA-AES256-SHA384"
"DHE-RSA-AES256-GCM-SHA384"
"DHE-RSA-AES256-SHA256"
"ECDHE-ECDSA-AES128-GCM-SHA256"
"ECDHE-RSA-AES128-GCM-SHA256"
"ECDHE-ECDSA-AES128-SHA256"
"ECDHE-RSA-AES128-SHA256"
"ECDHE-ECDSA-AES256-GCM-SHA384"
"ECDHE-RSA-AES256-GCM-SHA384"
"ECDHE-ECDSA-AES256-SHA384"
"ECDHE-RSA-AES256-SHA384"
"AES128-GCM-SHA256"
"AES128-SHA256"
"AES256-GCM-SHA384"
"AES256-SHA256"
Warning
With OPENSSL you must run the punchline on a compatible host, the underlying OpenSSL library and the Apache Portable Runtime (APR) must be installed.
Regarding security issues, these providers are tested using the "testssl" tool.
With OPENSSL, no vulnerability were found. With JDK, only one vulnerability is found, the "Secure Client-Initiated Renegotiation" which is a DoS threat (see this CVE for in-depth review).
Testing¶
For testing, you can generate a certificate and keys using :
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout punchplatform.key -out punchplatform.crt
To generate key files in PKCS8 format, use the following command to convert a non PKCS8 to PKCS8 key.
openssl pkcs8 -topk8 -nocrypt -in punchplatform.key -out punchplatform.key.pkcs8
Multi Stream Configurations¶
Using the publish-subscribe relationship you can invent arbitrary graph.
In addition some output nodes (in fact most) also support configuration options that allows you to have a single node in charge of dispatching the data received on respectively stream1 stream2 .. streamN to a (respectively) destination1 destination2 destinationN.
Here is an example using the syslog output node. Let us start with a regular mono-stream example.
{
"type" : "syslog_udp_node",
"settings" : {
"destination" : [ {"host" : "127.0.0.1", "port" : 9999}]
},
...
"subscribe" : [
{ "component" : "spout1", "stream" : "logs", "grouping": "localOrShuffle" }
]
}
Say now you have two streams , and you need to send data from to 127.0.0.1:9999 and from logs2 to 127.0.0.1:8888. Here is how to configure it:
{
"type" : "syslog_udp_bolt",
"settings" : {
"destinations" : [
{
"stream" : "logs1",
"destination" : [ {"host" : "127.0.0.1", "port" : 9999}]
},
{
"stream" : "logs2",
"destination" : [ {"host" : "127.0.0.1", "port" : 8888}]
}
]
},
...
"subscribe" : [
{ "component" : "spout1", "stream" : "logs1", "grouping": "localOrShuffle" }
{ "component" : "spout2", "stream" : "logs2", "grouping": "localOrShuffle" }
]
}