SSL/TLS and other Punchplatform security secrets deployment¶

In this documentation, we will describe all the steps to secure Punchplatform through TLS API Hardening. As the security policy is different for each platform, we will describe here an example in order to simplify the explanations.

However, the steps to follow are the same for all platforms.

Define your security policy¶

First, it is essential to define the security policy to apply to the platform:

Multi-tenants context ?
Technical means and expertise level of the expected attackers (physical access? network access)
Level of protection against internal attack means (SI users, UI operators, CLI Operators)

Depending on this, several solution design choices will have to be made to deploy Punchplatform:

Deployment layout (n-tiering, number of clusters and isolation, firewalling...)
- Do we separate ES Data for user/operation from ES 'business' data ?
- Do we provision isolation front-end layers and/or devices to mitigate hacking ease coming from a collection network or UI access ?
- Do we have a separate admin area where operation keys and/or master daemons are located ?
User authentification/roles/ACL
- Do we bind to an external user database (ldap/AD) ?
- What distinct functional roles do we need for accessing the data and configuring/operating the platform
- Authentification scheme for the command-line operators (single factor like ES-opendistro-security user/password, or also a per-capita TLS certificate) ?
Operation API hardening:
- monitor/secure access to the command-line environment (Bastion / logging of operator actions)
- use of TLS for ciphering and anti-man-in-the-middle protection of Punch internal/operation APIs against attacks through the internal network

This document presents the overall principles of deploying the TLS hardening of Punchplatform software interfaces (and optionnaly a per-user TLS authentication factor)..

In the rest of the document we take as example a secure deployment taking as assumptions:

1 business tenant (in fact, 2 technical tenants, as 'platform' us usually logically distinct from 'business' tenant ; but this does not imply separate rights or business-level multiple tenants with distinct operation technical operators groups).
multiple ( at least 2) distinct operators, with dedicated linux account for each (deployed here on only one operation server: server1)
Operators have an individual user account for accessing ES data (identification through user/password)
single cluster deployment for ES (server2), Zookeeper+ Kafka+ Shiva (server3) & Gateway (server2)
TLS certificates only used for server identification againsts network attacks and MiM, but certificates are the same for all the need of each given server (whatever client/server/operator process is using the certificate)
The application/pipelines daemons & process running on the punchplatform processing nodes all use a common 'technical' identity when accessing Elasticsearch (identification through user+password)

Prepare your TLS certificates¶

The Punchplatform deployer is not in charge of the certificate's provisioning themselves, but only of distributing them from the deployment machine to the target production servers.

The operator who deploys the Punchplatform should get all the digital certificates from a proper PKI. Check Punchplatform TLS certificates generation constraints to have more details about TLS certificates constraints.

In our example, we created :

a CA root (a single one for an entire platform to sign others keys)
a pair of key and certificate per linux server (signed with CA root, or by an intermediate CA signed by CA root)
a security admin certificate/key pair per Elasticsearch server (this is used only for configuration of the elasticsearch opendistro layer, and is not used in normal operation)

Because of technical constraints, these same information are packaged in two different type of files :

JKS (java keystores): used by kafka, zookeeper, shiva, punch gateway and punchplatform punchline processes
- a JKS for the 'keystore' per linux server containing the server certificate and key
- a JKS for the 'truststore' containing the CA public certificate (and any intermediate CA that may be needed to validate servers certificate)

These JKS files have to be password-protected. The password can be common to all the servers (because in fact the JKS and its password will both be protected by the same linux account ACLs , so generating unique passwords does not provide real additional security and will complicate the deployment configuration.)

PEM/CRT files (pkcs12):
a pair of key and certificate PEM Files for the for each Kibana or Elasticsearch server
a CA verification chain file in PEM format containing the CA public certificate (and any intermediate CA that may be needed to validate servers certificate)

To reduce configuration effort of punchplatform deployment, it is advised (on the deployer server filesystem) to have the servers certificates/keys to be named the same for each server, and stored in separate folders named after the server hostname.

At the end it gives a tree structure like this one on the deployer (ex with 3 server):

platform_certs
├── truststore.jks
├── fullchain.crt
├── server01
│   ├── admin.crt
│   ├── admin.pem
│   ├── server.crt
│   ├── server.jks
│   └── server.pem
├── server02
│   ├── admin.crt
│   ├── admin.pem
│   ├── server.crt
│   ├── server.jks
│   └── server.pem
└── server03
    ├── admin.crt
    ├── admin.pem
    ├── server.crt
    ├── server.jks
    └── server.pem

Warning

per server the names of the keys are the same but NOT the contents

Deployment secrets and Runtime secrets¶

In addition to the credential files (certificates, keys, JKS), some additional secret information (e.g. certificates key password, user/password for Elasticsearch access, JKS file passwords...) may be needed either at deployment time (when generating configuration files for the deployed daemons) or at runtime when a punch operator command is started, or when a shiva or gateway daemon starts a Punchline that will itself connect to ES/Kafka or some other TLS-secured backend.

All this secret information can be stored in 'secrets' files, in json format, separate from the rest of the deployment and runtime configuration. You just have to replace the password or other secret information in the 'punchplatform-deployment.settings' or in the punchlines configuration by special tags that reference this deployment or runtime secret.

Why would I want to hide these information in separate files ?

For multiple reasons :

You have to hide these secrets from non-admin users (e.g. an operator starting a punchline from an operator node as no need to know the password that the shiva-hosted punchline will use when connecting to Elasticsearch)
When a incident occurs, the Punch Professional Services Team will ask to you your punchplatform-deployment.settings or punchlines/channels configuration files to investigate.
If you transport a channel/punchline into an other platform, you would like it to work, even if this other platform has different applicable secrets (daemon accounts)

How do I actually protect (cipher on disk) deployment_secrets file (or certificats private keys) so that it is not stored in a readable form in the deployment configuration ?

Your deployment secret file contains secrets in plain text.

Passwords and Certificate keys are also sensitive information that you do not want anyone to retrieve from an archived deployment configuration (but you may need such an archive/git repository for quick redeployment of part or all your platform if needed)

You can cipher secrets file in the deployment configuration filesystem if needed by following the Ansible vault documentation.

This will mean a deciphering password will be asked by ansible/the punchplatform deployer each time the deployer is run and needs access to the ciphered files.

Deployment secrets¶

These parameters are ONLY used during deployment and are unnecessary afterwards (having been copied to daemons configuration, protected by linux filesystem security in the daemon account)

To achieve this, you need to create a deployment secret file in your deployment configuration called deployment_secrets.json. A unique root attribute of this document called 'deployment_secrets' is mandatory. Under this, you can store and structure whatever secret deployment information items you need, without mandatory naming.

for example :

{
  "deployment_secrets":{
    "platform": {
      "some_section": {
        "a_password": "hello",
        "some_other_password": "world"
      },
      "platform_truststore_pass": "hellotruststore"
    }
}

Then, you can reference these secret information in the punchplatform-deployment.settings file through tags of the form: ${DEPLOYMENT_SECRETS.xpath.to.secret.from.deployment_secrets.root}.

For example:

{
  "platform": {
    "platform_id": "punchplatform-primary",
    "setups_root": "/data/opt",
    "remote_data_root_directory": "/data",
    "remote_logs_root_directory": "/var/log/punchplatform",
    "punchplatform_daemons_user": "vagrant",
    "punchplatform_group": "vagrant",
    "binaries_version": "punch-binaries-6.3.0-SNAPSHOT",
    "platform_local_credentials_dir": "/home/punch/resources/security/platform_certs",
    "platform_local_common_secrets_filename": "platform_common_secrets.json",
    "platform_ca_name": "ca.pem",
    "platform_truststore_name": "truststore.jks",
    "platform_truststore_password": "@{DEPLOYMENT_SECRETS.platform.platform_truststore_pass}"
  }
}

During deployment, you have to use your deployment secret file using the command below :

punchplatform-deployer.sh --deploy -u vagrant -e @/home/punch/pp-conf/deployment_secrets.json

Info

If you have a ciphered deployment secret file, add this additional parameter : --ask-vault-pass

These deployment secrets are only used to generate production daemons configuration, for example shiva config :

# ssl part for Kafka
"ssl": {
    "ssl_truststore_location": "/home/vagrant/.secrets/truststore.jks",
    "ssl_truststore_pass": "secret123"
}

Info

The configuration of the daemons is protected with Unix rights. As a result, only the daemon linux account (and of course a linux superadministrator) can see the contents of the configuration in plain text.

Runtime secrets & resolver¶

Once components are deployed the deployment secrets are not used anymore, but you need some for runtime to authenticate punchlines & reporters for example

Depending on your security policy, you can have common user/passwords used by everyone on the platform (operators, punchlines..) or on the contrary multiple sets of secrets, applying to different operators, applications running in different shiva clusters...

To achieve this, you can store runtime secrets in as many 'secrets' json file as you need (in addition to the deployment secrets one).

Then you will control which secrets file(s) are to be deployed in each context (shiva, operator, gateway, operator accounts) through 2 types of deployment settings that appear at various levels in the punchplatform_deployment.settings.

path to local credential folders on the deployment environment where the deployer will try to find secrets and credentials to upload

These settings are most frequently called local_credentials_dir but may vary in some sections (like platform_local_credentials_dir) as detailed by the Deployment settings reference.

Note that when a local credentials dir is declared in some section, - it supplements the local crentials directories (optionnally) already declared in more general section (e.g. the platform credentials dir) It means a secret file will, at deployment time, be looked up for locally on the deployer environment from t his specific folder first, then from higher level more generic ones - it can be either an absolute folder path on the deployer filesystem or a path relative to $PUNCHPLATFORM_CONF_DIR.
names of additional/custom 'secrets' files that you want uploaded to the production context (shiva/operator/gateway)

These settings are most frequently called custom_additional_credentials_files but may vary in some sections (like platform_local_common_secrets_filename) as detailed by the Deployment settings reference.

Runtime secrets files content and usage after deployment¶

Each 'secret' file is a custom json storing as many information as you need in runtime usage. You will then be able, in channels/punchline configuration to reference secret information from these deployed secrets files using tags of the form: ${RUNTIME_SECRETS.some.path.inside.any.of.the.context.secrets.files}

The tags present in your channels/punchlines configuration files will be replaced at start time of the tasks by the 'resolver' mechanism. You can debug this process by using punchlinectl resolve subcommand in the appropriate context (operator, shiva node, gateway node) under the appropriate identiy (operator account, daemon account).

The priority order of information lookup in your secrets files (when there are several) is from the most 'precise' to the more 'wide' scope (like common to all platform). This is controlled through the PUNCHPLATFORM_RUNTIME_SECRETS environment variable automatically deployed by the punch deployer in the appropriate user/daemons environment configuration files.

Secrets/credentials files local lookup and upload rules¶

The deployer will upload both the custom files you explictely requested by 'custom/additional secrets/credentials'settings, and the implicit credential files you specified in other TLS deployment settings (like keystores,truststores, ca certificate configured for punch servers)
All secrets/credentials/certificates/stores settings are expected to be either
- Absolute path to some file present in the filesystem of the deployer server
- Relative path, applying to one of the provided 'local credentials folder' configured for the section
- Relative path for Per-server secrets/credentials are also looked up automatically in subfolder named as the target server hostname, inside the configured 'local credentials folders' applicable to the context section.
  
  Important
  
  So you DO NOT NEED to specify a 'local credential dir' for each server in a cluster if your credential dir is named as the server, and is located in one of the higher scopes local credential dir (specified at platform or cluster level)
You cannot configure WHERE your secrets will be deployed in each context The secrets files are located in the same place as other credentials file)
Whatever location the credential or secrets file is stored in on the deployer server filesystem, the production files are stored 'flat' in the production folder secrets folder
When you specify local credentials directories and credential/secrets file names in deployment settings, there is an additional logic between scope levels for all these settings.

This means that the credentials/secrets implicitely or explicitely specified for upload at any scope will be looked up by the deployer locally in any of the local credentials folder specified, whatever the scope (platform/cluster/server/user). The most detailed scope is looked up first. So for example: if you defined some common certificate or secrets file or folder at a cluster level, and then define a specific local credential directory at server level, then the common file name specified at cluster level will be looked up by the deployer first in the server specific

Operator secrets¶

Operator credentials/secrets Location¶

The Punch deployer will deploy each secret runtime file & certificates in a specific directory using a Punch environment variable called PUNCHPLATFORM_SECRETS_DIR with the value : $HOME/.secrets. Each resource is deployed with the appropriate Unix rights. In this way, secrets are protected and can only be read by the appropriate user.

For example, you will find in the home directory of the operator1, the secrets below :

operator1@server4:~/.secrets$ ll
total 72
drwxrwxr-- 2 operator1 operator1  4096 Dec 18 10:45 ./
drwxr-xr-x 7 operator1 operator1  4096 Dec 18 09:33 ../
-r-------- 1 operator1 operator1 22534 Dec 14 10:11 ca.pem
-r-------- 1 operator1 operator1    27 Dec 14 17:02 platform_secrets.json
-r-------- 1 operator1 operator1    27 Dec 14 17:02 operators_common_secrets.json
-r-------- 1 operator1 operator1  2610 Dec 14 10:11 server-cert.pem
-r-------- 1 operator1 operator1  3272 Dec 14 10:11 server-key.pem
-r-------- 1 operator1 operator1  5950 Dec 14 17:02 server-keystore.jks
-r-------- 1 operator1 operator1 15040 Dec 14 17:02 truststore.jks
-r-------- 1 operator1 operator1    74 Dec 15 15:23 users_secrets.json

Example:

a common platform_secrets.json providing a not-so-important-but-mandatory password for the JKS truststore containing the CA certificate:

{
  "platform_ca_truststore": "hellotruststore"
}

a runtime operators_common_secrets providing the password for the server certificate keystore:
```
{
  "operator_keystore": "hellokeystore"
}
```
a runtime user_secrets.json providing ES credentials for the operator command:

{
  "es": {
    "user": "operator1",
    "pass": "mypassword"
  }
}

And the operator environment will be initialized automatically with the following secrets priority list:

PUNCHPLATFORM_RUNTIME_SECRETS=/home/operator1/.secrets/user_secrets.json,/home/operator1/.secrets/operators_common_secrets.json,/home/operator1/.secrets/platform_secrets.json

Per-operator secrets file¶

Each individual operator can of course have personal secrets like ES user/password that are needed for punch operator command-line tools. For this, a user_secrets.json file is deployed in each of this user linux account ~/.secrets folder.

The operator can himself/herself edit this file to set his/her own credentials. Because of that, such secrets file are most often NOT provided in the deployment configuration.

But in any case:

this file is NEVER overwritten by the deployer if it exists.
If no custom content is provided by the deployment configuration, then an example file will be deployed if no previous file already exists in the target operator account.

Punchplatform operation commands¶

Punch commands (channelctl, platformctl ..) use a configuration file punchplatform.properties which defines :

Reporters for metrics and events generated by the commands
Clients configuration (kafka client, zookeeper client ..) to connect to punchplatform software components servers.

The TLS configuration for these reporters and clients are installed on the operator servers by the punch deployer, based on the punchplatform-deployment.settings file and more particularly to the punch_commands section and reporters section

In this way, Punch commands can connect to secure components transparently (without specifying all SSL parameters at the command line)

{
    "platform": {
    "platform_id" : "punchplatform-primary",
    "setups_root": "/data/opt",
    "remote_data_root_directory": "/data",
    "remote_logs_root_directory": "/var/log/punchplatform",
    "punchplatform_daemons_user": "punchplatform",
    "punchplatform_group": "punchplatform",
    "binaries_version": "punchplatform-binaries-6.1.0",
    "platform_local_credentials_dir": "/home/punch/platform_certs",
    "platform_local_common_secrets_filename": "platform_common_secrets.json",
    "platform_ca_name": "ca.pem",
    "platform_truststore_name": "truststore.jks",
    "platform_truststore_password": "@{DEPLOYMENT_SECRETS.platform.platform_truststore_pass}",
    "punch_commands": {
      "security": {
        "kafka_clients": {
          "local": {
            "ssl_enabled": true,
            "ssl_truststore_location": "@{PUNCHPLATFORM_SECRETS_DIR}/truststore.jks",
            "ssl_truststore_pass": "@{PUNCHPLATFORM_RUNTIME_SECRETS.secret}"
          }
        },
        "elasticsearch_clients": {
          "es_search": {
            "ssl_enabled": true,
            "ssl_client_certificate_authority": "@{PUNCHPLATFORM_SECRETS_DIR}/ca.pem",
            "ssl_client_certificate": "@{PUNCHPLATFORM_SECRETS_DIR}/server-cert.pem",
            "ssl_client_private_key": "@{PUNCHPLATFORM_SECRETS_DIR}/server-key.pem",
            "ssl_truststore_location": "@{PUNCHPLATFORM_SECRETS_DIR}/truststore.jks",
            "ssl_truststore_pass": "@{PUNCHPLATFORM_RUNTIME_SECRETS.secret}",
            "credentials": {
              "username": "@{PUNCHPLATFORM_RUNTIME_SECRETS.es.user}",
              "password": "@{PUNCHPLATFORM_RUNTIME_SECRETS.es.pass}"
            }
          }
        },
        "gateway_clients": {
          "gateway_32g": {
            "ssl_enabled": true,
            "ssl_truststore_location": "@{PUNCHPLATFORM_SECRETS_DIR}/truststore.jks",
            "ssl_truststore_pass": "@{PUNCHPLATFORM_RUNTIME_SECRETS.secret}"
          }
        },
        "zookeeper_clients": {
          "local": {
            "ssl_enabled": true,
            "ssl_truststore_location": "@{PUNCHPLATFORM_SECRETS_DIR}/truststore.jks",
            "ssl_truststore_pass": "@{PUNCHPLATFORM_RUNTIME_SECRETS.secret}"
          }
        }
      }
    }
  },
  "reporters": {
    "myreporter": {
      "type": "kafka",
      "brokers": "local",
      "topic": "platform-events",
      "reporting_interval": 30,
      "encoding": "json",
      "security.protocol": "SSL",
      "ssl.truststore.location": "@{PUNCHPLATFORM_SECRETS_DIR}/truststore.jks",
      "ssl.truststore.password": "@{PUNCHPLATFORM_RUNTIME_SECRETS.secret}"
    }
  }
}

Very Important

This section is the only section from punchplatform-deployment.settings file which references PUNCHPLATFORM_RUNTIME_SECRETS and not PUNCHPLATFORM_DEPLOYMENT_SECRETS, as in fact the tags values will be resolved only at runtime, when the user, gateway or shiva runs a punch command

Shiva secrets (and gateway secrets)¶

Info

These are the same principle as for an operator secrets, except only one linux account is used for all application run by shiva: the 'daemon' user configured by the punchplatform deployment settings at platform level.

Punch commands are used by shiva and gateways also !

When running applications/punchlines, shiva and gateways are also using the punchplatform commands configuration (for TLS security purpose), as configured by deployment settibgs, like for operators.

If in the deployment settings, any ${PUNCHPLATFORM_RUNTIME_SECRETS.xxx} tag is not matching any secret from the deployed secrets file(s) of the shiva/gateway context, then the associated shiva/gateway daemon may not start (with a daemon error mentionning the unresolved secret tags.)

Example of a typical shiva/gateway secrets set:

a common platform_secrets.json providing a not-so-important-but-mandatory password for the JKS truststore containing the CA certificate:

{
  "platform_ca_truststore": "hellotruststore"
}

a runtime shiva_secrets.json providing ES credentials for the applications on the shiva nodes

{
  "es": {
    "user": "shiva",
    "pass": "shiva"
  },
  "kafka_client":
  {
    "keystore_password": "hellokeystore",
    "hello"
  }
  "my_specific_need": {
    "some_password": "hello"
  }
}

Secrets/credential files layout example on the deployer filesystem¶

Runtime secrets files are defined on deployer server and created by a platform administrator. These files could also be encrypted using Ansible vault as explain above

As a result, here you can find an example of the tree structure on the deployer server:

platform_certs
├── truststore.jks
├── fullchain.crt
├── platform_secrets.json
├── operators_common_secrets.json
├── shiva_apps_secrets.json
├── gateway_tasks_secrets.json
├── fullchain.crt
├── server01
│   ├── admin.crt
│   ├── admin.pem
│   ├── server.crt
│   ├── server.jks
│   └── server.pem
├── server02
│   ├── admin.crt
│   ├── admin.pem
│   ├── server.crt
│   ├── server.jks
│   └── server.pem
└── server03
    ├── admin.crt
    ├── admin.pem
    ├── server.crt
    ├── server.jks
    └── server.pem

Simplifying punchlines configuration with resolver rules¶

Once all your secrets are deployed, you need to reference it in runtime resources (nodes, reporters ..).

To avoid having many TLS settings in each channel/punchline file, with many ${RUNTIME_SECRETS.xxx.yyy} tags in each file, you can leverage the generic resolver mechanism to inject additional settings in all these files in a generic way.

For example :

   // All ES input/output nodes (Storm nodes)
   elasticsearch_nodes:{
      selection:{
         tenant:*
         channel:*
         runtime:*
      }
      match:$.dag[?(@.type=='elasticsearch_output')].settings
      additional_values:{
         http_hosts:[
            {
               host: server2
               port:9200
            }
         ]

         credentials: {
            user: @{PUNCHPLATFORM_RUNTIME_SECRETS.es.user}
            password: @{PUNCHPLATFORM_RUNTIME_SECRETS.es.pass}
         }
         ssl: true
         ssl_keystore_pass: @{PUNCHPLATFORM_RUNTIME_SECRETS.secret}
         ssl_keystore_location: file://@{PUNCHPLATFORM_SECRETS_DIR}/server-keystore.jks
         ssl_truststore_location: file://@{PUNCHPLATFORM_SECRETS_DIR}/truststore.jks
         ssl_truststore_pass: @{PUNCHPLATFORM_RUNTIME_SECRETS.secret}

      }
   }
   extraction_node:{
      selection:{
         tenant: "*"
         channel: "*"
         runtime: "*"
      }
      match: "$.dag[?(@.type=='extraction_input')].settings"
      additional_values:{
         nodes:[
            localhost
         ]
         port: 9200
         elastic_settings:{
            es.net.http.auth.user: @{PUNCHPLATFORM_RUNTIME_SECRETS.elasticsearch.common.credentials.username}
            es.net.http.auth.pass: @{PUNCHPLATFORM_RUNTIME_SECRETS.elasticsearch.common.credentials.password}
            es.net.ssl: false
            es.net.ssl.keystore.pass: @{PUNCHPLATFORM_RUNTIME_SECRETS.elasticsearch.common.ssl_keystore_pass}
            es.net.ssl.keystore.location: file://{{ssl_resources}}/server-keystore.jks
            es.net.ssl.truststore.location: file://{{ssl_resources}}/truststore.jks
            es.net.ssl.truststore.pass: @{PUNCHPLATFORM_RUNTIME_SECRETS.elasticsearch.common.ssl_truststore_pass}
         }
      }
   }

Resolver file is defined on deployer server by the platform's administrator. This file will be deployed on an operator, gateway and shiva nodes and cannot be altered by operator

Test to validate a secure deployment¶

Check platform health¶

Once platform is fully deployed and configured, you can validate that each component is running properly using the command below :

platformctl health

Check runtime health¶

Once each component is running properly, you can validate that resolver rules are correctly set using the command below :

punchlinectl -t mytenant start -p my_punchline.json

If it did not work, you can use this command which is useful to debug a resolver rules issue :

punchlinectl -t mytenant resolve -p my_punchline.json