Skip to content

Punch Framework Deployer Guide

Configuration Prerequisites

The design starts by writing a json file named punchplatform-deployment.settings. There you declare what you need:

  • processing components : Storm, Spark
  • queuing components : Kafka
  • tndexation & visualisation components : Elasticsearch, Kibana
  • storage components : Ceph or Minio
  • orchestration and administration components : Shiva, operator environment.

Each components is described in the punchplatform-deployment.settings.

Then, you also have to write your resolv.hjson file. Check this documentation to have a full description

Install the Deployer

After making sure your deployment and target servers have the prerequisites, the deployer setup requires only two steps:

  • you must have the 'bin' subdirectory of the unzipped punchplatform deployer archive in your path. e.g.:

    mkdir -p /data/opt
    cd /data/opt
    unzip ~/Downloads/punch-deployer-x.y.z.zip
    
    # Ensure the deployer directory is NOT world-writeable (this is an Ansible requirement)
    chmod -R o-w .
    
    cd punch-deployer-x.y.z/bin
    # Add the bin folder to your path through a ~/.bashrc additional statement
    echo "export PATH=$(pwd):\$PATH" >> ~/.bashrc
    # Reload your ~/.bashrc in ALL your windows (are start new terminal windows)
    source ~∕.bashrc
    
  • you must have the root folder of your configuration ( where punchplatform-deployment.settings and resolv.hjson files are located) pointed at by the PUNCHPLATFORM_CONF_DIR environment variable. e.g.:

    cd ~/myplatform-conf
    echo "export PUNCHPLATFORM_CONF_DIR=$(pwd)" >> ~/.bashrc
    # Reload your ~/.bashrc in ALL your windows (are start new terminal windows)
    source ~∕.bashrc
    

Run the Deployer

After the few steps explained just before, you have a few additional commands available. The first to try is this one:

punchplatform-deployer.sh --generate-inventory

It will take some time. What does is do ? It reads your two configuration files (from $PUNCHPLATFORM_CONF_DIR), and generates a complete set of Ansible inventories to deploy the corresponding components on your target server(s). Check the generated ansible inventories, they are located in your deployment directory:

cd $PUNCHPLATFORM_CONF_DIR/generated_inventory

Starting from now you start thinking the ansible way. If the last command succeeded you can start the effective deployment:

punchplatform-deployer.sh --deploy -k -K

Important

If ansible informs you that it ignored your ansible.cfg file because of access permission issue, then your deployment will not work.

This is for security reasons (https://docs.ansible.com/ansible/latest/reference_appendices/config.html#avoiding-security-risks-with-ansible-cfg-in-the-current-directory) You must ensure that your deployer folder is not writeable by 'world' (i.e chmod -R o-w <deployerFolder>).

How do you test your deployment ? Simply retype the same command:

punchplatform-deployer.sh --deploy -k -K

It must be all green and yellow

Partial (re)deployment

As the deployer informs you, it possible to target or restrict the deployment by using the following group of hosts and tags:

group of hosts tags
zookeeper_servers zookeeper
elasticsearch_servers elasticsearch
kibana_servers kibana
storm_servers storm
kafka_servers kafka
metricbeat_servers metricbeat
ceph_servers ceph
punchplatform_operator_servers operator
git_bare_servers git
spark_servers spark
shiva_servers shiva
minio_servers minio

The Ansible argument to restrict deployment based on tags is --tags (or -t). The one to restrict a group of hosts is --limit (or -l). Note that you can use them both in the same command. You can also specify multiple values separated by comma. For example:

punchplatform-deployer.sh --deploy --tags zookeeper
punchplatform-deployer.sh --deploy --limit punchplatform_operator_servers --tags operator,git

Info

To list all the options provided by Ansible, run this command punchplatform-deployer.sh --deploy -h

High-Level Templating

For most advanced users, it might be very useful to generate the PunchPlatform deployment files themselves using a higher level templating file. The aim is to keep only the platform specific variables at one place in a simple JSON document.

This high-level templating is also helpful when the deployment of a large number of platforms is needed. For example, you need to deploy more than ten or hundred LTR platforms. Between several platforms, only a few differences will be noticeable (i.e. platform IDs, servers IP or some listening ports). Thanks to this high-level templating engine, you can write templates once and then simply use a single model file per platform.

Files Layout

The file that we want to generate is the punchplatform-deployment.settings. The associated template file must respectively be named punchplatform-deployment.settings.j2. Here, the .j2 file extension stands for "Jinja 2" which is the Python templating engine used underneath.

For example, put this template in a folder called platform_templates. The directory name is arbitrary, the only requirement is to put the template file in the same folder.

Here is a sample content of this file:

# punchplatform-deployment.settings.j2
{
  "platform": {
    "platform_id": "{{platform_id}}",
    "setups_root": "/data/opt",
    "remote_data_root_directory": "/data/opt",
    "remote_logs_root_directory": "/var/log/punchplatform",
    "punchplatform_daemons_user": "{{ daemon_username }}",
    "punchplatform_group": "{{ daemon_usergroup }}",
    "reporters" : ["myreporter"]
    },

   ...
}

As you can see, some special variables have been inserted in the document with {{ and }}. This structure follows the Jinja 2 data structure, to learn more about it, please see the official Jinja documentation. Each variable, will be replaced by its pre-defined value that comes from a JSON model file. To do so, create a new JSON file, it can be located anywhere. In our case, we will put it at pp-conf/platform_models/ltr-1-model.json. Here is its content:

{
  "platform_id": "my-ltr-1-unique-id",
  "target_server": "server1",
  "daemon_username": "vagrant",
  "daemon_usergroup": "vagrant"
}

Your deployer configuration directory layout should now look like this:

pp-conf
├── platform_templates
│   └── punchplatform-deployment.settings.j2
├── platform_models
│   └── ltr-1-model.json
└── tenants
    ├── platform
    │   ├── channels
    │   │   └── ...
    │   └── etc
    │       └── ...
    └── <your_other_tenant>
        ├── channels
        │   └── ...
        └── etc
            └── ...

Run the Generation

Now that all the configuration is setup, let's actually run the configuration file generation. From you terminal, execute:

punchplatform-deployer.sh --generate-platform-config --templates-dir platform_templates/ --model platform_models/ltr-1-model.json

If everything goes well, the command should output the absolute location of the generated file and your pp-conf directory should contains a punchplatform-deployment.settings file.

And that's it! You can now write your own template and model files to leverage the power of high-level templating.