Punch Framework Deployer Guide

Configuration Prerequisites

The design starts by writing a json file named punchplatform-deployment.settings. There you declare what you need:

• processing components : Storm, Spark
• queuing components : Kafka
• tndexation & visualisation components : Elasticsearch, Kibana
• storage components : Ceph or Minio
• orchestration and administration components : Shiva, operator environment.

Each components is described in the punchplatform-deployment.settings.

Then, you also have to write your resolv.hjson file. Check this documentation to have a full description

Install the Deployer

After making sure your deployment and target servers have the prerequisites, the deployer setup requires only two steps:

• you must have the 'bin' subdirectory of the unzipped punchplatform deployer archive in your path. e.g.:

  mkdir -p /data/opt
  cd /data/opt
  unzip ~/Downloads/punch-deployer-x.y.z.zip
  
  # Ensure the deployer directory is NOT world-writeable (this is an Ansible requirement)
  chmod -R o-w .
  
  cd punch-deployer-x.y.z/bin
  # Add the bin folder to your path through a ~/.bashrc additional statement
  echo "export PATH=$(pwd):\$PATH" >> ~/.bashrc
  # Reload your ~/.bashrc in ALL your windows (are start new terminal windows)
  source ~∕.bashrc
  

• you must have the root folder of your configuration ( where punchplatform-deployment.settings and resolv.hjson files are located) pointed at by the PUNCHPLATFORM_CONF_DIR environment variable. e.g.:

  cd ~/myplatform-conf
  echo "export PUNCHPLATFORM_CONF_DIR=$(pwd)" >> ~/.bashrc
  # Reload your ~/.bashrc in ALL your windows (are start new terminal windows)
  source ~∕.bashrc
  

Run the Deployer

After the few steps explained just before, you have a few additional commands available. The first to try is this one:

punchplatform-deployer.sh --generate-inventory

It will take some time. What does is do ? It reads your two configuration files (from $PUNCHPLATFORM_CONF_DIR), and generates a complete set of Ansible inventories to deploy the corresponding components on your target server(s). Check the generated ansible inventories, they are located in your deployment directory:

cd $PUNCHPLATFORM_CONF_DIR/generated_inventory

Starting from now you start thinking the ansible way. If the last command succeeded you can start the effective deployment:

punchplatform-deployer.sh --deploy -k -K

Important

If ansible informs you that it ignored your ansible.cfg file because of access permission issue, then your deployment will not work.

This is for security reasons (https://docs.ansible.com/ansible/latest/reference_appendices/config.html#avoiding-security-risks-with-ansible-cfg-in-the-current-directory) You must ensure that your deployer folder is not writeable by 'world' (i.e chmod -R o-w <deployerFolder>).

How do you test your deployment ? Simply retype the same command:

punchplatform-deployer.sh --deploy -k -K

It must be all green and yellow

Partial (re)deployment

As the deployer informs you, it possible to target or restrict the deployment by using the following group of hosts and tags:

group of hosts	tags
zookeeper_servers	zookeeper
elasticsearch_servers	elasticsearch
kibana_servers	kibana
storm_servers	storm
kafka_servers	kafka
metricbeat_servers	metricbeat
ceph_servers	ceph
punchplatform_operator_servers	operator
git_bare_servers	git
spark_servers	spark
shiva_servers	shiva
minio_servers	minio

The Ansible argument to restrict deployment based on tags is --tags (or -t). The one to restrict a group of hosts is --limit (or -l). Note that you can use them both in the same command. You can also specify multiple values separated by comma. For example:

punchplatform-deployer.sh --deploy --tags zookeeper
punchplatform-deployer.sh --deploy --limit punchplatform_operator_servers --tags operator,git

Info

To list all the options provided by Ansible, run this command punchplatform-deployer.sh --deploy -h

High-Level Templating

For most advanced users, it might be very useful to generate the PunchPlatform deployment files themselves using a higher level templating file. The aim is to keep only the platform specific variables at one place in a simple JSON document.

This high-level templating is also helpful when the deployment of a large number of platforms is needed. For example, you need to deploy more than ten or hundred LTR platforms. Between several platforms, only a few differences will be noticeable (i.e. platform IDs, servers IP or some listening ports). Thanks to this high-level templating engine, you can write templates once and then simply use a single model file per platform.

Files Layout

The file that we want to generate is the punchplatform-deployment.settings. The associated template file must respectively be named punchplatform-deployment.settings.j2. Here, the .j2 file extension stands for "Jinja 2" which is the Python templating engine used underneath.

For example, put this template in a folder called platform_templates. The directory name is arbitrary, the only requirement is to put the template file in the same folder.

Here is a sample content of this file:

# punchplatform-deployment.settings.j2
{
  "platform": {
    "platform_id": "{{platform_id}}",
    "setups_root": "/data/opt",
    "remote_data_root_directory": "/data/opt",
    "remote_logs_root_directory": "/var/log/punchplatform",
    "punchplatform_daemons_user": "{{ daemon_username }}",
    "punchplatform_group": "{{ daemon_usergroup }}",
    "reporters" : ["myreporter"]
    },

   ...
}

As you can see, some special variables have been inserted in the document with {{ and }}. This structure follows the Jinja 2 data structure, to learn more about it, please see the official Jinja documentation. Each variable, will be replaced by its pre-defined value that comes from a JSON model file. To do so, create a new JSON file, it can be located anywhere. In our case, we will put it at pp-conf/platform_models/ltr-1-model.json. Here is its content:

{
  "platform_id": "my-ltr-1-unique-id",
  "target_server": "server1",
  "daemon_username": "vagrant",
  "daemon_usergroup": "vagrant"
}

Your deployer configuration directory layout should now look like this:

pp-conf
├── platform_templates
│   └── punchplatform-deployment.settings.j2
├── platform_models
│   └── ltr-1-model.json
└── tenants
    ├── platform
    │   ├── channels
    │   │   └── ...
    │   └── etc
    │       └── ...
    └── <your_other_tenant>
        ├── channels
        │   └── ...
        └── etc
            └── ...

Run the Generation

Now that all the configuration is setup, let's actually run the configuration file generation. From you terminal, execute:

punchplatform-deployer.sh --generate-platform-config --templates-dir platform_templates/ --model platform_models/ltr-1-model.json

If everything goes well, the command should output the absolute location of the generated file and your pp-conf directory should contains a punchplatform-deployment.settings file.

And that's it! You can now write your own template and model files to leverage the power of high-level templating.