Custom Deployment

Design

The design starts by writing the punchplatform.properties file. The punchplatform is mix of several components:

• processing components : Storm, Spark
• queuing components : Kafka
• Indexation & visualisation components : Elasticsearch, Kibana
• Storage components : Ceph
• Administration components : Shiva, operator environment.

Each components is described in the punchplatform.properties.

Run the Deployer

After the few steps explained just before, you have a few additional commands available. The first to try is this one:

punchplatform-deployer.sh --generate-inventory

You can restrict the deployment by using the following group of hosts and tags:

group of hosts	tags
zookeeper_servers	zookeeper
elasticsearch_servers	elasticsearch
kibana_servers	kibana
storm_servers	storm
kafka_servers	kafka
metricbeat_servers	metricbeat
ceph_servers	ceph
punchplatform_operator_servers	operator
git_bare_servers	git
spark_servers	spark
shiva_servers	shiva

The Ansible argument to restrict deployment based on tags is --tags (or -t). The one to restrict a group of hosts is --limit (or -l). Note that you can use them both in the same command. You can also specify multiple values separated by comma. For example:

punchplatform-deployer.sh --deploy --tags zookeeper
punchplatform-deployer.sh --deploy --limit punchplatform_operator_servers --tags operator,git

Info

To list all the options provided by Ansible, run this command punchplatform-deployer.sh --deploy -h

It will take some time. What does is do ? It reads your two configuration files (from $PUNCHPLATFORM_CONF_DIR), and generates a complete set of Ansible inventories to deploy the corresponding components on your target server(s). Check the generated ansible inventories, they are located in your deployment directory:

cd $PUNCHPLATFORM_CONF_DIR/generated_inventory

Starting from now you start thinking the ansible way. If the last command succeeded you can start the effective deployment:

punchplatform-deployer.sh --deploy -k -K

How do you test your deployment ? Simply retype the same command:

punchplatform-deployer.sh --deploy -k -K

It must be all green and yellow.

High-Level Templating

For most advanced users, it might be very useful to generate the PunchPlatform deployment files themselves using a higher level templating file. The aim is to keep only the platform specific variables at one place in a simple JSON document.

This high-level templating is also helpful when the deployment of a large number of platforms is needed. For example, you need to deploy more than ten or hundred LTR platforms. Between several platforms, only a few differences will be noticeable (i.e. platform IDs, servers IP or some listening ports). Thanks to this high-level templating engine, you can write templates once and then simply use a single model file per platform.

Files Layout

The files that we want to generate are the punchplatform.properties and the punchplatform-deployment.settings. The associated template files must respectively be named punchplatform.properties.j2 and punchplatform-deployment.settings.j2. Here, the .j2 file extention stands for "Jinja 2" which is the Python templating engine used underneath.

For example, put these two templates in a folder called platform_templates. The directory name is arbitrary, the only requirement is to put all the template files in the same folder.

Here is a sample content of these files:

# punchplatform.properties.j2
{
  "platform": {
    "platform_id": "{{ platform_id }}"
  },
  "zookeeper": {
    "clusters": {
      "common": {
        "hosts": [
          "{{ target_server }}"
        ],
        "cluster_port": 2181,
        "punchplatform_root_node": "/punchplatform"
      }
    },
    "install_dir": "/data/opt/apache-zookeeper-3.5.5-bin"
  }
}

# punchplatform-deployment.settings.j2
{
  "setups_root": "/data/opt",
  "remote_data_root_directory": "/data",
  "remote_logs_root_directory": "/var/log/punchplatform",
  "punchplatform_daemons_user": "{{ daemon_username }}",
  "punchplatform_group": "{{ daemon_usergroup }}",
  "zookeeper_version": "apache-zookeeper-3.5.5-bin",
  "zookeeper_nodes_production_interface": "lo",
  "zookeeper_childopts": "-server -Xmx256m -Xms256m",
  "zookeeper_admin_cluster_name": "common",
  "punchplatform_operator_environment_version": "punchplatform-operator-environment-5.7.7",
  "punchplatform_operator": {
    "configuration_name_dir_from_home": "pp-conf",
    "punchplatform_conf_url": "{{ target_server }}:2181/punchplatform",
    "operators_username": {{ operators_username | to_json}},
    "servers": {
      "{{ target_server }}": {}
    }
  },
  "git_settings": {
    "synchro_mode": "ssh",
    "git_remote_user": "{{ daemon_username }}",
    "git_bare_server": "{{ target_server }}",
    "git_bare_server_address": "{{ target_server }}",
    "punchplatform_conf_repo_git_url": "/data/git-bare"
  }
}

As you can see, some special variables have been inserted in the document with {{ and }}. This structure follows the Jinja 2 data structure, to learn more about it, please see the official Jinja documentation. Each variable, will be replaced by its pre-defined value that comes from a JSON model file. To do so, create a new JSON file, it can be located anywhere. In our case, we will put it at pp-conf/platform_models/ltr-1-model.json. Here is its content:

{
  "platform_id": "my-ltr-1-unique-id",
  "target_server": "server1",
  "daemon_username": "vagrant",
  "daemon_usergroup": "vagrant",
  "operators_username": [ "vagrant" ]
}

Your deployer configuration directory layout should now look like this:

pp-conf
├── platform_templates
│   ├── punchplatform-deployment.settings.j2
│   └── punchplatform.properties.j2
├── platform_models
│   └── ltr-1-model.json
└── tenants
    ├── platform
    │   ├── channels
    │   │   └── ...
    │   └── etc
    │       └── ...
    └── <your_other_tenant>
        ├── channels
        │   └── ...
        └── etc
            └── ...

Run the Generation

Now that all the configuration is setup, let's actually run the configuration file generation. From you terminal, execute:

punchplatform-deployer.sh --generate-platform-config --templates-dir platform_templates/ --model platform_models/ltr-1-model.json

If everything goes well, the command should output the absolute location of the generated files and your pp-conf directory should contains a punchplatform.properties and a punchplatform-deployment.settings files.

And that's it! You can now write your own template and model files to leverage the power of high-level templating.