Custom Deployment¶
Design¶
The design starts by writing the punchplatform.properties file. The punchplatform is mix of several components:
- processing components : Storm, Spark
- queuing components : Kafka
- Indexation & visualisation components : Elasticsearch, Kibana
- Storage components : Ceph
- Administration components : Shiva, operator environment.
Each components is described in the punchplatform.properties.
Run the Deployer¶
After the few steps explained just before, you have a few additional commands available. The first to try is this one:
1 | punchplatform-deployer.sh --generate-inventory |
You can restrict the deployment by using the following group of hosts and tags:
group of hosts | tags |
---|---|
zookeeper_servers | zookeeper |
elasticsearch_servers | elasticsearch |
kibana_servers | kibana |
storm_servers | storm |
kafka_servers | kafka |
metricbeat_servers | metricbeat |
ceph_servers | ceph |
punchplatform_operator_servers | operator |
git_bare_servers | git |
spark_servers | spark |
shiva_servers | shiva |
The Ansible argument to restrict deployment based on tags is --tags
(or -t
). The one to restrict
a group of hosts is --limit
(or -l
). Note that you can use them both in the same command. You
can also specify multiple values separated by comma. For example:
1 2 | punchplatform-deployer.sh --deploy --tags zookeeper punchplatform-deployer.sh --deploy --limit punchplatform_operator_servers --tags operator,git |
Info
To list all the options provided by Ansible, run this command punchplatform-deployer.sh --deploy -h
It will take some time. What does is do ? It reads your two
configuration files (from $PUNCHPLATFORM_CONF_DIR
), and generates a
complete set of Ansible inventories to deploy the corresponding
components on your target server(s). Check the generated ansible
inventories, they are located in your deployment directory:
1 | cd $PUNCHPLATFORM_CONF_DIR/generated_inventory |
Starting from now you start thinking the ansible way. If the last command succeeded you can start the effective deployment:
1 | punchplatform-deployer.sh --deploy -k -K |
How do you test your deployment ? Simply retype the same command:
1 | punchplatform-deployer.sh --deploy -k -K |
It must be all green and yellow.
High-Level Templating¶
For most advanced users, it might be very useful to generate the PunchPlatform deployment files themselves using a higher level templating file. The aim is to keep only the platform specific variables at one place in a simple JSON document.
This high-level templating is also helpful when the deployment of a large number of platforms is needed. For example, you need to deploy more than ten or hundred LTR platforms. Between several platforms, only a few differences will be noticeable (i.e. platform IDs, servers IP or some listening ports). Thanks to this high-level templating engine, you can write templates once and then simply use a single model file per platform.
Files Layout¶
The files that we want to generate are the punchplatform.properties
and the punchplatform-deployment.settings
.
The associated template files must respectively be named punchplatform.properties.j2
and
punchplatform-deployment.settings.j2
. Here, the .j2
file extention stands for "Jinja 2" which is the Python
templating engine used underneath.
For example, put these two templates in a folder called platform_templates
. The directory name is arbitrary,
the only requirement is to put all the template files in the same folder.
Here is a sample content of these files:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # punchplatform.properties.j2 { "platform": { "platform_id": "{{ platform_id }}" }, "zookeeper": { "clusters": { "common": { "hosts": [ "{{ target_server }}" ], "cluster_port": 2181, "punchplatform_root_node": "/punchplatform" } }, "install_dir": "/data/opt/apache-zookeeper-3.5.5-bin" } } |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # punchplatform-deployment.settings.j2 { "setups_root": "/data/opt", "remote_data_root_directory": "/data", "remote_logs_root_directory": "/var/log/punchplatform", "punchplatform_daemons_user": "{{ daemon_username }}", "punchplatform_group": "{{ daemon_usergroup }}", "zookeeper_version": "apache-zookeeper-3.5.5-bin", "zookeeper_nodes_production_interface": "lo", "zookeeper_childopts": "-server -Xmx256m -Xms256m", "zookeeper_admin_cluster_name": "common", "punchplatform_operator_environment_version": "punchplatform-operator-environment-5.7.7", "punchplatform_operator": { "configuration_name_dir_from_home": "pp-conf", "punchplatform_conf_url": "{{ target_server }}:2181/punchplatform", "operators_username": {{ operators_username | to_json}}, "servers": { "{{ target_server }}": {} } }, "git_settings": { "synchro_mode": "ssh", "git_remote_user": "{{ daemon_username }}", "git_bare_server": "{{ target_server }}", "git_bare_server_address": "{{ target_server }}", "punchplatform_conf_repo_git_url": "/data/git-bare" } } |
As you can see, some special variables have been inserted in the document with {{
and }}
.
This structure follows the Jinja 2 data structure, to learn more about it, please see the
official Jinja documentation. Each variable,
will be replaced by its pre-defined value that comes from a JSON model file. To do so, create
a new JSON file, it can be located anywhere. In our case, we will put it at
pp-conf/platform_models/ltr-1-model.json
. Here is its content:
1 2 3 4 5 6 7 | { "platform_id": "my-ltr-1-unique-id", "target_server": "server1", "daemon_username": "vagrant", "daemon_usergroup": "vagrant", "operators_username": [ "vagrant" ] } |
Your deployer configuration directory layout should now look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | pp-conf ├── platform_templates │ ├── punchplatform-deployment.settings.j2 │ └── punchplatform.properties.j2 ├── platform_models │ └── ltr-1-model.json └── tenants ├── platform │ ├── channels │ │ └── ... │ └── etc │ └── ... └── <your_other_tenant> ├── channels │ └── ... └── etc └── ... |
Run the Generation¶
Now that all the configuration is setup, let's actually run the configuration file generation. From you terminal, execute:
1 | punchplatform-deployer.sh --generate-platform-config --templates-dir platform_templates/ --model platform_models/ltr-1-model.json |
If everything goes well, the command should output the absolute location of the generated files and your pp-conf
directory should contains a punchplatform.properties
and a punchplatform-deployment.settings
files.
And that's it! You can now write your own template and model files to leverage the power of high-level templating.