Track 1 Deployment procedure¶
Abstract
This track covers the fundamentals of punch deployment (with associated actions to conduct when using Punchbox tool as a quick-start infrastructure and configuration resources package)
Deployment overview¶
Refer to the Deployment Process and associated list of deployment steps of each steps to follow to have an operational deployed Punch platform
Key Points
-
Deployment of infrastructure prerequisites (including some system or custom solution setup INSIDE the production server Operating system) is not automated by Punch
-
There are many choices of what you want to deploy with the punch deployer and where (depending on the solution needs), and this requires solution engineering
-
Following a central deployment file prepared by the platform or solutionintegrator, the punch deployer tool installs the desired components of the punch framework on the target servers, and configures systemd so that the punch framework services are automatically running, clustered and interconnected.
-
After deploying the framework, channels configuration must be imported or created, to supply business-level pipelines, applications, dashboards, rules. This can happen both at deployment time (if the configuration has been prepared by the integrator beforehand), or enriched later by bringing additional or updated configuration channels and (re)starting the impacted channels.
In the deployment process, which steps must be done by the operator before executing deployment ?
Answer
- ensure the deployment machine has the local prerequisites installed
- ensure he has a remote sudoer account usable for remote-access from the deployment machine
- ensure that all the target prerequisites are deployed on the platform (NTP, DNS and interfaces resolution, disk space, firewalling rules, disks partitioning, standard packages repository available ...)
- get the deployer package and unzip it on the deployment machine
- configure deployer account environment variable to point at configuration (PUNCHPLATFORM_CONF_DIR) and deployer bin path.
- import factory-prepared or create locally the deployment files (punchplatform-deployment.settings and resolv.hjson)
Punchbox tool¶
Refer to the Deployer Getting Started to have a quick introduction of punchbox tool
Question
Which stages of the deployment procedure are managed by the punchbox tool for the operator in order to speed up and simplify the procedure ?
Exercise: deploy your platform (with punchbox quickstarter)¶
Important
In this exercise, we will deploy a Punch with the punchbox tool using Vagrant in order to create target VMs locally. Consequently, you must have at least 32GB of RAM on your computer in order to be able to perform the exercise without any problem. You must also have a deployment archive on your laptop. Finally you must have an internet access to do all this, to download the VMs images and dependencies.
Note that for actual production deployment, the internet access is not required, as the needed packages are provided either as part of the punch deployer archive, or by the standard packages repositories mirror that the platform must provide.
Without Vagrant, or even without the punchbox
If you do not want to use Vagrant to create your target VMs, you can still use the punchbox to generate an example configuration and ready-to-use deployment environment.
If you do not want/cannot use the punchbox, than you can do without. But then you are doing a "normal" deployment without any shorcuts
Although we advise use of punchbox for a first time deployment (or for later fast deployment of test environments on a single PC), the following exercise provide some 'Hints' about what to do without the punchbox help
Step 1 : Clone punchbox repository¶
Clone punchbox repository the branch associated with the version of your deployer
Step 2 : Install prerequisites on deployer¶
As you can see in the Deployment Prerequisites, some actions must be performed on deployer server before executing deployment.
With the punchbox tool these steps are wrapped into one single command to simplify this procedure
Simply execute this command to install all deployer prerequisites :
make install
Tip
As you may understand, your laptop will act in the following steps as the deployer server
Step 3 : Generate exemple platform configuration with punchbox template¶
Punchbox provides commands to quickly generate mandatory (example) configurations for deployment.
Run this punchbox command :
source activate.sh # This will report that no deployer is unzipped yet
punchbox --platform-config-file configurations/complete_punch_32G.json \
--generate-vagrantfile \
--punch-user-config punch/configurations/validation \
--deployer "<path_to_your_punchplatform_deployer_zip>"
source activate.sh # Now the deployer is unziped so we have full working environment
punchplatform-deployer.sh --generate-platform-config \
--templates-dir punch/deployment_template/ \
--model punch/build/model.json
Not part of the production deployment process
The above commands are not used when following the normal deployment process. They are a punchbox shortcut that provides you with a ready-to-go platform configuration example matching a chosen predefined test set of VM and punch components (here the 'complete_punch_32G').
What have been done on my laptop !?
- Your deployer archive has been unzipped
- A Vagrantfile has been generated
- The resolv.hjson has been created
- The punchplatform-deployment.settings has been generated
Check your generated configuration files in punch/build/pp-conf/ to see results.
Keep in mind that without Punchbox you have to write these
files and install prerequisites manually before deploying.
Important
In summary, all deployment prerequisites and manual configurations have been achieved with these punchbox commands But you cannot rely on this tool to setup the prerequisites for you on a production deployer server, because internet access is needed.
If you are not able to run the punchbox
Here are 6.3 configuration examples generated by the punchbox at step 3:
{
"platform": {
"platform_id": "punchplatform-primary",
"setups_root": "/data/opt",
"remote_data_root_directory": "/data",
"remote_logs_root_directory": "/var/log/punchplatform",
"punchplatform_daemons_user": "vagrant",
"punchplatform_group": "vagrant",
"binaries_version": "punch-binaries-6.3.0-SNAPSHOT"
},
"reporters": {
"myreporter": {
"type": "kafka",
"brokers": "local",
"topic": "platform-events",
"reporting_interval": 30,
"encoding": "json"
}
},
"shiva": {
"shiva_version": "punch-shiva-6.4.5-SNAPSHOT-SNAPSHOT",
"clusters": {
"local": {
"reporters": [
"myreporter"
],
"storage": {
"type": "kafka",
"kafka_cluster": "local"
},
"servers": {
"server4": {
"runner": true,
"can_be_master": true,
"tags": [
"local"
]
},
"server3": {
"runner": true,
"can_be_master": true,
"tags": [
"local"
]
}
}
}
}
},
"punchplatform_operator": {
"punchplatform_operator_environment_version": "punch-operator-6.4.5-SNAPSHOT-SNAPSHOT",
"configuration_name_dir_from_home": "pp-conf",
"reporters": [
"myreporter"
],
"operators_username": [
"vagrant"
],
"storage": {
"type": "kafka",
"kafka_cluster": "local"
},
"servers": {
"server1": {}
}
},
"spark": {
"punchplatform_analytics_deployment_version": "punch-spark-deployment-6.4.5-SNAPSHOT-SNAPSHOT",
"clusters": {
"local": {
"master": {
"servers": [
"server5"
],
"listen_interface": "enp0s8",
"master_port": 7077,
"rest_port": 6066,
"ui_port": 8081
},
"slaves": {
"server3": {
"listen_interface": "enp0s8",
"slave_port": 7078,
"webui_port": 8084
}
},
"spark_workers_by_punchplatform_spark": 1,
"zk_cluster": "local",
"zk_root": "spark-main",
"slaves_cpu": 2,
"slaves_memory": "1200M"
}
}
},
"zookeeper": {
"zookeeper_version": "apache-zookeeper-3.7.0-bin",
"zookeeper_nodes_production_interface": "enp0s8",
"zookeeper_childopts": "-server -Xmx128m -Xms128m",
"zookeeper_admin_cluster_name": "local",
"clusters": {
"local": {
"hosts": [
"server2"
],
"cluster_port": 2181,
"punchplatform_root_node": "/punchplatform-primary"
}
}
},
"gateway": {
"gateway_version": "6.3.0-SNAPSHOT",
"clusters": {
"gateway_32g": {
"tenant": "mytenant",
"modsecurity_enabled": false,
"servers": {
"server2": {
"inet_address": "172.28.128.22",
"port": 4242
}
},
"elasticsearch": {
"data_cluster": {
"cluster_id": "es_search",
"hosts": [
"server2:9200"
],
"settings": [
"es.index.read.missing.as.empty: yes",
"es.nodes.discovery: true"
]
},
"metric_cluster": {
"cluster_id": "es_search",
"hosts": [
"server2:9200"
],
"index_name": "mytenant-gateway-logs",
"settings": [
"es.index.read.missing.as.empty: yes",
"es.nodes.discovery: true"
]
}
},
"resources": {
"doc_dir": "/data/opt/punchplatform-documentation-6.3.0-SNAPSHOT/doc/html",
"archives_dir": "/tmp",
"manager": {
"metadata": [
{
"type": "elasticsearch",
"hosts": [
"server2:9200"
],
"index": "resources-metadata"
}
],
"data": [
{
"type": "file",
"root_path": "/tmp/punchplatform/resources"
}
]
}
},
"reporters": [
"myreporter"
]
}
}
},
"kafka": {
"kafka_version": "kafka_2.12-2.8.1",
"kafka_brokers_production_interface": "enp0s8",
"clusters": {
"local": {
"brokers_with_ids": [
{
"id": 1,
"broker": "server2:9092"
}
],
"zk_cluster": "local",
"zk_root": "kafka-local",
"brokers_config": "punchplatform-local-server.properties",
"default_replication_factor": 1,
"default_partitions": 2,
"partition_retention_bytes": 1073741824,
"partition_retention_hours": 24,
"kafka_brokers_jvm_xmx": "512M"
}
}
},
"elasticsearch": {
"elasticsearch_version": "7.10.2",
"clusters": {
"es_search": {
"nodes": {
"server2": {
"http_api_address": "server2",
"transport_address": "server2",
"bind_address": "_enp0s8_",
"rack_id": "1"
}
},
"http_api_port": 9200,
"cluster_production_transport_address": "server2",
"transport_port": 9300,
"minimum_master_nodes": 1,
"settings_by_type": {
"data_node": {
"max_memory": "512m",
"modsecurity_enabled": false,
"modsecurity_blocking_requests": false,
"script_execution_authorized": true,
"http_cors_enabled": true,
"readonly": true
}
}
}
}
},
"storm": {
"storm_version": "apache-storm-2.3.0",
"storm_nimbus_nodes_production_interface": "enp0s8",
"clusters": {
"local": {
"master": {
"servers": [
"server3"
],
"cluster_production_address": "server3"
},
"ui": {
"servers": [
"server3"
],
"cluster_admin_url": "server3:8080"
},
"slaves": [
"server3"
],
"zk_cluster": "local",
"zk_root": "storm-1.2.2-main",
"storm_workers_by_punchplatform_supervisor": 10,
"workers_childopts": "-Xmx128m",
"supervisor_memory_mb": 256,
"supervisor_cpu": 2
}
}
},
"minio": {
"minio_version": "RELEASE.2020-08-26T00-00-49Z",
"minio_access_key": "admin",
"minio_secret_key": "punchplatform",
"clusters": {
"local": {
"hosts": [
"server2"
],
"port": "9000"
}
}
},
"clickhouse": {
"clickhouse_version": "20.4.6.53",
"clusters": {
"common": {
"shards": [
{
"servers": [
"server5"
]
}
],
"zk_cluster": "local",
"zk_root": "clickhouse",
"http_port": 8123,
"tcp_port": 9100
}
}
},
"kibana": {
"kibana_version": "7.10.2",
"domains": {
"admin": {
"es_cluster_target": "es_search",
"es_type_of_nodes_targeted": "data_node",
"kibana_port": 5601,
"type": "administration"
}
},
"servers": {
"server1": {
"address": "server1"
}
},
"plugins": {
"punchplatform": {
"punchplatform_version": "6.3.0-SNAPSHOT",
"rest_api": {
"hosts": [
"http://server2:4242"
]
}
},
"punchplatform_feedback": {
"punchplatform_feedback_version": "6.3.0-SNAPSHOT"
}
}
},
"metricbeat": {
"metricbeat_version": "7.10.2",
"modules": {
"system": {
"high_frequency_system_metrics": {
"metricsets": [
"cpu",
"load",
"memory"
],
"reporting_interval": "30s"
},
"normal_frequency_system_metrics": {
"metricsets": [
"fsstat"
],
"reporting_interval": "5m"
},
"slow_frequency_system_metrics": {
"metricsets": [
"uptime"
],
"reporting_interval": "1h"
}
}
},
"elasticsearch": {
"cluster_id": "es_search"
}
}
}
{
// All ES output nodes (Storm & Spark nodes)
elasticsearch_output_nodes:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.dag[?(@.type=='elasticsearch_output' || @.type=='elastic_output')].settings
additional_values:{
http_hosts:[
{
host:server2
port:9200
}
]
}
}
// All ES input (Spark & Storm)
elasticsearch_input_nodes:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.dag[?(@.type=='elastic_input' || @.type=='elastic_query_stats' || @.type=='python_elastic_input' || @.type=='python_elastic_output' || @.type=='extraction_input')].settings
additional_values:{
nodes:[
server2
]
}
}
// All Kafka input/output nodes (Storm nodes)
kafka_input:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.dag[?(@.type=='kafka_input' || @.type=='kafka_output')].settings
additional_values:{
brokers: local
bootstrap.servers:server2:9092
metric_cluster_id:local
}
}
// All Kafka reporters
kafka_reporter:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.metrics.reporters[?(@.type=='kafka')]
additional_values:{
brokers: local
bootstrap.servers:server2:9092
topic: platform-events
reporting_interval: 30
encoding: json
}
}
// All Elasticsearch metrics reporters
es_metrics_reporter:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.metrics.reporters[?(@.type=='elasticsearch')]
additional_values:{
cluster_name: es_search
metric_document_field_name: log
reporting_interval: 30
http_hosts:[
{
host:server2
port:9200
}
]
}
}
// All plan persistence
persistence_plan_resolver:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.settings.persistence[?(@.type=='elasticsearch')]
additional_values:{
nodes:[
{
host:server2
port:9200
}
]
}
}
// Archive houeskeeping
archive_housekeeping:{
selection:{
tenant:*
channel:*
runtime:*
}
match:$.archiving_pools[*]
additional_values:{
es_cluster_id: es_search
}
}
// Channel monitoring
channel_monitoring:{
selection:{
tenant:*
channel: monitoring
runtime:*
}
match: .elasticsearch
additional_values:{
es_cluster_id: es_search
}
}
// Channel monitoring reporters
channel_monitoring_kafka_reporters:{
selection:{
tenant:*
channel: monitoring
runtime:*
}
match:$.reporters[?(@.type=='kafka')]
additional_values:{
brokers: local
bootstrap.servers:server2:9092
topic: platform-events
reporting_interval: 30
encoding: json
}
}
channel_monitoring_es_reporters:{
selection:{
tenant:*
channel: monitoring
runtime:*
}
match:$.reporters[?(@.type=='elasticsearch')]
additional_values:{
cluster_name: es_search
reporting_interval: 30
http_hosts:[
{
host:server2
port:9200
}
]
}
}
}
Step 4 : Start vagrant boxes VM¶
Simply run :
punchbox --start-vagrant
Tip
Your Vagrantfile have been generated in the previous step.
If you do not want or cannot use vagrant for generating and starting up VMs, then you can create (for this example) five Ubuntu 20.04 VMs named "server1" to "server5". All of them must have the prerequisites, including the ability to know each-other names and to access standard system packages repositories.
- server1, server2 and server5 must have at least 3GB RAM
- server 3 and server 4 must have at least 6GB RAM
- all these machines should have 30 GB of disk space, and 2vcpus
Step 5 : Deploy your Punch platform components¶
Refer to this documentation to have more information about deployment
To just generate the ansible inventory of your platform (this step is not actually needed when doint production deployment, because it is included in the following 'deploy' command.
The purpose here is just generating the ansible inventory (it also does some checks on your deployment configuration).
punchplatform-deployer.sh -gi
How is this deployer command in my linux PATH ?
If you followed above steps with punchbox, you initialized your CURRENT window environment when you typed 'source activate.sh'.
If you open a new window, you'll have to type again the 'source activate.sh'
If you are not using punchbox, or if you do not want to always type 'source activate.sh', you can define the appropriate environment variables in your ~/.bashrc file (see deployer setup).
Now you can have a look at the generated configuration folder (in $PUNCHPLATFORM_CONF_DIR/generated_inventory).
To actually deploy your platform software components:
# The '-u vagrant' is the name of the ssh (sudoer) user to use to connect to target servers to deploy
# If a password was needed for ssh/sudo, we would have added ansible '-kK' option
punchplatform-deployer.sh deploy -u vagrant
After some deployment actions, your platform is up to go. You can connect to your nodes and check your running services or import punch channels configuration
Tip
In this track deployer and targets are on your laptop thanks to Vagrant. But, with punchbox tools, you can also have remote targets instead of using Vagrant. Check punchbox documentation to have more details