Skip to content

Track 1 Deployment procedure

Abstract

This track covers the fundamentals of punch deployment (with associated actions to conduct when using Punchbox tool as a quick-start infrastructure and configuration resources package)

Deployment overview

Refer to the Deployment Process and associated list of deployment steps of each steps to follow to have an operational deployed Punch platform

Key Points
  • Deployment of infrastructure prerequisites (including some system or custom solution setup INSIDE the production server Operating system) is not automated by Punch

  • There are many choices of what you want to deploy with the punch deployer and where (depending on the solution needs), and this requires solution engineering

  • Following a central deployment file prepared by the platform or solutionintegrator, the punch deployer tool installs the desired components of the punch framework on the target servers, and configures systemd so that the punch framework services are automatically running, clustered and interconnected.

  • After deploying the framework, channels configuration must be imported or created, to supply business-level pipelines, applications, dashboards, rules. This can happen both at deployment time (if the configuration has been prepared by the integrator beforehand), or enriched later by bringing additional or updated configuration channels and (re)starting the impacted channels.

In the deployment process, which steps must be done by the operator before executing deployment ?

Answer
  • ensure the deployment machine has the local prerequisites installed
  • ensure he has a remote sudoer account usable for remote-access from the deployment machine
  • ensure that all the target prerequisites are deployed on the platform (NTP, DNS and interfaces resolution, disk space, firewalling rules, disks partitioning, standard packages repository available ...)
  • get the deployer package and unzip it on the deployment machine
  • configure deployer account environment variable to point at configuration (PUNCHPLATFORM_CONF_DIR) and deployer bin path.
  • import factory-prepared or create locally the deployment files (punchplatform-deployment.settings and resolv.hjson)

Punchbox tool

Refer to the Deployer Getting Started to have a quick introduction of punchbox tool

Question

Which stages of the deployment procedure are managed by the punchbox tool for the operator in order to speed up and simplify the procedure ?

Exercise: deploy your platform (with punchbox quickstarter)

Important

In this exercise, we will deploy a Punch with the punchbox tool using Vagrant in order to create target VMs locally. Consequently, you must have at least 32GB of RAM on your computer in order to be able to perform the exercise without any problem. You must also have a deployment archive on your laptop. Finally you must have an internet access to do all this, to download the VMs images and dependencies.

Note that for actual production deployment, the internet access is not required, as the needed packages are provided either as part of the punch deployer archive, or by the standard packages repositories mirror that the platform must provide.

Without Vagrant, or even without the punchbox

If you do not want to use Vagrant to create your target VMs, you can still use the punchbox to generate an example configuration and ready-to-use deployment environment.

If you do not want/cannot use the punchbox, than you can do without. But then you are doing a "normal" deployment without any shorcuts

Although we advise use of punchbox for a first time deployment (or for later fast deployment of test environments on a single PC), the following exercise provide some 'Hints' about what to do without the punchbox help

Step 1 : Clone punchbox repository

Clone punchbox repository the branch associated with the version of your deployer

Step 2 : Install prerequisites on deployer

As you can see in the Deployment Prerequisites, some actions must be performed on deployer server before executing deployment.

With the punchbox tool these steps are wrapped into one single command to simplify this procedure

Simply execute this command to install all deployer prerequisites :

make install

Tip

As you may understand, your laptop will act in the following steps as the deployer server

Step 3 : Generate exemple platform configuration with punchbox template

Punchbox provides commands to quickly generate mandatory (example) configurations for deployment.

Run this punchbox command :

source activate.sh # This will report that no deployer is unzipped yet

punchbox --platform-config-file configurations/complete_punch_32G.json \
        --generate-vagrantfile \
        --punch-user-config punch/configurations/validation \
        --deployer "<path_to_your_punchplatform_deployer_zip>"

source activate.sh # Now the deployer is unziped so we have full working environment

punchplatform-deployer.sh --generate-platform-config \
    --templates-dir punch/deployment_template/ \
    --model punch/build/model.json

Not part of the production deployment process

The above commands are not used when following the normal deployment process. They are a punchbox shortcut that provides you with a ready-to-go platform configuration example matching a chosen predefined test set of VM and punch components (here the 'complete_punch_32G').

What have been done on my laptop !?

Check your generated configuration files in punch/build/pp-conf/ to see results. Keep in mind that without Punchbox you have to write these files and install prerequisites manually before deploying.

Important

In summary, all deployment prerequisites and manual configurations have been achieved with these punchbox commands But you cannot rely on this tool to setup the prerequisites for you on a production deployer server, because internet access is needed.

If you are not able to run the punchbox

Here are 6.3 configuration examples generated by the punchbox at step 3:

{
  "platform": {
    "platform_id": "punchplatform-primary",
    "setups_root": "/data/opt",
    "remote_data_root_directory": "/data",
    "remote_logs_root_directory": "/var/log/punchplatform",
    "punchplatform_daemons_user": "vagrant",
    "punchplatform_group": "vagrant",
    "binaries_version": "punch-binaries-6.3.0-SNAPSHOT"
  },
  "reporters": {
    "myreporter": {
      "type": "kafka",
      "brokers": "local",
      "topic": "platform-events",
      "reporting_interval": 30,
      "encoding": "json"
    }
  },
  "shiva": {
    "shiva_version": "punch-shiva-6.4.5-SNAPSHOT-SNAPSHOT",
    "clusters": {
      "local": {
        "reporters": [
          "myreporter"
        ],
        "storage": {
          "type": "kafka",
          "kafka_cluster": "local"
        },
        "servers": {
          "server4": {
            "runner": true,
            "can_be_master": true,
            "tags": [
              "local"
            ]
          },
          "server3": {
            "runner": true,
            "can_be_master": true,
            "tags": [
              "local"
            ]
          }
        }
      }
    }
  },
  "punchplatform_operator": {
    "punchplatform_operator_environment_version": "punch-operator-6.4.5-SNAPSHOT-SNAPSHOT",
    "configuration_name_dir_from_home": "pp-conf",
    "reporters": [
      "myreporter"
    ],
    "operators_username": [
      "vagrant"
    ],
    "storage": {
      "type": "kafka",
      "kafka_cluster": "local"
    },
    "servers": {
      "server1": {}
    }
  },
  "spark": {
    "punchplatform_analytics_deployment_version": "punch-spark-deployment-6.4.5-SNAPSHOT-SNAPSHOT",
    "clusters": {
      "local": {
        "master": {
          "servers": [
            "server5"
          ],
          "listen_interface": "enp0s8",
          "master_port": 7077,
          "rest_port": 6066,
          "ui_port": 8081
        },
        "slaves": {
          "server3": {
            "listen_interface": "enp0s8",
            "slave_port": 7078,
            "webui_port": 8084
          }
        },
        "spark_workers_by_punchplatform_spark": 1,
        "zk_cluster": "local",
        "zk_root": "spark-main",
        "slaves_cpu": 2,
        "slaves_memory": "1200M"
      }
    }
  },
  "zookeeper": {
    "zookeeper_version": "apache-zookeeper-3.7.0-bin",
    "zookeeper_nodes_production_interface": "enp0s8",
    "zookeeper_childopts": "-server -Xmx128m -Xms128m",
    "zookeeper_admin_cluster_name": "local",
    "clusters": {
      "local": {
        "hosts": [
          "server2"
        ],
        "cluster_port": 2181,
        "punchplatform_root_node": "/punchplatform-primary"
      }
    }
  },
  "gateway": {
    "gateway_version": "6.3.0-SNAPSHOT",
    "clusters": {
      "gateway_32g": {
        "tenant": "mytenant",
        "modsecurity_enabled": false,
        "servers": {
          "server2": {
            "inet_address": "172.28.128.22",
            "port": 4242
          }
        },
        "elasticsearch": {
          "data_cluster": {
            "cluster_id": "es_search",
            "hosts": [
              "server2:9200"
            ],
            "settings": [
              "es.index.read.missing.as.empty: yes",
              "es.nodes.discovery: true"
            ]
          },
          "metric_cluster": {
            "cluster_id": "es_search",
            "hosts": [
              "server2:9200"
            ],
            "index_name": "mytenant-gateway-logs",
            "settings": [
              "es.index.read.missing.as.empty: yes",
              "es.nodes.discovery: true"
            ]
          }
        },
        "resources": {
          "doc_dir": "/data/opt/punchplatform-documentation-6.3.0-SNAPSHOT/doc/html",
          "archives_dir": "/tmp",
          "manager": {
            "metadata": [
              {
                "type": "elasticsearch",
                "hosts": [
                  "server2:9200"
                ],
                "index": "resources-metadata"
              }
            ],
            "data": [
              {
                "type": "file",
                "root_path": "/tmp/punchplatform/resources"
              }
            ]
          }
        },
        "reporters": [
          "myreporter"
        ]
      }
    }
  },
  "kafka": {
    "kafka_version": "kafka_2.12-2.8.1",
    "kafka_brokers_production_interface": "enp0s8",
    "clusters": {
      "local": {
        "brokers_with_ids": [
          {
            "id": 1,
            "broker": "server2:9092"
          }
        ],
        "zk_cluster": "local",
        "zk_root": "kafka-local",
        "brokers_config": "punchplatform-local-server.properties",
        "default_replication_factor": 1,
        "default_partitions": 2,
        "partition_retention_bytes": 1073741824,
        "partition_retention_hours": 24,
        "kafka_brokers_jvm_xmx": "512M"
      }
    }
  },
  "elasticsearch": {
    "elasticsearch_version": "7.10.2",
    "clusters": {
      "es_search": {
        "nodes": {
          "server2": {
            "http_api_address": "server2",
            "transport_address": "server2",
            "bind_address": "_enp0s8_",
            "rack_id": "1"
          }
        },
        "http_api_port": 9200,
        "cluster_production_transport_address": "server2",
        "transport_port": 9300,
        "minimum_master_nodes": 1,
        "settings_by_type": {
          "data_node": {
            "max_memory": "512m",
            "modsecurity_enabled": false,
            "modsecurity_blocking_requests": false,
            "script_execution_authorized": true,
            "http_cors_enabled": true,
            "readonly": true
          }
        }
      }
    }
  },
  "storm": {
    "storm_version": "apache-storm-2.3.0",
    "storm_nimbus_nodes_production_interface": "enp0s8",
    "clusters": {
      "local": {
        "master": {
          "servers": [
            "server3"
          ],
          "cluster_production_address": "server3"
        },
        "ui": {
          "servers": [
            "server3"
          ],
          "cluster_admin_url": "server3:8080"
        },
        "slaves": [
          "server3"
        ],
        "zk_cluster": "local",
        "zk_root": "storm-1.2.2-main",
        "storm_workers_by_punchplatform_supervisor": 10,
        "workers_childopts": "-Xmx128m",
        "supervisor_memory_mb": 256,
        "supervisor_cpu": 2
      }
    }
  },
  "minio": {
    "minio_version": "RELEASE.2020-08-26T00-00-49Z",
    "minio_access_key": "admin",
    "minio_secret_key": "punchplatform",
    "clusters": {
      "local": {
        "hosts": [
          "server2"
        ],
        "port": "9000"
      }
    }
  },
  "clickhouse": {
    "clickhouse_version": "20.4.6.53",
    "clusters": {
      "common": {
        "shards": [
          {
            "servers": [
              "server5"
            ]
          }
        ],
        "zk_cluster": "local",
        "zk_root": "clickhouse",
        "http_port": 8123,
        "tcp_port": 9100
      }
    }
  },
  "kibana": {
    "kibana_version": "7.10.2",
    "domains": {
      "admin": {
        "es_cluster_target": "es_search",
        "es_type_of_nodes_targeted": "data_node",
        "kibana_port": 5601,
        "type": "administration"
      }
    },
    "servers": {
      "server1": {
        "address": "server1"
      }
    },
    "plugins": {
      "punchplatform": {
        "punchplatform_version": "6.3.0-SNAPSHOT",
        "rest_api": {
          "hosts": [
            "http://server2:4242"
          ]
        }
      },
      "punchplatform_feedback": {
        "punchplatform_feedback_version": "6.3.0-SNAPSHOT"
      }
    }
  },
  "metricbeat": {
    "metricbeat_version": "7.10.2",
    "modules": {
      "system": {
        "high_frequency_system_metrics": {
          "metricsets": [
            "cpu",
            "load",
            "memory"
          ],
          "reporting_interval": "30s"
        },
        "normal_frequency_system_metrics": {
          "metricsets": [
            "fsstat"
          ],
          "reporting_interval": "5m"
        },
        "slow_frequency_system_metrics": {
          "metricsets": [
            "uptime"
          ],
          "reporting_interval": "1h"
        }
      }
    },
    "elasticsearch": {
      "cluster_id": "es_search"
    }
  }
}
{ 


   // All ES output nodes (Storm & Spark nodes)
   elasticsearch_output_nodes:{
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.dag[?(@.type=='elasticsearch_output' || @.type=='elastic_output')].settings
      additional_values:{ 
         http_hosts:[ 
            { 
               host:server2
               port:9200
            }
         ]

      }
   }

   // All ES input (Spark & Storm)
   elasticsearch_input_nodes:{
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.dag[?(@.type=='elastic_input' || @.type=='elastic_query_stats' || @.type=='python_elastic_input' || @.type=='python_elastic_output' || @.type=='extraction_input')].settings
      additional_values:{ 
         nodes:[ 
            server2
         ]

      }
   }




   // All Kafka input/output nodes (Storm nodes)
   kafka_input:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.dag[?(@.type=='kafka_input' || @.type=='kafka_output')].settings
      additional_values:{ 
         brokers: local
         bootstrap.servers:server2:9092
         metric_cluster_id:local
      }
   }



   // All Kafka reporters
   kafka_reporter:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.metrics.reporters[?(@.type=='kafka')]
      additional_values:{ 
         brokers: local
         bootstrap.servers:server2:9092
         topic: platform-events
         reporting_interval: 30
         encoding: json
      }
   }




   // All Elasticsearch metrics reporters
   es_metrics_reporter:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.metrics.reporters[?(@.type=='elasticsearch')]
      additional_values:{ 
         cluster_name: es_search
         metric_document_field_name: log
         reporting_interval: 30
         http_hosts:[ 
            { 
               host:server2
               port:9200
            }
         ]

      }
   }


   // All plan persistence
   persistence_plan_resolver:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.settings.persistence[?(@.type=='elasticsearch')]
      additional_values:{ 
         nodes:[ 
               {
               host:server2
               port:9200
              } 
         ]

      }
   }
   // Archive houeskeeping
   archive_housekeeping:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.archiving_pools[*]
      additional_values:{ 
         es_cluster_id: es_search

      }
   }
   // Channel monitoring 
   channel_monitoring:{ 
      selection:{ 
         tenant:*
         channel: monitoring
         runtime:*
      }
      match: .elasticsearch
      additional_values:{ 
         es_cluster_id: es_search

      }
   }
   // Channel monitoring reporters 
   channel_monitoring_kafka_reporters:{ 
      selection:{ 
         tenant:*
         channel: monitoring
         runtime:*
      }
      match:$.reporters[?(@.type=='kafka')]
      additional_values:{ 
       brokers: local
       bootstrap.servers:server2:9092

       topic: platform-events
       reporting_interval: 30
       encoding: json
      }
   }

    channel_monitoring_es_reporters:{ 
      selection:{ 
         tenant:*
         channel: monitoring
         runtime:*
      }
      match:$.reporters[?(@.type=='elasticsearch')]
      additional_values:{ 
         cluster_name: es_search

        reporting_interval: 30
         http_hosts:[ 
            { 
               host:server2
               port:9200
            }
         ]

      }
   }


}

Step 4 : Start vagrant boxes VM

Simply run :

punchbox --start-vagrant

Tip

Your Vagrantfile have been generated in the previous step.

If you do not want or cannot use vagrant for generating and starting up VMs, then you can create (for this example) five Ubuntu 20.04 VMs named "server1" to "server5". All of them must have the prerequisites, including the ability to know each-other names and to access standard system packages repositories.

  • server1, server2 and server5 must have at least 3GB RAM
  • server 3 and server 4 must have at least 6GB RAM
  • all these machines should have 30 GB of disk space, and 2vcpus

Step 5 : Deploy your Punch platform components

Refer to this documentation to have more information about deployment

To just generate the ansible inventory of your platform (this step is not actually needed when doint production deployment, because it is included in the following 'deploy' command.

The purpose here is just generating the ansible inventory (it also does some checks on your deployment configuration).

punchplatform-deployer.sh -gi
How is this deployer command in my linux PATH ?

If you followed above steps with punchbox, you initialized your CURRENT window environment when you typed 'source activate.sh'.

If you open a new window, you'll have to type again the 'source activate.sh'

If you are not using punchbox, or if you do not want to always type 'source activate.sh', you can define the appropriate environment variables in your ~/.bashrc file (see deployer setup).

Now you can have a look at the generated configuration folder (in $PUNCHPLATFORM_CONF_DIR/generated_inventory).

To actually deploy your platform software components:

# The '-u vagrant' is the name of the ssh (sudoer) user to use to connect to target servers to deploy
# If a password was needed for ssh/sudo, we would have added ansible '-kK' option
punchplatform-deployer.sh deploy -u vagrant

After some deployment actions, your platform is up to go. You can connect to your nodes and check your running services or import punch channels configuration

Tip

In this track deployer and targets are on your laptop thanks to Vagrant. But, with punchbox tools, you can also have remote targets instead of using Vagrant. Check punchbox documentation to have more details