Skip to content

Track 3 Deployed platform organisation and Patch procedure

Abstract

This track covers the fundamentals of the patching procedure on a running platform

Patchable and non-patchable components

Punchplatform provides different components on a deployed platform. All of them are not developed by Punch team, as a consequency, some of them are patchable and others not

Non-patchable components will imply an upgrade process for changing the version.

Deployed folders structure

In the deployment settings, we choose (among other important things):

  • a root folder for binaries setup (usually /data/opt or /opt).
  • a root folder for logs (usually /var/log/punchplatform)
  • a root folder for live application data (usually /data)

Tip

It is often a good idea to place the binaries in /data/opt, because /data is normally a dedicated partition, to avoid filling up '/' partition with big application data.

Because Punch may installed a few Gb of binaries for Each version, putting 'opt' in /data can help prevent filling up the '/' when deploying.

Here are typical results of deployment of punchplatform on the target nodes filesystem:

exemple tree of Zookeeper and kafka on the same node

    /data
    ├── kafka
    │   └── back
    ├── opt
    │   ├── apache-zookeeper-3.5.5-bin
    │   ├── kafka_2.12-2.8.1
    │   └── metricbeat-7.10.2-linux-x86_64
    └── zookeeper
        └── zookeeper-zkm

As we see, each binary package version has a dedicated folder here, so setting up a new kafka version would mean a new binaries (and configuration) folder, which leaves the previous version available for rollback.

But most of the time, there are live data associated to applications (Kafka queues, Zookeeper filesystem, Shiva management data stored in kafka cluster) that in fact can prevent rollbacking if you change the version of some component !

The active version of a framework component is decided by the systemd configuration installed by the punch deployer. So when you install a new version of a component using the deployer, the systemd monitor has to stop the previous running one, and restart using the new systemd configuration for the service

Exercise: view the running version of a service

    sudo systemctl cat zookeeper-zkm
This will dump you the current systemd configuration file, including the version of zookeeper that is wanted to run.

    [...]
      ExecStart=/data/opt/apache-zookeeper-3.5.5-bin/bin/zkServer.sh start-foreground
    [...]
What version is actually running ?

In fact, the only way to know is by running a 'ps' command. Because the deployer may have changed the systemd configuration file (what we just dumped), without actually restarting the application daemon.

Sometimes, this is unwanted (because the deployer failed and forgot to restart the daemon).

Sometimes, this is wanted (because we want to prepare all nodes for update, but then we will restart one node at a time at some other point, to ensure improved availability of the cluster)

So: 

sudo ps -eaf | grep apache-zookeeper

>>
      root       633     1  0 Oct28 ?        00:16:52 java -Dzookeeper.log.dir=/var/log/punchplatform/zookeeper -Dzookeeper.log.file=zookeeper--server-localhost.localdomain.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /data/opt/apache-zookeeper-3.5.5-bin/bin/../zoo [...]

Patch procedure

Refer to the Patch procedure

Exercise

Step 1 : Import your patch to deployer machine

Tip

According to the patch procedure documentation, on a running platform a patch will be delivered by Punchplatform team. However, for this exercice we are going to create a fake patch for the operator envrionment to simplify procedure

To create our patch, simply run : 

mkdir -p $PUNCHBOX_DIR/punch/build/punch-deployer-X.Y.Z/archives/patch
cd $PUNCHBOX_DIR/punch/build/punch-deployer-X.Y.Z/archives/
unzip punch-binaries-X.Y.Z.zip
cp punch-binaries-X.Y.Z/lib/punch-command-app-X.Y.Z-jar-with-dependencies.jar patch/

You have now a patch which is ready to be deployed on your target servers

Step 2 : Deploy your patch

Let's deploy our patch to operator nodes using associated tag

punchplatform-deployer.sh --deploy -u vagrant -t patch -l punchplatform_operator_servers

Tip

As you can see, when you launched this command a message such as WARNING: WARN: a binary patch is configured appeared which telling you that a patch is going to be deployed

Step 3 : Check that your patch have been correctly deployed

Log in to your operator node, and simply run :

punchlinectl -t mytenant start --punchline /home/vagrant/pp-conf/tenants/mytenant/channels/apache_httpd/input.json

Tip

As you can see, when you launched this command a message such as using patch : .. appeared which telling you that your patch is used

Tip

When patching a service such as shiva or gateway, this message will not appeared in your terminal but in service logs

Step 4 : Rollback procedure

You may now want to delete your patch, to achieve this, simply run : 

punchplatform-deployer.sh --ssh punchplatform_operator_servers "sudo rm -rf {{install_root}}/{{binaries_version}}/patch" -u vagrant

Tip

You can now re-run your punchline and see that your patch is not use anymore

Other operation usage of generated Ansible inventories

You can make use of the Ansible inventory available to run distributed commands on part of your platform for example for manual capacity checks, incident management, maintainance operation...

E.g. You want to quick check the remaining disk space on all your kafka data storage:

Tip

For available groups of machines that can be used with ansible, you can view $PUNCHPLATFORM_CONF_DIR/generated_inventory/current-deployment.inv

  • Using the punchplatform-deployer.sh ssh subcommand. e.g.:

    punchplatform-deployer.sh ssh kafka_servers "df -h /data"

>> server2 | CHANGED | rc=0 >>
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/sda1        39G  5.6G   34G  15% /

   server3 [...]

  • Using freeform Ansible (allows to use other module). E.g. creating /data/bkp on all nodes of a shiva cluster:

    ansible -i generated_inventory/current-deployment.inv shiva_local -m "file" -a "state=directory path=/data/bkp owner=vagrant"

Tip

As for all ansible commands, the punchplatform-deployer.ssh ssh may need connection additional options like '-u username' or '-kK' for password providing