Skip to content

Patch Procedure

Abstract

Patching refers to patch of a specific component or set of component after retrieving a patch file from the punch online site. This procedure is not a platform update or upgrade.

Patchable Components

The following components support patching:

  • Shiva : the shiva scheduler service can be patched. Refer to this procedure.
  • Shiva applications : shiva is actually used to start a number of punch applications (punchlines, logstash, monitoring apps etc..). These can be patched as well. Refer to this documentation.
  • Punch Gateway : the punch API and REST gateway together with its underlying actions can be patched. Refer to this procedure.
  • Operator environment : the operator unix environment and the channelctl, punchlinectl etc.. commands can be patched. Refer to this procedure documentation
  • Punch internal binaries : some punch binaries coming along spark, storm and pyspark official binaries can be patched. Refer to Storm procedure and Spark/Pyspark procedure documentation to have more informations. Note that in the special case of the punch built-in Storm components, those are updated by patching the operator nodes and restarting the related channels, not by patching the Storm cluster itself.

Non Patchable Components

The Storm, Spark, Elasticsearch, Zookepeer, Minio, Clickhouse clusters and the Kibana application cannot be patched. This does not mean you cannot conduct an update or upgrade procedure with a rollback strategy, it simply means the punch does not provide a custom lighweight patch strategy.

To address the update of these components, get in touch with the punch professional service team.

Patch Procedure

The punch makes it easy and safe to patch a production platform. But even so, patching a production platform is a critical action. Make sure you have a clear idea of:

  • who is responsible ? : patching a component always have consequences, sometimes unexpected. Make sure the impacts are identified, reviewed and accepted by the platform owner.
  • what is the downtime ? : most often there is no or little downtime. This said it all depends on the component you patch. Make sure you kown what will be impacted and for how long. When patching multiple nodes of a highly-available cluster addressed by a single Virtual IP (shiva nodes holding network listeners, gateway servers) you may want to limit update command to only one node at a time (using -l ), and manage mannually the Virtual IP location (through your platform system-level Virtual IP management commands) consistently so that the IP is always served by a node which you are not currently updating.
  • will it generate monitoring alerts ? patching a component requires some sort of restart. Make sure the monitoring plane will not generate intempestive alarms
  • are you ready to rollback ? : the punch patching procedure is designed to be simple and easy to rollback. Make sure you know how to do it before patching.

If these are cleared up, proceed to the following procedure. Make sure you complete all the steps in order.

1. Platform Check

First, check the version of your platform:

# Go to an operator device and type
$ punchplatform-version.sh
# You must have this kind of output:
 PunchPlatform package:
 X.Y.Z

Next, check if a patch is already installed on your platform. It is a key point to check. Go to an operator, shiva and gateway servers and check if a

/data/opt/punchplatform-binaries-X.Y.Z/patch

folder exists. If you have some jars or libraries in there, first check that the latest stable punch release includes the issue that necessitated the deployment of the patch(es). Refer to the online documentation and/or email the support team at support@punchplatform.com. It might be necessary to first update your platform to the latest minor release that corresponds to your release so as to get rid of unecessary patches.

Assuming you have no patch or you have updated your platform as just explained, check the platform status and make sure :

  • All the services are green in punch monitoring dashboard
  • your supervisor (nagios or any other monitoring components) reports a green status especially with respect to the System, storage, memory, CPU monitoring.

Do not proceed is that status is unclear. Check your local support or in turn the punch support team.

2. Patch Preparation

Download the patch, most often a zip file or a jar file from the online punch download area.

Next update the deployer folder. That folder is project specific and located on the deployer server (i.e. the server from where you deployed your platform).

Create if not already there the following folder :

cd <deployer_path>
$ mkdir archives/patch/

Copy in that folder the downloaded patch. Create a git tag before patching

Important

It can be a simple jar or a zip with a subdirectory (i.e. storm, spark ..) containing some jars or pex files In this case you must unzip this patch and keep tree structure in your patch directory (archives/patch/spark/mypatch.jar)

Next, still on the same deployer server, go to $PUNCHPLATFORM_CONF_DIR and tag the version

cd $PUNCHPLATFORM_CONF_DIR
git pull
git tag -a v<CURRENT_VERSION> -m "before migrate to <NEW_VERSION>"
git push --tags

Generate the deployment configuration with the following command. these files has no impact of the running platform, it is only a local operation.

punchplatform-deployer.sh --generate-inventory

That command regenerates the various inventories and variables used by ansible to deploy the punch components. This command has no effect on the running platform. If errors are reported, please check the punchplatform-deployment.settings configuration, and contact the punch support.

Push the new configuration. This operation has no no impact on the running platform.

git push

Important

Before going to the next step, make sure you communicated your procedure to the platform stakeholders.

3. Patch Deployment

Deploy the patch to all punch servers. Depending on what you patch (operator, shiva, gateway, storm, spark ..) execute :

punchplatform-deployer.sh --deploy -Kk -t patch [-l punchplatform_operator_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l shiva_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l gateway_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l storm_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l spark_servers]

Next test the new patch by restarting all or some channels. It is of course much safer to restart only a single channel and check it runs as expected:

channelctl stop --channel <channel>
channelctl start --channel <channel>

Make sure :

  • the channel stop and start was ok
  • that the behaviour of the channel is correct : eps rate, failure ratea, content of the data in Elasticsearch, in the SIEM etc ...
  • check the monitoring status.

Tip

If errors appears, no panic! Rollback your environment with the old deployer (rollback deployment environment variable and redeploy operators )

4. Final Checks

Tag the end of migration from the operator device:

cd $PUNCHPLATFORM_CONF_DIR
git pull
git tag -a v<NEW_VERSION> -m "end of migration to <NEW_VERSION>"
git push --tags

Announce the end of the patching to all stakeholders.

5. Rollback procedure

Remove the patch directory on target nodes. Depending on what you have patched (operator, shiva, gateway, storm, spark ..) execute :

$ punchplatform-deployer.sh --ssh punchplatform_operator_servers "rm -rf {{install_root}}/{{binaries_version}}/patch"
or:
$ punchplatform-deployer.sh --ssh shiva_servers "rm -rf {{install_root}}/{{binaries_version}}/patch"
or:
$ punchplatform-deployer.sh --ssh gateway_servers "rm -rf {{install_root}}/{{binaries_version}}/patch"
or:
$ punchplatform-deployer.sh --ssh storm_servers "rm -rf {{install_root}}/{{binaries_version}}/patch"
or:
$ punchplatform-deployer.sh --ssh spark_servers "rm -rf {{install_root}}/{{binaries_version}}/patch"

Then update patch directory on the deployment device with the previous patches or remove existing patches and redeploy :

$ punchplatform-deployer.sh --deploy -Kk -t patch [-l punchplatform_operator_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l shiva_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l gateway_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l storm_servers]
or:
$ punchplatform-deployer.sh --deploy -Kk -t patch [-l spark_servers]

Create a patch

Check this documentation if you want to create a patch