Skip to content

HOWTO deploy an external library

Why do that ?

After conclusive tests on developing customs applications based on our public API, you want to deploy your jar/pex on your platform.

For instance:

  • Adding a custom node for Spark/Pyspark or Storm in Java or Python.
  • Adding a custom application for Shiva in Java or Python.
  • Providing additional libraries to Punchlines (spark, storm, pyspark) during runtime executions.
  • Providing external resources, e.g. configuration files...

Compatible components

The following components support additional libraries :

Deployment Procedure

Punch makes it easy and safe to add an external library to a production platform.

1. Prerequisite

You must have your external library available on your deployer server (your custom node jar or pex file for example)

2. Pre Deployment (Deployer Server)

Important

  • We consider that $DEPLOYER_DIR points to the deployer installed path on your deployed platform.
  • $RUNTIME can either be storm, pyspark, spark or shiva
  • We expect that all the below instructions will be executed on the server used to deploy your production platform

Instructions

Below instructions are applicable for all $RUNTIME (storm, pyspark, spark and shiva)

# storm value can be replaced by spark, shiva or pyspark
RUNTIME=storm
mkdir -p $DEPLOYER_DIR/archives/extlib/$RUNTIME
mkdir -p $DEPLOYER_DIR/archives/extresource/$RUNTIME
  • all external libraries (jar/pex) should be copied under $DEPLOYER_DIR/archives/extlib/$RUNTIME
  • all external resources (e.g. custom nodes meta-file generated by punchplatform-inspect-node.sh) should be copied under $DEPLOYER_DIR/archives/extresource/$RUNTIME

Warning

Before proceeding, You should TAG all changes with git tag command.

cd $PUNCHPLATFORM_CONF_DIR
git pull
git tag -a v<CURRENT_VERSION> -m "before adding new library <MY_LIBRARY>"
git push --tags

Note

Deployment configuration exists only locally and are not gitted.

Before deploying, you will have to update your local deployment configuration (punchplatform-deployment.settings), and the deployer ansible inventories. These configurations will be used afterwards for deploying...

The below command should be executed flawlessly, if any error(s) are encountered, you may contact our support team.

punchplatform-deployer.sh --generate-inventory

Push the new configuration. This operation has no impact on the running platform.

git push

Important

Before going to the next step, make sure you communicated your procedure to the platform stakeholders.

3. Deployment

Deploy the library to all target servers. Depending on what you want to update (storm, spark, pyspark or shiva), execute :

# storm value can be replaced by spark, shiva or pyspark
RUNTIME=storm
punchplatform-deployer.sh --deploy -Kk -t $RUNTIME

4. Post Deployment (Deployed server)

Execute the same instructions that were used during your development phase to see if your custom application is working as expected.

Tip

If errors appears, no panic! Rollback your environment with the old deployer (rollback deployment environment variable and redeploy operators )

4. Final Checks

Tag the end of migration from the operator device:

cd $PUNCHPLATFORM_CONF_DIR
git pull
git tag -a v<NEW_VERSION> -m "end of migration to <NEW_VERSION>"
git push --tags

5. Rollback procedure

# external libraries for our supported runtimes are located under
$PUNCHPLATFORM_INSTALL_DIR/extlib/pyspark
$PUNCHPLATFORM_INSTALL_DIR/extlib/spark
$PUNCHPLATFORM_INSTALL_DIR/extlib/storm
$PUNCHPLATFORM_INSTALL_DIR/extlib/shiva

# external resources for our supported runtimes are located under
$PUNCHPLATFORM_INSTALL_DIR/extresource/pyspark
$PUNCHPLATFORM_INSTALL_DIR/extresource/spark
$PUNCHPLATFORM_INSTALL_DIR/extresource/storm
$PUNCHPLATFORM_INSTALL_DIR/extresource/shiva

Removal of extlib sub-directories based on runtime. Depending on what you have updated (storm, spark or pyspark), execute :

# storm value can be replaced by spark, shiva or pyspark
RUNTIME=storm
SERVER=storm_servers
punchplatform-deployer.sh --ssh $SERVER "rm -rf {{install_root}}/{{binaries_version}}/extlib/$RUNTIME/*"
punchplatform-deployer.sh --ssh $SERVER "rm -rf {{install_root}}/{{binaries_version}}/extresource/$RUNTIME/*"

Then update external library directory on the deployment device with the previous libraries or remove existing libraries and re-deploy :

# storm value can be replaced by spark, shiva or pyspark
RUNTIME=storm
punchplatform-deployer.sh --deploy -Kk -t $RUNTIME