Shiva

Abstract

Shiva is punch core service. Unlike most of the other services it is developed and maintained by the punch team. This page aims to explain shiva layout and how to patch it in case of issues

Overview

Shiva is an application scheduler that lets you schedule and run various applications on top of a punch cluster.

Refer to the shiva documentation for understanding shiva architecture and role.

Tip

If you are familiar with kubernetes, see shiva as a lightweigth application scheduler just like kubernetes is a container scheduler. Shiva is much simpler of course, and does not provide any of the (many) kuberne services such as kubernetes networking capabilities. Yet it allows you to deploy production-ready applications on physical or virtual servers.

Shiva configuration

A shiva node is a server equipped with shiva packages. It can act as a shiva leader or worker. Only workers run application. Here is the folder layout of shiva:

punchplatform-shiva-6.1.0/
├── activate.sh
├── bin
│   ├── archives-housekeeping
│   ├── channels-monitoring
│   ├── commons-lib.sh
│   ├── elastalert
│   ├── elasticsearch-housekeeping
│   ├── hjson-manager
│   ├── logstash
│   ├── mlflow_api
│   ├── planctl
│   ├── platform-monitoring
│   ├── punchlinectl
│   ├── punchplatform-env.sh
│   ├── punchplatform-lib.sh
│   └── sparkctl
├── conf
│   └── shiva.conf
├── data
├── log4j2
│   ├── log4j2.properties
│   ├── log4j2-punchline.xml
│   ├── log4j2-shiva-daemon.properties
│   ├── log4j2-shiva-daemon.xml
│   ├── log4j2-sparkctl.properties
│   ├── log4j2-sparkctl.xml
│   └── log4j2-stdout.xml
├── punchplatform.properties
└── resolv.hjson

This folder contains shiva configuration files together with all the startup (shell) command for applications possibly launched by shiva. There are also two punch platform-level configuration files (resolv.hjson and punchplatform.properties) generated at deploy time. These contain the runtime information required to run shiva applications.

Shiva binaries

The binaries and libraries (i.e. executable) files are provided in a separate folder. These are provided by a punch uber binary package named punchplatform-binaries-*. The punch deployer installs it on each punch node. Here is its layout:

punchplatform-binaries-6.1.0/
├── apache-storm-2.1.0
├── lib
│   ├── spark/
│   ├── pyspark/
│   ├── storm/
│   ├── punchplatform-command-cli-6.1.0-jar-with-dependencies.jar
│   ├── punchplatform-gateway-app-6.1.0.jar
│   ├── punchplatform-log-injector-6.1.0-jar-with-dependencies.jar
│   ├── punchplatform-monitoring-service-6.1.0-jar-with-dependencies.jar
│   ├── punchplatform-puncher-6.1.0-jar-with-dependencies.jar
│   ├── punchplatform-shiva-app-6.1.0-jar-with-dependencies.jar
│   └── ...
├── logstash-7.4.2
├── resources
└── spark-2.4.3-bin-hadoop2.7

As you can see this folder contains both punchplatform-shiva-app-*-jar-with-dependencies.jar which is Shiva service library but also for example logstash binaries which are necessary to launch logstash as a Shiva application

Patching Shiva

To patch shiva or any of its internal applications, refer to this documentation.