Package org.thales.punch.apps.shiva

Shiva Punch Scheduler

See: Description

Interface Summary

Interface	Description
IController	Top level interface to start new controlles.
ILeader	This interface completely defines the leader input callback : add or remove a new worker, add or remove an application.
IWorker	Each worker has a fairly simple design.

Class Summary

Class	Description
Main	Shiva daemon main class.

Package org.thales.punch.apps.shiva Description

Shiva Punch Scheduler

Shiva is a daemon you start on a bunch of servers. It provides a distributed application scheduler with basically the role of starting or stopping punch or your own applications.

We designed Shiva with operational simplicity in mind. It is nothing more than a damonized variant of the punch admin (terminal) environment. I.e. when Shiva starts a (say) punchline, it does it exactly as yourself from the command line. The only difference is that Shiva has no user home directory and runs as a punch userland daemon.

Shiva Design

Shiva is implemented on top of a file-system like API. On single node setup, an actual file system is used, and Shiva watches for local files to be notified upon start and stop client commands.

On a distributed punch, Kafka is used instead. Shiva use three topic (control, data and application) to implement a very simple gossip protocol.

Leaders and Workers

If you start (say) 5 daemons, the default setup is that each is both a leader and a worker. Only one will actually be granted the leader role, this is implemented using a sigle partition topic. Each leader subscribes to that topic sinle partition, only one is granted by the Kafka broker.

That leader regularly publishes a hello message catched by the workers. Each worker responds with its id and assignement. If a worker does not respond it is removed from the set of known workers. As simple as that. On single node setup nothing of that is required, there is a single daemon both leader and worker.

Refer to org.thales.punch.apps.shiva.kafka for further informations about the kafka distributed mode.

Workers Design

Workers are in charge of launching applications : punchlines, plan, your own apps, elastalert etc.. How is each worker server equipped with the required binaries and configuration files ?

Binaries are meant to be pre-installed on each Shiva worker. On on-prem installations, the is the role of the punch deployer. The Shiva package is self-contained and includes all the punch supported applications binaries : logstash, elastalert, spark and storm punchline jars etc..

This is not good enough if you want to start punchlines, logstash or elastalert you need to provide configuration files. These are present in the user (operator) configuration directory on its admin server. What happens is actually quite simple, when you (the operator) you start an shiva application, the configuration directory is broadcasted to all workers. At the end, each shiva worker (server) is equuipeed with the same configuration directory than the one of the operator.

Technically speaking this is achieved by sending zip archives through a compacted Kafka topics. This scheme has many benefits, namely :

• Troubleshooting shiva is as easy as using the operator command line commands. Simply sshh to a shiva server and issue the same command you would execute on yor operator server.
• The punch resolver logic that helps decorate the various configuration files right before running the apps is exactly the same than on the operator server as well. No complex or subtle difference between the operator and the shiva way of working.
• All in all Shiva is damned simple to understand.
• It only requires Kafka and no other external service. Again damned simple. No more zookeeper.

Shiva Packaging

The Shiva zip package, as explained before, contains everything required to run all punch apps. In fact the shiva package is the punch uber single package that is required on both shiva and the punch operator servers.

True this package is big (because it embarks spark and other jars), but again simplicity is the key design driver.

Of course this does not mean it is not modulasr. You can only use logstash and nothing else.