HOWTO check Shiva services

Why do that

You want to check if your shiva cluster is up and running. Shiva runs punch applications both data processing apps and administrative services. It is thus key to ensure is it properly up and running.

What to do

Check shiva service health

Ensure you shiva cluster is running. On a standalone platform use the following command:

alice@laptop:~$ punchplatform-shiva.sh --status

On a deployed punch, the shiva server runs as a systemctl service:

sudo systemctl status shiva-runner.service

Check shiva logs

Runner daemon logs location

The runner/leader daemon logs are usually in /var/log/punch*/shiva/shiva-runner-daemon.log.

Task logs

A Shiva task/application can logs events in two ways:

1. Simply log message to stdout/stderr.
2. Log to a specified file using its own logic, for example using its own log4j configuration file.

Case 1: Output to stdout/stderr

The shiva runner daemon captures such logs, and send them - locally on the runner daemon, into /var/log/punch*/shiva/shiva-runner-subprocess.log - optionnally into the configured 'reporter' associated to this shiva cluster. So this depends on your platform setup, but it is a good production practice to - send these events (through a kafka reporter and an events dispatcher punchline) into a central Elasticsearch indice (usually called platform-logs-).

Check out the [platform-log-*] index using your admin Kibana. You will see there your task logs.

Case 2: Writing to a file

You must analyze the task type and its specific arguments (technology? logger configuration?..), to know where its local log files are located on the runner node. Shiva is not in charge of those log files.. Shiva nodes log all their actions. It is likely you have a shiva kibana dashboard at hand. If not locate the [shiva-logs] index and use the discover dashboard to check what is executed where.

Check shiva submissions

You may also want to check what is now scheduled in shiva. You can do that with the following terminal command :

channelctl -t mytenant status

Check shiva kafka topics and assignements

Follow this guide to have more informations

How to run manually a submitted task to reproduce errors ?

When a task is submitted, the shiva runner loads the task command and arguments to a local folder on the assigned runner node. That folder is defined as one of the shiva startup argument. By default is use the local operating system temporary folder. There you will find a tree folder structure obeying the tenant/channel hierarchy. For example assuming you submitted the hello world task as explained in the Shiva guide :

.
└── punchplatform
  └── mytenant
      ├── channels
          └── hello_world_shiva
              └── my_hello_world_task
                  ├── CONTENT-1
                  └── hello_world.sh

To debug your command script, you can call it from this directory to see what happens.

To do so:

• Connect on the shiva node that is supposed to run your task, and log in as the same user that runs the shiva daemon (this is the 'platform.punchplatform_daemons_user' from deployment settings)
• run a 'bash' shell
• run source /data/opt/punch-shiva-<yourversion>/activate.sh in order to define punch environment variables (path, conf...)
• change directory to the temporary folder of your task : /tmp/shiva/punchplatform//channels///
• run your task command line (with same command and args as indicated in the channel_structure file for the channel)