Skip to content

Configuration

Abstract

This chapter provides configuration management information that only apply to punch running on the native punch orchestrator.

Make sure you read the general configuration reference guide first.

Shiva Application Management

Standalone

The standalone punch is a single-node easy setup that you can install and run in minutes. It is also a good example Here is how the standalone application control and monitoring plane is implemented on a standalone to communicate commands and operator configuration to the runtime environment.

image

With this simple setup the local filesystem is used. The administration store is also used to request application start or stop commands. A punch daemon called shiva is in charge of executing the applications.

Production Punch

A production punch requires a high-available administration store. Instead of using the local filesystem, kafka is used to provide the required storage engine for commands and operator configuration snapshots. This is illustrated next :

image

  1. the configuration folder is on the operator local filesystem. It is located in the installation directory conf folder.
  2. operator is using hisr (linux|macos) terminal to issue administrative command (e.g. application or channels submission)
  3. in the Kafka cluster, three topics are used :
    • (4) standalone-shiva-ctl for scheduling applications to shiva
    • (5) standalone-shiva-data for scheduling applications to shiva
    • (6) standalone-admin to retain the current application(s) status
  4. a shiva cluster : this is the punch proprietary application scheduler.
  5. a spark cluster to run spark and pyspark applications.
  6. a storm cluster to run streaming punch processing applications.

The configuration is stored inside a configuration snapshot, sent through Kafka to the shiva cluster. The runner node will extract this snapshot to provide a consistent configuration view to the punchline when it runs on the shiva runner node.

This configuration is therefore immutable during the lifetime of the application submission (i.e. until an operator stops and restarts the application again). An exception: if the punchline refers to resource stored in an url-accessible resource back-end, then dynamic reload of new resource versions is possible without the application restart by an operator.

Platform Properties File

The applications and punch components often need to know the layout/functions of the platform servers.

The punchplatform.properties file is generated during deployment time, based on the punchplatform-deployment.settings configuration file provided to the deployer tool. The punchplatform.properties is deployed on the various platform nodes that need this information. This file holds essential informations required by punch components to interact with each other ; it can also be viewed by the operator to better understand the platform structure, and easily locate services when investigating incidents.

In an operator environment, the file is pointed by the PUNCHPLATFORM_PROPERTIES_FILE environment variable.

Here is an example from the standalone punchplatform :

"platform" : {
    "platform_id" : "my-unique-platform-id",
        "admin" : {
            "type": "kafka",
            "cluster" : "local",
            "topics" : {
                "admin_topic" : {
                    "name" : "standalone-admin"
                }
            }
        }
    },
    "kafka" : {
        "clusters" : {
            "local" : {
                "brokers_with_ids" : [ { "id" : 0, "broker" : "localhost:9092" } ],
            }
        }
    },
    "shiva": {
        "clusters" : {
            "platform" : {
                "type" : "kafka",
                "cluster" : "local", 
                "topics" : {
                    "control_topic" : {
                        "name" : "standalone-shiva-ctl"
                    },
                    "data_topic" : {
                        "name" : "standalone-shiva-data"
                    }
                }
            }
        }
    }
}

Tip

The punchplatform is highly modular and lightweight. Here the example platform has only the internal punch application scheduler called shiva that allows the execution of many simple and useful applications such as logstash, punch lightweigth pipelines or you own python application. Of course you can configure it with more components such as a Spark, Storm engine, plus Elasticsearch. It all depends on your use case but the principles are the same.

This file is distributed on platform servers holding Shiva, Gateway and Operator components and located in each associated package. Check component's documentation to have more details

Platform Resolver

The resolv.yml file contains important informations used by the operator, the gateway and shiva components at runtime. It supersedes information from your application and channels, to adapt them for the actual platform if needed.

This file is prepared by the integration, and deployed alongside the punchplatform.properties file on all requiring nodes of the platform. In an operator environment, a copy of this file is pointed by $PUNCHPLATFORM_RESOLV_FILE environment variable.

Check this documentation to have a full description of this file and how it works.

Here is an example from the standalone punchplatform :

{
   // All ES input/output nodes (Storm nodes)
   elasticsearch_nodes:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.dag[?(@.type=='elasticsearch_input' || @.type=='elasticsearch_output')].settings
      additional_values:{ 
         http_hosts:[ 
            { 
               host:localhost
               port:9200
            }
         ]
      }
   }

   // All ES spark nodes 
   elastic_nodes:{ 
      selection:{ 
         tenant:*
         channel:*
         runtime:*
      }
      match:$.dag[?(@.type=='elastic_batch_input' || @.type=='elastic_batch_output' || @.type=='elastic_stream_output' || @.type=='elastic_input' || @.type=='elastic_query_stats' || @.type=='python_elastic_input' || @.type=='python_elastic_output')].settings
      additional_values:{ 
         nodes:[ 
            localhost
         ]
      }
   }
}

This file is required for Shiva, Gateway and Operator components and located in each associated package. Check component's documentation to have more details.