Skip to content

6.0 to 6.1 Upgrade Notes

This document explains what configuration changes have to be performed during a PunchPlatform update from version 6.0.x to 6.1.0, and other changes you need to know about (e.g. command-line changes).

Command lines

The punchlinectl and planctl command all support an additional '-c' or '--channel' option. You can use this option to force the channel name of a submitted punchline or plan when executed outside the scope of a tenant.

Configurations

Punchlines

The name, tenant and channel properties of punchlines are deprecated. They will be automatically set for you using the correct tenant channel and punchline file name when starting an application or channel using 'channelctl'

Punchlets and Punchline Meta

The world tuple available to punchlets is improved. It automatically contains the following information :

{
    "tenant" : "mytenant",
    "channel" : "mychannel",
    "name" : "input.json",
    "meta" : {
        .. any additional explicit value you declare in your punchline meta dictionary ..
    }
}

Make sure you leverage this new improved usage as follows in your punchlets:

    // these are always set
    [logs][log][tenant] = world:[tenant];
    [logs][log][channel] = world:[channel];
    // this one is set only if explicitly defined in the punchline meta section
    if (world:[meta][vendor]) {
     [logs][log][vendor] = world:[meta][vendor];
    } else {
      [logs][log][vendor] = "unknown";
    }

Note

Punchlets backward compatibility has been preserved by keeping the tenant and channel properties part of the meta dictionary as well.

Plan configuration

We deprecated plan_settings for settings in plan.hjson.

For example, you have to write your plan.json as follow :

{
  name: test
  model:{
  ...
  }
  settings: {
    cron: "*/1 * * * *"
    persistence: [
      {
        type: elasticsearch
        index_name: platform-plan-cursor
      }
    ]
  }
  metrics: {
    reporters: [
      {
        type: kafka
      }
    ]
  }
}

Channel Structure

When starting punchlines from shiva, the "-t" argument is not allowed anymore. The Shiva format to start a punchline from shiva is as follows :

{
    "version" : "6.0",
    "start_by_tenant" : true,
    "stop_by_tenant" : true,
    "applications": [
        {
            "name": "input",
            "runtime" : "shiva",
            "command" : "punchlinectl",
            # the tenant and channel name are automatically set for you. 
            "args": [ "start", "--punchline", "input.json" ],
            "shiva_runner_tags": [ "local" ],
            "cluster": "local",
            "reload_action": "kill_then_start"
        }
    ]
}

If you set use the '-t' argument you will be thrown an illegal argument exception error.

Deployment

Merge punchplatform.properties & punchplatform-deployment.settings

We merged punchplatform.properties & punchplatform-deployment.settings to one single file punchplatform-deployment.settings

For example to deploy kafka, you had to write in your punchplatform.properties :

{
  ...
  "kafka_version" : "kafka_2.11-2.4.0",
  "kafka_brokers_production_interface" : "eth0"
  ...
}

And in your punchplatform-deployment.settings :

"kafka" : {
    "clusters" : {
        "local" : {
            "brokers_with_ids" : [
              {"id" : 1, "broker" : "node01:9092" },
              {"id" : 2, "broker" : "node02:9092" },
              {"id" : 3, "broker" : "node03:9092" }
            ],
            "zk_cluster" : "common",
            "zk_root" : "kafka-local",
            "brokers_config" : "punchplatform-local-server.properties",
            "default_replication_factor" : 1,
            "default_partitions" : 2,
            "partition_retention_bytes" : 1073741824,
            "partition_retention_hours" : 24,
            "kafka_brokers_jvm_xmx": "512M"
        }
    },
    "install_dir" : "/data/opt/kafka_2.11-2.4.0"
}

Now, you have to write in the unique punchplatform-deployment.settings deployment file something like :

"kafka" : {
    "kafka_version" : "kafka_2.11-2.4.0",
    "kafka_brokers_production_interface" : "eth0",
    "clusters" : {
        "local" : {
            "brokers_with_ids" : [
              {"id" : 1, "broker" : "node01:9092" },
              {"id" : 2, "broker" : "node02:9092" },
              {"id" : 3, "broker" : "node03:9092" }
            ],
            "zk_cluster" : "common",
            "zk_root" : "kafka-local",
            "brokers_config" : "punchplatform-local-server.properties",
            "default_replication_factor" : 1,
            "default_partitions" : 2,
            "partition_retention_bytes" : 1073741824,
            "partition_retention_hours" : 24,
            "kafka_brokers_jvm_xmx": "512M"
        }
    }
}

As you can see, all keys from old punchplatform.properties keep the same name but have been displaced in punchplatform-deployment.settings into the right component section.

To migrate, you can apply this procedure for almost all components except a few special cases (check next sections)

Install_dir parameter is now deprecated

In old Punch versions, install_dir was a mandatory parameter in each component section in punchplatform.properties.

It is now deprecated, you have to remove it from all your previous configurations

Revamp platform section in punchplatform-deployment.settings

The platform section is have been revamp in punchplatform-deployment.settings file. It regroups some of old parameters presents in punchplatform.properties file and some new ones like binaries_version and reporters. Check Platform section in the documentation for the complete list

Revamp shiva section in punchplatform-deployment.settings

The shiva section is have been revamp in punchplatform-deployment.settings file. Indeed, we have completely revised the implementation of Shiva which has led to a modification of the parameters exposed. Check Shiva section in the documentation for the complete list

Among others, we have deleted the plugins section from Shiva configuration and added a storage configuration.

New PunchPlatform Operator section in punchplatform-deployment.settings

A new punchplatform_operator section is now mandatory in punchplatform-deployment.settings file. It regroups some of old parameters presents in punchplatform.properties file and some new ones like storage section. Check Platform section in the documentation for the complete list

New reporter section in punchplatform-deployment.settings

A new reporter section is now mandatory in punchplatform-deployment.settings file. As you have to specify a reporter in many component section (operator, gateway, shiva ..), we decided to regroup them in one single section. Then, you have to specify your reporter per component using a unique reporter id. By this way, we reduced verbosity in deployment configuration. Check Reporter section in the documentation to see how to write this new section.

For example, if you declared a reporter like :

"reporters": {
      "central_reporter" : {
        "type": "kafka",
        "bootstrap.servers": "node02:9092",
        "topic": "reporter-topic",
        "metric_document_field_name": "log",
        "reporting_interval": 30,
        "encoding": "lumberjack"
      }
}

And you want to use it for shiva, you have to declared it using your reporter id :

"shiva": {
    "shiva_version": "punchplatform-shiva-6.1.0",
    "clusters": {
        "common": {
            "reporters": ["central_reporter"],
            "storage":{
                "type": "kafka",
                "kafka_cluster": "common"
            },
            "servers": {
                "localhost": {
                    "runner": true,
                    "can_be_master": true,
                    "tags": []
                }
            }
        }
    }
}

Watchout, three components now require a reporter declaration :

Note

All old configurations which declared a reporter directly in component section (for example Gateway) are not supported anymore

Pyspark section has been removed from deployment configuration

You can remove pyspark parameters from all your previous deployment configurations. It is now deploy and configure thanks to spark section

Gateway, Elasticsearch & Kibana sections have been update

In addition of simply merging punchplatform.properties & punchplatform-deployment.settings, we also made some configuration adjustments in order to reduce verbosity. Check each specific section in documentation to be sure that your old configuration is still compatible

New mandatory file for deployment : resolv.hjson

A tool called 'resolver' has been added, to change/complement channels file at start time Resolver to reduce configuration verbosity and help make it more platform-independent.

This file is now mandatory for deployment. You have to write it from scratch following the documentation before deploying