6.0 to 6.1 Upgrade Notes¶
This document explains what configuration changes have to be performed during a PunchPlatform update from version 6.0.x to 6.1.0, and other changes you need to know about (e.g. command-line changes).
Command lines¶
The punchlinectl and planctl command all support an additional '-c' or '--channel' option. You can use this option to force the channel name of a submitted punchline or plan when executed outside the scope of a tenant.
Configurations¶
Punchlines¶
The name, tenant and channel properties of punchlines are deprecated. They will be automatically
set for you using the correct tenant channel and punchline file name when starting an application or channel using 'channelctl'
Punchlets and Punchline Meta¶
The world tuple available to punchlets is improved. It automatically contains the following information :
{
"tenant" : "mytenant",
"channel" : "mychannel",
"name" : "input.json",
"meta" : {
.. any additional explicit value you declare in your punchline meta dictionary ..
}
}
Make sure you leverage this new improved usage as follows in your punchlets:
// these are always set
[logs][log][tenant] = world:[tenant];
[logs][log][channel] = world:[channel];
// this one is set only if explictly defined in the punchline meta section
if (world:[meta][vendor]) {
[logs][log][vendor] = world:[meta][vendor];
} else {
[logs][log][vendor] = "unknown";
}
Note
Punchlets backward compatibility has been preserved by keeping the tenant and channel properties part of the meta dictionar as well.
Plan configuration¶
We deprecated plan_settings for settings in plan.hjson.
For example, you have to write your plan.json as follow :
{
name: test
model:{
...
}
settings: {
cron: "*/1 * * * *"
persistence: [
{
type: elasticsearch
index_name: platform-plan-cursor
}
]
}
metrics: {
reporters: [
{
type: kafka
}
]
}
}
Channel Structure¶
When starting punchlines from shiva, the "-t" argument is not allowed anymore. The Shiva format to start a punchline from shiva is as follows :
{
"version" : "6.0",
"start_by_tenant" : true,
"stop_by_tenant" : true,
"applications": [
{
"name": "input",
"runtime" : "shiva",
"command" : "punchlinectl",
# the tenant and channel name are automatically set for you.
"args": [ "start", "--punchline", "input.json" ],
"shiva_runner_tags": [ "local" ],
"cluster": "local",
"reload_action": "kill_then_start"
}
]
}
If you set use the '-t' argument you will be thrown an illegal argument exception error.
Deployment¶
Merge punchplatform.properties & punchplatform-deployment.settings¶
We merged punchplatform.properties & punchplatform-deployment.settings to one single file punchplatform-deployment.settings
For example to deploy kafka, you had to write in your punchplatform.properties :
{
...
"kafka_version" : "kafka_2.11-2.4.0",
"kafka_brokers_production_interface" : "eth0"
...
}
And in your punchplatform-deployment.settings :
"kafka" : {
"clusters" : {
"local" : {
"brokers_with_ids" : [
{"id" : 1, "broker" : "node01:9092" },
{"id" : 2, "broker" : "node02:9092" },
{"id" : 3, "broker" : "node03:9092" }
],
"zk_cluster" : "common",
"zk_root" : "kafka-local",
"brokers_config" : "punchplatform-local-server.properties",
"default_replication_factor" : 1,
"default_partitions" : 2,
"partition_retention_bytes" : 1073741824,
"partition_retention_hours" : 24,
"kafka_brokers_jvm_xmx": "512M"
}
},
"install_dir" : "/data/opt/kafka_2.11-2.4.0"
}
Now, you have to write in the unique punchplatform-deployment.settings deployment file something like :
"kafka" : {
"kafka_version" : "kafka_2.11-2.4.0",
"kafka_brokers_production_interface" : "eth0",
"clusters" : {
"local" : {
"brokers_with_ids" : [
{"id" : 1, "broker" : "node01:9092" },
{"id" : 2, "broker" : "node02:9092" },
{"id" : 3, "broker" : "node03:9092" }
],
"zk_cluster" : "common",
"zk_root" : "kafka-local",
"brokers_config" : "punchplatform-local-server.properties",
"default_replication_factor" : 1,
"default_partitions" : 2,
"partition_retention_bytes" : 1073741824,
"partition_retention_hours" : 24,
"kafka_brokers_jvm_xmx": "512M"
}
}
}
As you can see, all keys from old punchplatform.properties keep the same name but have been displaced in punchplatform-deployment.settings into the right component section.
To migrate, you can apply this procedure for almost all components except a few special cases (check next sections)
Install_dir parameter is now deprecated¶
In old Punch versions, install_dir was a mandatory parameter in each component section in punchplatform.properties.
It is now deprecated, you have to remove it from all your previous confiugrations
Revamp platform section in punchplatform-deployment.settings¶
The platform section is have been revamp in punchplatform-deployment.settings file. It regroups some of old parameters presents in punchplatform.properties file and some new ones like binaries_version and reporters. Check Platform section in the documentation for the complete list
Revamp shiva section in punchplatform-deployment.settings¶
The shiva section is have been revamp in punchplatform-deployment.settings file. Indeed, we have completely revised the implementation of Shiva which has led to a modification of the parameters exposed. Check Shiva section in the documentation for the complete list
Among others, we have deleted the plugins section from Shiva configuration and added a storage configuration.
New PunchPlatform Operator section in punchplatform-deployment.settings¶
A new punchplatform_operator section is now mandatory in punchplatform-deployment.settings file. It regroups some of old parameters presents in punchplatform.properties file and some new ones like storage section. Check Platform section in the documentation for the complete list
New reporter section in punchplatform-deployment.settings¶
A new reporter section is now mandatory in punchplatform-deployment.settings file.
As you have to specify a reporter in many component section (operator, gateway, shiva ..), we decided to regroup them in one single section. Then, you have to specify your reporter per component using a unique reporter id. By this way, we reduced verbosity in deployment configuration. Check Reporter section in the documentation to see how to write this new section.
For example, if you declared a reporter like :
"reporters": {
"central_reporter" : {
"type": "kafka",
"bootstrap.servers": "node02:9092",
"topic": "reporter-topic",
"metric_document_field_name": "log",
"reporting_interval": 30,
"encoding": "lumberjack"
}
}
And you want to use it for shiva, you have to declared it using your reporter id :
"shiva": {
"shiva_version": "punchplatform-shiva-6.1.0",
"clusters": {
"common": {
"reporters": ["central_reporter"],
"storage":{
"type": "kafka",
"kafka_cluster": "common"
},
"servers": {
"localhost": {
"runner": true,
"can_be_master": true,
"tags": []
}
}
}
}
}
Watchout, three components now require a reporter declaration :
Note
All old configurations which declared a reporter directly in component section (for example Gateway) are not supported anymore
Pyspark section has been removed from deployment configuration¶
You can remove pyspark parameters from all your previous deployment configurations. It is now deploy and configure thanks to spark section
Gateway, Elasticsearch & Kibana sections have been update¶
In addition of simply merging punchplatform.properties & punchplatform-deployment.settings, we also made some configuration adjustments in order to reduce verbosity. Check each specific section in documentation to be sure that your old configuration is still compatible
New mandatory file for deployment : resolv.hjson¶
A tool called 'resolver' has been added, to change/complement channels file at start time Resolver to reduce configuration verbosity and help make it more platform-independent.
This file is now mandatory for deployment. You have to write it from scratch following the documentation before deploying