Skip to content

6.3 to 6.4 Upgrade Notes

This document explains what configuration changes have to be performed during a PunchPlatform update from version 6.3.x to 6.4.X, and other changes you need to know about (e.g. command-line changes).

COTS

  • Apache Storm : upgraded from 2.2.0 to 2.3.0. No breaking change in Punch configuration.
  • Apache Kafka : upgraded from 2.4.0 to 2.8.0. No breaking change in Punch configuration.
  • Apache Zookeeper : upgraded from 3.5.7 to 3.7.0. No breaking change in Punch configuration.
  • Elasticsearch : upgraded from 7.8.0 to 7.10.2. Many breaking changes in Punch Nodes configuration.
  • Kibana : upgraded from 7.8.0 to 7.10.2. Many breaking changes in Punch Plugin configurations.
  • Opendistro Security for Kibana : upgraded from 1.9.0.0 to 1.13.0.1
  • Opendistro Security for Elasticsearch : upgraded from 1.9.0.0 to 1.13.1.0
  • Opendistro SQL for Elasticsearch : upgraded from 1.9.0.0 to 1.13.2.0

Nodes

Consequence of the Elasticsearch upgrade is the deprecation of some elasticsearch nodes, both in Spark and Storm :

In Storm :

  • elasticsearch_input must now be replaced with extraction_input.

In Spark :

  • elastic_batch_input must now be replaced with elastic_input.
  • elastic_batch_output must now be replaced with elastic_output.
  • elastic_stream_output is not available anymore.

Kibana Punch Plugin

Kibana Punch plugin is not available on Kibana 7.10.2.

Kibana plugin API has major breaking changes from 7.8.0 to 7.10.2 version. Rather than migrating the whole Punch plugin, we kept only the features used on our customer production platforms.

Punch Plugin Features :

Plugins are now developed independently of one another. As a result, they can be installed independently, but each has its own version. You can find the plugin version for your Punch version in Punch Plugins list.

Punch Documentation

The Punch Documentation plugin displays official documentation offline. The documentation is embedded within the plugin. It does not require to deploy the Gateway to serve the documentation.

Standalone

The plugin is installed by default.

Deployer

The plugin can be installed by adding the following section in kibana.plugins or in kibana.domains.<domain>.plugins :

{
  "kibana": {
    "plugins": {
      "punch_documentation": {
        "version": "1.0.1",
        "documentation_version": "6.4.1"
      }
    }
  }
}
Resources

Data Extraction

The Data Extraction plugin makes it possible to extract data from Elasticsearch. It requires the Punch Gateway to launch an extraction job in the background.

Standalone

The plugin is installed by default.

Deployer

The plugin can be installed by adding the following section in kibana.plugins or in kibana.domains.<domain>.plugins :

{
  "kibana": {
    "plugins": {
      "data_extraction": {
        "version": "1.2.1",
        "enabled": true,
        "use_legacy": false,
        "tenant": "reftenant",
        "rest_api": {
          "hosts": [
            "https://centralback1:4242",
            "https://centralback2:4242"
          ],
          "ssl_enabled": true
        }
      }
    }
  }
}

As for the Gateway, its configuration stays the same to launch extraction job in the background.

Resources

Data Feedback

The Data Feedback plugin makes it possible to annotate data from Elasticsearch. It does not require the Punch Gateway.

Compared to the Feedback plugin in <6.4 versions, there are no default option for input types parameters in kibana.yml or in punchplatorm-deployment.settings.

If you do not set the input types parameters, then all inputs will be available and configurable in Kibana UI.

On the other hand, if you specify input types parameters as in the example, the related inputs will not be configurable in Kibana UI.

Also, the prefix parameter in kibana.yml have changed from punchplatform-feedback.<parameter> to data_feedback.<parameter>. Some parameters also have new names.

Here is the new parameters with sample values : 

data_feedback.tenant: reftenant
data_feedback.saveInNewIndex: false
data_feedback.feedbackIndex: "feedbacks"
data_feedback.feedbackType: "text"
data_feedback.select.options: ["green","red","blue"]
data_feedback.range.min: 0
data_feedback.range.max: 10
data_feedback.checkbox.label: "False positive"
data_feedback.maximumNumberOfTags: 1

Standalone

The plugin is installed by default.

Deployer

The plugin can be installed by adding the following section in kibana.plugins or in kibana.domains.<domain>.plugins :

Compared to parameters in kibana.yml, parameters in punchplatform-deployment.settings have not changed names.

{
  "kibana": {
    "plugins": {
      "punchplatform_feedback": {
        "punchplatform_feedback_version": "2.1.0",
        "tenant": "reftenant",
        "select_options": ["blue", "green", "red"],
        "max_tags": 1,
        "range_min": 0,
        "range_max": 10,
        "checkbox_label": "False Positive",
        "save_in_new_index": true,
        "feedback_index": "feedbacks"
      }
    }
  }
}
Resources

Storectl

The storectl command was removed from Punchplatform starting 6.4.3 version. It had many bugs and was hard to use on production platforms.

It is easier to build Kibana dashboards based on indexes containing archives metadata, rather than querying stats through storectl application.

Archives Housekeeping

The Archives Housekeeping application was refactored without breaking changes. However, a new parameter was introduced to manage the metadata carefully.

Before this version, when an archive metadata was processed, it was deleted from Elasticsearch. This deletion happened even if some devices were not cleaned, or if the application failed to clean a device.

Starting 6.4.3 version, the delete_metadata parameter makes it possible to change this behavior. This parameter can have the following values:

  • always: metadata is always deleted after processing devices. In order to avoid any breaking change, this is the default behavior when the parameter is not set.
  • never: metadata is never deleted after processing devices.
  • all_devices: metadata is deleted only when all devices specified in metadata are cleaned by archives housekeeping. If some devices were not cleaned (because they were not specified in archives housekeeping configuration or because the application failed to clean the device) then archives housekeeping updates metadata with the devices where the archive is still present. This is the recommended behavior.

You should add this parameter to your archives-housekeeping.yaml file.