Skip to content

DAVE-6.4.1 release notes

This document is a summary of content, changes, limitations and fixes of this release as compared to DAVE-6.4.0 release.

Refer to the the Punchplatform official site if you look for the complete documentations and resources.

The documentation for this release can be found inside the deployment archives (standalone and deployer versions), and at https://doc.punchplatform.com/6.4.1/index.html.

The documentation for the most recent release can be found at https://doc.punchplatform.com.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your clusters. The upgrade notes (e.g. upgrade from 6.3 to 6.4) discuss every critical information about incompatibilities and breaking changes, performance changes, and any other configuration changes that might impact your production deployment of Punchplatform.

Changes

This release is the 6.4.X LTS release. Compared to a 6.3.x, the 6.4.0 releases mainly leverages Kafka 2.8.1, Apache Storm 2.3.0, and Zookeeper 3.7.0 versions in order to benefit from their latest java 1.8 releases.

In this 6.4.1 version, Elasticsearch suite is upgraded to 7.10.2 to benefit from the latest Opensource version. As a result, Opendistro plugins also have been upgraded to 1.13.X.X. Check Migration Guide for more details.

Opensearch will be included as an alternative to Elasticsearch suite in the next 6.4 version.

Breaking changes

Nodes

With Elasticsearch update, we had to deprecate some Elasticsearch nodes, both in Spark and Storm :

In Storm :

  • elasticsearch_input must be replaced with extraction_input.

In Spark :

  • elastic_batch_input must be replaced with elastic_input.
  • elastic_batch_output must be replaced with elastic_output.
  • elastic_stream_output is not available anymore.

A few minor improvements have also been introduced, refer to the list below.

Kibana Punch Plugin

Kibana Punch plugin is not available on Kibana 7.10.2.

Kibana plugin API has major breaking changes from 7.8.0 to 7.10.2 version. Rather than migrating the whole Punch plugin, we kept only the features used on production platforms.

Punch Plugin Features :

  • Punch Documentation : AVAILABLE in Punch Documentation plugin.
  • Elastic Extraction : AVAILABLE in Data Extraction plugin.
  • Feedback Visualization : AVAILABLE in Data Feedback visualization plugin.
  • Punchline Editor : REMOVED
  • Resource Manager UI : REMOVED
  • Configuration Management : REMOVED

PLugins are now developped independently from one another. As a result, they can be installed independently, but each has its own version. You can find a version compatibility map between Punch and plugins in Punch Plugins list.

More information about the Kibana Punch Plugin in Migration Guide.

Improvements

  • #0001 increase elasticsearch jvm heap size standalone
  • #1758 improve exception handling in file_output
  • #1771 migrate punch plugin app for kibana 7.10.2
  • #0001 improve the extraction punchline template
  • #1752 upgrade elasticsearch and remove elastic-hadoop connector
  • #1773 add Elastic Output in Spark
  • #1774 migrate elasticsearch nodes

Bug Fixes

  • #1711 clickhouse output node multiplies records when connection to server is temporarily unavailable
  • #1756 unarchive ansible task does not set the correct user and group
  • #1761 fix failure on kibana deployment when server bind address is 0.0.0.0
  • #1780 correct deployment errors when having several resolv file in the PUNCHPLATFORM_CONF_DIR
  • #1784 fix shiva logs enriched plan with credentials in clear
  • #1785 zookeeper service is running with root user instead of daemon user
  • #1787 elastic beats roles are changing root user principal group
  • #1788 clickhouse data wrongly located inside install directory
  • #0001 use json instead of yaml for elasticsearch-housekeeping configuration

Security Fixes

  • #1786 remove log4j vulerabilities - CVE-2021-45046 - CVE-2021-44228 - CVE-2021-45105 - CVE-2021-44832

Internal tasks

  • #0000 add dependencies provided by punch api when developing custom nodes
  • #0001 update robot extraction files
  • #0000 enhance logstash documentation
  • #0000 update extraction input examples
  • #0000 correct extraction_input samples in documentation
  • #0000 add documentation on shiva_runner_tags