Skip to content

DAVE 6.0.0

Below is a summary of the JIRA issues addressed in the DAVE-6.0.0 release of Punchplatform. For full documentation of the release, a guide to get started, and information about the project, see the Punchplatform project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Punchplatform.

The documentation for the most recent release can be found at https://doc.punchplatform.com.

Release summary

This release brings major fixes to version 5.0.0. It also brings deployment improvements and provides Kibana plugin, Archiving and Ceph.

Important changes

  • Elastic stack (Elasticsearch, Kibana, Beats..) upgraded to 7.4.2
  • Spark upgraded to 2.4.3
  • Storm upgraded to 2.1.0
  • Kafka upgraded to 2.11_2.4.0
  • Python 3.6.8

Please read Migration Guide from 5.x to 6.0.0

Release notes

Release notes - Punch - Version Dave-6.0.0

Overview

Elasticsearch

New SSL/TLS configurations for Opendistro Security plugin

Due to a change to a major version number of elasticsearch, multiple breaking changes from elasticsearch were introduced. I.E removal of multiple doc types from an index... which was done to avoid bad practices !

More information of other potential breaking changes: Read here

Kibana

Opendistro Security plugin's configuration section is not necessary anymore and has been removed.

New SSL/TLS configurations per domain

Opendistro

Opendistro Security plugin support for version 1.4.0.0 compatible with Elasticsearch and Kibana 7.4.2.

New SSL/TLS configurations for Elasticsearch security plugin

Rest API Gateway

New endpoints for tenants :

  • /tenant
  • /extraction
  • /registry
  • /punchline

The configuration has changed to configure

Pyspark

  • added the ability to specify a single pex per punchline execution, this can drastically reduce execution time and more importantly offers the possibility to use multiple version of the same python module across different punchlines !
  • added the ability to use pyspark udf functions, see here for an example on how to use it.

Epic

  • [PP-1546] - Configuration Management
  • [PP-2853] - Upgrade externals (kbn/es/beat, storm, spark...)
  • [PP-3381] - Application Design
  • [PP-3396] - Cybersecurity Modules
  • [PP-3451] - Mini Punch
  • [PP-3452] - Punchline Editor

New Feature

  • [PP-3832] - [Curator] Remove Alias from index when index are closing

Story

  • [PP-2411] - Python 3 compatibility for deployer
  • [PP-3085] - Deprecate suspervisord in DAVE
  • [PP-3260] - Functionnal Opendistro Security plugin alongside ES 7
  • [PP-3429] - Support for Java 11
  • [PP-3453] - Airflow
  • [PP-3459] - Leveraging UDFs
  • [PP-3467] - Model save and restore
  • [PP-3512] - window based lumberjack protocol
  • [PP-3513] - Dave
  • [PP-3530] - machine learning demo
  • [PP-3570] - Disable loading of spark when using only python and improve packaging
  • [PP-3645] - Airflow getting started documentation
  • [PP-3650] - Starter kit for airflow, pyspark on public github
  • [PP-3653] - Airflow + celery or airflow + dask and airflow HA
  • [PP-3659] - Spike: Try to use Airflow with ClickHouse
  • [PP-3702] - Parquet Archiving in SparkStructuredStreaming
  • [PP-3704] - Create JDBC Connectors
  • [PP-3707] - Refactor cluster_name, es_cluster etc... to cluster_id
  • [PP-3869] - Improve Nifi Processor
  • [PP-3892] - Publish analytics job artifact on maven central and starter for PML java

Task

  • [PP-3059] - Provide platform specific standalone archive for linux and darwin
  • [PP-3232] - Update the Zookeeper config template
  • [PP-3430] - Using Jackson instead of Boon for JSON parsing
  • [PP-3603] - starter kit for custom PML node
  • [PP-3610] - ES benchmark tools
  • [PP-3640] - Dave rollback to java 8
  • [PP-3646] - Pyspark deployer for craig/dave release
  • [PP-3778] - Dave : Upgrade to latest Kafka release

Improvement

  • [PP-3872] - Kibana Punch UI user experience improvements for extraction
  • [PP-3901] - Adapt Plan code with resolve.conf

Bug

  • [PP-2537] - PunchBolt wrong setting name
  • [PP-3630] - Spark doesn't support Java11
  • [PP-3742] - Metrics reporters
  • [PP-3773] - Topology errors when using Storm 2.1.0
  • [PP-3805] - Elastic7.x breaking change for EsinputNode