Below is a summary of the JIRA issues addressed in the DAVE-6.0.0 release of Punchplatform. For full documentation of the release, a guide to get started, and information about the project, see the Punchplatform project site.
Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Punchplatform.
The documentation for the most recent release can be found at https://doc.punchplatform.com.
This release brings major fixes to version 5.0.0. It also brings deployment improvements and provides Kibana plugin, Archiving and Ceph.
- Elastic stack (Elasticsearch, Kibana, Beats..) upgraded to 7.4.2
- Spark upgraded to 2.4.3
- Storm upgraded to 2.1.0
- Kafka upgraded to 2.11_2.4.0
- Python 3.6.8
Please read Migration Guide from 5.x to 6.0.0
Release notes - Punch - Version Dave-6.0.0
New SSL/TLS configurations for Opendistro Security plugin
Due to a change to a major version number of elasticsearch, multiple breaking changes from elasticsearch were introduced. I.E removal of multiple doc types from an index... which was done to avoid bad practices !
More information of other potential breaking changes: Read here
Opendistro Security plugin's configuration section is not necessary anymore and has been removed.
New SSL/TLS configurations per domain
Opendistro Security plugin support for version
188.8.131.52 compatible with Elasticsearch and Kibana 7.4.2.
New SSL/TLS configurations for Elasticsearch security plugin
Rest API Gateway¶
New endpoints for tenants :
The configuration has changed to configure
- added the ability to specify a single pex per punchline execution, this can drastically reduce execution time and more importantly offers the possibility to use multiple version of the same python module across different punchlines !
- added the ability to use pyspark udf functions, see here for an example on how to use it.
- [PP-1546] - Configuration Management
- [PP-2853] - Upgrade externals (kbn/es/beat, storm, spark...)
- [PP-3381] - Application Design
- [PP-3396] - Cybersecurity Modules
- [PP-3451] - Mini Punch
- [PP-3452] - Punchline Editor
- [PP-3832] - [Curator] Remove Alias from index when index are closing
- [PP-2411] - Python 3 compatibility for deployer
- [PP-3085] - Deprecate suspervisord in DAVE
- [PP-3260] - Functionnal Opendistro Security plugin alongside ES 7
- [PP-3429] - Support for Java 11
- [PP-3453] - Airflow
- [PP-3459] - Leveraging UDFs
- [PP-3467] - Model save and restore
- [PP-3512] - window based lumberjack protocol
- [PP-3513] - Dave
- [PP-3530] - machine learning demo
- [PP-3570] - Disable loading of spark when using only python and improve packaging
- [PP-3645] - Airflow getting started documentation
- [PP-3650] - Starter kit for airflow, pyspark on public github
- [PP-3653] - Airflow + celery or airflow + dask and airflow HA
- [PP-3659] - Spike: Try to use Airflow with ClickHouse
- [PP-3702] - Parquet Archiving in SparkStructuredStreaming
- [PP-3704] - Create JDBC Connectors
- [PP-3707] - Refactor cluster_name, es_cluster etc... to cluster_id
- [PP-3869] - Improve Nifi Processor
- [PP-3892] - Publish analytics job artifact on maven central and starter for PML java
- [PP-3059] - Provide platform specific standalone archive for linux and darwin
- [PP-3232] - Update the Zookeeper config template
- [PP-3430] - Using Jackson instead of Boon for JSON parsing
- [PP-3603] - starter kit for custom PML node
- [PP-3610] - ES benchmark tools
- [PP-3640] - Dave rollback to java 8
- [PP-3646] - Pyspark deployer for craig/dave release
- [PP-3778] - Dave : Upgrade to latest Kafka release
- [PP-3872] - Kibana Punch UI user experience improvements for extraction
- [PP-3901] - Adapt Plan code with resolve.conf
- [PP-2537] - PunchBolt wrong setting name
- [PP-3630] - Spark doesn't support Java11
- [PP-3742] - Metrics reporters
- [PP-3773] - Topology errors when using Storm 2.1.0
- [PP-3805] - Elastic7.x breaking change for EsinputNode