Skip to content

DAVE 6.0.1

Below is a summary of the JIRA issues addressed in the DAVE-6.0.1 release of Punchplatform. For full documentation of the release, a guide to get started, and information about the project, see the Punchplatform project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Punchplatform.

The documentation for the most recent release can be found at https://doc.punchplatform.com.

Release summary

New Kibana visualization named "feedback visualization". Improve resource manager and graphical catalog editor, Improve data cluster filtering by punchlet with context metrics Can deploy MinIO (requires at least 4 nodes) Multiples major fixes (Syslog, UDP, Shiva)

Important changes

  • JDK 8 minor version >= 241

Release notes

Release notes - Punch - Version DAVE-6.0.1

Security

Warning

New security features for DAVE-6.0.1 needs at least JDK 8 with a version superior or equal to 241

The standalone can be installed with a security option :

./install.sh --with-security

Check security documentation at Getting Started - Security.

Epic

  • [PP-1863] - Kibana Punch plugin as Tenant Editor
  • [PP-4076] - ENF - IHM Feedback

New Feature

  • [PP-3832] - Curator Remove Alias from index when index are closing
  • [PP-3959] - Indexer interface for metadata
  • [PP-4048] - Reissue of an extraction (save as)
  • [PP-4052] - KafkaHeaders and other useful metadata publications by KafkaSpout
  • [PP-4102] - new cidr range operator
  • [PP-4137] - kibana visualization to enrich data on the fly
  • [PP-4199] - efficient multi threading punchlet dag
  • [PP-4212] - efficient multi threading pipeline

Improvement

  • [PP-3085] - Deprecate suspervisord in DAVE
  • [PP-3530] - machine learning demo
  • [PP-3587] - Improved ES_housekeeper template to allow "index_settings" action
  • [PP-3701] - Punch Kibana plugin uses Punch Gateway
  • [PP-3702] - Parquet Archiving in SparkStructuredStreaming
  • [PP-3834] - Add action to curator template ( remove or add an alias in index, reassign with tag)
  • [PP-3869] - Improve Nifi Processor
  • [PP-3872] - Kibana Punch UI user experience improvements for extraction
  • [PP-3901] - Adapt Plan code with resolve.conf
  • [PP-3910] - Updating standalone shells
  • [PP-3912] - Modular environment for Rest API/gateway features
  • [PP-3914] - Prefix all Systemd service description by Punchplatform
  • [PP-3941] - SSL deployer improvements
  • [PP-3953] - Webdav registry route for gateway
  • [PP-3957] - pp-documentation packaging improvements & analytics doc reference guide revamp
  • [PP-3974] - Create storm light main to replace old TopologyMain
  • [PP-3977] - Re-package nifi processor into one nar
  • [PP-4040] - ElasticsearchClientWrapper Sliced Scroll
  • [PP-4047] - Merge the "jobs" and "executions" screens into one
  • [PP-4069] - Extraction with topolight (new elastic node & existing fileoutput)
  • [PP-4084] - kafka log injector with custom headers
  • [PP-4085] - Inject a log file with the log injector
  • [PP-4086] - Gateway/kibana to use new extraction
  • [PP-4093] - Gateway endpoint for executing spark
  • [PP-4101] - improve punch tuple to json performance
  • [PP-4104] - Punchline executions from Gateway should be asynchronous
  • [PP-4107] - Improve Gateway endpoints documentation
  • [PP-4110] - Remove deadcode and refactor gateway
  • [PP-4178] - Resources Manager additional API endpoints
  • [PP-4160] - Improve plugin: add notifications + bug fixes
  • [PP-4180] - Spark | PML Submit | Can't read file specified in spark.files config
  • [PP-4183] - Resource manager backport to 6.0
  • [PP-4197] - Add support for Resource manager on docker images
  • [PP-4202] - ByteArray clients - Allow compression
  • [PP-4219] - Report forwarding punchlet execution
  • [PP-4224] - Embedded resource manager in gateway

Task

  • [PP-3430] - Using Jackson instead of Boon for JSON parsing
  • [PP-3603] - starter kit for custom PML node
  • [PP-3641] - 1 to N for python es input
  • [PP-3770] - OpenDistro doesn't work on Dave
  • [PP-3936] - Code cleaning for analytics
  • [PP-3938] - Forward credentials to ES through gateway
  • [PP-3939] - PP-documentation bad hierarchy
  • [PP-3962] - Add csv settings for remote resources
  • [PP-3966] - Kibana plugin migration to 7.4
  • [PP-3990] - improve the shiva development experience
  • [PP-4054] - Punchplatform installation
  • [PP-4055] - Convert craig pipelines in dave
  • [PP-4062] - Configuration template to scale
  • [PP-4067] - Re-integrate Elastalert
  • [PP-4068] - Fix git traceability for punchlang-lib between craig and dave
  • [PP-4099] - Improve code quality
  • [PP-4103] - Punh plugin security deployment is failing for different conf per domain
  • [PP-4105] - Resolver file for deployer
  • [PP-4163] - fixe the archiving regression in 6.0
  • [PP-4179] - Resources Manager API Documentation

Bug

  • [PP-2537] - PunchBolt wrong setting name
  • [PP-2792] - Platform health should include Kafka ISR checks / Leader checks
  • [PP-3264] - Kafka daemon not starting due to lack of broker id
  • [PP-3586] - monitoring/metrics hardcoded ES platform index
  • [PP-3609] - ElasticsearchBolt nullpointer
  • [PP-3742] - Metrics reporters
  • [PP-3890] - offset committed stops in case of many failures in mult-batch context
  • [PP-3919] - Kibana plugin crash when using Punchline Editor
  • [PP-3926] - Documentation getting started requirements
  • [PP-3927] - Shiva task not captured if fast failure
  • [PP-3930] - Rest API's punchline executions reports cannot be requested
  • [PP-3931] - Standalone installation does not work properly when uninstalling es and kibana plugins
  • [PP-3937] - Opendistro Alerting cannot be deployed with kibana 7 anymore
  • [PP-3940] - outdated Requirements in Before you start (Getting started)
  • [PP-3948] - channel-monitoring not working with custom value of es_platform_log_index
  • [PP-3951] - concurrent exception in syslog servers
  • [PP-3971] - bad zk connection string in kafka-topics.sh
  • [PP-3973] - Incorrect punchctl --configure result on standalone sourcefire channel
  • [PP-3982] - Non-interactive mode is not possible anymore in install.sh
  • [PP-3989] - Extraction doesn't work when no filter is specified in Kibana plugin
  • [PP-4005] - Missing options for elastic_input-based extraction in extraction-plugin-generated PML
  • [PP-4008] - exists filter not supported by extraction plugin
  • [PP-4009] - Csv operator automatically escape " with \, leading to unwanted behaviour in some situation
  • [PP-4014] - Gateway ansible implementation issue
  • [PP-4018] - Third Party not working in 6.0
  • [PP-4019] - Issue with new 6.0 format
  • [PP-4036] - Dave Spark-setup deployment
  • [PP-4045] - The filter of executions and jobs in the plugin does not work
  • [PP-4072] - Worker memory isn't taken into account with punchlinectl command
  • [PP-4073] - Shiva issue on permission
  • [PP-4077] - standalone scripts output wrong colored output
  • [PP-4087] - File_output streaming mode doesn't write as csv
  • [PP-4090] - Storm nimbus needs python
  • [PP-4091] - Can't overide storm settings with topology file
  • [PP-4095] - Log-injector punchlets test is more efficient than a punch_node
  • [PP-4106] - Punchplatform-pyspark-scan.sh is not used by gateway endpoint
  • [PP-4108] - Pyspark added missing dependencies: pyarrow
  • [PP-4112] - Rejected indexation in elasticsearch
  • [PP-4136] - CsvOperator - Escape Character should have empty option
  • [PP-4139] - Punchline Pyspark loading all custom dependencies at runtime
  • [PP-4141] - Pyspark better exception handling & remove dead code & cleaning & udf
  • [PP-4146] - Airflow PunchOperator bad start command
  • [PP-4152] - Update ansible for python3 offline installation
  • [PP-4167] - ElasticSearchInput (Storm) forces type "doc"
  • [PP-4169] - punchplatform-zookeeper-console.sh unfriendly error when multiple zk clusters
  • [PP-4170] - Elasticsearch masters list is wrongly built when using long names for transport addresses
  • [PP-4185] - Resource manager configuration improperly exported in environment
  • [PP-4186] - Wrong template for resource metadata in elastichsearch metadata index
  • [PP-4195] - Pyspark: Sql statement list not registering temporary tables
  • [PP-4206] - Multiple punch_node are not allowed
  • [PP-4218] - Some resources are altered by resolver during shiva execution
  • [PP-4239] - Forwarding needs resource manager for local system storage
  • [PP-3649] - implement ansible role for pyspark