DAVE-6.3.3 release notes¶

This document is a summary of content, changes, limitations and fixes of this release compared to DAVE-6.3.2 release

Refer to the the punchplatform official site if you look for the complete documentations and resources.

The documentation for this release can be found inside the deployment archives (standalone and deployer versions), and at https://doc.punchplatform.com/6.3.3/index.html.

The documentation for the most recent release can be found at https://doc.punchplatform.com.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your clusters. The upgrade notes (e.g. upgrade from 6.2 to 6.3) discuss every critical information about incompatibilities and breaking changes, performance changes, and any other configuration changes that might impact your production deployment of Punchplatform.

Changes¶

Kibana plugins conf¶

Any plugin configuration can now be written inside a domain section only (even the plugin version configuration)
Any plugin can now be disabled with the configuration enabled: false (default true)

Dissect¶

The default behavior of the dissect operator change. It now adds the matched fields into the target tuple instead of overwritting it. If you need the overwriting you can use the overwrite method as follows:

'''java dissect(\"%{host}:%{port}\") .overwrite() .on(\"myhost:8080\") .into([target]); '''

Gateway¶

Add Minio support for resource manager data backend
New route /v1/<TENANT>/resources/update/<RESOURCE> to edit resource metadata properties.

Storm¶

return of published hostname settings in the storm configuration

Improvements¶

#1330 Allow additive secrets/credentials sources for platform/cluster/node level for storm and spark
#1346 customize book listing terminated application timeout.
#1417 Add success and errors stream to archive reader. (See usage documentation)

Zookeeper¶

# 1354 Removed deprecated deployer role 'zookeeper-nodes-creator'

Warning¶

Extraction¶

Prior to version 6.3.3, consider extraction feature as not working although in some very simple use-case you might be getting the desired behavior.

Bug Fixes¶

#813 modsecurity and kibana domain conflict
#1091 misordered fields when using kafka input non lumberjack decoders
#1219 gateway doesn't only return last version of resource when queried with filters
#1241 dissect punch operator overwrites existing subfields
#1265 filter node configuration regression
#1299 punch kafka-topics shell cannot alter topics
#1314 non-idempotent steps in kibana plugins setup
#1329 switch from runtime spark to storm for extraction use case. A new endpoint has been added for that purpose.
#1329 fix channel_structure, book_structure ignoring YAML extension file
#1336 Modsecurity is not deployable
#1343 storectl does not work with elastic ssl client auth
#1345 settings section in book_structure files is ignored
#1347 Modsecurit does not consider the kiban.domains..index setting
#1352 fixed extraction input node not parsing properly elasticsearch query from punchline
#1353 It's not possible to disable modsecurity for a specific kibana domain
#1355 missing ssl parameters for kafka reporters in gateway configuration
#1359 spark nodes do not support kafka ssl with client authentication
#1360 Modsecurity additionals requests are blocked unlegimately by modsecurity
#1362 punchlinectl resolve command does not work with selected runtimes in resolver
#1373 Kibana (punch) plugin deployment fails when only part of the kibana domains have this plugin configured
#1378 Missing required Open distro security audit settings to allow housekeeping of audit indices and customization of audited events
#1381 Extraction not working with modsecurity following punch doc
#1382 queries and filter section is not taken into account in extraction
#1390 Deprecated and ambiguous custom_headers configuration in punchplatform ui plugin
#1392 The punchplatform feedback plugin does not recognize any configuration anymore
#1393 Kibana secrets are not deployed on every configured servers
#1400 wrong output file format for csv extraction
#1401 no output file for json extraction
#1403 light operator without cots
#1424 Gateway list all versions of all resources instead of last version of each one of them
#1430 channelctl does not resolve yaml or hjson channel structure
#1435 resolve secrets for any config files (Shiva)
#1459 add punchline name in gateway punchline execution service
#1462 unflat tuple operator does not work
#1465 elastic settings are not taken into account in gateway configuration
#1468 add headers in csv extraction in spark runtime
#1469 add to_yaml jinja filter to convert map to yaml
#1494 Deployment fails if resolver is not a hjson file
#1498 Edit visualization and dashboards not working with modsecurity
#1504 channelctl configure for hjson and yaml files
#1501 Fix resolver management for shiva applications
#1500 fix spark app data cleanup policy
#1484 fix zookeeper secrets deployment when platform user does not exists
#1490 Fix kibana.yml to support multiple gateway hosts
#1488 fix storm ui url
#1485 fix multiple user's operator secrets on multiple operator servers
#1486 fix kibana deployment template
#1448 Fix doc for syslog_output with tls
#1414 Fix exception message in ES Output
#1431 Fix storm workers logs in /var/log/punchplatform/storm and workers.artifacts folder
#1471 fix array publishing in punchnode
#1337 fix security breach in archive housekeeping logs
#1341 zookeeper command inlining and robot test fixing
#1358 Fix resource deletion when using ES metadata backend
#1377 fix channelctl not accepting cluster type pyspark
#1306 fixed pyspark test still being ignored
#1306 fixed plan test with new entrypoint
#1034 fixed lumberjack thread stop

Others changes¶

#1492 review all samples and documentation
#1504 channelctl configure for hjson and yaml files
#1468 "true" as default value for "add_header" field
#1505 Handle Utf8 objects in archive extraction
#1499 improve TLS documentation for certificates specifications
#1502 allow channels to have sub folders
#1288 hide Kafka keystore/truststore configuration logs
#1432 Storm parameters missing from storm roles
#1461 resolver uses env variable to get application name if not provided
#1491 check all examples 6.3 and 6.4
#1427 removing unused csp rules + add conf in deployment settings to disable csp strict for Kibana
#1447 Gateway is forced to populate users configurations with the deployer
#1473 "Download" button from Punch Plugin Extraction features doesn't redirect to Gateway itself anymore
#1474 not all directories can hold a conf dir
#1467 get extraction credentials through http headers
#1464 Add multiply operation to punchlets
#1456 improve the puncher to deal with new parser packaging
#1455 add a new shortcut to the punch language to access the single child
#1421 ArchiveReader should emit metadata with output line
#1406 Add headers in csv extraction
#1420 create resource manager shell
#1403 light operator standalone without cots
#1412 minio implementation for resource manager
#1333 Add charset argument to ArchiveReader and FileOutput
#1415 Improve resource manager API and Gateway endpoint
#1382 queries and filter section is not taken into account
#1399 upload a single file when resource manager is empty fails
#1389 reworked archiving test to make use of inlined exit conditions
#1376 set worker cleanup to false to avoid deleting arrow files for running spark application
#1223 improve strategy in case of failure in clickhouse output node
#1368 improve the log injector to generate from and relative to dates
#1319 Tuple object from json string does not raise Exception
#1356 wrong documented prerequisites for deployer
#1306 add support for user defined runtime
#1357 add ssl parameters in nodes documentation
#1350 rename "target.type" of gateway mapping to "event_type"
#1304 manual validation process for housekeeping and storectl
#1306 improved reader writer code
#1349 improve-build-scope
#1225 improve the error handling of bad configured punchlines stream and fields
#1280 documented the zookeeper upgrade
#1346 Set exit_application_timeout to 1 seconds instead of 1 ms
#1306 Housekeeping robot tests
#1346 release note edit
#1346 Add exit_application_timeout setting to book_structure
#1344 Injector Timeout doc
#1330 add dave 6.3.3 release note
#1334 Unify all punch components for secrets and credentials with the same upload behavior
#1330 storm and spark additive credentials files upload
#1306 Manage punch secrets for Robot HTTP Request library
#1331 implement robot test from manual validation
#1302 create validation documentation

Internal tasks¶

#0000 - Fix es mapping issues for spark monitoring dashboard
#0000 - Resolve "add inactivity timeout parameter in log injector"
#0000 - fix deployer when using symbolic link for resolver file
#0000 - fix extraction credentials + resolver security template for standalone
#0000 - improve resolver performances
#0000 - Fix kafkactl required argument exception
#0000 - keep compatibility for platform monitoring configuration
#0000 - fix docker image for gateway
#0000 - Gitlab CI improvements
#0000 - attempt to use profiles to avoid building sub modules

Known issues¶

punchplatform-kafka-topic.sh --topic mytopic --alter --config myconf=myvalue does not work in 6.3.3 version. Use kafkactl alter-topic --cluster mycluster --topic mytopic --config myconf=myvalue instead.