DAVE-6.3.0 release notes¶

This document is a summary of content, changes, limitations and fixes of this release as compared to DAVE-6.2.2 release

Refer to the the punchplatform official site if you look for the complete documentations and resources.

The documentation for this release can be found inside the deployment archives (standalone and deployer versions), and at https://doc.punchplatform.com/6.3.0/index.html.

The documentation for the most recent release can be found at https://doc.punchplatform.com.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your clusters. The upgrade notes (e.g. upgrade from 6.2 to 6.3) discuss every critical information about incompatibilities and breaking changes, performance changes, and any other configuration changes that might impact your production deployment of Punchplatform.

Release main features and enhancements¶

Important

The 6.3.0 release is elected as punch LTS release. Check the roadmap documentation for details.

Platform reporters has been moved to operator section (#1075, #1076)¶

Reporters used by operator must now be declared in punchplatform_operator section in punchplatform-deployment.settings and not anymore in the platform section. Check the punchplatform_operator section in punchplatform-deployment.settings documentation to have more details.

This reporter section is now mandatory and is checked by the punch deployer. In addition, the Platform Monitoring documentation has been updated to be more complete.

Avro Parquet extraction in ArchiveReader¶

#1027 : You can now extract your archives encoded in Avro and Parquet using the Archive Reader. Check the extraction guide.

Elastalert¶

#1092 : Added a new parameter for elasticsearch rule loader, which enables you to select a field on the retrieved documents as the rule.yaml file.

Punchplatform cli¶

#1174 add platformctl command to get platform health in command-line. Similarly a new kafkactl command provides access to kafka topics and consumers.
#1164 More verbosity needed for channelctl reload/start
#951 shiva command-line tool for viewing task assignment
#868 re-implemented the reload action

Patch procedure¶

#1081 : Patch procedures have been updated to be able to update a remote patch if necessary. Check Patch procedure documentation to see details.

Custom Node API¶

#1106 : Provided a test framework for custom nodes in Storm. Check the API Storm Custom Nodes documentation to see details.
#1198 : Converted the ClickHouse Output to leverage the custom node API. The legacy node is still available.

Elasticsearch¶

#1126 : Added a new parameter for elasticsearch clusters, which enables you to select an alternative temporary directory to prevent issues due to /tmp partition mount with noexec

Spark and Pyspark¶

#1270 : a new parameter has been added to submit spark applications to your spark cluster. --spark-cluster. A valid spark url to use as master url is generated from your punchplatform.properties. Prefer using --spark-cluster to --spark-master.

Main documentation improvements:¶

HLI and CHA modules training material were significantly improved.
Additional prerequisites/advices provided on target servers operating system (partitioning, swap management).

Extraction¶

Prior to version 6.3.3, consider extraction feature as not working although in some very simple use-case you might be getting the desired behavior.

Security¶

#1206 deal with ssl credentials with a new punch-command-config, a secrets-file and resolver resolution
#1203 resolver enhancement: credential tag support
#1202 Unify secrets upload by deployer
#1200 Move secrets out of deployer roles and deployment.settings
#1179 Upgrading Zookeeper 3.5.5 to 3.5.7
#1178 Add SSL support for Zookeeper
#1176 Deploy certificate and refactoring for better secret management
#1166 Deployer update ES cluster certificates documentation
#1159 Implement a secret manager for punch commands
#1130 Deploy spark with certificates for punchline with SSL
#1122 Add SSL support for Kafka
#1116 Deploy storm with certificates for punchline with SSL
#1115 Deploy shiva with certificates for punchline with SSL
#1107 Deployer inventories for ELK and Gateway with SSL

Other Improvements¶

#1228 be able to declare a custom node as a standard node
#1229 enhance user experience on plugin
#1230 Kafka and Zookeeper use mTLS when SSL is enabled
#1236 Elastalert: kafka alerter & Es rule loader SSL
#1240 Install RELP module in packaged logstash on shiva nodes
#1242 Systematically provide in all punch binaries environments the hostname in PUNCHPLATFORM_HOSTNAME env variable
#1253 make shiva capable of filtering child logs based on rates
#1256 upgrade kafka lib to 2.7
#1257 spark jackson dependency is dangerously unspecified
#1259 split shiva subprocess logs in separated files
#1264 improve punch operator shells to deal with ssl parameter automatically
#1267 Deployer : Kafka and zookeeper mTLS configurations
#1226 : provide a direct memory allocation bytebuffer strategy for very large resources
#1227 : new elasticsearch output option to control the index suffix
#1228 : be able to declare a custom node as a standard node
#987 : operator activate.sh must be a common deployed file, and not a per-operator file
#962 : Improve the punchline editor in the Punch plugin
#1084 : storm custom nodes can use reserved stream _ppf_metrics.
#1030 : Archive Reader can now filter lines using grep and match_bloom settings.
#1104 : reworked jar and pex instrospection for custom nodes. punchplatform-analytics-scan.sh is deprecated in favor for punchplatform-inspect-node.sh. The newly added shell can generate storm, pyspark or spark custom nodes meta-data as json. Futhermore, a slight change to Storm/Spark public API, it is now mandatory to implement a new method which takes as parameter respectively StormNodePubSub or SparkNodePubSub.
#1113 : Display the Punch configuration files in the Resource Manager
#1129 : New Archive Reader Node in Storm. See node documentation.
#1104 : reworked jar and pex instrospection for custom nodes. punchplatform-analytics-scan.sh is deprecated in favor for punchplatform-inspect-node.sh. The newly added shell can generate storm, pyspark or spark custom nodes meta-data as json. Futhermore, a slight change to Storm/Spark public API, it is now mandatory to implement a new method which takes as parameter respectively StormNodePubSub or SparkNodePubSub.
#1180 : add custom file type when creating a resource in the Resource Manager
#1196 : Display replays and extractions in a timeline in the Kibana plugin
#1207 : added a new parameter in channel_structure.json that can be defined within an application (apply_resolver_on: [ {resolve: "file_name.json", selection: { host: , file: , ... }} ])
#1211 : improve plugin debugging and tests
#1204 : create zabbix output node
#1199 : supports bookctl and channelctl configure command with directory path
#1198 : convert clickhouse output node to new api
#1192 : elasticsearch output should publish tuple in error stream
#1191 : Channelctl configure is not verbose enough
#1189 : avoid the use of slidingTimeWindowHistogram as it keeps to many points in memory
#1187 : Resource Manager improve upload performance
#1184 : add metrics to Archive Reader
#1183 : big json resource loading must benefit from a streaming strategy
#1180 : add custom file type when creating resource in resources manager
#1162 : extraction & replay with Kibana UI through gateway using dagctl cli
#1158 : implement a modulo based load balancing for punchlet dag engine
#1154 : protect punchlang parser from invalid 160 whitespaces
#1150 : resolver enhancement: new selection feature
#1148 : add rotating map as a native punchlamg facility
#1147 : elasticsearch bol must be protected against runtime exceptions and intempestive logging
#1146 : support an hourly index pattern configuration for the elasticsearch output
#1138 : implement the compact mode for csv resource files
#1129 : create archive reader node in storm
#1113 : display configuration in resource manager
#1106 : unit and integration test framework for custom nodes
#1104 : Spark and Pyspark scanner should use static resources for internal nodes
#1103 : kibana ui for data replay
#1097 : Python application (Housekeeping, elastalert, ...) except pyspark does not work with resolve.conf
#1083 : Capacity to replay Archive (parsed log) => Elatic with 200MB archive files
#1057 : MLFLOW | Add To Deployer (Ansible)
#1030 : filter archive when extracting
#1056 : need shiva logs capture max size to prevent larsen effect
#862 : add ssl support for resource manager elasticsearch backend

Bug Fixes¶

#1276 Unable to delete resource when using ES as metadata backend
#1274 Gateway fails to execute/list books
#1273 move shiva working directory from /tmp dir
#1271 ambiguous Storm Syslog bolt UDP
#1269 Gateway cannot launch shells because of book-template
#1268 Tenant error when executing punchline through Gateway
#1266 Shiva deployment not possible with multiple tags per nodes
#1263 reduce fail timeouts for platformctl health command
#1262 Resource manager can't upload/download heavy files
#1260 errors when importing kibana dashboards in command-line
#1254 kafka output duplicate all its tuple acks
#1251 (undocumented) breaking rename of es output node 'cluster_id' setting into 'cluster_name'
#1250 channelctl reload does not update shiva punchline
#1249 wrongly required 'settings' section in gateway elasticsearch cluster
#1248 opendistro_security section is required although it should be optional
#1245 kibana extraction plugin fails to display more than 10 extraction
#1239 gateway doesn't retrieve book executions
#1238 punchline editor doesn't handle execution in storm environment
#1237 Documentation for security deployment
#1235 channelctl status fails after shiva task start was previously attempted on non-existing cluster
#1233 platformctl health reports error for elasticsearch on standalone due to wrong standalone naming of es node
#1232 log records sent from admin command may include null values
#1221 Elasticsearch fail to deploy on multi ip servers with active iptable control
#1218 errors caused by incorrect resource files prevent subsequent loading
#1214 wrong reporter..bootstrap.server in generated punchplatform.properties
#1212 IMPROVE Storm deployment setting to solve noexec /tmp problem
#1006 wrong keys in resolver for elasticsearch storm nodes
#1235 : channelctl status fails after shiva task start was previously attempted on non-existing cluster
#1073 : Unix right error with punchplatform-deployer.sh --copy-configuration
#1088 : Fix SSL for elasticsearch input and output nodes in storm environment
#1150 : resolv.hjson now supports two additional fields selection (host and file).
#1150 : resolving with unknown selection keys will now throw a runtime exception.
#1193 : fix custom processing node that called onClose() method after it processed a Tuple.
#1220 : fixed metric reporter type console not publishing on STDOUT
#1217 : spark install dir empty when spark is not deployed
#1201 : high shiva cpu after scheduling plan
#1197 : Shiva health wrong RED status "The end instant must be greater the start"
#1195 : plugin crash when closing last tab in rm
#1193 : custom processing node calls onclose method for each tuple
#1190 : json array codec does not deal with escaping
#1188 : shiva does not kill child process
#1182 : craig suffer from urllib python dependency issues
#1165 : Storm certificate error during deployment
#1163 : cannot select a search when creating a feedback table in Kibana
#1157 : shiva worker does not clean local application folders
#1156 : very first channelctl status is wrong
#1155 : Minio cluster deployment/documentation issue
#1152 : node documentation in punchline editor is broken
#1151 : punchline editor doesn't handle every node setting
#1145 : remote resource are loaded only after the first cron expression fires
#1144 : Storm Monitoring not Working (undocumented/obsolete cluster_admin_url and http_api_ports settings)
#1142 : last patch jar is not taken into account
#1140 : punch compilation errors are reported at wrong line number
#1137 : Gateway cannot log (and produces huge log files) due to Incorrect use of bootstrap.servers in gateway group_vars
#1136 : Redhat deployment missing os_RedHat.yml
#1135 : Decrease deployer archive size
#1132 : 6.2.1 fetch csv operator regression after new compact mode
#1131 : Difference between kafka topics created during deployment and those declared in generated punchplatform.properties
#1125 : plan cursor commands do not work with opendistro
#1123 : Missing standard elasticsearch templates in deployer package
#1121 : file output does not handle ppf metrics
#1119 : spark nodes properties are not retrieved [UI bug]
#1111 : Gateway Spring Boot does not erase temporary uploaded files
#1109 : Wrong 'changed' metricbeat status when idempotent redeployment
#1101 : Many false cluster healths in platform health document
#1089 : channelctl -t $TENANT should raise proper error message if $TENANT does not exists in $PUNCHPLATFORM_CONF_DIR/tenants
#1078 : Topology launching failure when kafka_input have not the bootstrap.servers setttings
#1077 : Monitoring failure at startup on standalone due to events dispatcher not reading the topic from the start
#991 : OpenDistro credentials not updated in Audit and FileBeat config
#985 : OpenDistro cannot be deployed with hostname verification
#926 : Gateway punchlet filtering is blocking es forwarding if ssl is enabled
#922 : Gateway cluster-based configuration prevent any server-based configuration for SSL
#891 : No error context on punchline nullpointer
#821 : false positive using unit test in puncher library