DAVE-6.3.4 release notes¶

This document is a summary of content, changes, limitations and fixes of this release compared to the previous DAVE-6.3.3 release.

Refer to the the punchplatform official site if you look for the complete documentation sets and resources.

The documentation for this release can be found inside the deployment archives (standalone and deployer versions), and at https://doc.punchplatform.com/6.3.4/index.html.

The documentation for the most recent release can be found at https://doc.punchplatform.com. Note that this latest documentation might be a beta release.

Changes¶

Out Of Memory detection¶

The default value to detect OOM on stormlines has changed. OOM detection is controlled using a JVM startup property (punchplatform.oom_exit_treshold) whose value was changed from 0.9 to 0.0. As a result, 6.3.4 stormlines will not exit promptly but will experience a standard Out Of Memory exception.

The reason for that change is that in case of concurrent gc invocations, the detection algorithm was fooled. This caused intempestive Stormline exit.

Important

If you used the Out Of Memory protection in production (i.e. did not set any particular value) and are happy with it, and/or if you see new OOM failures after you updated your Punch to 6.3.4, reset your threshold to its previous default values. You can do this by adding the following jvm option to your stormlines :

settings :
  topology.worker.childopts : "-server -Xms100m -Xmx100m -Dpunchplatform.oom_exit_treshold=0.90"

Documentation¶

A few documentation errors have been corrected:

The default value of various input nodes rcv_queue_size parameter is documented as 100000. The actual default value is actually 10000. This documentation bug is now corrected. In any case there should be no impact to your punchlines.
the documentation of socket input and output node were still referring to deprecated parameters:
queue_size instead of rcv_queue_size or send_queue_size
queue_flush_size instead of send_queue_flush_size
queue_flush_interval_ms instead of send_queue_flush_interval_ms
a queue_size parameter was documented as part of the socket UDP listen section. That parameter has no effect and has been removed.

Important

Deprecated parameters are dealt with properly to guarantee backward compatibility. Should you use them it will only cause warning logs. The documentation corrections listed here have no effect whatsoever on your actual production configurations. I.e. these are only documentation fixes, there are no changes in the 6.3.4 runtime.

Improvements¶

#896 : octet counting autodetection feature in TCP syslog spout
#1175 : make ResourceAwareScheduler as default storm scheduler
#1380 : improve extraction deployment configuration in gateway
#1456 : improve the puncher for executing several versus single tests
#1458 : kibana configurations for timeouts to elasticsearch
#1480 : yq support for deployer
#1492 : review all examples
#1512 : added method for checking java 11 or 1.8 compatibility
#1513 : improve shiva Kafka protocol to deal with larger files
#1533 : plan support for immediate starting and stopping
#1540 : gateway should be able to execute punchlets with test headers
#1545 : support server.basepath when extracting data on kibana
#1546 : add extraction name in punchline ID
#1556 : empty status for extractions
#1558 : display extractions even if their punchlines are unavailable
#1561 : sort extraction results
#1589 : improve the kv operator for json content
#1672 : add extra opendistro parameters in deployer
#1683 : support special characters in column names in Clickhouse Stormline Output node
#1684 : make xms xmx options dynamic in archives housekeeping
#1685 : clickhouse output node configuration (optional) simplification through column list retrieval from incoming storm tuple
#1687 : clickhouse output node configuration (optional) simplification through single storm-tuple field usage
#1688 : support implicit boolean-to-int conversion for clickhouse stormline output node
#1691 : in place remote resource loading support for punchlang fetch operators
#1701 : support S3 to load punch resources
#1709 : make xms xmx options dynamic in shells for java applications
#1714 : new bulk exception setting in elasticsearch output (bulk_failure_action)
#1734 : add log management platform configuration to examples

Bug Fixes¶

#861 : added mandatory support for auto-detected octet counting syslog frame format
#1125 : fix planctl with tls
#1171 : kafka output fails on ppf metrics when multiple topics
#1477 : punch kafka-topics shell --add-config and --delete-config options in punchplatform-kafka-topics.sh, including TLS support
#1495 : remaining @PUNCH@ after key value operator handled an escaped string value
#1510 : platformctl storm health returns false negative
#1517 : Extraction through Punch Plugin does not work with multi ES cluster
#1520 : do not wait tick to enable the clickhouse jdbc handle
#1523 : update resolver on platform does not restart shiva and gateway anymore
#1524 : Cannot retrieve saved punchlines using the resource manager
#1531 : SSL/TLS Kafka reporter settings are ignored by the gateway
#1532 : improve the robustness of the kafka reporter
#1537 : id field not extracted with plugin extraction
#1539 : punchplatform runtime secrets tag cannot be used without ssl
#1546 : Replace .pml with .json for punchlines in resource manager
#1571 : Fixed monitor_zombie_topologies.sh that was not updated for 6.3 and reported and/or killed incorrect topologies
#1599 : Fixed unavailability of kafka clusters on failover of leader, due to deactivated kafka cleaner/compaction mechanism, causing an ever-increasing consumer offsets topic.
#1601 : Syslog Input Error when log in octet_counting and log end_of_line on same worker
#1630 : set enable.auto.commit default to false
#1641 : make channelctl report the correct errors for invalid channels
#1652 : take into account the jvm opts and improve command line error messages
#1669 : fixed not working --offsets subcommand (for non-TLS kafka only) in punchplatform-kafka-topics.sh command.
#1679 : fixed punch python applications compatibility issues with python 3.6.9 (elasticsearch housekeeping, elastalert, pyspark)
#1690 : fixed potential logs loss in clickhouse stormline output node : was acking tuples before batch is sent
#1695 : emit metadata in one by one mode of file output
#1708 : disable the out of memory protection by default
#1727 : added support for null values in punch resources
#1741 : tls documentation error for all output nodes
#1740 : fixed server side only TLS authentication for output socket nodes

Other changes¶

#1551 : make injector started without any environment variable
#1528 : removed the siddhi dependencies
#0001 : added a parser getting started for standalone
#0001 : added configuration error if channel name contains dots
#0000 : add kafka auto reset offset parameter in doc
#0000 : troubleshooting page for kafka commit failure
#1665 : update lodash to 4.17.21

Internal tasks¶

The training material was improved.

#0001 : improve error message of puncher
#0001 : kafka storage type by default for shiva and operator in standalone
#0001 : improve error messages of corrupted channels
#1508 : improve tests execution time (plans)
#1549 : punchline tenant channel and name inconsistency
#1551 : stormlines must start without any punch environment variables
#1674 : punchlinectl --punchline returns 0 when command is badly parsed
#1680 : LTS validation tests on punchbox
#1681 : Deployment inventory confusion between ES and GATEWAY homonymous clusters prevents punchbox deployment
#1720 : integrate common punchlets from artifactory in standalone