############################# Release Notes ############################# Please read the following notes carefully to precisely identify your version : 1. if you have a running platform, the version number can be determined by navigating to the 'online documentation' link from the PunchPlatform AdminService Graphical User Interface. 2. Starting at version 2.0.0, a new version scheme has been defined, =using logical name ('**ahmad**'). Redmine bug tracker was also used. Now it is Jira. Starting from Ahmad, the FIXED changes indicated as part of each release refer to the corresponding Redmine ticket. The indicated "Build version" (Vx.y.z) are suffixed to the following LMC deployment artifacts : * Clustered LMC deployer "punchplatform-deployer-x.y.z.zip" * Standalone LMC deployer "punchplatform-standalone-x.y.z.zip" * Installed LMC Administration/Monitoring service "punchplatform-admin-node-x.y.z.zip" 4. a SNAPSHOT in the name of the version below indicates that it is under development and/or testing. The last stable version is the first one in this document without SNAPSHOT. Feel free to visit the Jira : https://punchplatformfactory.atlassian.net/projects/PP The latest version of the documentation is available here : https://documentation.gp.sef.cloudomc.ts-agora.fr/pp/dist .. important:: if you see a below, you are working on a development branch ******************** Brad Release Changes ******************** Brad v4.0.2 Release Changes ==================================== Bug fixes --------- - [PP-1841] - fixe the Kafka JmxReporter leak - [PP-1576] - Archiving performance reduction on topic containing months of data - [PP-1550] - Transactional Kafka incorrect reuse of persisted batch Id when not reusing persisted kafka offset - [PP-1472] - Batch failure management error in Kafka Spout transactional/batch mode - [PP-1667] - Fixed incorrect parsing of error documents Brad v4.0.1 Release Changes =========================== New features ------------ - [PP-1219] - export2csv plugin adjustment - [PP-1237] - Package enduser documentation in the product - [PP-1239] - provide user with a nice light and colored punchlet editor - [PP-1375] - siddhi punch integration - [PP-1317] - Platform configuration control Enhancements ------------ - [PP-741] - Add supervisord support for Shiva supervision - [PP-970] - Shiva monitoring - [PP-1105] - Test services/tasks killing behavior - [PP-1121] - Port of export to CSV for Brad (Kibana 5) - [PP-1231] - Allow Single topology not to have output section - [PP-1246] - PML - binaries size - [PP-1250] - Add javadoc in PML code - [PP-1313] - Shiva UI and CLI beautifuling - [PP-1340] - PML - Doc change - [PP-1351] - Change a pygregator configuration parameter name - [PP-1379] - Check values consistency in deployer configuration audit - [PP-1400] - Installer standalone optimization - [PP-1414] - Parser SOGo - [PP-1429] - On ceph machines, the updatedb process cause troubles for application - [PP-1435] - kafka server config deployer - check coherence between default and deployer template - [PP-1499] - Procedure to install Ceph on deployment machine missing - [PP-1504] - Kibana deployment fails - [PP-1509] - Ceph manager deployment fails - [PP-1510] - An install procedure to install ceph on deployment machine must be available - [PP-744] - Allow resilient cron task start/stop/status in Shiva - [PP-835] - wrong location of kafka gc log - [PP-942] - Pygregator : check after aggregation request templatization if some nested aggregation names are identical - [PP-953] - Remove servers section of zookeeper - [PP-1085] - Check the punchplatform configuration dependencies with the deployer - [PP-1194] - PML - dependencie conflict handling - [PP-1240] - need a sublime editor - [PP-1241] - Upgrade elastic components to 5.6.4 - [PP-1277] - PML - Use cases - [PP-1294] - Choose for each kibana instance the server place - [PP-1308] - Aggregation requests support in Batch Requests service (e.g. Top X) - [PP-1359] - Test spark 2.2.1 on deployer - [PP-1380] - Add outputFields, type and index in elastic batch node - [PP-1466] - when statements ansible - [PP-1519] - simplify kibana plugin installation with punchplatform-cluster.sh Bug fixes --------- - [PP-745] - Améliorations de mapping et de recherche Kibana - [PP-1003] - Update punchplatform glossary (e.g. graphite references out of date) - [PP-1053] - log warning in punchplatform-ml.sh - [PP-1103] - Shiva cluster config update too permissive - [PP-1173] - Java heap space zookeeper - [PP-1205] - punchplatform-command.sh is not well protected - [PP-1209] - PML - zookeeper crash - [PP-1221] - Unable to deploy operator when pp-admin and pp-operator are not on the same device - [PP-1223] - Kibana 5 very slow when save visu or dashboards - [PP-1226] - Incomprehensible error message on Shiva UI due to an exception when a task has never started - [PP-1227] - Rename punchplatform-shiva.sh start-task to submit-task - [PP-1233] - Exception catcher isn't compliant with Brad new indices format - [PP-1234] - File descriptors leak on Shiva - [PP-1242] - Punchplatform-heartbeat.sh doesn't work on standalone - [PP-1243] - punchplatform-packetbeat.sh for standalone - [PP-1245] - Update Doc's Kibana dashboards - [PP-1270] - Apache parser do not raise an error when logs are not properly parsed - [PP-1288] - gaussian type for log injector doesn't work - [PP-1301] - Deployer audit code fails with some configuration values - [PP-1303] - Pinchplatform-setup-kibana fails when proxy and unary operator error - [PP-1304] - Punch language always interpret Tuple as boolean in conditionals - [PP-1319] - Elastalert deployment fails - [PP-1327] - Conflict between kibana and elasticsearch bind address - [PP-1329] - Shiva service does not start after VM reboot - [PP-1335] - Audit configuration fails on specific cases - [PP-1357] - Light topo waiting time not used - [PP-1367] - Non immediate SUBMIT status in Batch Service - [PP-1388] - Housekeeping : 404 ERRORS for admin service buttons (REST API) - [PP-1427] - Documentation bug for testing parser - [PP-1430] - Extraction of archived logs to console does not work - [PP-1439] - etc/conf.json settings for housekeeping ceph - [PP-1459] - Archiving ceph issues with patch - [PP-1461] - Archiving system internal summaries inconsistency causing incomplete Housekeeping or Extraction - [PP-1483] - avishai RC5 chroot issues - [PP-1486] - Unit tests of archiving client KO following PP-1461 - [PP-1487] - Bug doc dev - how to build elastalert dependencies - [PP-1493] - documentation on pygregator - [PP-1501] - Disturbing behavior of Git Bare deployment - [PP-1506] - Deployment of export2csv in chroot mode : Unable to find zip - [PP-1512] - "Troubleshooting export2csv kibana plugin" out of date - [PP-1518] - [log-injector] lumberjack-client mode : Missing mandatory property 'PunchPlatformSettingsMapImpl.destination'. - [PP-1533] - [batch-request-plugin] command: line 29: loop: command not found Brad v4.0.0 Release Changes =========================== New features ------------ - **IMPROVED** PP-1004 - Upgrade Elasticsearch stack to 5.6.0 - **IMPROVED** PP-1022 - Ceph Luminous 12.2.0 Support - **FIXED** PP-1155 - Upgrade Kafka to 0.11.0.0 - **IMPROVED** PP-1102 - Variable gaussian distribution into the PunchPlatform log injector Depreciations ------------- - **IMPROVEMENT** PP-1193 - Grafana now legacy Breaking changes ---------------- - **IMPROVED** #897 - Update default index naming on Brad to -events-YYYY.MM.DD Enhancements ------------ - **FIXED** PP-1048 - PunchPlatform Admin (java) fails to start - **FIXED** PP-1108 - Modsecurity rules improved - **FIXED** PP-1110 - Update getting started of Punchplatform, ML and Pygregator - **FIXED** PP-1112 - Kibana default index pattern mytenant-events- is not taken into account - **FIXED** PP-1113 - Kibana 5 fails when add index pattern - Request Entity Too Large - **FIXED** PP-1116 - Wrong software version in punchplatform-deployment.settings documentation - **FIXED** PP-1122 - Pygregator PyUnit tests KO - **FIXED** PP-1124 - Admin Behave Unit tests KO (housekeeper/elasticsearch) - **FIXED** PP-1126 - Clean hard coded version maven dependencies - **FIXED** PP-1132 - Add punchplatform-* commands to manual pages (documentation) - **FIXED** PP-1134 - Warnings during compilation of Documentation - **FIXED** PP-1135 - PDF generation of documentation failed - **FIXED** PP-1136 - PML missing punchplatform-analytics-plan-4.0.0.jar - **FIXED** PP-1138 - Getting started standalone inconsistent - **FIXED** PP-1142 - Documentation: add XFS dependency for CEPH install - **FIXED** PP-1147 - Template lmc for light topologies refactor - **FIXED** PP-1149 - Update Pygregator getting started, focus on websense channel - **FIXED** PP-1158 - Broken link in documentation's Getting Started - **FIXED** PP-1163 - punchplatform-channel.sh, --verbose does nothing - **FIXED** PP-1164 - punchplatform-channel.sh --help (or -help) does nothing - **FIXED** PP-1174 - PML - dependencie problem with master zookeeper synchronization - **FIXED** PP-1175 - Documentation : Blocking tests for releases out of date - **FIXED** PP-1176 - Documentation : Licensing out of date - **FIXED** PP-1182 - Man pages are ok for development doc but ko for packaging doc - **FIXED** PP-1185 - Documentation : clean workspace before compile the doc - **FIXED** PP-1186 - Standalone install.sh interactive install fails - **FIXED** PP-1187 - Unwanted dependency on Storm in Ceph deployment - **FIXED** PP-1188 - Elastalert deployment fails - **FIXED** PP-1190 - Unwanted dependency on Storm in Ceph deployment - **FIXED** PP-1192 - Unable to stop spark process on standalone - **FIXED** PP-1199 - punchplatform-setup-kibana.sh prints clear errors when kibana and ES are down - **FIXED** PP-1202 - Cephbeat deployment fails - **FIXED** PP-1203 - mapping_global_standalone.json : template error - **FIXED** PP-1208 - Deprecated type field are still present in elasticsearch templates - **FIXED** PP-760 - Ceph deployer improvement for check mode support and maintainability - **FIXED** PP-992 - Turn Deployer idempotent back again - **IMPROVED** #878 - Fix punchplatform-deployment.settings soft version - **IMPROVED** #909 - Upgrade Elasticsearch and Kibana from 5.5.1 to 5.5.2 - **IMPROVED** PP-1000 - Update default namping index on Brad to -events-YYYY.MM.DD - **IMPROVED** PP-1006 - When an index is not created at the time an alias is created by the curator script, this index is not included in the alias - **IMPROVED** PP-1044 - Add official Ceph Luminous documentation in Brad version - **IMPROVED** PP-1083 - Jar deployment, analytics plan on driver node - **IMPROVED** PP-1084 - Type log field error in parsing error not indexed - **IMPROVED** PP-1086 - Add spark binaries to operator nodes in deployment mode - **IMPROVED** PP-1087 - Handle REST mode spark application submission - **IMPROVED** PP-1127 - Spark can be deployed without metric reporter feature - **IMPROVED** PP-1129 - Add Spark and ML terms in glossary (documentation) - **IMPROVED** PP-1130 - Add CyberSecurity, Parsing and Log Management terms in glossary (documentation) - **IMPROVED** PP-1131 - punchplatform-log-injector manual pages with --punchlets and others outdated - **IMPROVED** PP-1133 - Split the dependencies between kafka, zookeeper and PunchPlatform operator in deployer - **IMPROVED** PP-1137 - Light standalone: metricbeat have to write metrics only in elasticsearch - **IMPROVED** PP-1148 - Improve PML deployment - **IMPROVED** PP-1183 - Quality Management doc update - **IMPROVED** PP-1184 - Delete Platform Monitoring dashboard in Kibana - **IMPROVED** PP-813 - add HOWTO deal with detection punchlet - **IMPROVED** PP-965 - Shiva cluster should be configured on standalone so as not to be in YELLOW health - **IMPROVEMENT** PP-1191 - Improve install.sh for standalone to print UI urls - **IMPROVEMENT** PP-1201 - Add Security Audit documentation *********************** Avishai Release Changes *********************** Avishai v3.3.6 Release Changes ============================== - **FIXED** PP-1461 - Archiving system internal summaries inconsistency causing incomplete Housekeeping or Extraction - **IMPROVED** PP-1458 Provided new 'change_ciphering_key' archive processing action to allow efficient in-place change of archiving keys without massive load/rewrite of objects data. - **FIXED** PP-1457 re-cipher using archive processor bolt from archives that contain unciphered batchesCould - **IMPROVED** PP-766 Improved and safer ciphered archiving procedures using interactive input of secrets (keys and keystore password) + advises on keys and secrets management (added in Archiving guide of Administration documentation) + end-to-end demo steps on a standalone to understand the process - **FIXED** PP-1453 Could not use multiple executors for ciphering/deciphering (single executor worked OK) - **FIXED** PP-1432 Some use cases failures when de-ciphering archived logs using punchplatform-objects-storage.sh command-line tool. - **FIXED** PP-1405 punch date formatters must change when year changes - **FIXED** PP-1401 - Housekeeping : wrong request in javascript part and wrong http verb in admin-service - **FIXED** PP-1388 - Housekeeping : 404 ERRORS for admin service buttons (REST API) - **IMPROVED** PP-1225 - SSL server in spouts has vulnerabilities - **IMPROVED** PP-727 - [Punch] Implement a Cloneable interface for Tuples. - **IMPROVED** PP-750 - Archiving housekeeping is slow despite indexation and causes OutOfMemory even erasure of non-existing timescope - **FIXED** PP-758 - Bug doc on Ceph - **FIXED** PP-765 - Choose the name of the public key in deployer - **IMPROVED** PP-786 - Elasticsearch snapshotting not allow by deployer - **IMPROVED** PP-790 - EPS reporting - **IMPROVED** PP-817 - Improve performance of syslog header parsing - **FIXED** PP-839 - Kafka spout settings issues in system metric topology - **FIXED** PP-851 - Log extracts by end-user - **FIXED** PP-867 - Netty metrics with netty as prefix but in the documentation it is syslog_spout prefix - **FIXED** PP-968 - Shiva does not stop properly - **IMPROVED** PP-969 - Shiva housekeeping - **IMPROVED** PP-974 - Shiva & light topologies deployment - **FIXED** PP-982 - Storm worker logs are not cleaned up for killed topologies - **FIXED** PP-1010 - WinAD : Ajouter champs Workstation & Category - **IMPROVED** PP-1016 - A finite number of unsubmitted tasks history must be kept in zookeeper - **FIXED** PP-1017 - Shiva data folders shall be purged of too old tasks folders - **FIXED** PP-1025 - Shiva documentation should explain cluster configuration - **IMPROVED** PP-1035 - Build unit tests for Punch programming simply and directly - **FIXED** PP-1039 - [Juniper] Tweak action field - **FIXED** PP-1041 - [Juniper] RT_Flow/Flow_reassembled_succeeded - **FIXED** PP-1055 - NullPointerException in ElasticsearchBolt in case of error logs when exception_handler punchlet is misconfigured - **FIXED** PP-1057 - Operational facts on Shiva - **FIXED** PP-1058 - Pygregator not defensive enough during configuration & logging check - **FIXED** PP-1059 - [Modsecurity] Extend PCRE size limit - **FIXED** PP-1065 - [Export2csv] Partial export for a query - **FIXED** PP-1066 - [Export2csv] Query fails if querying for indices for more than 7 days - **FIXED** PP-1069 - Full integration of pygregator in shiva - **FIXED** PP-1070 - Shiva CLI logs misplaced or bad rights - **FIXED** PP-1074 - Shiva tasks working directory inconsistency - **IMPROVED** PP-1090 - Lumberjack client/server with window frames - **IMPROVED** PP-1091 - Shiva status inconsistent in stand-alone (UNKNOWN when --with-shiva is NOT used, else YELLOW) - **IMPROVED** PP-1093 - Shiva CLI description inconsistent - **IMPROVED** PP-1100 - Shiva cluster configuration parameters must be overridable in deployment configuration file - **FIXED** PP-1189 - Option update-config of Shiva CLI reports same output event with or without change - **FIXED** PP-1206 - Pygregator and lights topologies fail when launched with shiva in a channel - **FIXED** PP-779 - Documentation kafka bolt insufficient - **FIXED** PP-1170 - Temp sharing in shiva command wrappers fails for several operators - **FIXED** PP-1159 - Getting started Shiva without install pre-requisites - **FIXED** PP-1104 - Shiva stores tasks working directories in /tmp - **FIXED** PP-1080 - Unable to deploy spark with PunchPlatform deployer - **IMPROVED** PP-1068 - Support pygregator deployment (official deployer) - **IMPROVED** PP-1056 - [WinAD] : Set upo proper unit tests - **FIXED** PP-1026 - Modsecurity blocks the requests when there is too many filters - **FIXED** PP-1021 - Legacy reverse proxy sections in the product documentation - **FIXED** PP-1019 - Unable to deploy a PunchPlatform with the git when the first deployment fails - **FIXED** PP-1040 - [Bluecoat] Parse system logs - **FIXED** PP-1037 - [WinAD][Parser] Correction for certain EventID - **FIXED** PP-1077 - Export2csv escaping to comply with CSV standard. - **FIXED** PP-1096 - [Admin] Monitor Kafka topic health - **FIXED** PP-948 - Remove --autotest from punchplatform-channel.sh because it is not supported anymore and update man pages - **FIXED** PP-825 - incorrect self_monitoring documentation - **FIXED** PP-941 - punchplatform-log-injector.sh generate wrong rep.ts - **FIXED** PP-1090 - Lumberjack client/server with window frames - **FIXED** PP-1072 - Export2CSV kibana plugin improved progress tracking/end condition checking to improve stability and reliability - **FIXED** PP-1071 - Export2CSV now supports inclusion of _id, _index and _type in search, query and CSV export - **FIXED** PP-1064 - Export2CSV now supports standalone deployment (manual install) even when using an external configuration directory - **FIXED** PP-1063 - Export2CSV now uses appropriate implicit timestamp when switching between indexes patterns. - **FIXED** PP-793 - Export2CSV now supports outputing explictely in the request the implicit timestamp column of the search - **IMPROVED** PP-1067 - Add a bolt to verify the sequence of received logs - **IMPROVED** PP-1031 - More user-friendly command-line help on punchplatform-shiva.sh and punchplatform-objects-storage.sh - **IMPROVED** PP-818 - Improve punchplatform-objects-storage.sh batches-tasks-list to retrieve es cluster data from punchplatform.properties and cluster id - **IMPROVED** PP-749 - Provided 'HOWTO reinject documents/logs from archive' with associated kibana resources and standalone sample - **FIXED** #964 - more user-friendly PunchPlatform admin service error message when 'punchplatform_admin' section is missing from punchplatform.properties - **NEW** #961 - Support ES nodes tagging using node.box_type ES parameter - **IMPROVED** #960 - Add an optional section in application deployment sections to configure supervisord logrotate - **FIXED** #955 - Ceph admin node deployment fails if ceph is deployed alone - **FIXED** #944 - Fixed slow unarchiving/archive housekeeping due to bad internal usage of archiving indexation structures. - **IMPROVED** #942 - make the lumberjack netty server able to cope with buggy logstash senders - **FIXED** #919 - Archive processor Bolt does not support replay of files without header." - **FIXED** PP-748 / #202 - Archive processor Bolt now supports replay of files without header. - **FIXED** #892 - FilterBolt : wrong parsing of configuration for "exclude_substring" setting - **NEW** #868 - Allow batch signature/digest sending by FileBolt (to enable later data integrity checks of archived data). - **NEW** #888 - Add scripts to request ES for EPS metrics and to transform the result in CSV format - **NEW** #887 - EPS max metric - **IMPROVED** #859 - Add new parameters in Pygregator: safety delay, term_or_value filter, exists filter and query_string for flexible queries Avishai v3.3.5 Release Changes ============================== - **FIXED** #825 - wrong casting between Integer vs Long and Float versus Double - **FIXED** #930 - Pygregator time frames computing wrong - **FIXED** #918 - Unable to unarchive empty raw logs with Archive propcessor bolt - **FIXED** #912 - Pygregator results are wrong in some conditions (range in term buckets) - **FIXED** #870 - nullPointer exception in FileBolt when no report tuple emission is configured - **FIXED** #858 - Pygregator doesn't return aggregations results when one or several specified fields are missing - **FIXED** #856 - Pygregator Getting started doesn't work - **FIXED** #855 - Pygregator: force_replay option not working outside frames - **FIXED** #860 - Wrong Dependencies (task-scheduler) injected directly in Storm - **FIXED** #836 - Shiva where are the logs in standalone ? - **FIXED** #832 - Shiva unable to stop a process - **FIXED** #833 - Shiva java.lang.NullPointerException - **FIXED** #835 - Shiva autocompletion isn't working - **FIXED** #826 - Lack of information about 'failed to send invalid messages to error index' log - **FIXED** #867 - Incorrect MD5 sum in md5 files output by FileBolt - **NEW** #925 - Add pipeline aggregations in pygregator - **NEW** #868 - Allow batch signature/digest sending by FileBolt (to enable later data integrity checks of archived data). - **NEW** #851 - Spark Metrics - **NEW** #904 - PML - Standalone Getting Started - **NEW** #846 - PML (PunchPlatform Machine-Learning) - Spark machine-learning library MLlib integration - **NEW** #839 - added duplicate method to type Tuple to provide punchlets with a deep copy semantics - **NEW** #842 - add an optional exit action to lumberjack spout should clients timeouts - **NEW** #845 - add support for lumberjack keep alives from the spout to the bolt - **IMPROVED** #914 - Move Pygregator in pp-core - **IMPROVED** #841 - Install.sh add possibility to set a specific hostname with option --hostname - **IMPROVED** #862 - Add tag with aggregation of task id and custom tag - **IMPROVED** #861 - Add tag --no-pygregator for installation script - **IMPROVED** #840 - Metrics : add a periodic tuple.ack metric "tick.tuple.ack" - **IMPROVED** #838 - Improve Archive Processor Bolt to merge data in case of multiple files - **IMPROVED** #829 - Minor Upgrade elasticsearch (2.4.0 to 2.4.6) and kibana (4.6.1 to 4.6.5) - **IMPROVED** #830 - Documentation structure improvment - **IMPROVED** #831 - Add EquipmentsSupervisionRule to Elastalert - **IMPROVED** #834 - Reduce metricbeat logging on standalone - **IMPROVED** #837 - Pygregator release 1.0.10 with the es output format indices-YYYY.MM.DD - **IMPROVED** #843 - Minor upgrade of Kibana for security reason to 4.6.6 - **IMPROVED** #844 - Enable kafka compression by default, disable the topic auto creation - **IMPROVED** #890 - Add metricbeat System module documentation - **IMPROVED** #913 - add ignore_above settings in mapping_events to be more efficient - **IMPROVED** #859 - Add new parameters in Pygregator: safety delay, term_or_value filter, exists filter and query_string for flexible queries - **IMPROVED** #857 - Add multiple metrics aggregations support into Pygregator Avishai v3.3.4 Release Changes ============================== - **FIXED** #821 - Elasticsearch modsecurity deployment regression with aliases - **FIXED** #822 - Add compatibility to single zookeeper deployment and multi Avishai v3.3.3 Release Changes ============================== - **FIXED** #814 - Shiva : in memory archiving for arguments - **FIXED** #813 - Filtered aggregations fail - **NEW** #736 - Shiva tasks manager is now available for distributed/resilient running of services or periodic tasks, both for PunchPlatform monitoring/housekeeping purpose, and also as part of PunchPlatform channels. - **IMPROVED** #807 - Homogenous command logs - **FIXED** #806 - [Standalone] Non-JSON standalone-generated punchplatform.properties - **FIXED** #805 - Deployment of Admin-service : PUNCHPLATFORM_CONF_DIR is not correctly set - **FIXED** #804 - Ceph deployment broken: client keyring path not overriden in ceph conf files - **IMPROVED** #803 - Zookeeper : start is too slow - **FIXED** #802 - Deactivate Filebeat by default in standalone - **FIXED** #797 - Admin : --force option for channel configuration doesn't work anymore - **FIXED** #799 - Archiving HouseKeeping Service Logging Problem - **FIXED** #815 - punchplatform-elasticsearch.sh do not use the max memory defined in punchplatform.properties on standalone - **FIXED** #816 - Define settings in punchplatform-deployment.settings to configure jam size of storm components - **BUG** #795 - Lack of explanations for 3.3.2 Elasticsearch new features - **BUG** #817 - Multi zookeeper instances on one device - **BUG** #818 - Multi elastalert deployment - **IMPROVED** #819 - Improve metricbeat logging - **NEW** #820 - windowed census - **FIXED** #798 - Pygregator fails when a periodic aggregation is executed Avishai v3.3.2 Release Changes ============================== - **FIXED** #794 - Pygregator fails when a nested bucket aggregation is executed - **FIXED** #790 - Remove cloning in JSONSettingsMap and JsonSettingsList constructor method - **FIXED** #646 - When an elasticsearch mapping template is not correct JSON, no clear error message in admin service GUI - **IMPROVED** #791 - Add new settings for elasticsearch to optimize recovery - **IMPROVED** #792 - Increase the stopwaitsecs for elasticsearch nodes in supervisord - **FIXED** #789 - Pygregator : replace tab by spaces, line 69 of config.yml Avishai v3.3.1 Release Changes ============================== - **IMPROVED** #788 - Admin-service : insertion of Elasticsearch templates is very slow (> 50s) - **IMPROVED** #786 - Limit the Kafka and Elasticsearch memory consumption in standalone - **FIXED** #785 - Admin-service : KafkaConsumer memory consumption - **IMPROVED** #778 - SSL LumberjackSpout Documentation - **NEW** #775 - Low finite operation duration for standalone release - **NEW** #779 - Supervision : New module with cruncher - **FIXED** #758 - Spark API Massive Logging - **FIXED** #755 - Downgrade apache storm to 1.0.2 - **FIXED** #768 - Update Elastalert to 0.1.16 - **FIXED** #774 - Fixed PYYAML dependency packaging (was not correct for some hos environments) - **FIXED** #766 - LumberjackBolt doesn't take into account ssl configuration - **FIXED** #765 - Zookeeper uses too much memory - **FIXED** #760 - Admin-service : channel health is green but one the path latency is red, channel start timestamp - **FIXED** #769 - Spark API Low Kafka Injection - **IMPROVED** #777 - Upgrade Grafana to 4.4.1 - **FIXED** #780 - Unable to install punchplatform-admin python - **IMPROVED** #781 - Upgrade metricbeat to 5.5.0 - **IMPROVED** #782 - Upgrade zeppelin to 0.7.2 only for standalone - **IMPROVED** #783 - Add ansible_inventory_settings to punchplatform-deployment.settings - **FIXED** #776 - do not dump jvm in case of OOM Avishai v3.3.0 Release Changes ============================== - **IMPROVED** #755 - Upgrade apache storm to 1.0.3 - **IMPROVED** #754 - Improve filebeat integration in punchplatform-standalone Avishai v3.2.11 Release Changes =============================== - **IMPROVED** #756 - Upgrade spark to 2.0.1 and minor corrections Avishai v3.2.10 Release Changes =============================== - **IMPROVED** #739 - Warning in case of dangerous implicit error publication configuration + setting to allow custom publication to a selection of streams - **FIXED** #744 - Punchplatform-channel configure option not operational - **FIXED** #743 - Extract2Csv plugin does not handle aliases - **FIXED** #741 - Bug in Job command when channel starts - **IMPROVED** #740 - Always write date on 2 digits (months, days, ...) in paths (ContentsFiles publication action in FileBolt) - **FIXED** #738 - Admin-service uses 2G of memory but Xmx is 256m - **NEW** #733 - support for live extension of CEPH objects storage/archiving cluster by node addition - **IMPROVED** #742 - Spark-API, add batch mode Avishai v3.2.9 Release Changes ============================== - **FIXED** #732 BASH completion fails for Linux - **FIXED** #731 Admin-service : start/stop/reload instabilities - **FIXED** #730 expiration tuples does not allow a punchless to generate forwarded tuples - **IMPROVED** Kafka compression is now supported (includes a fix for out-of sequence offsets of messages) - **IMPROVED** #729 - Metrics reporter : possibility to declared multiple nodes for Elasticsearch reporter - **FIXED** #721 - PunchPlatform admin logging : add 'maxFileSize' property Avishai v3.2.8 Release Changes ============================== - **IMPROVED** #728 - Admin service : possibility to deactivate the Storm request for calculation of channel health - **FIXED** #727 - FilterBolt shall publish _ppf_metric_logs - **IMPROVED** #724 - Admin service : Start/Stop/Reload through REST API - **FIXED** #723 - Admin service : Multiple thread of the same service are alive (Archiving service) - **FIXED** #716 - ArchiveProcessorBolt does not work with deciphering enabled in cluster mode - **FIXED** #722 - Allow the same topology to be started to different clusters Avishai v3.2.7 Release Changes ============================== - **FIXED** #713 - Ceph systemd configuration - **NEW** #711 - New ArchiveProcessorBolt publication action : tuples emission (for Kafka/Elasticsearch/whatever injection) Avishai v3.2.6 Release Changes ============================== - **NEW** #712 - External UDP enabler/disabler of ElasticsearchSpout and KafkaSpout for custom backpressure scenarios - **NEW** #700 - Replay from a timestamp feature : 'from' configuration for kafkaSpout - **NEW** #695 - Add KafkaRange library. Provides methods to convert Kafka offset to timestamp and vice versa. - **NEW** #694 - Add spark to PunchPlatform deployer - **NEW** #688 - Spark API Variable Transformation - **NEW** #687 - Spark API Variable Input Configuration - **NEW** #680 - Spark API Invariant Transformation implementation - **NEW** #682 - Admin-service : metrics reporter for kafka and channel with admin grafana dashboard - **NEW** #681 - Add Apache Cassandra to PunchPlatform deployer - **NEW** #694 - Add Apache Spark to PunchPlatform deployer - **NEW** #697 - Add support Ansible 2.3 to PunchPlatform deployer - **NEW** #698 - Add punchplatform-setup-kibana.sh to load dashboard - **IMPROVED** #701 - default Housekeeping frequency is now 1 minute and not 20 minutes + it is now not needed to provide parameter 'archive_max_age_in_days' in punchplatform.properties when archivehiving is not configured - **IMPROVED** #703 - Archiving scalablity through concurrent write support (multiple file bolts) - **IMPROVED** #708 - Add master_role type to elasticsearch node function - **IMPROVED** #690 - Platform identifier attribute in punchplatform.properties - **FIXED** #678 - ArchiveProcessorBolt now functional ((un-)compression, en/de-ciphering, keystores, errors reporting, doc) - **FIXED** #710 - Fixed Filter Bolt (High CPU due to bad acknowledgemnt order) - **FIXED** #709 - Fixed Kafka exception hiding by nullpointer exception in KafkaBolt - **FIXED** #707 - If es client of metric reporter has a wrong url then the topology must still start - **FIXED** #706 - LTR latency metrics generates error on lmr_in component - **FIXED** #699 - FileSpout doesn't generate autotest latencies - **FIXED** #696 - 'man punchplatform-synchronization.sh' doesn't work - **FIXED** #691 - Admin service doesn't start if we make a link punchplatform.properties to an external file and without -pp args in start command - **FIXED** #689 - FileBolt does not write data with filesystem publication - **FIXED** #702 - toFlatTuple operator exposed as toFlatUple - **FIXED** #683 - Doc packaging with sphinx : it was included twice (target dir), reduce generation by two times Avishai v3.2.5 Release Changes ============================== - **NEW** #679 - K-Means integration with Spark API (standalone) - **NEW** #677 - Java API integration making easy conversion from text to numeric (for analytics features) - **NEW** #676 - Integration of Spark-Streaming in PunchPlatform channels (standalone) - **NEW** #656 - FilesReaderBolt for tuples extraction/injection from files + improvements of ElasticsearchBolt to allow tasks management for long-duration resilient jobs - **IMPROVED** #674 - Admin-service : for kafka, display the insertion timestamp associated to an offset - **IMPROVED** #673 - Ciphering : use KeyStore instead of x509 files to manage keys - **IMPROVED** #670 - Upgrade metricbeat&filebeat from 5.2.0 to 5.3.0 - **IMPROVED** #664 - multi-level templating in punchplatform-channel.sh command - **IMPROVED** #653 - Direct buffers (FileBolt publication methods) - applied to other file publication actions (file, md5) - **IMPROVED** #650 - Fast-forward of Elasticsearch Spout when no documents in current extraction timeslice to next event in Elasticsearch - **IMPROVED** #505 - Automatic housekeeping of archive topics - **IMPROVED** #500 - Archiving documentation - **FIXED** #675 - no more Elasticsearch bolt crash if multiple additional fields configured, or if additional field is of numeric type - **FIXED** #669 - Update elastalert version to 0.1.8 with elasticsearch 2.X dependencies - **FIXED** #668 - ./apache_httpd_perf.sh --punchlet gives a missing class Java error - **FIXED** #667 - punchplatform-channel.sh --start doesn't failed if the topology file doesn't exist - **FIXED** #665 - --show argument on a punchplatform-channel command doesn't work - **FIXED** #661 - Empty stored files in CEPH - **FIXED** #660 - nullPointer exception in FileBolt if null value present in input tuple - **FIXED** #634 - Operator deployment regression - **FIXED** #633 - Elasticsearch bind address regression on deployer Avishai v3.2.4 Release Changes ============================== - **FIXED** #648 - Watchdog should kill process but doesn't - **IMPROVED** #652 - Update export2csv kibana plugin - **IMPROVED** #653 - Direct buffers (FileBolt publication methods) - first step Avishai v3.2.3 Release Changes ============================== - **NEW** #501 - storing in archiving system of "parsing error" documents for later retrieval - **IMPROVED** #645 - Add possibility to submit a light jar : --without-dependencies args for punchplatform-channel.sh - **IMPROVED** #647 - Reduced frequency of zookeeper commits in Kafka Spout - **FIXED** #643 - Admin-service : curator lock timeout is not configurable and by default set to 1 second - **FIXED** #642 - if punchplatform-channel.sh --repair --start failed for one topology then the topologies of the channel are stopped - **FIXED** #637 - Faulty batch commit algorithm in Partition Manager - **FIXED** #501 - Check and manages store/retrieval in CEPH of "parsing error" documents Avishai v3.2.2 Release Changes ============================== - **FIXED** #641 - log-balancing bug in kafka bolt (only one partition used) Avishai v3.2.1 Release Changes ============================== - **NEW** #609: Filesystem-backed objects storage (for archiving on NFS) - supports concurrent writing even to a same objects pool topic/partition - **NEW** #611 - Archive processor bolt to allow actions on archived files (extract, copy to other archive) - **NEW** #593 Fast extraction of raw log compressed files from CEPH - **FIXED** #632 - punchplatform-admin --show gives "PARTIAL" result with the start command - **IMPROVED** #638 - Metrics are using too much disk space : avoid using _source field - **IMPROVED** #631 -'Single Thread' admin-service to multithread (one thread by service) - **FIXED** #620 - Unwanted Archive overwrite in archive space in case of Zookeeper data loss due to Kafka batch Id regeneration to 1 - **FIXED** #628 - Add timeout for elasticsearch request for the channel health service (10 seconds) - **FIXED** #626 - If the "Patch" directory is empty then the user obtains an "ambiguous topology jars" - **FIXED** #624 - Punchplatform-channel.sh doesn't take into account parameter '-timeout xx' - **FIXED** #623 - Admin service should request elasticsearch on cluster_admin_url - **FIXED** #621 - filter bolt default stream regression - **FIXED** #625 - Slow archiving to CEPH and high memory consumption of File Bolt - **FIXED** #627 - the patching of the platform using the patch directory is broken - **FIXED** #617 - CEPH deployer parametrization of PGs and Resilience + prerequisites documentation - **FIXED** #640 - dashboard push through punchplatform-setup-grafana.sh now supports both "saved by hand" and "pulled" dashboard format Avishai v3.2.0 Release changes ============================== - **IMPROVED** #509 - New Java Admin integrating CEPH and METRICS monitoring - **IMPROVED** #584 - New GUI integration ******************* Art Release Changes ******************* Art v3.0.24 Release Changes =========================== - **FIXED** #588 - Unwanted "logger" output on console for jobs CLI - **FIXED** #597 - Offset committing problem in transactional kafka spout when errors documents are present in topic - **FIXED** #610 - Reduce memory consumption of objects-storage layer - **FIXED** #526 - ceph deployment: osd partitions have NOT to be XFS-formatted - **FIXED** #511 - Filesystem and MD5 publication (FileBolt) doesn't work (fail during topology launching) - **FIXED** #510 - Cephbeat fails when it's deployed on admin machines - **FIXED** #525 - Cephbeat does not log in appropriate folder - **NEW** #502 - Geneneric dashboard for CEPH - **NEW** #498 - add automatic multi-pools support in cephbeat - **IMPROVED** #497 - replace ceph config file arg by ceph cluster name - **IMPROVED** #600 - Add filebeat to deployer - **IMPROVED** #582 - Better error message when connection to CEPH fails Art v3.0.23 Release Changes =========================== - **NEW** #594 - additional root fields can be added by Elasticsearch bolt from storm fields, both for normal documents and errors - **FIXED** #595 - logs repeated at slow traffic because of timeout queue of KafkaBolt - **FIXED** #432 - Custom metrics tags (implemented them at storm spout/bolt settings level again, and not only at topology settings level) - **FIXED** #512 - install.sh can now again be launched out of install.sh directory - **FIXED** #581 - too verbose punchplatform-channel.sh --start + SLF4J warning in command lines (Object storage, jobs) - **FIXED** #512 - standalone install.sh not working propery (storm.yaml) when not run locally - **IMPROVED** #596 - kafka spout watchdog timeout set by default to 2 minutes - **IMPROVED** #580 - Update grafana version to 4.1.1 - **IMPROVED** #583 - Deployment refactor - **IMPROVED** #601 - Upgrade metricbeat to 5.2.0 - **NEW** #591 KafkaBolt can publish data to third party brokers - **NEW** #579 KafkaBolt can publish plain JSON - **NEW** #590 Punch resource files can be stored under tenant or channel sub-folders - **NEW** #602 add geoip support on full deployed platform - **MERGED** Ahmad v2.1.22 Art v3.0.21 Release Changes =========================== - **FIXED** #507 RythmerBolt correctly registered as default static bolt - **NEW** #508 added Mistral Sophos Sourcefire and Stonesoft parsers Art v3.0.20 Release Changes =========================== - **FIXED** #486 uncatched exception in kafka spout can lead to a zombie reader - **FIXED** #470 Mlockall Memory Elasticsearch configuration issue - **NEW** #312-added ceph cluster deployment for objects storage + punchplatform-objects-storage.sh + File bolt capability to write into CEPH + Tasks List Spout capable of reading tuples from CEPH objects + sample tenant/topologies to store/extract tuples in CEPH + punchplatform-jobs.sh to list/clear jobs list + update of administration service to display jobs and allow to clear them. - **IMPROVED** #448-removed standalone management commands on cluster admin node in cluster deployer - **IMPROVED** #482 configurable queue size for TCP netty client, hence syslog bolts - **NEW** #484 Punch ipmatch operator for black and whitelisting ips - **NEW** #483 filter bolt, for deploying high performance filtering in LTRs Art v3.0.19 Release Changes =========================== - **IMPROVED** #447 Bower and nodes dependencies as maven external artifacts - **IMPROVED** #452 added 'punchplatform_conf_repo_git_url' deployment setting for customization of configuration branch to deploy (default is 'production') - **IMPROVED** #463 Upgrade metricbeat to 5.0.1 - **IMPROVED** #453 added 'bind_address' optional setting to elasticsearch nodes in punchplatform.properties, to allow listening onto specific addresses (for example 0.0.0.0 when VIP-listening is required) - **IMPROVED** #462 added a syslog_udp_bolt to the punchplatform - **IMPROVED** #464 refactor punchplatform-admin-user deployment for ltr use case - **IMPROVED** #466 Active/Passive grafana is needed when synchronizing dashboards through DRBD - **FIXED** #451 Elasticsearch spout component should publish the field _ppf_metric_logs - **FIXED** #405 Metricbeat always installed on linux, regardless of --no-metricbeat Art v3.0.18 Release Changes =========================== - **IMPROVED** #231/#432 - Allow additional custom metrics tags in spouts/bolts metrics context, by user configuration in bolts/spouts settings - **FIXED** #444 - reduced log level from warning to information level for 'added Kafka broker address' message popping when using punchplatform-channel and punchplatform-topology commands Art v3.0.17 Release Changes =========================== - **FIXED** #236 Updated storm options in punchplatform-topology to avoid message "Storm : Using deprecated config nimbus.host for backward compatibility". Art v3.0.16 Release Changes =========================== - **FIXED** #442 Wrong value of kafka message-backlog in Grafana - **FIXED** #441 - Sometimes there are no metrics in Grafana and ES Art v3.0.15 Release Changes =========================== - **IMPROVED** #396 - Improve punchplatform-metricbeat.sh - **FIXED** #429 - Log reporter for metrics now reports autotest latencies as non-errors and also allows tags values reporting - **IMPROVED** #428 - Elasticsearch metrics reporter now stores metric name as document type in Elasticsearch, allowing specific mapping (especially specific TTL settings for automatic metric purge) - **IMPROVED** #427 - Elasticsearch metrics reporter now generates metrics index with date suffix, allowing automatic housekeeping by PunchPlatform admin service - **FIXED** #420 - Third party topologies fail to start Art v3.0.14 Release Changes =========================== - **NEW** #410 - Integration of Zeppelin in standalone - **NEW** #409 - a new Punch 'containsOneOf' operator is available - **FIXED** #412 - openjdk8 set default alternatives ubuntu - **FIXED** #24 - compatibility with external Openjdk deployment - **IMPROVED** #385 - punchplatform-channel.sh now checks json of generated files when using --configure - **IMPROVED** #414 - Deduplication of error logs (as warning) when configuration is repeatedly unavailable (to allow for DRDB providing git repository without admin logs flooding on slave node) - **IMPROVED** #380 - Added Elastalert to the standalone Art v3.0.13 Release Changes =========================== - **FIXED** #407 metricbeat template synchronisation fails in PunchPlatform admin python - **IMPROVED** #395 - Automatic json check on channel generated files - **IMPROVED** #406 log rotate on elasticsearch logs - **IMPROVED** #397 log rotate on kafka logs - **IMPROVED** #388 kafka spout should read JSON Art v3.0.12 Release Changes =========================== - **FIXED** #401 kafkabolt can be stuck forever - **IMPROVED** #398 Punchlet errors are dispatched potentially on all error streams and not specifically on the error stream associated to the stream which has raised the error - **IMPROVED** #395 Automatic json check on channel generated files Art v3.0.11 Release Changes =========================== - **FIXED** #379 Os_system dashboard on grafana doesn't work properly with metricbeat - **IMPROVED** #376 replace topbeat/collectd by metricbeat - **IMPROVED** #373 Add a keepalive for lumberjack connection - **FIXED** #369 In ElasticConsumer, the 'refresh' thread may be stopped if the coordinator is null - **FIXED** #392 kafkabolt load balancing is now plain round robin - **FIXED** #393 log injector fail to handle several injection files Art v3.0.10 Release Changes =========================== - **FIXED** #364 Control of the Double.NaN in the metrics value - **FIXED** #363 Initialization of shared curator beetween ElasticConsumers : no zookeeper coordinator connected - **FIXED** #365 Added 'recv_buffer_size' setting to syslog spout in order to actually size reception buffer - this fixes wrong (too small) reception buffer size that may cause more packet loss than usual (for UDP) - **FIXED** #366 - counter metrics (like uptime) are not reported - **IMPROVED** #361 Kill topologies after only 5 seconds by default Art v3.0.9 Release Changes ========================== - **IMPROVED** #339 Update storm to 1.0.2 - **FIXED** #345 NullPointerException in KafkaBolt if the latency metrics are not availables Art v3.0.8 Release Changes ========================== - **IMPROVED** #344 LMR and LTR configuration - **IMPROVED** #343 Added per stream options to both syslog and lumberjack bolts Art v3.0.7 Release Changes ========================== - **FIXED** #342 The health of a channel should be Green if it is active and has no autotests defined. Art v3.0.6 Release Changes ========================== - **IMPROVED** #336 Refactor metrics : no more propagation/proxy and improve conception - **IMPROVED** #334 Upgrade grafana from 3.0.0beta to 3.1.1 only for linux platform - **IMPROVED** #333 Reduce the logging of modsecurity - **FIXED** #332 Kibana & Grafana slowness when not connected to internet - **FIXED** #328 Conflict of metrics name if there is multiple destinations for lumberjackBolt Art v3.0.5 Release Changes ========================== - **FIXED** #317 Wrong initialization of health variable for channel structure autotest control. - **FIXED** #269 storm logging configuration that had all workers write in the single 'worker.xml' file - **IMPROVED** #319 Add tempo metrics counter in syslogSpout which gives a reference for metrics propagation (only if log level is trace) Art v3.0.4 Release Changes ========================== - **FIXED** #308 Health of channel is red if the topology files haven't metrics/metrics_source property - **FIXED** #296 Kibana VIP are wrongly deployed also on LMC admin nodes - **FIXED** #295 SyslogBolt not acking tuples - **FIXED** #294 Removed supervisord from pacemaker control scope on admin/nimbus node because nimbus is now active-active Art v3.0.3 Release Changes ========================== - **IMPROVED** #267 The autotest_latency can be configured directly in the topology files - **IMPROVED** #266 Add the possibility to configure multiple reporters of the same type (elasticsearch) - **FIXED** #274 The tenant health is GREEN when the channels has an UNKNOWN health - **FIXED** #272 Documentation of metrics with submetric types - **FIXED** #255 In proxy mode, some metrics are not propagated : "Unable to retrieve context" - **FIXED** #277 Syslog Bolt does not start when multiple destination are configured - **FIXED** #275 Support for uppercase in storm nodes hostnames (induced wrong metrics names including latency autotest metrics) Art v3.0.2 Release Changes ========================== - **IMPROVED** #247 Spark version 1.6.0 to 1.6.1 - **FIXED** #250 Remove bulk request for metrics and replace with specific requests for each channel. - **FIXED** #245 Metrics proxying is not transparent for multi-values metrics sent to multi-Elasticsearch backend - **FIXED** #228 Update all sample/test configuration of STORM to reflect new multi-master feature of STORM V1.0.1 - **FIXED** #220 Virtualenv not activated on new Mac standalone deployment Art v3.0.1 Release Changes ========================== - **FIXED** #249 Add zookeeper property in kafka custom configuration Art v3.0.0 Release Changes ========================== - **IMPROVED** #238 Reduced amount of error traces when python api service get errors during forwarding of API calls to Java admin service - **IMPROVED** #233 Update zookeeper version 3.4.6 to 3.4.8 - **IMPROVED** #226 Add the possibility to configure the elasticsearch client for metrics. - **IMPROVED** #192 Added a regex filter - **IMPROVED** #190 Added the possibility to choose the delimiter for metrics. - **IMPROVED** #189 Added Grafana dashboards for selfmonitoring metrics - **IMPROVED** #187 Added Grafana dashboards for the classic metrics (not selfmonitoring metrics) from ES. - **NEW** #218 move to Storm 1.0.1 - **NEW** #211 Store metrics in ES a structured field to allow completion in grafana and metric-specific mappings - **NEW** #186 Possibility to calculate the health of channel from metrics of ES - **NEW** #131 new GraphiteReporter with the possibility to choose between Graphite and ES for all metrics - **NEW** #179 Ahmad is not Art - **NEW** #162 Add Flink in pp-standalone - **NEW** #81 Kibana 4 for clients - **FIXED** #246 Fixed lumberjack multithreading corruption - **FIXED** #240 Log the metric if the context is null and it is not send. - **FIXED** #237 Date retrieval/parsing error when admin service sycnchronizes elasticsearch templates - **FIXED** #132 Standalone installation failed with the new version of virtualenv (MacOS) - **FIXED** #74 Check if _source storage is still needed with kibana 4 - **FIXED** #62 Tweak ES required eligible master quorum to prevent split brain - **FIXED** #24 Oracle JDK not delivered as part of standalone. **************************** Ahmad Release Changes **************************** Ahmad v2.1.22 Release Changes ============================= - **FIXED** #598 escaping problems when handling Punch runtime error data Ahmad v2.1.21 Release Changes ============================= - **FIXED** #403 PunchPlatform Admin Channel Tab : no metric from graphite - **IMPROVED** #402 Documented parameter 'punchplatform_admin.max_monitored_kafka_topics_frames_size_in_bytes" that must be used to cope with "consumerfetchsizetoosmall" exception in PunchPlatform self monitoring service. - **IMPROVED** #397 Log4j configuration to rotate kafka logs - **IMPROVED** #406 Log4j configuration to rotate elasticsearch logs - **IMPROVED** #461 Elasticsearch spout extraction timeslice width is now customizable for faster small extractions Ahmad v2.1.20 Release Changes ============================= - **IMPROVED** #394 upgrade grafana from 1.9.0 to 3.1.1 - **FIXED** #370 kibana & grafana in https - **FIXED** #227 LMC log in partition /var/log Ahmad v2.1.19 Release Changes ============================= - **FIXED** #341 lumberjack client fail to ack some tuple under high load - **FIXED** #340 syslog bolt should not close to quickly its socket to slow peer - **FIXED** #338 Punchplatform-admin-python : division of None by int if the age of an index cannot be calculated - **FIXED** #337 Punchplatform-admin doesn't show channels - **IMPROVED** #340 Punchplatform-admin-python : syslog bolt keep up longer with slow servers. Ahmad v2.1.18 Release Changes ============================= - **FIXED** #331 Punchplatform-admin, wrong age for es indexes - **FIXED** #329 Punchplatform-admin-java uses too much thread Ahmad v2.1.17 Release Changes ============================= - **NEW** #313 Add ioc example in standalone - **IMPROVED** #318 Add the possibility to change the name of the optional fields (local_host, local_port, etc.) by configuration in spout_settings - **IMPROVED** #314 Add filter in jinja template for environment variable 'env_override' - **FIXED** #315 Syslog spout: when listening in UDP, logs seems to be truncated after 2048 characters - **FIXED** #310 Overfly readibility on latency indicator square - **FIXED** #322 Kafka Consumer does not consume correctly out of order batches Ahmad v2.1.16 Release Changes ============================= - **FIXED** #307 Punch escaping of KV operator doesn't work if the field end with '\' - **FIXED** #304 ConcurrentModificationException in OffsetTree class - **FIXED** #302 Even if the propagation is disabled, a proxy reporter is instantiated -> out of memory ! - **FIXED** #301 The prefix "punchplatform" is hard coded for autotests - **FIXED** #298 Remove tracermetrics.punch and concurrency problems for autotests - **FIXED** #297 Kafka Spout fail if its autotest fail Ahmad v2.1.15 Release Changes ============================= - **FIXED** #289 Kafka consumer can fail when Zookeeper is unavailable - **FIXED** #290 KV and CSV fail with inner quotes. - **IMPROVED** #288 Replace localOrShuffleGrouping by shuffleGrouping for metrics topology - **IMPROVED** #292 KafkaSpout traffic decreasing is now optional and configurable Ahmad v2.1.14 Release Changes ============================= - **FIXED** #279 Unused channel persisted structure when stopping/restarting channel - **FIXED** #276 Infinite recursive call in TupleImple class (Punch) - **FIXED** #271 Wrong behaviour of Kafka Spout group (bad zookeeper Rule computing) - **FIXED** #254 Punchplatform-channel.sh --restart stops only the first channel - **FIXED** #225 Wrong behaviour of channel restart when channel structure has evolved Ahmad v2.1.13 Release Changes ============================= - **IMPROVED** #261 improves memory usage and data recopy. - **FIXED** #265 multi destination groups have poor performance when a destination is missing. Ahmad v2.1.11 Release Changes ============================= - **FIXED** #252 one log out of n is lost when autometrics are turned on. Regression of v2.1.9. - **FIXED** #253 LumberjackSpout overwrites the local_uuid and similar fields - **FIXED** #256 added option to disable escaping in the KV operator Ahmad v2.1.10 Release Changes ============================= - **NEW** #235 smart spout and bolt placement - **FIXED** #246 Lumberjack Spout packet corruption when more than one client is connected - **FIXED** #221 Lumberjack Spout must support uique id and timestamp - **FIXED** #138 With the search tools of sphinx documentation, we can retrieve the development doc in the packaging doc. - **FIXED** #89 Doc: duplicate title "HOWTO debug admin service python code" with different content - **FIXED** #79 Punch API documentation is not available through the UserGuid - **FIXED** #9 logs injected through punchplatform-log-injector.sh should not have user-locale-dependent date format. - **IMPROVED** #234 socket endpoints can be configured with a read_timeout_ms property Ahmad v2.1.9 Release Changes ============================ - **NEW** #216 multi destination group for geo-redundant/dual room setup - **NEW** #197 multiline codec now accepts a starts-with configuration. - **NEW** #194 XML codec available from FileSpout and sockets spouts - **IMPROVED** #188 All the metrics are propagated by the proxy mode - **IMPROVED** #176 addressed permission denied on graphite storage - **IMPROVED** #150 Support of "negative" latency due to Time deynchronization between LTR and LMC - **IMPROVED** #191 All the spouts have the possibility to launch metrics - **IMPROVED** #189 Removed the 'by component' latency health from the algorithm which calculates the health of a channel - **IMPROVED** #217 Socket Bolts configurable timeouts for initial connection to server(s) - **IMPROVED** #215 the file spout is capable of reemiting failed tuples - **IMPROVED** #200 Added a metrics section with property "propagation_enabled" in topology files - **FIXED** #206 The latencies of the path in the output toplogies are not displayed - **FIXED** #201 ssl certificate (client or server role) propagation - **FIXED** #193 remove the health per storm component indicator - **FIXED** #185 Java admin service returns empty channels when one of the tenants has no channels configured (unhandled exception in loop) - **FIXED** #183 punchplatform-channel.sh --force-local-version and --persist-structure not working - **FIXED** #87 Missing field from a tuple does not trigger a unit test failure - **FIXED** #56 per tenant retention configuration documentation Ahmad v2.1.8 Release Changes ============================ - **FIXED** #195 Memory leak for the selfmonitoring metrics - **FIXED** #196 Multiple sending of metrics (selfmonitoring metrics) to the reporters (graphite and slj4f) Ahmad v2.1.7 Release Changes ============================ - **FIXED** #184 PunchPlatform --reload doesn't persist the channel structure - Factory setting for latency metrics selection scope for channel health monitoring set to 10 minutes. Ahmad v2.1.6 Release Changes ============================ - **NEW** Feature #181: Reduced the number of metrics - **IMPROVED** #188 The timeout of graphite client must be increased in consistency with the cache configuration for heavy graphite latency (2 minutes). Ahmad v2.1.5 Release Changes ============================ - **NEW** Feature #178: Metrics must be injected by the syslogspout. Removes the injector spout "_tracer" (see Ahmad v2.1.3 below for impact on older version channels configuration.) - **FIXED** #177 Admin API and GUI freeze when graphite latency raises Ahmad v2.1.4 Release Changes ============================ - **NEW** Feature #174: FileBolt is available. It lets you dumps your logs on external storage in simple CSV raw format. - **IMPROVED** Feature #168: Add metrics to extractor spout Ahmad v2.1.3 Release Changes ============================ NEW FEATURES AND FIXES ---------------------- - **NEW** Feature #172: FailerBolt is available with various tuple failure mode - **NEW** Feature #151: Filtering request in elasticsearch spout - **FIXED** #171 Kafka offset disorder acknowledgements - **FIXED** #113 Too verbose punchplatform-channel.sh --start Migrating from Ahmad V2.1.2 --------------------------- Because of Feature #178 in progress (latency metrics simplification), in order to activate the metrics in this version (and all following versions), you have to add a new 'self_monitoring.activation' property in your topology file for each syslog_spout. .. code-block:: bash "spouts" : [ { "type" : "syslog_spout", [...] "spout_settings" : { [...] "self_monitoring.activation" : true, [...] } } ] The "self_monitoring.activation" allows the syslog_spout to inject a metric : autotest.[punchplatform_X].[tenant_X].[channel_X].[topology_X].[component_X].[punchplatform_Y].[tenant_Y].[channel_Y].[topology_Y].[component_Y] where X is the injector and Y the receiver which launches the latency metrics between the injection time and the current time. component_X is the name of the injector syslog_spout. (Before it was the name of the topology which injects the selfmonitoring metrics) The old section "self_monitoring" is not necessary in the topology files and will be ignored if present. Because of this change, all channel structure files defining latency monitoring paths have to be corrected : start points based on a component name identical to the topology name should be replaced by the component name of the first topology spout. This is required to provide health monitoring of latency in the channels tab of the admin service gui. Ahmad v2.1.2 Release Changes ============================ NEW FEATURES AND IMPROVEMENTS ----------------------------- - **NEW** Feature #12: Channels monitoring and autotesting do not rely on injector (needed for LTR V2) - **NEW** Feature #13: Channel health uses latency measure to provide more progressive status (YELLOW) Feature #34: Integration of the "integration test" artifact in automatic tests - **NEW** Feature #72: Elasticsearch Extract/Replay jobs - **NEW** Feature #99: [PUNCH] grok operator supports several patterns - **NEW** Feature #125: uuid and timestamp insertion without punchlets (performance optimization) - **NEW** the standalone support Spark for testing LMC platform openess to third party tools. - **Improvement** #119: Integration of bower/grunt in maven to complete automatic build - **Improvement** #128: Adapted deployer for LTR V2 - **Improvement** #129: Ansible used to deploy the new admin Java service (no firewall rules change needed) to prepare refactoring/maintainability of admin service - **Improvement** the KafkaSpout supports a batch transactional mode. - **Improvement** a new AggregatorBolt is available to perform various counts. - **NEW** log extract function included in kibana 3 - **IMPROVED** better documentation, performance tools delivered as part of the standalone. ISSUES SOLVED BY THIS RELEASE ----------------------------- - **FIXED** #116 Kafka offset documentation - **FIXED** Bug #71: Delete information about Security Center Client - **FIXED** Bug #86: CefOperator not usable in punchlet - **FIXED** Bug #93: Response time of Admin REST API degrading over time - **FIXED** Bug #94: Dependency instability of log (sl4fj, log4j, log4j2, logback) - **FIXED** Bug #98: Missing suggested listening in documentation - **FIXED** Bug #104: Metrics slow down topologies - **FIXED** Bug #105: Indexes in housekeeping error are filtered in admin GUI - **FIXED** Bug #106: Reactivation of archives housekeeping after end of failure does not work - **FIXED** Bug #108: Channel reload uses old channel configuration - **FIXED** Bug #112: Grouping declaration of punchlets results in bug in IHM Admin. - **FIXED** Bug #121: Demonstration graphite container in standalone distribution spawns multiple collectd processes - **FIXED** Bug #126: Graphite sender for the selfmonitoring metrics must be launched in new Thread - **FIXED** Bug #130: compression metrics are wrong - **FIXED** Bug #133: Standalone setup does not apply memory limit to elasticsearch - **FIXED** Bug #137: Suppression of the certificate and key when building the netty server. - **FIXED** Bug #158: Missing favicon.ico in PunchPlatform admin - **FIXED** Bug #146: With some settings, the restore snaphot function doesn't work - **FIXED** Bug #161: When archiving indexes, housekeeping health is non-nominal, or archiving is shifted by up to 23 hours Migrating from Ahmad V1 ----------------------- configuration changes ^^^^^^^^^^^^^^^^^^^^^ From Ahmad V1 or earlier LMC version, mandatory configuration changes are needed in - punchplatform.properties - punchplatform-deployment.settings - grafana dashboards Please refer to the following document in earlier PunchPlatform distributions : README.migration_1.1.4_TO_2.1.0.rst In addition, optional optimization can be used by changingle channels templates and configuration file for : - Reduction of CPU consumption and IOs at main Storm cluster level, through grouping and execution of multiple Punch files within a single Punch bolt in the parsing/filter topology - Reduction of CPU consumption at the input/LTR level through moving the input punchlet from the input topology to the parser topology first Punch bolet (new input punchlet based on the assignment of timestamp, unique id and input data to storm fields by the first LMC spout, instead of assigning them within the json document by an initial punchlet) - More efficient parsing/punch error handling by replacing explicit management of error logs in the main storm stream by using 'throw new PunchRuntimeException("message")' construct and leaving the stream management/log sending to elasticsearch to the LMC implicit error streams. technical components update ^^^^^^^^^^^^^^^^^^^^^^^^^^^ This release includes changes of LMC Administration/Monitoring service, associated Java libraries used to start/restart/reload log channels, and LMC deployers (standalone and cluster modes). Ahmad V2 relies on a **Required** upgraded version of Apache Storm : apache-storm-0.10.0 Although This release does not require an upgrade of Kafka framework, the default internal LMC frame format within Kafka is changed. For suggested migration principle in order to avoid mixing of old and new formats, and to ensure no log loss and reduced real-time down-time at Elasticsearch level during migration Known Issues in this release ---------------------------- Bug # 113 : too verbose punchplatform-channel.sh --start Bug #132 : Standalone installation fails with the new version of virtualenv (MacOS) Bug # 102 : Bad ES Health evaluation by HAPROXY when ES responding too slowly Documentation Bug #153 : Two setup cluster setup sections in documentation Documentation Bug #111 : PDF documentation generation errors Documentation Bug #89 : duplicate title "HOWTO debug admin service python code" with different contents Bug #15 : Completion not working for punchplatform-cluster.sh Bug #9 : Logs injector date format behaviour should not change depending on user locale Documentation Bug #138 : With the search tools of sphinx documentation, we can retrieve the development doc in the packaging doc. Ahmad v2.0.0 Release Changes ============================ - **IMPROVED** CidrMatch operator does not throw exception in case of empty or invalid input tuple - **NEW** the KafkaSpout can filter out PunchPlatform streams (error, metrics, autotest) by configuration - **NEW** FindByInterval operator supports result ordering, by smallest or largest interval range - **NEW** Version Management based on logical name (Ahmad) logical version (Ahmad-v1) and archives version number (v2.0.0). - **NEW** Topologies are visually graphed in the gui - **NEW** New auto metrics injected from within topologies, and reported to graphite along the way. Using this strategy the monitoring is always available even if the final topology or elasticsearch is unavailable. - **NEW** Fields are propagated from one topology to another using a lumberjack encoding. This result in a significant CPU improvement are no parsing is necessary to emit the right fields in the topology. - **IMPROVED** the reference time attached to a log can be set in a punchlet anywhere in the channel. No more configuration is required in the admin service in order to deduce the timestamp associated to a log stored in Kafka. - **NEW** Lumberjack protocol v2 support. This makes it possible to use Elasticsearch beats log injector to send their data to the PunchPlatform syslog bolt. Many beats are available including the packetbeats to capture traffic data on the wire. - **NEW** Compression is provided in Kafka topics. - **NEW** a new File spout is provided. It allows you to read one or several files, line by line, run an optional punchlet on each line, and ingest it in a topology. - **NEW** PunchPlatform Admin GUI and REST API has a new information in "kafka" section : in addition of the offsets of "oldest" and "newest" messages stored in each topic partition, the service also provides the input timestamp of the message (i.e. the time when the message first entered punchplatform). - **NEW** Houskeeping has a new "online with archive" retention phase : archiving can start while indexes are still online. Archives are then incrementally updated daily until end of online retention. This allows for handling late arrival of logs, while still having the benefit of the end-of-the-day archive (and available on the first day, regardless of the online retention period) - **NEW** Housekeeping enables tenant-specific retention phase settings - **IMPROVED** Housekeeping is now less verbose, and allows for more frequent housekeeper cycles (typically 1 cycle each minute) without flooding the admin service log. This improves system reaction time to user requests for reloading index archives. Activation requires configuration update, although the configuration change is backward compatible. - **NEW** All spouts accept a rate limiter configuration, allowing you to either limit the topology rate to a fixed value expressed in tuple emitted per seconds, or to only request the spout to slow down tuple ingestion should the topology experience an overload situqtion. This feqtures prevents your topologies to be overloqded and qt the end, shot down and restarted by the Storm supervisor. - **#48** **NEW** Metrics have been revamped and renamed so as to limit the number of new metrics sent to Graphite. Refer to the migration guide for adapting your Grafana dashboards - **NEW** more metrics are available in particular to monitor incoming and outgoing socket traffic and compression - **NEW** the Lumberjack compression can be set at socket level and/or at Lumberjack frame level. - **IMPROVED** the syslog bolt will not be affected by a slow remote peer when configured with the "drop_if_queue_full" property set to on. - **NEW** inner classes are allowed in Punchlet using the explicit Java API - **#96** **IMPROVED** Punch uuid - **#95** **IMPROVED** error message end-to-end trip - **#77** **IMPROVED** CPU usage for serializing JSON to Punch Tuple in one pass. - **IMPROVED** Lumberjack unit test - **#55** **FIXED** KV operator bug - **#41** **FIXED** CSV wrong error reporting - **#100** **FIXED** findByInterval and asArray should not return an empty list in case the Punch Tuple is empty - **#27** **FIXED** Undefined variable kibana_distrib for the role kibana-server - **#31** **FIXED** Graphite injection of self monitoring test metrics Ahmad v1.1.7 Release Changes ============================ - **IMPROVED** Documentation - **FIXED [SYN-3019]** Make the topology fail if a declared resource file is not found - **FIXED [SYN-3018]** ElasticsearchBolt supports Tuple pruning. - **FIXED [SYN-3017]** SyslogBolt supports Tuple pruning. - **FIXED [SYN-3016]** ElasticsearchBolt non acknowledgement of tuples in case of partial batch sending Ahmad v1.1.6 Release Changes ============================ - **IMPROVED** Documentation - **NEW** lumberjack based LTR. - **NEW** generic incoming and outgoing traffic summary metrics. - **FIXED [SYN-3015]** allow several syslog spouts in the same topology. - **FIXED [SYN-3014]** correct lookup strategy for loading json file resources. Ahmad v1.1.5 Release Changes ============================ -**IMPROVED** Documentation reworked to provide administrator and maintenance guide sections, including cluster deployment procedure, overview of administration/maintenance tools and metrics documentation Ahmad v1.1.4 Release Changes ============================ - **NEW** KafkaSpout only reemit failed tuples - **NEW** support localOrShuffle grouping storm strategy - **NEW** Punch getHostIp method favors IPV4 over IPV6 addresses - **NEW** Extended standard log parsers - **IMPROVED** parser SDK allow grok overloading and JSON resource files - **IMPROVED** parser SDK support a new less verbose test API - **FIXED [SYN-3013]** PunchRuntimeException makes the log indexed in the error Elasticsearch type. Ahmad v1.1.3 Release Changes ============================ - **NEW** Compatibility ensured to elasticsearch 1.5.2 - **NEW** EsperBolt for real time correlation - **IMPROVED** GUI made much more usable (thanks to Pierre for great contribution) - **NEW** Embedded Elasticsearch for document on the fly enrichment fully supported. Ahmad v1.1.2 Release Changes ============================ - **FIXED [SYN-3012]** conflicts when loading several resource json files with the same name length. - **IMPROVED** Syslog bolt can be configured with an adaptative overload control algorithm to prevent topology flooding. - **IMPROVED** More documentation, better structure - **NEW** NEW Punch CEF operator to extract arsight CEF messages and transform them into Punch Tuples. Ahmad v1.1.1 Release Changes ============================ - ** FEATURE [SYN-3155]** Elasticsearch mappings/index templates are now automatically sychronized/checked by admin service periodically, with tracking of GIT revision inside metadata available in Elasticsearch, and with synchronization details available in "Config" status page of Admin service. - **IMPROVED [SYN-3013]** TCP Syslog spout and Kafka Bolt now implement message acknowledgement and buffer/flow control in order to protect messages against destruction in case of failure in exchanges with Kafka and in order to prevent out of memory during input flooding - **IMPROVED [SYN-3150]** Kafka partition health now includes yellow state for underreplication. - **FIXED [SYN-3012]** IMplemented a defaulting insertion method in case of elasticsearch injection repeated rejection (for example, discrepancy between document and mapping) : errors are inserted in a default document type with raw, escaped document. - **FIXED [SYN-3274]** Kafka is now using production network to communicate with zookeeper, and between brokers - **FIXED [SYN3158]** Supervisord is now autostarted at boot on LTRs - **FIXED [SYN3157]** Apache2 is now fully under pacemaker control (if present) on graphite servers (therefore not autostarted on boot) -**IMPROVED [SYN-3031]** based on Kafka 0.8.2 and associated python client v0.9.3, YELLOW status is now provided for KAFKA topics in Admin API and HMI when failed brokers cause under-replication of partitions. - **NEW** multi-port injector enabling finer logs customization and defining injection session templates with multiple technologies. Ahmad v1.1.0 Release Changes ============================ -**DEPENDENCIES UPGRADES*** - Elasticsearch Version 1.4.4 solving througput performance issue when archiving indexes. - Kafka Version 0.8.2,**FIXING CRITICAL BUG [SYN-3150]*** - Kibana 3.1.2, bringing additional features -**FIXED [SYN-3151]** Admin service elasticsearch monitoring now reports individual elasticsearch clusters health even when one of them is unreachable, or has improperly configured/mounted archiving repository. Ahmad v1.0.0 Release Changes ============================ - **NEW** Administration Service includes elasticsearch data housekeeping (archiving/destroying of indexes depending on their age) - **NEW** Administration Service REST API and User Interface provide details on elasticsearch clusters and indexes health and status - **NEW** ElasticsearchBolt fully configurable. - **NEW** ElasticsearchBolt sends failed documents into an error index - **NEW** NEW naming using punchplatform/PunchPlatform. - **NEW** NEW PunchPlatform deployer, ansible-based allowing full automated deployment of a PunchPlatform cluster, including High-Availability and Tenants Isolation patterns, and metrics collection and graphing both at operating system and applicative levels - **NEW** load balancing for elasticsearch clusters implemented in punchplatform-cluster deployment through ansible- impact on punchplatform.properties files - **NEW** Git based configuration directory with automatic configuration synchronization and check by the PunchPlatform admin services - **NEW** trace/autotest message also reported by Kafka spout - **NEW** zen unicast discovery mode supported by Elasticsearch bolt. - **CHANGED** logger names consistent and documented - **CHANGED** elasticsearch cluster parameter renamed in elastic bolt settings (impacts on existing configured channels) - **NEW** Punch CSV operator handles single quote - **FIXED** Punch smaller/greater/smallerEq/greaterEq operators - **FIXED** Punch allows '-' in property name Ahmad v0.5.0 Release Changes ============================ - **NEW** support for multiple storm clusters (upgrade needed in punchplatform.properties) - **NEW** compatibility with ansible deployment - ** CHANGED** "install -a" mode is now available for setup with an existing configuration Ahmad v0.4.0 Release Changes ============================ - **NEW** Kafka status monitoring through PunchPlatform REST API, including under-replication alerts when kafka nodes are in non-nominal status but channels are still operationals. - **NEW** Nominal transport for elasticsearch client is "node" mode Ahmad v0.3.5 Release Changes ============================ -**NEW [SYN-3031]** PunchPlatform admin service REST API and associated GUI now report - detailed kafka cluster, kafka topics and partition offsets and - overall PunchPlatform health, with salient alerts displayed in header page - individual channels health, computed by automatic autotesting by PunchPlatform admin REST API service. -**FIXED [SYN-3016]** PunchPlatform admin service GUI does not expects internet access for CSS fetching anymore. -**FIXED** Punch divide operator -**NEW** autotest/trace supported for arbitrary topologies -**NEW** upgraded to elasticsearch 1.4.1 Ahmad v0.3.4 Release Changes ============================ -**NEW** upgraded storm to 0.9.3 -**NEW** all storm topologies are supported. This includes multi-stream and multi-fields groupings. -**NEW** Punch can drop Tuples, redirect them to various streams, add fields. -**NEW** Csv operator is smarter, and deals with escaping. -**NEW** redmine bugtrack system in place -**NEW** new Kafka supervision rest API and GUI. -**NEW** [SYN-3009] zsh support -**FIXED** trim for Csv operator. -**FIXED** Punch bolt Storm Tuple anchoring is effective. HOW TO UPGRADE FROM v0.3. :: - Patch PunchPlatform admin server(s) and servers running local-mode topologies. - unzip and install the new archive - No mandatory stop/restart or reload Checks : after the upgrade, run :: > punchplatform-channel.sh --autotest / Success means that your channel structure is correctly updated. .. rubric:: v0.3.3 -**IMPROVED** PunchPlatform shell script logs are now rotating to avoid ever-increasing size in case of use for automatic monitoring -**LIMITATION** incorrectly advertised as upgraded storm to 0.9.3 .. rubric:: v0.3.2 -**FIXED** slow Punch compilation with many grok patterns -**LIMITATION** incorrectly advertised as upgraded storm to 0.9.3 .. rubric:: v0.3.1 -**NEW** syslog bolt with acknowledgement capabilities. -**NEW** FindByKey operator capable of building index for innerfields -**FIXED** automatic auto test for supervision -**FIXED** admin REST API and GUI blocking -**FIXED** punchplatform-punch.sh v0.3.0 regression. .. rubric:: v0.3.0 -**IMPROVED** punchplatform-inject.sh can now inject traffic directly into kafka topics. Using the PunchPlatform format. -**IMPROVED** better error/warrning messages in case of unconfigured or misconfigured punchplatform\_admin section in PunchPlatform properties file -**IMPROVED** 'punchplatform-channel.sh --autotest' now works also for channels which input points are not syslog tcp listening "local-mode" topologies, but are instead kafka topic(s) -**IMPROVED** 'punchplatform-channel.sh --autotest' now works faster, independently on the ES refresh parameter, because one specific document is fetched by URL with id from ES instead of issuing a search request. -**IMPROVED** punchlet resource files loading more efficient, and compressed. -**FIXED** jvm metrics are now published to graphite with correct naming for "output" topologies using elasticsearch bolt -**FIXED** PunchPlatform admin/monitoring api service now logs into a rotating log to avoid file system fill-up .. rubric:: v0.2.10 -**KNOWN BUG** punchplatform-channel.sh --autotest does not work for channels without a syslog spout. -**IMPROVED** PunchPlatform admin service shows uptime of channel -**NEW** custo directory to make it easier to patch jars -**FIXED** key value operator. .. rubric:: v0.2.9 -**NEW**: non linear topology -**NEW**: metric punchlet API -**NEW**: PunchPlatform administration REST API and simple gui -**FIXED**: key value operator .. rubric:: v0.2.8 -**FIXED**: new parameter for last elasticsearch -**FIXED**: removed doc references to rivers -**NEW**: punchlet can be full Java classes .. rubric:: v0.2.7 -**NEW**: extensive Punch documentation addition. Delivered as generated Html from javadoc -**NEW**: elasticsearch 1.4 -**NEW**: Punch new lookupByKey works with multi index -**FIXED**: Punch operator is().in() was not working on sets -**FIXED**: Punch is now camelCase consistent -**IMPROVED**: Punch if () statements works with more complex statements -**FIXED** es\_id is now taken into account by elastic bolt -**PATCH** there is an embeded watchdog in kafka consumer metrics layer to kill kafka bolt jvm if metrics are not updated frequently enough (this is a patch while waiting correction of STORM hangs) ==> NIMBUS will restart topology element so as to ensure availability. .. rubric:: v0.2.6 -**FIXED**: escaping keys when using KV operator -**FIXED**: Settings enriched for kafka bolt, to allow for distinct startup offset policy when NO offset has been stored in ZK, and when the last processing offset is known from ZK ==> default settings now processes all existing messages in Kafka topic when first starting a consumer group from scratch, instead of loosing the messages already there. .. rubric:: v0.2.5 -**FIXED**: STORM hangs removed due to too big batch processing in PunchPlatform Spouts .. rubric:: v0.2.4 -**NEW**: new metrics to track kafka offsets evolution -**NEW**: moved to Elasticsearch bolt instead of Elasticsearch rivers .. rubric:: V0.2.3 -**FIXED**: configuration directory layout using resources directories. -**NEW**: logstash dialect supported for dynamic expressions using %{} .. rubric:: V0.2.2 -**NEW**: add and append Punch Tuple functions -**NEW**: logstash dialect supported by punch -**NEW**: grok patterns now loaded from tenants, tenants/aTenant, tenants/aTenant/aChannel groks directories -**NEW**: updated patterns to latest logstash -**FIXED**: grok bug when chaining patterns without separations -**FIXED**: grok bug lossing matches if using reaptingly the same destination Tuple -**FIXED**: Punch compilation error in case of inner class calls .. rubric:: V0.2.1 -**NEW**: punchplatform-cluster.sh tool for cluster deployment and monitoring (beta) -**NEW**: punchplatform-channel.sh reload option implemented -**NEW**: punchplatform-channel.sh works remotely through ssh. -**NEW**: manual pages, channel, troubleshooting and various other documentations -**NEW**: elastic search index and type now fully user configured through punch. Kafka bolt is now elasticsearch agnostic. -**NEW**: new Punch operators: *puid(), ruid(), ecount()* -**NEW**: auto test and trace capabilities of punchplatform-channel -**NEW**: configuration simplified. The river related encoding (plain versus elasticsearch) disappeared. -**NEW**: local topologies are now managed. -**NEW**: graphite metrics. -**NEW**: configuration is ulti-zone ready : multi elastic search, multi zookeeper, multi kafka -**NEW**:syslog spout does not use zookeeper anymore (only optionally) -**FIXED**: elastic river and spout damage partial Kafka cluster with missing leaders. -**FIXED**: no more hardcoded fields : punchplatform\_id, punchplatform\_version, punchplatform\_timestamp. .. rubric:: V0.2.0 -**NEW**: new Punch operators find\_by\_key, find\_by\_interval -**NEW**: improved toDate operator -**NEW**: new elastic Kafka consumer, Spout and River -**FIXED**: allow '@' in Punch identifiers .. rubric:: V0.1.5 -**NEW**: new Punch operators: *coalesce, cidrmatch* -**FIXED**: allow '\_' in Punch identifiers .. rubric:: V0.1.4 -**NEW**: safe generation of uuid in syslog-spout. Zookeeper based -**NEW**: safe offset restart in elasticsearch river for resiliency .. rubric:: V0.1.3 -**IMPROVED** git and maven release. Version and packaging strategy defined. .. rubric:: V0.1.2 -**NEW** allows named regex in punch, not only grok patterns. .. rubric:: V0.1.1 -**NEW** Punch operators : *toDate, toLong, toBoolean, toDouble* -**NEW** **type inference for KV and grok operators** is now configurable, off by default -**NEW** **elasticsearch index and type** now configurable in the PunchBolt -**NEW** Punchlet can be called from syslog spout, not only from Punchlet bolt -**NEW** Grok operator allows for multi target destination -**NEW** Support for** arbitrary number of Kafka clusters** -**NEW** Support templating and automatic generation of channel configurations files using j2 -**FIXED** operator kv() with empty value results in a mess -**FIXED** operator kv() with quoted string values should not include the quotes -**FIXED** Kafka spout does not work for more than one channel : used a hardcoded client id .. rubric:: V0.1.0 First delivered version based on finalized poc.