Release Notes

Please read the following notes carefully to precisely identify your version :

  1. if you have a running platform, the version number can be determined by navigating to the ‘online documentation’ link from the PunchPlatform AdminService Graphical User Interface.
  2. Starting at version 2.0.0, a new version scheme has been defined, =using logical name (‘ahmad’). Redmine bug tracker was also used. Now it is Jira. Starting from Ahmad, the FIXED changes indicated as part of each release refer to the corresponding Redmine ticket.

The indicated “Build version” (Vx.y.z) are suffixed to the following LMC deployment artifacts :

  • Clustered LMC deployer “punchplatform-deployer-x.y.z.zip”
  • Standalone LMC deployer “punchplatform-standalone-x.y.z.zip”
  • Installed LMC Administration/Monitoring service “punchplatform-admin-node-x.y.z.zip”
  1. a SNAPSHOT in the name of the version below indicates that it is under development and/or testing. The last stable version is the first one in this document without SNAPSHOT.

Feel free to visit the Jira : https://punchplatformfactory.atlassian.net/projects/PP

The latest version of the documentation is available here : https://documentation.gp.sef.cloudomc.ts-agora.fr/pp/dist

Important

if you see a below, you are working on a development branch

Brad Release Changes

Brad v4.0.2 Release Changes

Bug fixes

  • [PP-1841] - fixe the Kafka JmxReporter leak
  • [PP-1576] - Archiving performance reduction on topic containing months of data
  • [PP-1550] - Transactional Kafka incorrect reuse of persisted batch Id when not reusing persisted kafka offset
  • [PP-1472] - Batch failure management error in Kafka Spout transactional/batch mode
  • [PP-1667] - Fixed incorrect parsing of error documents

Brad v4.0.1 Release Changes

New features

  • [PP-1219] - export2csv plugin adjustment
  • [PP-1237] - Package enduser documentation in the product
  • [PP-1239] - provide user with a nice light and colored punchlet editor
  • [PP-1375] - siddhi punch integration
  • [PP-1317] - Platform configuration control

Enhancements

  • [PP-741] - Add supervisord support for Shiva supervision
  • [PP-970] - Shiva monitoring
  • [PP-1105] - Test services/tasks killing behavior
  • [PP-1121] - Port of export to CSV for Brad (Kibana 5)
  • [PP-1231] - Allow Single topology not to have output section
  • [PP-1246] - PML - binaries size
  • [PP-1250] - Add javadoc in PML code
  • [PP-1313] - Shiva UI and CLI beautifuling
  • [PP-1340] - PML - Doc change
  • [PP-1351] - Change a pygregator configuration parameter name
  • [PP-1379] - Check values consistency in deployer configuration audit
  • [PP-1400] - Installer standalone optimization
  • [PP-1414] - Parser SOGo
  • [PP-1429] - On ceph machines, the updatedb process cause troubles for application
  • [PP-1435] - kafka server config deployer - check coherence between default and deployer template
  • [PP-1499] - Procedure to install Ceph on deployment machine missing
  • [PP-1504] - Kibana deployment fails
  • [PP-1509] - Ceph manager deployment fails
  • [PP-1510] - An install procedure to install ceph on deployment machine must be available
  • [PP-744] - Allow resilient cron task start/stop/status in Shiva
  • [PP-835] - wrong location of kafka gc log
  • [PP-942] - Pygregator : check after aggregation request templatization if some nested aggregation names are identical
  • [PP-953] - Remove servers section of zookeeper
  • [PP-1085] - Check the punchplatform configuration dependencies with the deployer
  • [PP-1194] - PML - dependencie conflict handling
  • [PP-1240] - need a sublime editor
  • [PP-1241] - Upgrade elastic components to 5.6.4
  • [PP-1277] - PML - Use cases
  • [PP-1294] - Choose for each kibana instance the server place
  • [PP-1308] - Aggregation requests support in Batch Requests service (e.g. Top X)
  • [PP-1359] - Test spark 2.2.1 on deployer
  • [PP-1380] - Add outputFields, type and index in elastic batch node
  • [PP-1466] - when statements ansible
  • [PP-1519] - simplify kibana plugin installation with punchplatform-cluster.sh

Bug fixes

  • [PP-745] - Améliorations de mapping et de recherche Kibana
  • [PP-1003] - Update punchplatform glossary (e.g. graphite references out of date)
  • [PP-1053] - log warning in punchplatform-ml.sh
  • [PP-1103] - Shiva cluster config update too permissive
  • [PP-1173] - Java heap space zookeeper
  • [PP-1205] - punchplatform-command.sh is not well protected
  • [PP-1209] - PML - zookeeper crash
  • [PP-1221] - Unable to deploy operator when pp-admin and pp-operator are not on the same device
  • [PP-1223] - Kibana 5 very slow when save visu or dashboards
  • [PP-1226] - Incomprehensible error message on Shiva UI due to an exception when a task has never started
  • [PP-1227] - Rename punchplatform-shiva.sh start-task to submit-task
  • [PP-1233] - Exception catcher isn’t compliant with Brad new indices format
  • [PP-1234] - File descriptors leak on Shiva
  • [PP-1242] - Punchplatform-heartbeat.sh doesn’t work on standalone
  • [PP-1243] - punchplatform-packetbeat.sh for standalone
  • [PP-1245] - Update Doc’s Kibana dashboards
  • [PP-1270] - Apache parser do not raise an error when logs are not properly parsed
  • [PP-1288] - gaussian type for log injector doesn’t work
  • [PP-1301] - Deployer audit code fails with some configuration values
  • [PP-1303] - Pinchplatform-setup-kibana fails when proxy and unary operator error
  • [PP-1304] - Punch language always interpret Tuple as boolean in conditionals
  • [PP-1319] - Elastalert deployment fails
  • [PP-1327] - Conflict between kibana and elasticsearch bind address
  • [PP-1329] - Shiva service does not start after VM reboot
  • [PP-1335] - Audit configuration fails on specific cases
  • [PP-1357] - Light topo waiting time not used
  • [PP-1367] - Non immediate SUBMIT status in Batch Service
  • [PP-1388] - Housekeeping : 404 ERRORS for admin service buttons (REST API)
  • [PP-1427] - Documentation bug for testing parser
  • [PP-1430] - Extraction of archived logs to console does not work
  • [PP-1439] - etc/conf.json settings for housekeeping ceph
  • [PP-1459] - Archiving ceph issues with patch
  • [PP-1461] - Archiving system internal summaries inconsistency causing incomplete Housekeeping or Extraction
  • [PP-1483] - avishai RC5 chroot issues
  • [PP-1486] - Unit tests of archiving client KO following PP-1461
  • [PP-1487] - Bug doc dev - how to build elastalert dependencies
  • [PP-1493] - documentation on pygregator
  • [PP-1501] - Disturbing behavior of Git Bare deployment
  • [PP-1506] - Deployment of export2csv in chroot mode : Unable to find zip
  • [PP-1512] - “Troubleshooting export2csv kibana plugin” out of date
  • [PP-1518] - [log-injector] lumberjack-client mode : Missing mandatory property ‘PunchPlatformSettingsMapImpl.destination’.
  • [PP-1533] - [batch-request-plugin] command: line 29: loop: command not found

Brad v4.0.0 Release Changes

New features

  • IMPROVED PP-1004 - Upgrade Elasticsearch stack to 5.6.0
  • IMPROVED PP-1022 - Ceph Luminous 12.2.0 Support
  • FIXED PP-1155 - Upgrade Kafka to 0.11.0.0
  • IMPROVED PP-1102 - Variable gaussian distribution into the PunchPlatform log injector

Depreciations

  • IMPROVEMENT PP-1193 - Grafana now legacy

Breaking changes

  • IMPROVED #897 - Update default index naming on Brad to <tenant>-events-YYYY.MM.DD

Enhancements

  • FIXED PP-1048 - PunchPlatform Admin (java) fails to start
  • FIXED PP-1108 - Modsecurity rules improved
  • FIXED PP-1110 - Update getting started of Punchplatform, ML and Pygregator
  • FIXED PP-1112 - Kibana default index pattern mytenant-events- is not taken into account
  • FIXED PP-1113 - Kibana 5 fails when add index pattern - Request Entity Too Large
  • FIXED PP-1116 - Wrong software version in punchplatform-deployment.settings documentation
  • FIXED PP-1122 - Pygregator PyUnit tests KO
  • FIXED PP-1124 - Admin Behave Unit tests KO (housekeeper/elasticsearch)
  • FIXED PP-1126 - Clean hard coded version maven dependencies
  • FIXED PP-1132 - Add punchplatform-* commands to manual pages (documentation)
  • FIXED PP-1134 - Warnings during compilation of Documentation
  • FIXED PP-1135 - PDF generation of documentation failed
  • FIXED PP-1136 - PML missing punchplatform-analytics-plan-4.0.0.jar
  • FIXED PP-1138 - Getting started standalone inconsistent
  • FIXED PP-1142 - Documentation: add XFS dependency for CEPH install
  • FIXED PP-1147 - Template lmc for light topologies refactor
  • FIXED PP-1149 - Update Pygregator getting started, focus on websense channel
  • FIXED PP-1158 - Broken link in documentation’s Getting Started
  • FIXED PP-1163 - punchplatform-channel.sh, –verbose does nothing
  • FIXED PP-1164 - punchplatform-channel.sh –help (or -help) does nothing
  • FIXED PP-1174 - PML - dependencie problem with master zookeeper synchronization
  • FIXED PP-1175 - Documentation : Blocking tests for releases out of date
  • FIXED PP-1176 - Documentation : Licensing out of date
  • FIXED PP-1182 - Man pages are ok for development doc but ko for packaging doc
  • FIXED PP-1185 - Documentation : clean workspace before compile the doc
  • FIXED PP-1186 - Standalone install.sh interactive install fails
  • FIXED PP-1187 - Unwanted dependency on Storm in Ceph deployment
  • FIXED PP-1188 - Elastalert deployment fails
  • FIXED PP-1190 - Unwanted dependency on Storm in Ceph deployment
  • FIXED PP-1192 - Unable to stop spark process on standalone
  • FIXED PP-1199 - punchplatform-setup-kibana.sh prints clear errors when kibana and ES are down
  • FIXED PP-1202 - Cephbeat deployment fails
  • FIXED PP-1203 - mapping_global_standalone.json : template error
  • FIXED PP-1208 - Deprecated type field are still present in elasticsearch templates
  • FIXED PP-760 - Ceph deployer improvement for check mode support and maintainability
  • FIXED PP-992 - Turn Deployer idempotent back again
  • IMPROVED #878 - Fix punchplatform-deployment.settings soft version
  • IMPROVED #909 - Upgrade Elasticsearch and Kibana from 5.5.1 to 5.5.2
  • IMPROVED PP-1000 - Update default namping index on Brad to <tenant>-events-YYYY.MM.DD
  • IMPROVED PP-1006 - When an index is not created at the time an alias is created by the curator script, this index is not included in the alias
  • IMPROVED PP-1044 - Add official Ceph Luminous documentation in Brad version
  • IMPROVED PP-1083 - Jar deployment, analytics plan on driver node
  • IMPROVED PP-1084 - Type log field error in parsing error not indexed
  • IMPROVED PP-1086 - Add spark binaries to operator nodes in deployment mode
  • IMPROVED PP-1087 - Handle REST mode spark application submission
  • IMPROVED PP-1127 - Spark can be deployed without metric reporter feature
  • IMPROVED PP-1129 - Add Spark and ML terms in glossary (documentation)
  • IMPROVED PP-1130 - Add CyberSecurity, Parsing and Log Management terms in glossary (documentation)
  • IMPROVED PP-1131 - punchplatform-log-injector manual pages with –punchlets and others outdated
  • IMPROVED PP-1133 - Split the dependencies between kafka, zookeeper and PunchPlatform operator in deployer
  • IMPROVED PP-1137 - Light standalone: metricbeat have to write metrics only in elasticsearch
  • IMPROVED PP-1148 - Improve PML deployment
  • IMPROVED PP-1183 - Quality Management doc update
  • IMPROVED PP-1184 - Delete Platform Monitoring dashboard in Kibana
  • IMPROVED PP-813 - add HOWTO deal with detection punchlet
  • IMPROVED PP-965 - Shiva cluster should be configured on standalone so as not to be in YELLOW health
  • IMPROVEMENT PP-1191 - Improve install.sh for standalone to print UI urls
  • IMPROVEMENT PP-1201 - Add Security Audit documentation

Avishai Release Changes

Avishai v3.3.6 Release Changes

  • FIXED PP-1461 - Archiving system internal summaries inconsistency causing incomplete Housekeeping or Extraction
  • IMPROVED PP-1458 Provided new ‘change_ciphering_key’ archive processing action to allow efficient in-place change of archiving keys without massive load/rewrite of objects data.
  • FIXED PP-1457 re-cipher using archive processor bolt from archives that contain unciphered batchesCould
  • IMPROVED PP-766 Improved and safer ciphered archiving procedures using interactive input of secrets (keys and keystore password) + advises on keys and secrets management (added in Archiving guide of Administration documentation) + end-to-end demo steps on a standalone to understand the process
  • FIXED PP-1453 Could not use multiple executors for ciphering/deciphering (single executor worked OK)
  • FIXED PP-1432 Some use cases failures when de-ciphering archived logs using punchplatform-objects-storage.sh command-line tool.
  • FIXED PP-1405 punch date formatters must change when year changes
  • FIXED PP-1401 - Housekeeping : wrong request in javascript part and wrong http verb in admin-service
  • FIXED PP-1388 - Housekeeping : 404 ERRORS for admin service buttons (REST API)
  • IMPROVED PP-1225 - SSL server in spouts has vulnerabilities
  • IMPROVED PP-727 - [Punch] Implement a Cloneable interface for Tuples.
  • IMPROVED PP-750 - Archiving housekeeping is slow despite indexation and causes OutOfMemory even erasure of non-existing timescope
  • FIXED PP-758 - Bug doc on Ceph
  • FIXED PP-765 - Choose the name of the public key in deployer
  • IMPROVED PP-786 - Elasticsearch snapshotting not allow by deployer
  • IMPROVED PP-790 - EPS reporting
  • IMPROVED PP-817 - Improve performance of syslog header parsing
  • FIXED PP-839 - Kafka spout settings issues in system metric topology
  • FIXED PP-851 - Log extracts by end-user
  • FIXED PP-867 - Netty metrics with netty as prefix but in the documentation it is syslog_spout prefix
  • FIXED PP-968 - Shiva does not stop properly
  • IMPROVED PP-969 - Shiva housekeeping
  • IMPROVED PP-974 - Shiva & light topologies deployment
  • FIXED PP-982 - Storm worker logs are not cleaned up for killed topologies
  • FIXED PP-1010 - WinAD : Ajouter champs Workstation & Category
  • IMPROVED PP-1016 - A finite number of unsubmitted tasks history must be kept in zookeeper
  • FIXED PP-1017 - Shiva data folders shall be purged of too old tasks folders
  • FIXED PP-1025 - Shiva documentation should explain cluster configuration
  • IMPROVED PP-1035 - Build unit tests for Punch programming simply and directly
  • FIXED PP-1039 - [Juniper] Tweak action field
  • FIXED PP-1041 - [Juniper] RT_Flow/Flow_reassembled_succeeded
  • FIXED PP-1055 - NullPointerException in ElasticsearchBolt in case of error logs when exception_handler punchlet is misconfigured
  • FIXED PP-1057 - Operational facts on Shiva
  • FIXED PP-1058 - Pygregator not defensive enough during configuration & logging check
  • FIXED PP-1059 - [Modsecurity] Extend PCRE size limit
  • FIXED PP-1065 - [Export2csv] Partial export for a query
  • FIXED PP-1066 - [Export2csv] Query fails if querying for indices for more than 7 days
  • FIXED PP-1069 - Full integration of pygregator in shiva
  • FIXED PP-1070 - Shiva CLI logs misplaced or bad rights
  • FIXED PP-1074 - Shiva tasks working directory inconsistency
  • IMPROVED PP-1090 - Lumberjack client/server with window frames
  • IMPROVED PP-1091 - Shiva status inconsistent in stand-alone (UNKNOWN when –with-shiva is NOT used, else YELLOW)
  • IMPROVED PP-1093 - Shiva CLI description inconsistent
  • IMPROVED PP-1100 - Shiva cluster configuration parameters must be overridable in deployment configuration file
  • FIXED PP-1189 - Option update-config of Shiva CLI reports same output event with or without change
  • FIXED PP-1206 - Pygregator and lights topologies fail when launched with shiva in a channel
  • FIXED PP-779 - Documentation kafka bolt insufficient
  • FIXED PP-1170 - Temp sharing in shiva command wrappers fails for several operators
  • FIXED PP-1159 - Getting started Shiva without install pre-requisites
  • FIXED PP-1104 - Shiva stores tasks working directories in /tmp
  • FIXED PP-1080 - Unable to deploy spark with PunchPlatform deployer
  • IMPROVED PP-1068 - Support pygregator deployment (official deployer)
  • IMPROVED PP-1056 - [WinAD] : Set upo proper unit tests
  • FIXED PP-1026 - Modsecurity blocks the requests when there is too many filters
  • FIXED PP-1021 - Legacy reverse proxy sections in the product documentation
  • FIXED PP-1019 - Unable to deploy a PunchPlatform with the git when the first deployment fails
  • FIXED PP-1040 - [Bluecoat] Parse system logs
  • FIXED PP-1037 - [WinAD][Parser] Correction for certain EventID
  • FIXED PP-1077 - Export2csv escaping to comply with CSV standard.
  • FIXED PP-1096 - [Admin] Monitor Kafka topic health
  • FIXED PP-948 - Remove –autotest from punchplatform-channel.sh because it is not supported anymore and update man pages
  • FIXED PP-825 - incorrect self_monitoring documentation
  • FIXED PP-941 - punchplatform-log-injector.sh generate wrong rep.ts
  • FIXED PP-1090 - Lumberjack client/server with window frames
  • FIXED PP-1072 - Export2CSV kibana plugin improved progress tracking/end condition checking to improve stability and reliability
  • FIXED PP-1071 - Export2CSV now supports inclusion of _id, _index and _type in search, query and CSV export
  • FIXED PP-1064 - Export2CSV now supports standalone deployment (manual install) even when using an external configuration directory
  • FIXED PP-1063 - Export2CSV now uses appropriate implicit timestamp when switching between indexes patterns.
  • FIXED PP-793 - Export2CSV now supports outputing explictely in the request the implicit timestamp column of the search
  • IMPROVED PP-1067 - Add a bolt to verify the sequence of received logs
  • IMPROVED PP-1031 - More user-friendly command-line help on punchplatform-shiva.sh and punchplatform-objects-storage.sh
  • IMPROVED PP-818 - Improve punchplatform-objects-storage.sh batches-tasks-list to retrieve es cluster data from punchplatform.properties and cluster id
  • IMPROVED PP-749 - Provided ‘HOWTO reinject documents/logs from archive’ with associated kibana resources and standalone sample
  • FIXED #964 - more user-friendly PunchPlatform admin service error message when ‘punchplatform_admin’ section is missing from punchplatform.properties
  • NEW #961 - Support ES nodes tagging using node.box_type ES parameter
  • IMPROVED #960 - Add an optional section in application deployment sections to configure supervisord logrotate
  • FIXED #955 - Ceph admin node deployment fails if ceph is deployed alone
  • FIXED #944 - Fixed slow unarchiving/archive housekeeping due to bad internal usage of archiving indexation structures.
  • IMPROVED #942 - make the lumberjack netty server able to cope with buggy logstash senders
  • FIXED #919 - Archive processor Bolt does not support replay of files without header.”
  • FIXED PP-748 / #202 - Archive processor Bolt now supports replay of files without header.
  • FIXED #892 - FilterBolt : wrong parsing of configuration for “exclude_substring” setting
  • NEW #868 - Allow batch signature/digest sending by FileBolt (to enable later data integrity checks of archived data).
  • NEW #888 - Add scripts to request ES for EPS metrics and to transform the result in CSV format
  • NEW #887 - EPS max metric
  • IMPROVED #859 - Add new parameters in Pygregator: safety delay, term_or_value filter, exists filter and query_string for flexible queries

Avishai v3.3.5 Release Changes

  • FIXED #825 - wrong casting between Integer vs Long and Float versus Double
  • FIXED #930 - Pygregator time frames computing wrong
  • FIXED #918 - Unable to unarchive empty raw logs with Archive propcessor bolt
  • FIXED #912 - Pygregator results are wrong in some conditions (range in term buckets)
  • FIXED #870 - nullPointer exception in FileBolt when no report tuple emission is configured
  • FIXED #858 - Pygregator doesn’t return aggregations results when one or several specified fields are missing
  • FIXED #856 - Pygregator Getting started doesn’t work
  • FIXED #855 - Pygregator: force_replay option not working outside frames
  • FIXED #860 - Wrong Dependencies (task-scheduler) injected directly in Storm
  • FIXED #836 - Shiva where are the logs in standalone ?
  • FIXED #832 - Shiva unable to stop a process
  • FIXED #833 - Shiva java.lang.NullPointerException
  • FIXED #835 - Shiva autocompletion isn’t working
  • FIXED #826 - Lack of information about ‘failed to send invalid messages to error index’ log
  • FIXED #867 - Incorrect MD5 sum in md5 files output by FileBolt
  • NEW #925 - Add pipeline aggregations in pygregator
  • NEW #868 - Allow batch signature/digest sending by FileBolt (to enable later data integrity checks of archived data).
  • NEW #851 - Spark Metrics
  • NEW #904 - PML - Standalone Getting Started
  • NEW #846 - PML (PunchPlatform Machine-Learning) - Spark machine-learning library MLlib integration
  • NEW #839 - added duplicate method to type Tuple to provide punchlets with a deep copy semantics
  • NEW #842 - add an optional exit action to lumberjack spout should clients timeouts
  • NEW #845 - add support for lumberjack keep alives from the spout to the bolt
  • IMPROVED #914 - Move Pygregator in pp-core
  • IMPROVED #841 - Install.sh add possibility to set a specific hostname with option –hostname
  • IMPROVED #862 - Add tag with aggregation of task id and custom tag
  • IMPROVED #861 - Add tag –no-pygregator for installation script
  • IMPROVED #840 - Metrics : add a periodic tuple.ack metric “tick.tuple.ack”
  • IMPROVED #838 - Improve Archive Processor Bolt to merge data in case of multiple files
  • IMPROVED #829 - Minor Upgrade elasticsearch (2.4.0 to 2.4.6) and kibana (4.6.1 to 4.6.5)
  • IMPROVED #830 - Documentation structure improvment
  • IMPROVED #831 - Add EquipmentsSupervisionRule to Elastalert
  • IMPROVED #834 - Reduce metricbeat logging on standalone
  • IMPROVED #837 - Pygregator release 1.0.10 with the es output format indices-YYYY.MM.DD
  • IMPROVED #843 - Minor upgrade of Kibana for security reason to 4.6.6
  • IMPROVED #844 - Enable kafka compression by default, disable the topic auto creation
  • IMPROVED #890 - Add metricbeat System module documentation
  • IMPROVED #913 - add ignore_above settings in mapping_events to be more efficient
  • IMPROVED #859 - Add new parameters in Pygregator: safety delay, term_or_value filter, exists filter and query_string for flexible queries
  • IMPROVED #857 - Add multiple metrics aggregations support into Pygregator

Avishai v3.3.4 Release Changes

  • FIXED #821 - Elasticsearch modsecurity deployment regression with aliases
  • FIXED #822 - Add compatibility to single zookeeper deployment and multi

Avishai v3.3.3 Release Changes

  • FIXED #814 - Shiva : in memory archiving for arguments
  • FIXED #813 - Filtered aggregations fail
  • NEW #736 - Shiva tasks manager is now available for distributed/resilient running of services or periodic tasks, both for PunchPlatform monitoring/housekeeping purpose, and also as part of PunchPlatform channels.
  • IMPROVED #807 - Homogenous command logs
  • FIXED #806 - [Standalone] Non-JSON standalone-generated punchplatform.properties
  • FIXED #805 - Deployment of Admin-service : PUNCHPLATFORM_CONF_DIR is not correctly set
  • FIXED #804 - Ceph deployment broken: client keyring path not overriden in ceph conf files
  • IMPROVED #803 - Zookeeper : start is too slow
  • FIXED #802 - Deactivate Filebeat by default in standalone
  • FIXED #797 - Admin : –force option for channel configuration doesn’t work anymore
  • FIXED #799 - Archiving HouseKeeping Service Logging Problem
  • FIXED #815 - punchplatform-elasticsearch.sh do not use the max memory defined in punchplatform.properties on standalone
  • FIXED #816 - Define settings in punchplatform-deployment.settings to configure jam size of storm components
  • BUG #795 - Lack of explanations for 3.3.2 Elasticsearch new features
  • BUG #817 - Multi zookeeper instances on one device
  • BUG #818 - Multi elastalert deployment
  • IMPROVED #819 - Improve metricbeat logging
  • NEW #820 - windowed census
  • FIXED #798 - Pygregator fails when a periodic aggregation is executed

Avishai v3.3.2 Release Changes

  • FIXED #794 - Pygregator fails when a nested bucket aggregation is executed
  • FIXED #790 - Remove cloning in JSONSettingsMap and JsonSettingsList constructor method
  • FIXED #646 - When an elasticsearch mapping template is not correct JSON, no clear error message in admin service GUI
  • IMPROVED #791 - Add new settings for elasticsearch to optimize recovery
  • IMPROVED #792 - Increase the stopwaitsecs for elasticsearch nodes in supervisord
  • FIXED #789 - Pygregator : replace tab by spaces, line 69 of config.yml

Avishai v3.3.1 Release Changes

  • IMPROVED #788 - Admin-service : insertion of Elasticsearch templates is very slow (> 50s)
  • IMPROVED #786 - Limit the Kafka and Elasticsearch memory consumption in standalone
  • FIXED #785 - Admin-service : KafkaConsumer memory consumption
  • IMPROVED #778 - SSL LumberjackSpout Documentation
  • NEW #775 - Low finite operation duration for standalone release
  • NEW #779 - Supervision : New module with cruncher
  • FIXED #758 - Spark API Massive Logging
  • FIXED #755 - Downgrade apache storm to 1.0.2
  • FIXED #768 - Update Elastalert to 0.1.16
  • FIXED #774 - Fixed PYYAML dependency packaging (was not correct for some hos environments)
  • FIXED #766 - LumberjackBolt doesn’t take into account ssl configuration
  • FIXED #765 - Zookeeper uses too much memory
  • FIXED #760 - Admin-service : channel health is green but one the path latency is red, channel start timestamp
  • FIXED #769 - Spark API Low Kafka Injection
  • IMPROVED #777 - Upgrade Grafana to 4.4.1
  • FIXED #780 - Unable to install punchplatform-admin python
  • IMPROVED #781 - Upgrade metricbeat to 5.5.0
  • IMPROVED #782 - Upgrade zeppelin to 0.7.2 only for standalone
  • IMPROVED #783 - Add ansible_inventory_settings to punchplatform-deployment.settings
  • FIXED #776 - do not dump jvm in case of OOM

Avishai v3.3.0 Release Changes

  • IMPROVED #755 - Upgrade apache storm to 1.0.3
  • IMPROVED #754 - Improve filebeat integration in punchplatform-standalone

Avishai v3.2.11 Release Changes

  • IMPROVED #756 - Upgrade spark to 2.0.1 and minor corrections

Avishai v3.2.10 Release Changes

  • IMPROVED #739 - Warning in case of dangerous implicit error publication configuration + setting to allow custom publication to a selection of streams
  • FIXED #744 - Punchplatform-channel configure option not operational
  • FIXED #743 - Extract2Csv plugin does not handle aliases
  • FIXED #741 - Bug in Job command when channel starts
  • IMPROVED #740 - Always write date on 2 digits (months, days, …) in paths (ContentsFiles publication action in FileBolt)
  • FIXED #738 - Admin-service uses 2G of memory but Xmx is 256m
  • NEW #733 - support for live extension of CEPH objects storage/archiving cluster by node addition
  • IMPROVED #742 - Spark-API, add batch mode

Avishai v3.2.9 Release Changes

  • FIXED #732 BASH completion fails for Linux
  • FIXED #731 Admin-service : start/stop/reload instabilities
  • FIXED #730 expiration tuples does not allow a punchless to generate forwarded tuples
  • IMPROVED Kafka compression is now supported (includes a fix for out-of sequence offsets of messages)
  • IMPROVED #729 - Metrics reporter : possibility to declared multiple nodes for Elasticsearch reporter
  • FIXED #721 - PunchPlatform admin logging : add ‘maxFileSize’ property

Avishai v3.2.8 Release Changes

  • IMPROVED #728 - Admin service : possibility to deactivate the Storm request for calculation of channel health
  • FIXED #727 - FilterBolt shall publish _ppf_metric_logs
  • IMPROVED #724 - Admin service : Start/Stop/Reload through REST API
  • FIXED #723 - Admin service : Multiple thread of the same service are alive (Archiving service)
  • FIXED #716 - ArchiveProcessorBolt does not work with deciphering enabled in cluster mode
  • FIXED #722 - Allow the same topology to be started to different clusters

Avishai v3.2.7 Release Changes

  • FIXED #713 - Ceph systemd configuration
  • NEW #711 - New ArchiveProcessorBolt publication action : tuples emission (for Kafka/Elasticsearch/whatever injection)

Avishai v3.2.6 Release Changes

  • NEW #712 - External UDP enabler/disabler of ElasticsearchSpout and KafkaSpout for custom backpressure scenarios
  • NEW #700 - Replay from a timestamp feature : ‘from’ configuration for kafkaSpout
  • NEW #695 - Add KafkaRange library. Provides methods to convert Kafka offset to timestamp and vice versa.
  • NEW #694 - Add spark to PunchPlatform deployer
  • NEW #688 - Spark API Variable Transformation
  • NEW #687 - Spark API Variable Input Configuration
  • NEW #680 - Spark API Invariant Transformation implementation
  • NEW #682 - Admin-service : metrics reporter for kafka and channel with admin grafana dashboard
  • NEW #681 - Add Apache Cassandra to PunchPlatform deployer
  • NEW #694 - Add Apache Spark to PunchPlatform deployer
  • NEW #697 - Add support Ansible 2.3 to PunchPlatform deployer
  • NEW #698 - Add punchplatform-setup-kibana.sh to load dashboard
  • IMPROVED #701 - default Housekeeping frequency is now 1 minute and not 20 minutes + it is now not needed to provide parameter ‘archive_max_age_in_days’ in punchplatform.properties when archivehiving is not configured
  • IMPROVED #703 - Archiving scalablity through concurrent write support (multiple file bolts)
  • IMPROVED #708 - Add master_role type to elasticsearch node function
  • IMPROVED #690 - Platform identifier attribute in punchplatform.properties
  • FIXED #678 - ArchiveProcessorBolt now functional ((un-)compression, en/de-ciphering, keystores, errors reporting, doc)
  • FIXED #710 - Fixed Filter Bolt (High CPU due to bad acknowledgemnt order)
  • FIXED #709 - Fixed Kafka exception hiding by nullpointer exception in KafkaBolt
  • FIXED #707 - If es client of metric reporter has a wrong url then the topology must still start
  • FIXED #706 - LTR latency metrics generates error on lmr_in component
  • FIXED #699 - FileSpout doesn’t generate autotest latencies
  • FIXED #696 - ‘man punchplatform-synchronization.sh’ doesn’t work
  • FIXED #691 - Admin service doesn’t start if we make a link punchplatform.properties to an external file and without -pp args in start command
  • FIXED #689 - FileBolt does not write data with filesystem publication
  • FIXED #702 - toFlatTuple operator exposed as toFlatUple
  • FIXED #683 - Doc packaging with sphinx : it was included twice (target dir), reduce generation by two times

Avishai v3.2.5 Release Changes

  • NEW #679 - K-Means integration with Spark API (standalone)
  • NEW #677 - Java API integration making easy conversion from text to numeric (for analytics features)
  • NEW #676 - Integration of Spark-Streaming in PunchPlatform channels (standalone)
  • NEW #656 - FilesReaderBolt for tuples extraction/injection from files + improvements of ElasticsearchBolt to allow tasks management for long-duration resilient jobs
  • IMPROVED #674 - Admin-service : for kafka, display the insertion timestamp associated to an offset
  • IMPROVED #673 - Ciphering : use KeyStore instead of x509 files to manage keys
  • IMPROVED #670 - Upgrade metricbeat&filebeat from 5.2.0 to 5.3.0
  • IMPROVED #664 - multi-level templating in punchplatform-channel.sh command
  • IMPROVED #653 - Direct buffers (FileBolt publication methods) - applied to other file publication actions (file, md5)
  • IMPROVED #650 - Fast-forward of Elasticsearch Spout when no documents in current extraction timeslice to next event in Elasticsearch
  • IMPROVED #505 - Automatic housekeeping of archive topics
  • IMPROVED #500 - Archiving documentation
  • FIXED #675 - no more Elasticsearch bolt crash if multiple additional fields configured, or if additional field is of numeric type
  • FIXED #669 - Update elastalert version to 0.1.8 with elasticsearch 2.X dependencies
  • FIXED #668 - ./apache_httpd_perf.sh –punchlet gives a missing class Java error
  • FIXED #667 - punchplatform-channel.sh –start doesn’t failed if the topology file doesn’t exist
  • FIXED #665 - –show argument on a punchplatform-channel command doesn’t work
  • FIXED #661 - Empty stored files in CEPH
  • FIXED #660 - nullPointer exception in FileBolt if null value present in input tuple
  • FIXED #634 - Operator deployment regression
  • FIXED #633 - Elasticsearch bind address regression on deployer

Avishai v3.2.4 Release Changes

  • FIXED #648 - Watchdog should kill process but doesn’t
  • IMPROVED #652 - Update export2csv kibana plugin
  • IMPROVED #653 - Direct buffers (FileBolt publication methods) - first step

Avishai v3.2.3 Release Changes

  • NEW #501 - storing in archiving system of “parsing error” documents for later retrieval
  • IMPROVED #645 - Add possibility to submit a light jar : –without-dependencies args for punchplatform-channel.sh
  • IMPROVED #647 - Reduced frequency of zookeeper commits in Kafka Spout
  • FIXED #643 - Admin-service : curator lock timeout is not configurable and by default set to 1 second
  • FIXED #642 - if punchplatform-channel.sh –repair –start failed for one topology then the topologies of the channel are stopped
  • FIXED #637 - Faulty batch commit algorithm in Partition Manager
  • FIXED #501 - Check and manages store/retrieval in CEPH of “parsing error” documents

Avishai v3.2.2 Release Changes

  • FIXED #641 - log-balancing bug in kafka bolt (only one partition used)

Avishai v3.2.1 Release Changes

  • NEW #609: Filesystem-backed objects storage (for archiving on NFS) - supports concurrent writing even to a same objects pool topic/partition
  • NEW #611 - Archive processor bolt to allow actions on archived files (extract, copy to other archive)
  • NEW #593 Fast extraction of raw log compressed files from CEPH
  • FIXED #632 - punchplatform-admin –show gives “PARTIAL” result with the start command
  • IMPROVED #638 - Metrics are using too much disk space : avoid using _source field
  • IMPROVED #631 -‘Single Thread’ admin-service to multithread (one thread by service)
  • FIXED #620 - Unwanted Archive overwrite in archive space in case of Zookeeper data loss due to Kafka batch Id regeneration to 1
  • FIXED #628 - Add timeout for elasticsearch request for the channel health service (10 seconds)
  • FIXED #626 - If the “Patch” directory is empty then the user obtains an “ambiguous topology jars”
  • FIXED #624 - Punchplatform-channel.sh doesn’t take into account parameter ‘-timeout xx’
  • FIXED #623 - Admin service should request elasticsearch on cluster_admin_url
  • FIXED #621 - filter bolt default stream regression
  • FIXED #625 - Slow archiving to CEPH and high memory consumption of File Bolt
  • FIXED #627 - the patching of the platform using the patch directory is broken
  • FIXED #617 - CEPH deployer parametrization of PGs and Resilience + prerequisites documentation
  • FIXED #640 - dashboard push through punchplatform-setup-grafana.sh now supports both “saved by hand” and “pulled” dashboard format

Avishai v3.2.0 Release changes

  • IMPROVED #509 - New Java Admin integrating CEPH and METRICS monitoring
  • IMPROVED #584 - New GUI integration

Art Release Changes

Art v3.0.24 Release Changes

  • FIXED #588 - Unwanted “logger” output on console for jobs CLI
  • FIXED #597 - Offset committing problem in transactional kafka spout when errors documents are present in topic
  • FIXED #610 - Reduce memory consumption of objects-storage layer
  • FIXED #526 - ceph deployment: osd partitions have NOT to be XFS-formatted
  • FIXED #511 - Filesystem and MD5 publication (FileBolt) doesn’t work (fail during topology launching)
  • FIXED #510 - Cephbeat fails when it’s deployed on admin machines
  • FIXED #525 - Cephbeat does not log in appropriate folder
  • NEW #502 - Geneneric dashboard for CEPH
  • NEW #498 - add automatic multi-pools support in cephbeat
  • IMPROVED #497 - replace ceph config file arg by ceph cluster name
  • IMPROVED #600 - Add filebeat to deployer
  • IMPROVED #582 - Better error message when connection to CEPH fails

Art v3.0.23 Release Changes

  • NEW #594 - additional root fields can be added by Elasticsearch bolt from storm fields, both for normal documents and errors
  • FIXED #595 - logs repeated at slow traffic because of timeout queue of KafkaBolt
  • FIXED #432 - Custom metrics tags (implemented them at storm spout/bolt settings level again, and not only at topology settings level)
  • FIXED #512 - install.sh can now again be launched out of install.sh directory
  • FIXED #581 - too verbose punchplatform-channel.sh –start + SLF4J warning in command lines (Object storage, jobs)
  • FIXED #512 - standalone install.sh not working propery (storm.yaml) when not run locally
  • IMPROVED #596 - kafka spout watchdog timeout set by default to 2 minutes
  • IMPROVED #580 - Update grafana version to 4.1.1
  • IMPROVED #583 - Deployment refactor
  • IMPROVED #601 - Upgrade metricbeat to 5.2.0
  • NEW #591 KafkaBolt can publish data to third party brokers
  • NEW #579 KafkaBolt can publish plain JSON
  • NEW #590 Punch resource files can be stored under tenant or channel sub-folders
  • NEW #602 add geoip support on full deployed platform
  • MERGED Ahmad v2.1.22

Art v3.0.21 Release Changes

  • FIXED #507 RythmerBolt correctly registered as default static bolt
  • NEW #508 added Mistral Sophos Sourcefire and Stonesoft parsers

Art v3.0.20 Release Changes

  • FIXED #486 uncatched exception in kafka spout can lead to a zombie reader
  • FIXED #470 Mlockall Memory Elasticsearch configuration issue
  • NEW #312-added ceph cluster deployment for objects storage + punchplatform-objects-storage.sh + File bolt capability to write into CEPH + Tasks List Spout capable of reading tuples from CEPH objects + sample tenant/topologies to store/extract tuples in CEPH + punchplatform-jobs.sh to list/clear jobs list + update of administration service to display jobs and allow to clear them.
  • IMPROVED #448-removed standalone management commands on cluster admin node in cluster deployer
  • IMPROVED #482 configurable queue size for TCP netty client, hence syslog bolts
  • NEW #484 Punch ipmatch operator for black and whitelisting ips
  • NEW #483 filter bolt, for deploying high performance filtering in LTRs

Art v3.0.19 Release Changes

  • IMPROVED #447 Bower and nodes dependencies as maven external artifacts
  • IMPROVED #452 added ‘punchplatform_conf_repo_git_url’ deployment setting for customization of configuration branch to deploy (default is ‘production’)
  • IMPROVED #463 Upgrade metricbeat to 5.0.1
  • IMPROVED #453 added ‘bind_address’ optional setting to elasticsearch nodes in punchplatform.properties, to allow listening onto specific addresses (for example 0.0.0.0 when VIP-listening is required)
  • IMPROVED #462 added a syslog_udp_bolt to the punchplatform
  • IMPROVED #464 refactor punchplatform-admin-user deployment for ltr use case
  • IMPROVED #466 Active/Passive grafana is needed when synchronizing dashboards through DRBD
  • FIXED #451 Elasticsearch spout component should publish the field _ppf_metric_logs
  • FIXED #405 Metricbeat always installed on linux, regardless of –no-metricbeat

Art v3.0.18 Release Changes

  • IMPROVED #231/#432 - Allow additional custom metrics tags in spouts/bolts metrics context, by user configuration in bolts/spouts settings
  • FIXED #444 - reduced log level from warning to information level for ‘added Kafka broker address’ message popping when using punchplatform-channel and punchplatform-topology commands

Art v3.0.17 Release Changes

  • FIXED #236 Updated storm options in punchplatform-topology to avoid message “Storm : Using deprecated config nimbus.host for backward compatibility”.

Art v3.0.16 Release Changes

  • FIXED #442 Wrong value of kafka message-backlog in Grafana
  • FIXED #441 - Sometimes there are no metrics in Grafana and ES

Art v3.0.15 Release Changes

  • IMPROVED #396 - Improve punchplatform-metricbeat.sh
  • FIXED #429 - Log reporter for metrics now reports autotest latencies as non-errors and also allows tags values reporting
  • IMPROVED #428 - Elasticsearch metrics reporter now stores metric name as document type in Elasticsearch, allowing specific mapping (especially specific TTL settings for automatic metric purge)
  • IMPROVED #427 - Elasticsearch metrics reporter now generates metrics index with date suffix, allowing automatic housekeeping by PunchPlatform admin service
  • FIXED #420 - Third party topologies fail to start

Art v3.0.14 Release Changes

  • NEW #410 - Integration of Zeppelin in standalone
  • NEW #409 - a new Punch ‘containsOneOf’ operator is available
  • FIXED #412 - openjdk8 set default alternatives ubuntu
  • FIXED #24 - compatibility with external Openjdk deployment
  • IMPROVED #385 - punchplatform-channel.sh now checks json of generated files when using –configure
  • IMPROVED #414 - Deduplication of error logs (as warning) when configuration is repeatedly unavailable (to allow for DRDB providing git repository without admin logs flooding on slave node)
  • IMPROVED #380 - Added Elastalert to the standalone

Art v3.0.13 Release Changes

  • FIXED #407 metricbeat template synchronisation fails in PunchPlatform admin python
  • IMPROVED #395 - Automatic json check on channel generated files
  • IMPROVED #406 log rotate on elasticsearch logs
  • IMPROVED #397 log rotate on kafka logs
  • IMPROVED #388 kafka spout should read JSON

Art v3.0.12 Release Changes

  • FIXED #401 kafkabolt can be stuck forever
  • IMPROVED #398 Punchlet errors are dispatched potentially on all error streams and not specifically on the error stream associated to the stream which has raised the error
  • IMPROVED #395 Automatic json check on channel generated files

Art v3.0.11 Release Changes

  • FIXED #379 Os_system dashboard on grafana doesn’t work properly with metricbeat
  • IMPROVED #376 replace topbeat/collectd by metricbeat
  • IMPROVED #373 Add a keepalive for lumberjack connection
  • FIXED #369 In ElasticConsumer, the ‘refresh’ thread may be stopped if the coordinator is null
  • FIXED #392 kafkabolt load balancing is now plain round robin
  • FIXED #393 log injector fail to handle several injection files

Art v3.0.10 Release Changes

  • FIXED #364 Control of the Double.NaN in the metrics value
  • FIXED #363 Initialization of shared curator beetween ElasticConsumers : no zookeeper coordinator connected
  • FIXED #365 Added ‘recv_buffer_size’ setting to syslog spout in order to actually size reception buffer - this fixes wrong (too small) reception buffer size that may cause more packet loss than usual (for UDP)
  • FIXED #366 - counter metrics (like uptime) are not reported
  • IMPROVED #361 Kill topologies after only 5 seconds by default

Art v3.0.9 Release Changes

  • IMPROVED #339 Update storm to 1.0.2
  • FIXED #345 NullPointerException in KafkaBolt if the latency metrics are not availables

Art v3.0.8 Release Changes

  • IMPROVED #344 LMR and LTR configuration
  • IMPROVED #343 Added per stream options to both syslog and lumberjack bolts

Art v3.0.7 Release Changes

  • FIXED #342 The health of a channel should be Green if it is active and has no autotests defined.

Art v3.0.6 Release Changes

  • IMPROVED #336 Refactor metrics : no more propagation/proxy and improve conception
  • IMPROVED #334 Upgrade grafana from 3.0.0beta to 3.1.1 only for linux platform
  • IMPROVED #333 Reduce the logging of modsecurity
  • FIXED #332 Kibana & Grafana slowness when not connected to internet
  • FIXED #328 Conflict of metrics name if there is multiple destinations for lumberjackBolt

Art v3.0.5 Release Changes

  • FIXED #317 Wrong initialization of health variable for channel structure autotest control.
  • FIXED #269 storm logging configuration that had all workers write in the single ‘worker.xml’ file
  • IMPROVED #319 Add tempo metrics counter in syslogSpout which gives a reference for metrics propagation (only if log level is trace)

Art v3.0.4 Release Changes

  • FIXED #308 Health of channel is red if the topology files haven’t metrics/metrics_source property
  • FIXED #296 Kibana VIP are wrongly deployed also on LMC admin nodes
  • FIXED #295 SyslogBolt not acking tuples
  • FIXED #294 Removed supervisord from pacemaker control scope on admin/nimbus node because nimbus is now active-active

Art v3.0.3 Release Changes

  • IMPROVED #267 The autotest_latency can be configured directly in the topology files
  • IMPROVED #266 Add the possibility to configure multiple reporters of the same type (elasticsearch)
  • FIXED #274 The tenant health is GREEN when the channels has an UNKNOWN health
  • FIXED #272 Documentation of metrics with submetric types
  • FIXED #255 In proxy mode, some metrics are not propagated : “Unable to retrieve context”
  • FIXED #277 Syslog Bolt does not start when multiple destination are configured
  • FIXED #275 Support for uppercase in storm nodes hostnames (induced wrong metrics names including latency autotest metrics)

Art v3.0.2 Release Changes

  • IMPROVED #247 Spark version 1.6.0 to 1.6.1
  • FIXED #250 Remove bulk request for metrics and replace with specific requests for each channel.
  • FIXED #245 Metrics proxying is not transparent for multi-values metrics sent to multi-Elasticsearch backend
  • FIXED #228 Update all sample/test configuration of STORM to reflect new multi-master feature of STORM V1.0.1
  • FIXED #220 Virtualenv not activated on new Mac standalone deployment

Art v3.0.1 Release Changes

  • FIXED #249 Add zookeeper property in kafka custom configuration

Art v3.0.0 Release Changes

  • IMPROVED #238 Reduced amount of error traces when python api service get errors during forwarding of API calls to Java admin service
  • IMPROVED #233 Update zookeeper version 3.4.6 to 3.4.8
  • IMPROVED #226 Add the possibility to configure the elasticsearch client for metrics.
  • IMPROVED #192 Added a regex filter
  • IMPROVED #190 Added the possibility to choose the delimiter for metrics.
  • IMPROVED #189 Added Grafana dashboards for selfmonitoring metrics
  • IMPROVED #187 Added Grafana dashboards for the classic metrics (not selfmonitoring metrics) from ES.
  • NEW #218 move to Storm 1.0.1
  • NEW #211 Store metrics in ES a structured field to allow completion in grafana and metric-specific mappings
  • NEW #186 Possibility to calculate the health of channel from metrics of ES
  • NEW #131 new GraphiteReporter with the possibility to choose between Graphite and ES for all metrics
  • NEW #179 Ahmad is not Art
  • NEW #162 Add Flink in pp-standalone
  • NEW #81 Kibana 4 for clients
  • FIXED #246 Fixed lumberjack multithreading corruption
  • FIXED #240 Log the metric if the context is null and it is not send.
  • FIXED #237 Date retrieval/parsing error when admin service sycnchronizes elasticsearch templates
  • FIXED #132 Standalone installation failed with the new version of virtualenv (MacOS)
  • FIXED #74 Check if _source storage is still needed with kibana 4
  • FIXED #62 Tweak ES required eligible master quorum to prevent split brain
  • FIXED #24 Oracle JDK not delivered as part of standalone.

Ahmad Release Changes

Ahmad v2.1.22 Release Changes

  • FIXED #598 escaping problems when handling Punch runtime error data

Ahmad v2.1.21 Release Changes

  • FIXED #403 PunchPlatform Admin Channel Tab : no metric from graphite
  • IMPROVED #402 Documented parameter ‘punchplatform_admin.max_monitored_kafka_topics_frames_size_in_bytes” that must be used to cope with “consumerfetchsizetoosmall” exception in PunchPlatform self monitoring service.
  • IMPROVED #397 Log4j configuration to rotate kafka logs
  • IMPROVED #406 Log4j configuration to rotate elasticsearch logs
  • IMPROVED #461 Elasticsearch spout extraction timeslice width is now customizable for faster small extractions

Ahmad v2.1.20 Release Changes

  • IMPROVED #394 upgrade grafana from 1.9.0 to 3.1.1
  • FIXED #370 kibana & grafana in https
  • FIXED #227 LMC log in partition /var/log

Ahmad v2.1.19 Release Changes

  • FIXED #341 lumberjack client fail to ack some tuple under high load
  • FIXED #340 syslog bolt should not close to quickly its socket to slow peer
  • FIXED #338 Punchplatform-admin-python : division of None by int if the age of an index cannot be calculated
  • FIXED #337 Punchplatform-admin doesn’t show channels
  • IMPROVED #340 Punchplatform-admin-python : syslog bolt keep up longer with slow servers.

Ahmad v2.1.18 Release Changes

  • FIXED #331 Punchplatform-admin, wrong age for es indexes
  • FIXED #329 Punchplatform-admin-java uses too much thread

Ahmad v2.1.17 Release Changes

  • NEW #313 Add ioc example in standalone
  • IMPROVED #318 Add the possibility to change the name of the optional fields (local_host, local_port, etc.) by configuration in spout_settings
  • IMPROVED #314 Add filter in jinja template for environment variable ‘env_override’
  • FIXED #315 Syslog spout: when listening in UDP, logs seems to be truncated after 2048 characters
  • FIXED #310 Overfly readibility on latency indicator square
  • FIXED #322 Kafka Consumer does not consume correctly out of order batches

Ahmad v2.1.16 Release Changes

  • FIXED #307 Punch escaping of KV operator doesn’t work if the field end with ‘’
  • FIXED #304 ConcurrentModificationException in OffsetTree class
  • FIXED #302 Even if the propagation is disabled, a proxy reporter is instantiated -> out of memory !
  • FIXED #301 The prefix “punchplatform” is hard coded for autotests
  • FIXED #298 Remove tracermetrics.punch and concurrency problems for autotests
  • FIXED #297 Kafka Spout fail if its autotest fail

Ahmad v2.1.15 Release Changes

  • FIXED #289 Kafka consumer can fail when Zookeeper is unavailable
  • FIXED #290 KV and CSV fail with inner quotes.
  • IMPROVED #288 Replace localOrShuffleGrouping by shuffleGrouping for metrics topology
  • IMPROVED #292 KafkaSpout traffic decreasing is now optional and configurable

Ahmad v2.1.14 Release Changes

  • FIXED #279 Unused channel persisted structure when stopping/restarting channel
  • FIXED #276 Infinite recursive call in TupleImple class (Punch)
  • FIXED #271 Wrong behaviour of Kafka Spout group (bad zookeeper Rule computing)
  • FIXED #254 Punchplatform-channel.sh –restart stops only the first channel
  • FIXED #225 Wrong behaviour of channel restart when channel structure has evolved

Ahmad v2.1.13 Release Changes

  • IMPROVED #261 improves memory usage and data recopy.
  • FIXED #265 multi destination groups have poor performance when a destination is missing.

Ahmad v2.1.11 Release Changes

  • FIXED #252 one log out of n is lost when autometrics are turned on. Regression of v2.1.9.
  • FIXED #253 LumberjackSpout overwrites the local_uuid and similar fields
  • FIXED #256 added option to disable escaping in the KV operator

Ahmad v2.1.10 Release Changes

  • NEW #235 smart spout and bolt placement
  • FIXED #246 Lumberjack Spout packet corruption when more than one client is connected
  • FIXED #221 Lumberjack Spout must support uique id and timestamp
  • FIXED #138 With the search tools of sphinx documentation, we can retrieve the development doc in the packaging doc.
  • FIXED #89 Doc: duplicate title “HOWTO debug admin service python code” with different content
  • FIXED #79 Punch API documentation is not available through the UserGuid
  • FIXED #9 logs injected through punchplatform-log-injector.sh should not have user-locale-dependent date format.
  • IMPROVED #234 socket endpoints can be configured with a read_timeout_ms property

Ahmad v2.1.9 Release Changes

  • NEW #216 multi destination group for geo-redundant/dual room setup
  • NEW #197 multiline codec now accepts a starts-with configuration.
  • NEW #194 XML codec available from FileSpout and sockets spouts
  • IMPROVED #188 All the metrics are propagated by the proxy mode
  • IMPROVED #176 addressed permission denied on graphite storage
  • IMPROVED #150 Support of “negative” latency due to Time deynchronization between LTR and LMC
  • IMPROVED #191 All the spouts have the possibility to launch metrics
  • IMPROVED #189 Removed the ‘by component’ latency health from the algorithm which calculates the health of a channel
  • IMPROVED #217 Socket Bolts configurable timeouts for initial connection to server(s)
  • IMPROVED #215 the file spout is capable of reemiting failed tuples
  • IMPROVED #200 Added a metrics section with property “propagation_enabled” in topology files
  • FIXED #206 The latencies of the path in the output toplogies are not displayed
  • FIXED #201 ssl certificate (client or server role) propagation
  • FIXED #193 remove the health per storm component indicator
  • FIXED #185 Java admin service returns empty channels when one of the tenants has no channels configured (unhandled exception in loop)
  • FIXED #183 punchplatform-channel.sh –force-local-version and –persist-structure not working
  • FIXED #87 Missing field from a tuple does not trigger a unit test failure
  • FIXED #56 per tenant retention configuration documentation

Ahmad v2.1.8 Release Changes

  • FIXED #195 Memory leak for the selfmonitoring metrics
  • FIXED #196 Multiple sending of metrics (selfmonitoring metrics) to the reporters (graphite and slj4f)

Ahmad v2.1.7 Release Changes

  • FIXED #184 PunchPlatform –reload doesn’t persist the channel structure
  • Factory setting for latency metrics selection scope for channel health monitoring set to 10 minutes.

Ahmad v2.1.6 Release Changes

  • NEW Feature #181: Reduced the number of metrics
  • IMPROVED #188 The timeout of graphite client must be increased in consistency with the cache configuration for heavy graphite latency (2 minutes).

Ahmad v2.1.5 Release Changes

  • NEW Feature #178: Metrics must be injected by the syslogspout. Removes the injector spout “_tracer” (see Ahmad v2.1.3 below for impact on older version channels configuration.)
  • FIXED #177 Admin API and GUI freeze when graphite latency raises

Ahmad v2.1.4 Release Changes

  • NEW Feature #174: FileBolt is available. It lets you dumps your logs on external storage in simple CSV raw format.
  • IMPROVED Feature #168: Add metrics to extractor spout

Ahmad v2.1.3 Release Changes

NEW FEATURES AND FIXES

  • NEW Feature #172: FailerBolt is available with various tuple failure mode
  • NEW Feature #151: Filtering request in elasticsearch spout
  • FIXED #171 Kafka offset disorder acknowledgements
  • FIXED #113 Too verbose punchplatform-channel.sh –start

Migrating from Ahmad V2.1.2

Because of Feature #178 in progress (latency metrics simplification), in order to activate the metrics in this version (and all following versions), you have to add a new ‘self_monitoring.activation’ property in your topology file for each syslog_spout.

"spouts" : [
        {
            "type" : "syslog_spout",
            [...]
             "spout_settings" : {
                    [...]
                    "self_monitoring.activation" : true,
                    [...]
             }
        }
]

The “self_monitoring.activation” allows the syslog_spout to inject a metric :

autotest.[punchplatform_X].[tenant_X].[channel_X].[topology_X].[component_X].[punchplatform_Y].[tenant_Y].[channel_Y].[topology_Y].[component_Y] where X is the injector and Y the receiver which launches the latency metrics between the injection time and the current time. component_X is the name of the injector syslog_spout. (Before it was the name of the topology which injects the selfmonitoring metrics)

The old section “self_monitoring” is not necessary in the topology files and will be ignored if present.

Because of this change, all channel structure files defining latency monitoring paths have to be corrected : start points based on a component name identical to the topology name should be replaced by the component name of the first topology spout. This is required to provide health monitoring of latency in the channels tab of the admin service gui.

Ahmad v2.1.2 Release Changes

NEW FEATURES AND IMPROVEMENTS

  • NEW Feature #12: Channels monitoring and autotesting do not rely on injector (needed for LTR V2)
  • NEW Feature #13: Channel health uses latency measure to provide more progressive status (YELLOW)

Feature #34: Integration of the “integration test” artifact in automatic tests

  • NEW Feature #72: Elasticsearch Extract/Replay jobs
  • NEW Feature #99: [PUNCH] grok operator supports several patterns
  • NEW Feature #125: uuid and timestamp insertion without punchlets (performance optimization)
  • NEW the standalone support Spark for testing LMC platform openess to third party tools.
  • Improvement #119: Integration of bower/grunt in maven to complete automatic build
  • Improvement #128: Adapted deployer for LTR V2
  • Improvement #129: Ansible used to deploy the new admin Java service (no firewall rules change needed) to prepare refactoring/maintainability of admin service
  • Improvement the KafkaSpout supports a batch transactional mode.
  • Improvement a new AggregatorBolt is available to perform various counts.
  • NEW log extract function included in kibana 3
  • IMPROVED better documentation, performance tools delivered as part of the standalone.

ISSUES SOLVED BY THIS RELEASE

  • FIXED #116 Kafka offset documentation
  • FIXED Bug #71: Delete information about Security Center Client
  • FIXED Bug #86: CefOperator not usable in punchlet
  • FIXED Bug #93: Response time of Admin REST API degrading over time
  • FIXED Bug #94: Dependency instability of log (sl4fj, log4j, log4j2, logback)
  • FIXED Bug #98: Missing suggested listening in documentation
  • FIXED Bug #104: Metrics slow down topologies
  • FIXED Bug #105: Indexes in housekeeping error are filtered in admin GUI
  • FIXED Bug #106: Reactivation of archives housekeeping after end of failure does not work
  • FIXED Bug #108: Channel reload uses old channel configuration
  • FIXED Bug #112: Grouping declaration of punchlets results in bug in IHM Admin.
  • FIXED Bug #121: Demonstration graphite container in standalone distribution spawns multiple collectd processes
  • FIXED Bug #126: Graphite sender for the selfmonitoring metrics must be launched in new Thread
  • FIXED Bug #130: compression metrics are wrong
  • FIXED Bug #133: Standalone setup does not apply memory limit to elasticsearch
  • FIXED Bug #137: Suppression of the certificate and key when building the netty server.
  • FIXED Bug #158: Missing favicon.ico in PunchPlatform admin
  • FIXED Bug #146: With some settings, the restore snaphot function doesn’t work
  • FIXED Bug #161: When archiving indexes, housekeeping health is non-nominal, or archiving is shifted by up to 23 hours

Migrating from Ahmad V1

configuration changes

From Ahmad V1 or earlier LMC version, mandatory configuration changes are needed in

  • punchplatform.properties
  • punchplatform-deployment.settings
  • grafana dashboards

Please refer to the following document in earlier PunchPlatform distributions : README.migration_1.1.4_TO_2.1.0.rst

In addition, optional optimization can be used by changingle channels templates and configuration file for : - Reduction of CPU consumption and IOs at main Storm cluster level, through grouping and execution of multiple Punch files within a single Punch bolt in the parsing/filter topology - Reduction of CPU consumption at the input/LTR level through moving the input punchlet from the input topology to the parser topology first Punch bolet (new input punchlet based on the assignment of timestamp, unique id and input data to storm fields by the first LMC spout, instead of assigning them within the json document by an initial punchlet) - More efficient parsing/punch error handling by replacing explicit management of error logs in the main storm stream by using ‘throw new PunchRuntimeException(“message”)’ construct and leaving the stream management/log sending to elasticsearch to the LMC implicit error streams.

technical components update

This release includes changes of LMC Administration/Monitoring service, associated Java libraries used to start/restart/reload log channels, and LMC deployers (standalone and cluster modes).

Ahmad V2 relies on a Required upgraded version of Apache Storm : apache-storm-0.10.0

Although This release does not require an upgrade of Kafka framework, the default internal LMC frame format within Kafka is changed.

For suggested migration principle in order to avoid mixing of old and new formats, and to ensure no log loss and reduced real-time down-time at Elasticsearch level during migration

Known Issues in this release

Bug # 113 : too verbose punchplatform-channel.sh –start Bug #132 : Standalone installation fails with the new version of virtualenv (MacOS) Bug # 102 : Bad ES Health evaluation by HAPROXY when ES responding too slowly Documentation Bug #153 : Two setup cluster setup sections in documentation Documentation Bug #111 : PDF documentation generation errors Documentation Bug #89 : duplicate title “HOWTO debug admin service python code” with different contents Bug #15 : Completion not working for punchplatform-cluster.sh Bug #9 : Logs injector date format behaviour should not change depending on user locale Documentation Bug #138 : With the search tools of sphinx documentation, we can retrieve the development doc in the packaging doc.

Ahmad v2.0.0 Release Changes

  • IMPROVED CidrMatch operator does not throw exception in case of empty or invalid input tuple
  • NEW the KafkaSpout can filter out PunchPlatform streams (error, metrics, autotest) by configuration
  • NEW FindByInterval operator supports result ordering, by smallest or largest interval range
  • NEW Version Management based on logical name (Ahmad) logical version (Ahmad-v1) and archives version number (v2.0.0).
  • NEW Topologies are visually graphed in the gui
  • NEW New auto metrics injected from within topologies, and reported to graphite along the way. Using this strategy the monitoring is always available even if the final topology or elasticsearch is unavailable.
  • NEW Fields are propagated from one topology to another using a lumberjack encoding. This result in a significant CPU improvement are no parsing is necessary to emit the right fields in the topology.
  • IMPROVED the reference time attached to a log can be set in a punchlet anywhere in the channel. No more configuration is required in the admin service in order to deduce the timestamp associated to a log stored in Kafka.
  • NEW Lumberjack protocol v2 support. This makes it possible to use Elasticsearch beats log injector to send their data to the PunchPlatform syslog bolt. Many beats are available including the packetbeats to capture traffic data on the wire.
  • NEW Compression is provided in Kafka topics.
  • NEW a new File spout is provided. It allows you to read one or several files, line by line, run an optional punchlet on each line, and ingest it in a topology.
  • NEW PunchPlatform Admin GUI and REST API has a new information in “kafka” section : in addition of the offsets of “oldest” and “newest” messages stored in each topic partition, the service also provides the input timestamp of the message (i.e. the time when the message first entered punchplatform).
  • NEW Houskeeping has a new “online with archive” retention phase : archiving can start while indexes are still online. Archives are then incrementally updated daily until end of online retention. This allows for handling late arrival of logs, while still having the benefit of the end-of-the-day archive (and available on the first day, regardless of the online retention period)
  • NEW Housekeeping enables tenant-specific retention phase settings
  • IMPROVED Housekeeping is now less verbose, and allows for more frequent housekeeper cycles (typically 1 cycle each minute) without flooding the admin service log. This improves system reaction time to user requests for reloading index archives. Activation requires configuration update, although the configuration change is backward compatible.
  • NEW All spouts accept a rate limiter configuration, allowing you to either limit the topology rate to a fixed value expressed in tuple emitted per seconds, or to only request the spout to slow down tuple ingestion should the topology experience an overload situqtion. This feqtures prevents your topologies to be overloqded and qt the end, shot down and restarted by the Storm supervisor.
  • #48 NEW Metrics have been revamped and renamed so as to limit the number of new metrics sent to Graphite. Refer to the migration guide for adapting your Grafana dashboards
  • NEW more metrics are available in particular to monitor incoming and outgoing socket traffic and compression
  • NEW the Lumberjack compression can be set at socket level and/or at Lumberjack frame level.
  • IMPROVED the syslog bolt will not be affected by a slow remote peer when configured with the “drop_if_queue_full” property set to on.
  • NEW inner classes are allowed in Punchlet using the explicit Java API
  • #96 IMPROVED Punch uuid
  • #95 IMPROVED error message end-to-end trip
  • #77 IMPROVED CPU usage for serializing JSON to Punch Tuple in one pass.
  • IMPROVED Lumberjack unit test
  • #55 FIXED KV operator bug
  • #41 FIXED CSV wrong error reporting
  • #100 FIXED findByInterval and asArray should not return an empty list in case the Punch Tuple is empty
  • #27 FIXED Undefined variable kibana_distrib for the role kibana-server
  • #31 FIXED Graphite injection of self monitoring test metrics

Ahmad v1.1.7 Release Changes

  • IMPROVED Documentation
  • FIXED [SYN-3019] Make the topology fail if a declared resource file is not found
  • FIXED [SYN-3018] ElasticsearchBolt supports Tuple pruning.
  • FIXED [SYN-3017] SyslogBolt supports Tuple pruning.
  • FIXED [SYN-3016] ElasticsearchBolt non acknowledgement of tuples in case of partial batch sending

Ahmad v1.1.6 Release Changes

  • IMPROVED Documentation
  • NEW lumberjack based LTR.
  • NEW generic incoming and outgoing traffic summary metrics.
  • FIXED [SYN-3015] allow several syslog spouts in the same topology.
  • FIXED [SYN-3014] correct lookup strategy for loading json file resources.

Ahmad v1.1.5 Release Changes

-IMPROVED Documentation reworked to provide administrator and maintenance guide sections, including cluster deployment procedure, overview of administration/maintenance tools and metrics documentation

Ahmad v1.1.4 Release Changes

  • NEW KafkaSpout only reemit failed tuples
  • NEW support localOrShuffle grouping storm strategy
  • NEW Punch getHostIp method favors IPV4 over IPV6 addresses
  • NEW Extended standard log parsers
  • IMPROVED parser SDK allow grok overloading and JSON resource files
  • IMPROVED parser SDK support a new less verbose test API
  • FIXED [SYN-3013] PunchRuntimeException makes the log indexed in the error Elasticsearch type.

Ahmad v1.1.3 Release Changes

  • NEW Compatibility ensured to elasticsearch 1.5.2
  • NEW EsperBolt for real time correlation
  • IMPROVED GUI made much more usable (thanks to Pierre for great contribution)
  • NEW Embedded Elasticsearch for document on the fly enrichment fully supported.

Ahmad v1.1.2 Release Changes

  • FIXED [SYN-3012] conflicts when loading several resource json files with the same name length.
  • IMPROVED Syslog bolt can be configured with an adaptative overload control algorithm to prevent topology flooding.
  • IMPROVED More documentation, better structure
  • NEW NEW Punch CEF operator to extract arsight CEF messages and transform them into Punch Tuples.

Ahmad v1.1.1 Release Changes

  • ** FEATURE [SYN-3155]** Elasticsearch mappings/index
    templates are now automatically sychronized/checked by admin service periodically, with tracking of GIT revision inside metadata available in Elasticsearch, and with synchronization details available in “Config” status page of Admin service.
  • IMPROVED [SYN-3013] TCP Syslog spout and Kafka Bolt now
    implement message acknowledgement and buffer/flow control in order to protect messages against destruction in case of failure in exchanges with Kafka and in order to prevent out of memory during input flooding
  • IMPROVED [SYN-3150] Kafka partition health now includes
    yellow state for underreplication.
  • FIXED [SYN-3012] IMplemented a defaulting insertion
    method in case of elasticsearch injection repeated rejection (for example, discrepancy between document and mapping) : errors are inserted in a default document type with raw, escaped document.
  • FIXED [SYN-3274] Kafka is now using production network
    to communicate with zookeeper, and between brokers
  • FIXED [SYN3158] Supervisord is now autostarted at boot
    on LTRs
  • FIXED [SYN3157] Apache2 is now fully under pacemaker
    control (if present) on graphite servers (therefore not autostarted on boot)
-IMPROVED [SYN-3031] based on Kafka 0.8.2 and associated
python client v0.9.3, YELLOW status is now provided for KAFKA topics in Admin API and HMI when failed brokers cause under-replication of partitions.
  • NEW multi-port injector enabling finer logs
    customization and defining injection session templates with multiple technologies.

Ahmad v1.1.0 Release Changes

-DEPENDENCIES UPGRADES*

  • Elasticsearch Version 1.4.4 solving througput performance issue when archiving indexes.
  • Kafka Version 0.8.2,**FIXING CRITICAL BUG [SYN-3150]***
  • Kibana 3.1.2, bringing additional features
-FIXED [SYN-3151] Admin service elasticsearch monitoring now
reports individual elasticsearch clusters health even when one of them is unreachable, or has improperly configured/mounted archiving repository.

Ahmad v1.0.0 Release Changes

  • NEW Administration Service includes elasticsearch data
    housekeeping (archiving/destroying of indexes depending on their age)
  • NEW Administration Service REST API and User Interface
    provide details on elasticsearch clusters and indexes health and status
  • NEW ElasticsearchBolt fully configurable.
  • NEW ElasticsearchBolt sends failed documents into an
    error index
  • NEW NEW naming using punchplatform/PunchPlatform.
  • NEW NEW PunchPlatform deployer, ansible-based allowing
    full automated deployment of a PunchPlatform cluster, including High-Availability and Tenants Isolation patterns, and metrics collection and graphing both at operating system and applicative levels
  • NEW load balancing for elasticsearch clusters
    implemented in punchplatform-cluster deployment through ansible- impact on punchplatform.properties files
  • NEW Git based configuration directory with automatic
    configuration synchronization and check by the PunchPlatform admin services
  • NEW trace/autotest message also reported by Kafka spout
  • NEW zen unicast discovery mode supported by
    Elasticsearch bolt.
  • CHANGED logger names consistent and documented
  • CHANGED elasticsearch cluster parameter renamed in
    elastic bolt settings (impacts on existing configured channels)
  • NEW Punch CSV operator handles single quote
  • FIXED Punch smaller/greater/smallerEq/greaterEq
    operators
  • FIXED Punch allows ‘-‘ in property name

Ahmad v0.5.0 Release Changes

  • NEW support for multiple storm clusters (upgrade needed
    in punchplatform.properties)
  • NEW compatibility with ansible deployment
  • ** CHANGED** “install -a” mode is now available for setup
    with an existing configuration

Ahmad v0.4.0 Release Changes

  • NEW Kafka status monitoring through PunchPlatform REST
    API, including under-replication alerts when kafka nodes are in non-nominal status but channels are still operationals.
  • NEW Nominal transport for elasticsearch client is
    “node” mode

Ahmad v0.3.5 Release Changes

-NEW [SYN-3031] PunchPlatform admin service REST API and

associated GUI now report

  • detailed kafka cluster, kafka topics and partition offsets and
  • overall PunchPlatform health, with salient alerts displayed in header page
  • individual channels health, computed by automatic autotesting by PunchPlatform admin REST API service.

-FIXED [SYN-3016] PunchPlatform admin service GUI does not expects internet access for CSS fetching anymore.

-FIXED Punch divide operator -NEW autotest/trace supported for arbitrary topologies -NEW upgraded to elasticsearch 1.4.1

Ahmad v0.3.4 Release Changes

-NEW upgraded storm to 0.9.3 -NEW all storm topologies are supported. This includes multi-stream and multi-fields groupings. -NEW Punch can drop Tuples, redirect them to various streams, add fields. -NEW Csv operator is smarter, and deals with escaping. -NEW redmine bugtrack system in place -NEW new Kafka supervision rest API and GUI. -NEW [SYN-3009] zsh support

-FIXED trim for Csv operator. -FIXED Punch bolt Storm Tuple anchoring is effective.

HOW TO UPGRADE FROM v0.3.

- Patch PunchPlatform admin server(s) and servers running local-mode topologies.
    - unzip and install the new archive
- No mandatory stop/restart or reload

Checks : after the upgrade, run

> punchplatform-channel.sh --autotest <tenant>/<channel>

Success means that your channel structure is correctly updated.

v0.3.3

-IMPROVED PunchPlatform shell script logs are now rotating to avoid ever-increasing size in case of use for automatic monitoring

-LIMITATION incorrectly advertised as upgraded storm to 0.9.3

v0.3.2

-FIXED slow Punch compilation with many grok patterns -LIMITATION incorrectly advertised as upgraded storm to 0.9.3

v0.3.1

-NEW syslog bolt with acknowledgement capabilities. -NEW FindByKey operator capable of building index for innerfields -FIXED automatic auto test for supervision -FIXED admin REST API and GUI blocking -FIXED punchplatform-punch.sh v0.3.0 regression.

v0.3.0

-IMPROVED punchplatform-inject.sh can now inject traffic
directly into kafka topics. Using the PunchPlatform format.
-IMPROVED better error/warrning messages in case of
unconfigured or misconfigured punchplatform_admin section in PunchPlatform properties file
-IMPROVED ‘punchplatform-channel.sh –autotest’ now works also
for channels which input points are not syslog tcp listening “local-mode” topologies, but are instead kafka topic(s)

-IMPROVED ‘punchplatform-channel.sh –autotest’ now works faster, independently on the ES refresh parameter, because one specific document is fetched by URL with id from ES instead of issuing a search request.

-IMPROVED punchlet resource files loading more efficient, and compressed.

-FIXED jvm metrics are now published to graphite with correct naming for “output” topologies using elasticsearch bolt

-FIXED PunchPlatform admin/monitoring api service now logs into a rotating log to avoid file system fill-up

v0.2.10

-KNOWN BUG punchplatform-channel.sh –autotest does not work for channels without a syslog spout.

-IMPROVED PunchPlatform admin service shows uptime of channel

-NEW custo directory to make it easier to patch jars

-FIXED key value operator.

v0.2.9

-NEW: non linear topology

-NEW: metric punchlet API

-NEW: PunchPlatform administration REST API and simple gui

-FIXED: key value operator

v0.2.8

-FIXED: new parameter for last elasticsearch

-FIXED: removed doc references to rivers

-NEW: punchlet can be full Java classes

v0.2.7

-NEW: extensive Punch documentation addition. Delivered as generated Html from javadoc

-NEW: elasticsearch 1.4

-NEW: Punch new lookupByKey works with multi index

-FIXED: Punch operator is().in() was not working on sets

-FIXED: Punch is now camelCase consistent

-IMPROVED: Punch if () statements works with more complex statements

-FIXED es_id is now taken into account by elastic bolt

-PATCH there is an embeded watchdog in kafka consumer metrics layer to kill kafka bolt jvm if metrics are not updated frequently enough (this is a patch while waiting correction of STORM hangs) ==> NIMBUS will restart topology element so as to ensure availability.

v0.2.6

-FIXED: escaping keys when using KV operator

-FIXED: Settings enriched for kafka bolt, to allow for distinct startup offset policy when NO offset has been stored in ZK, and when the last processing offset is known from ZK ==> default settings now processes all existing messages in Kafka topic when first starting a consumer group from scratch, instead of loosing the messages already there.

v0.2.5

-FIXED: STORM hangs removed due to too big batch processing in PunchPlatform Spouts

v0.2.4

-NEW: new metrics to track kafka offsets evolution

-NEW: moved to Elasticsearch bolt instead of Elasticsearch rivers

V0.2.3

-FIXED: configuration directory layout using resources directories.

-NEW: logstash dialect supported for dynamic expressions using %{}

V0.2.2

-NEW: add and append Punch Tuple functions

-NEW: logstash dialect supported by punch

-NEW: grok patterns now loaded from tenants, tenants/aTenant, tenants/aTenant/aChannel groks directories

-NEW: updated patterns to latest logstash

-FIXED: grok bug when chaining patterns without separations

-FIXED: grok bug lossing matches if using reaptingly the same destination Tuple

-FIXED: Punch compilation error in case of inner class calls

V0.2.1

-NEW: punchplatform-cluster.sh tool for cluster deployment and monitoring (beta)

-NEW: punchplatform-channel.sh reload option implemented

-NEW: punchplatform-channel.sh works remotely through ssh.

-NEW: manual pages, channel, troubleshooting and various other documentations

-NEW: elastic search index and type now fully user configured through punch. Kafka bolt is now elasticsearch agnostic.

-NEW: new Punch operators: puid(), ruid(), ecount()

-NEW: auto test and trace capabilities of punchplatform-channel

-NEW: configuration simplified. The river related encoding (plain versus elasticsearch) disappeared.

-NEW: local topologies are now managed.

-NEW: graphite metrics.

-NEW: configuration is ulti-zone ready : multi elastic search, multi zookeeper, multi kafka

-NEW:syslog spout does not use zookeeper anymore (only optionally)

-FIXED: elastic river and spout damage partial Kafka cluster with missing leaders.

-FIXED: no more hardcoded fields : punchplatform_id, punchplatform_version, punchplatform_timestamp.

V0.2.0

-NEW: new Punch operators find_by_key, find_by_interval

-NEW: improved toDate operator

-NEW: new elastic Kafka consumer, Spout and River

-FIXED: allow ‘@’ in Punch identifiers

V0.1.5

-NEW: new Punch operators: coalesce, cidrmatch -FIXED: allow ‘_’ in Punch identifiers

V0.1.4

-NEW: safe generation of uuid in syslog-spout. Zookeeper based

-NEW: safe offset restart in elasticsearch river for resiliency

V0.1.3

-IMPROVED git and maven release. Version and packaging strategy defined.

V0.1.2

-NEW allows named regex in punch, not only grok patterns.

V0.1.1

-NEW Punch operators : toDate, toLong, toBoolean, toDouble

-NEW type inference for KV and grok operators is now configurable, off by default

-NEW elasticsearch index and type now configurable in the PunchBolt

-NEW Punchlet can be called from syslog spout, not only from Punchlet bolt

-NEW Grok operator allows for multi target destination

-NEW Support for** arbitrary number of Kafka clusters**

-NEW Support templating and automatic generation of channel configurations files using j2

-FIXED operator kv() with empty value results in a mess

-FIXED operator kv() with quoted string values should not include the quotes

-FIXED Kafka spout does not work for more than one channel : used a hardcoded client id

V0.1.0

First delivered version based on finalized poc.