Skip to content

CRAIG 5.7.4

Below is a summary of the JIRA issues addressed in the CRAIG-5.7.4 release of Punchplatform. For full documentation of the release, a guide to get started, and information about the project, see the Punchplatform project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Punchplatform.

The documentation for the most recent release can be found at https://doc.punchplatform.com.

Release summary

Operator

The punchplatform.properties file is now pushable by the operator deployment role, using the configuration :

  • punchplatform_operator.servers.<server_id>.properties_source_path

By provinding the absolute path to punchplatform.properties on the deployer system, the file will be placed inside configuration_name_dir_from_home of each operator on the aimed server

Kibana

Kibana plugins can be configured inside each domain section in punchplatform.properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "kibana" : {
    "domains" : {
      "admin" : {
        "plugins": {
          "punchplatform": {}
        }   
      } 
    }
  }
}

The punchplaform.properties configuration file can now be pushed alongside the Punchplatform Plugin using its configuration, on the deployed machine.
The file will be placed inside the tmp_path location, and this feature is disabled by default (false). This configuration can also be defined inside the global punch plugin section under kibana.plugins.punchplatform :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "kibana" : {
    "domains" : {
      "admin" : {
        "plugins": {
          "punchplatform": {
             "tmp_path": "/home/vagrant/tmp",
              "push_punch_properties": true
          }
        }   
      } 
    }
  }
}

New configurations have been added

  • kibana.plugins.punchplatform.extraction_es_data_cluster_type
  • kibana.plugins.punchplatform.extraction_es_data_cluster_hosts
  • kibana.plugins.punchplatform.extraction_es_data_cluster_port
  • kibana.plugins.punchplatform.extraction_es_data_cluster_settings
  • kibana.plugins.punchplatform.extraction_es_metric_cluster_type
  • kibana.plugins.punchplatform.extraction_es_metric_cluster_hosts
  • kibana.plugins.punchplatform.extraction_es_metric_cluster_port
  • kibana.plugins.punchplatform.extraction_es_metric_cluster_index
  • kibana.plugins.punchplatform.extraction_es_metric_cluster_settings

Visit the Punch Properties Documentation for more information.

Punch Rest API Gateway

The Punchplatform Gateway is a Restful service placed in front of the Punchplatform services such as Elasticsearch, Punchctl client for channels or PML jobs. It provides a transparency access to the Punchplatform features by an endpoint design for any external client.

Visit the Getting Started documentation for this component !

Visit the Punch Properties Documentation for this component !

Visit the Deployment Settings Documentation for this component !

Kibana Punch Plugin

The extraction performed by the Kibana Punch plugin has been reviewed. The "jobs" and "executions" screens have been merged into one. And additional useful features have been added such as "kill running, replay (save as / execute)".

Release notes

Release notes - Punch - Version Craig-5.7.4

New Feature

  • [PP-3878] - Punch EsRestClient ssl support for 5.7
  • [PP-3981] - normalize elastic_input parameters with elastic_batch_input
  • [PP-3987] - Support required() instead of loadAtStartup() for remote resources
  • [PP-4022] - Additional metric in UDP SyslogSpout reporting packets loss
  • [PP-4046] - Ability to kill an extraction by the GUI.
  • [PP-4048] - Reissue of an extraction (save as)
  • [PP-4052] - KafkaHeaders and other useful metadata publications by KafkaSpout

Improvement

  • [PP-3975] - Improve Spark logger
  • [PP-4044] - Trace operator actions and calculate the extraction duration accordingly.
  • [PP-4047] - Merge the "jobs" and "executions" screens into one
  • [PP-4084] - kafka log injector with custom headers
  • [PP-3587] - Improved ES_housekeeper template to allow "index_settings" action
  • [PP-4012] - standalone Data extraction application
  • [PP-4133] - Backport Gateway on Craig branches

Bug

  • [PP-3264] - Kafka daemon not starting due to lack of broker id
  • [PP-3586] - monitoring/metrics hardcoded ES platform index
  • [PP-3609] - ElasticsearchBolt nullpointer
  • [PP-3717] - csv operator doesn't work as expected when using | as separator
  • [PP-3948] - channel-monitoring not working with custom value of es_platform_log_index
  • [PP-3951] - concurrent exception in syslog servers
  • [PP-3952] - non-overridable 24h default kafka offsets retention
  • [PP-3958] - punchlinectl should not depend on punchplatform.properties on kibana nodes
  • [PP-3963] - Regression on default doc type in ElasticBatchInput PML node
  • [PP-3965] - Plans fail to save their cursors in es
  • [PP-3971] - bad zk connection string in kafka-topics.sh
  • [PP-3973] - Incorrect punchctl --configure result on standalone sourcefire channel
  • [PP-3976] - Extraction file formats configuration is ignored by punch plugin for kibana with deployer
  • [PP-3978] - elastic_input & elastic_batch_input: type ==null, empty_datasets for sql, file_output save empty dataset
  • [PP-3980] - Undetailed resource fetch error (Json fetching by PunchBolt)
  • [PP-3985] - Too small hard-coded read timeout in resource fetching
  • [PP-3988] - Kibana punch plugin doc path configuration is ignored by deployer
  • [PP-3989] - Extraction doesn't work when no filter is specified in Kibana plugin
  • [PP-3996] - ES prefix not included in scroll followup request in InputNode
  • [PP-4000] - extraction error due to socket timeout too short in ElasticInput node
  • [PP-4003] - [PLAN] wrong from and to timestamp after the first iteration
  • [PP-4005] - Missing options for elastic_input-based extraction in extraction-plugin-generated PML
  • [PP-4007] - Dependency to punchplatform.properties for extraction plugin but deployer does not install a copy in plugin dir
  • [PP-4008] - exists filter not supported by extraction plugin
  • [PP-4009] - Csv operator automatically escape " with \, leading to unwanted behaviour in some situation
  • [PP-4014] - Gateway ansible implementation issue
  • [PP-4017] - Deployer fails for a new kibana domain installation next to an other existing kibana domain
  • [PP-4020] - Confusion between ElasticRestClientBuilder es_socket_timeout and es_connect_timeout
  • [PP-4021] - Refactor elasticsettings: connect.timeout to es.http.timeout
  • [PP-4038] - FileBolt Tags feature does not comply with specified behaviour
  • [PP-4045] - The filter of executions and jobs in the plugin does not work
  • [PP-4088] - ElasticClientWrapper es.scroll: false does not deactivate scroll
  • [PP-4118] - Forwarding doesn't work when using Gateway for Kibana's ES hosts
  • [PP-4136] - CsvOperator - Escape Character should have empty option