Overview

A complete punch validation campaign is performed for each major release. It consists of four different sets of test:

1. Unit tests: unit tests are run after each commit. The report is attached with each release. These tests can be played by anyone with access to the inner source punch repository.
2. Robot tests: robot tests are delivered as part of the punch standalone. They consist in various end-to-end tests that require the punch services to be deployed and up and running. Robot test can be played by anyone equipped with a standalone platform. They are run on deployed punchplatforms as well.
3. Punchbox integration tests: the public punchbox project provides a validation tool that leverage ElastAlert to play sophisticated end-to-end tests, and publish a test report using an elastalert rule. Refer to the punchbox github project for details.
4. Manual integration tests: manual tests are executed in order to complete the two previous test sets to provide additional coverage.

Minor and patch releases are tested using unit, robot and punchbox tests. Depending of the scope of the patch or the minor release, some or all of the manual tests are played as well.

Variability

In addition to testing each punch component, the punch test suite cover the following variability.

OS

The punch can be deployed on three operating system families: Ubuntu, Debian and RedHat. In addition the standalone is supported on MacOs.

Security

The following components are tested with additional TLS security configurations:

• Kafka, Zookeeper, Shiva
• Kibana, Elasticsearch, Gateway : the Gateway is required by the Kibana plugins.
• Metric Reporters : towards Elasticsearch or Kafka.
• Operator command line tools
• Beats

Patch

Patching procedures are an integral part of the punch. Patching is thus also part of the test campaigns.