Skip to content

Manual Test Suite

This page lists the punch manual tests.

Tip

If you encounter an issue on your platform, performing the relevant test(s) may help your troubleshooting.

Zookeeper

Check Zookeeper punchplatform-deployment.settings

  • Prerequisite

    Section for Zookeeper.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    A successful playbook.
    Another check to do.

Deploy Zookeeper without SSL

Deploy Zookeeper with SSL

Check Zookeeper configuration file

  • Prerequisite

    Deploy Zookeeper without SSL or Deploy Zookeeper with SSL

  • Test

    cat /data/opt/apache-zookeeper-3.7.0-bin/conf/zoo.cfg

    cat /data/opt/apache-zookeeper-3.7.0-bin/conf/java.env

  • Expected

    No syntax error.
    All instances in cluster.
    SSL parameters if activated.
    Data dir=/data/zookeeper.
    4 letters words API allowed.

Check Zookeeper service file

Check Zookeeper service status

Check Zookeeper logs

Check Zookeeper Native CLI

Check Zookeeper Punch CLI

  • Prerequisite

    Deploy Zookeeper without SSL or Deploy Zookeeper with SSL

  • Test

    punchplatform-zookeeper-console.sh 

    punchplatform-zookeeper-console.sh --cluster local 

    punchplatform-zookeeper-console.sh --servers demo1:2181

  • Expected

    Opens Zookeeper CLI.
    Zookeeper commands like ls / work.

  • Robot test

    zookeeper.robot

Remote check with 4-letters keyword

Check with platformctl

Kafka

Check Kafka punchplatform-deployment.settings

  • Prerequisite

    Section for Kafka.

  • Test 

    punchplatform-deployer.sh --generate-inventory

  • Expected

    The Ansible playbook should complete without error.

Deploy Kafka without SSL

Deploy Kafka with SSL

Check Kafka configuration files

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL

  • Test

    cat /data/opt/kafka_2.12-2.8.1/config/punchplatform-local-server.properties

    cat /data/opt/kafka_2.12-2.8.1/config/kafka-topic.properties

  • Expected

    No syntax error.
    SSL parameters if activated.
    All Zookeeper nodes in zookeeper.connect.
    Each broker has a unique id.

Check Kafka service file

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL

  • Test

    cat /usr/lib/systemd/system/kafka-local.service

  • Expected

    No syntax error.
    Description=Punchplatform Kafka-{cluster-name}.
    User={punchplatform-daemon}.
    Restart=always.

Check Kafka Platform Reporters

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL
    Have at least one Kafka Platform Reporter.
    Deploy Operator.

  • Test

    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic platform-events -v

  • Expected

    Platform logs in the topic

Create Kafka topic

  • Prerequisite

    Kafka deployed

  • Test

    punchplatform-kafka-topics.sh --create --topic test-topic

  • Expected

    Topic created

Describe Kafka topic

  • Prerequisite

    Create Kafka topic

  • Test

    punchplatform-kafka-topics.sh --describe --topic test-topic

  • Expected

    Topic described

Add config to Kafka topic

  • Prerequisite

    Create Kafka topic

  • Test

    punchplatform-kafka-topics.sh --add-config retention.ms=15000000 --topic test-topic

  • Expected

    Topic described

Check offsets in Kafka topic

  • Prerequisite

    Create Kafka topic

  • Test

    punchplatform-kafka-topics.sh --offsets --topic test-topic

  • Expected

    Offsets described

Reset consumer offset for Kafka Consumer

  • Prerequisite

    channelctl -t platform start --channel monitoring channelctl -t platform stop --application monitoring/common/local_events_dispatcher

  • Test

    punchplatform-kafka-consumers.sh --shift-offsets -1000 --group platform.monitoring.local_events_dispatcher.kafka_input
punchplatform-kafka-consumers.sh --reset-to-earliest --group platform.monitoring.local_events_dispatcher.kafka_input
punchplatform-kafka-consumers.sh --reset-to-latest --group platform.monitoring.local_events_dispatcher.kafka_input

  • Expected

    Offsets changed

Check Kafka Output Node in Storm environment

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL
    Deploy Storm or Shiva (if SSL enabled).
    Modify apache_httpd channel to work in Storm or Shiva (if SSL enabled).

  • Test

    Launch channel : 

    channelctl -t mytenant start --channel apache_httpd
    Inject logs : 
    punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant/apache_httpd_injector.json
    Read Kafka topic : 
    punchplatform-log-injector.sh --kafka-consumer -brokers common -topic mytenant_apache_httpd_archiving --earliest

  • Expected

    Logs inserted in the Kafka topic.

Check Kafka Input Node in Storm environment

  • Prerequisite

    Check Kafka Output Nodes in Storm environment
    Change start_offset_strategy: earliest in archiving.hjson of apache channel.

  • Test

    Launch channel : 

    channelctl -t mytenant start --channel apache_httpd
    Check logs are archived on the assigned server : 
    tree /tmp/archive-logs/storage/mytenant/apache_httpd

  • Expected

    Archiving logs

Check Kafka Batch Output Node in Spark environment

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL
    Deploy Spark.
    Copy example in a file.

  • Test

    Launch punchline : 

    punchlinectl --tenant mytenant start -p /path/to/your/example/file.hjson
    Read Kafka topic : 
    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic jonathan -v

  • Expected

    Logs inserted in the Kafka topic.

Check Kafka Batch Input Node in Spark environment

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL
    Deploy Spark.
    Copy example in a file.

  • Test

    Launch punchline : 

    punchlinectl --tenant mytenant start -p /path/to/your/example/file.hjson
    Check Elasticsearch index kafka-offset-2020.06.08 : 
    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic jonathan -v

  • Expected

    Logs inserted in the Kafka topic.

Check Kafka Stream Output Node in Spark environment

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL
    Deploy Spark.
    Copy example in a file.

  • Test

    Launch punchline : 

    punchlinectl --tenant mytenant start -p /path/to/your/example/file.hjson
    Check Elasticsearch index kafka-offset-2020.06.08 : 
    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic jonathan -v

  • Expected

    Logs inserted in the Kafka topic.

Check Kafka Stream Input Node in Spark environment

  • Prerequisite

    Deploy Kafka without SSL or Deploy Kafka with SSL
    Deploy Spark.
    Copy example in a file.

  • Test

    Launch punchline : 

    punchlinectl --tenant mytenant start -p /path/to/your/example/file.hjson
    Check Elasticsearch index kafka-offset-2020.06.08 : 
    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic jonathan -v

  • Expected

    Logs inserted in the Kafka topic.

## Storm ### Launch channel in Storm

  • Prerequisite

    Deploy Storm Deploy Operator Have apache_httpd channel from standalone

  • Test

    channelctl -t mytenant start --channel apache_httpd

Shiva

Check Shiva punchplatform-deployment.settings

  • Prerequisite

    Section for Shiva.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Shiva without SSL

  • Prerequisite

    Section for Gateway.

  • Test

    punchplatform-deployer.sh --deploy -t shiva

  • Expected

    Successful playbook.

Deploy Shiva with SSL

  • Prerequisite

    Section for Gateway.

  • Test

    punchplatform-deployer.sh --deploy -t shiva --become -e  @/path/to/deployment_secrets.json

  • Expected

    Successful playbook.

Check Shiva configuration file

  • Prerequisite

    Deploy Shiva without SSL or Deploy Shiva with SSL

  • Test

    cat /data/opt/punch-shiva-6.3.0/conf/shiva.conf

  • Expected

    Each node have unique identifier.
    Reporters with all required settings.
    SSL parameters if activated.
    Control topic : platform-shiva-{cluster}-ctl.
    Command topic : platform-shiva-{cluster}-cmd.
    Assignement topic : platform-shiva-{cluster}-assignement.
    Worker tags match at least cluster name and hostname.

Check Shiva service file

  • Prerequisite

    Deploy Shiva without SSL or Deploy Shiva with SSL

  • Test

    cat /usr/lib/systemd/system/shiva-runner.service

  • Expected

    No syntax error.
    Description=Punchplatform Shiva.
    User={punchplatform-daemon}.
    Restart=on-failure.
    Hostname variable is right.

Check Shiva service status

Check Shiva logs

  • Prerequisite

    Deploy Shiva without SSL or Deploy Shiva with SSL

  • Test

    sudo journalctl -lf -u shiva-runner.service

    tail -f /var/log/punchplatform/shiva/shiva-runner-daemon.log

  • Expected

    No error logs.
    No deprecation logs.
    Regular updates for assignements.

Check Shiva with platformctl

Start Shiva channel

Stop Shiva channel

Check Shiva Applications logs

Shutdown a Shiva worker

Deploy Shiva without SSL or Deploy Shiva with SSL
Start Shiva channel

  • Test

    sudo systemctl stop shiva-runner.service

  • Expected

    Tasks are reassigned to other workers.

Shutdown a Shiva leader

Deploy Shiva without SSL or Deploy Shiva with SSL
Start Shiva channel

  • Test

    sudo systemctl stop shiva-runner.service

  • Expected

    New leader is elected.
    Tasks are reassigned to other workers.

Elasticsearch

Check Elasticsearch punchplatform-deployment.settings

  • Prerequisite

    Section for Elasticsearch.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Elasticsearch without SSL

  • Prerequisite

    Section for Elasticsearch.

  • Test

    punchplatform-deployer.sh --deploy -t elasticsearch

  • Expected

    Successful playbook.

Deploy Elasticsearch with Opendistro

  • Prerequisite

    Section for Opendistro.

  • Test

    punchplatform-deployer.sh --deploy -t elasticsearch

  • Expected

    Successful playbook.

Deploy Elasticsearch with SSL

  • Prerequisite

    Section for Gateway.

  • Test

    punchplatform-deployer.sh --deploy -t elasticsearch --become -e  @/path/to/deployment_secrets.json

  • Expected

    Successful playbook.

Check Elasticsearch configuration file

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL

  • Test

    cat /data/opt/elasticsearch-7.10.2/config/elasticsearch.yml

    cat /data/opt/elasticsearch-7.10.2/config/jvm.options

  • Expected

    Each node have unique identifier.
    All nodes are in cluster.
    Minimum master nodes = nb_nodes / 2 + 1.
    Opendistro parameter if activated.
    SSL parameters if activated.
    Xms = Xmx

Check Elasticsearch service file

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL

  • Test

    cat /usr/lib/systemd/system/elasticsearch.service

  • Expected

    No syntax error.
    Description=Punchplatform Elasticsearch.
    User={punchplatform-daemon}.
    Group={punchplatform-daemon-group}.
    Restart=on-failure.
    Hostname variable is right.

Check Elasticsearch service status

Check Elasticsearch logs

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL

  • Test

    sudo journalctl -lf -u elasticsearch.service

    tail -f /var/log/punchplatform/elasticsearch/es_search.log

  • Expected

    No error logs.
    No deprecation logs.
    SSL Problem Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16) -> Not an issue for Elasticsearch cluster.

Check Elasticsearch with platformctl

Push Elasticsearch templates

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL Deploy Operator.

  • Test

    Without Opendistro : 

    punchplatform-push-es-templates.sh -d $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/templates/ -l http://demo3:9200
    With Opendistro : 
    punchplatform-push-es-templates.sh -d $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/templates/ -u {opendistro.user} -p {opendistro.password} -l http://demo3:9200
    With SSL and Opendistro : 
    punchplatform-push-es-templates.sh -d $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/templates/ -u {opendistro.user} -p {opendistro.password} --ssl -ca $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt -ce $PUNCHPLATFORM_SECRETS_DIR/server.crt -pk $PUNCHPLATFORM_SECRETS_DIR/server.pem -l https://demo3:9200

  • Expected

    Mappings successfully pushed.

Curl cluster state

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL

  • Test

    Without Opendistro : 

    curl -v http://demo1:9200/_cat/nodes?v
    With Opendistro : curl -v http://demo1:9200/_cat/nodes?v -u {opendistro.user}:{opendistro.password} 
    With Opendistro and SSL :
curl -vk https://demo1:9200/_cat/nodes?v -u {opendistro.user}:{opendistro.password} --key $PUNCHPLATFORM_SECRETS_DIR/server.pem --cert $PUNCHPLATFORM_SECRETS_DIR/server.crt --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt

  • Expected

    All nodes are in cluster.

Create new Opendistro user

  • Prerequisite

    Deploy Elasticsearch with Opendistro.
    Generate a hash password with /data/opt/elasticsearch-7.10.2/plugins/opendistro_security/tools/hash.sh.

  • Test

    Add user in /data/opt/elasticsearch-7.10.2/plugins/opendistro_security/securityconfig/internal_users.yml. Update Opendistro configuration : 

    /data/opt/elasticsearch-7.10.2/plugins/opendistro_security/tools/securityadmin.sh -cd ../securityconfig/ -icl -nhnv -h demo3 -cacert ../../../config/fullchain.crt -cert ../../../config/admin.crt -key ../../../config/admin.pem

  • Expected

    No error. New user in Opendistro.

Check Extraction Input Node in Storm

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL
    Deploy Operator.
    Have elasticsearch_to_stdout.hjson from standalone.

  • Test

    Resolve file to check parameters : 

    punchlinectl -t mytenant resolve -f /path/to/punchline/elasticsearch_to_stdout.hjson
    Launch punchline : 
    punchlinectl -t mytenant start -p /path/to/punchline/elasticsearch_to_stdout.hjson

  • Expected

    Logs in the targetted ES index.

Check Elasticsearch Output Node in Storm

  • Prerequisite

    Deploy Elasticsearch without SSL or Deploy Elasticsearch with SSL
    Deploy Operator.
    Have stormshield_networksecurity channel.

  • Test

    Resolve file to check parameters : 

    punchlinectl -t mytenant resolve -f $PUNCHPLATFORM_CONF_DIR/tenants/mytenant/channels/stormshield_networksecurity/input.hjson
    Launch punchline : 
    punchlinectl -t mytenant start -p $PUNCHPLATFORM_CONF_DIR/tenants/mytenant/channels/stormshield_networksecurity/input.hjson
    Launch injector : 
    punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant/stormshield_networksecurity_injector.json

  • Expected

    Logs in the targetted ES index.

Check Elasticsearch Input Node in Spark

Check Elasticsearch Output Node in Spark

Check Elasticsearch Input Node in Spark

Kibana

Check Kibana punchplatform-deployment.settings

  • Prerequisite

    Section for Kibana.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Kibana without SSL

  • Prerequisite

    Section for Kibana.

  • Test

    punchplatform-deployer.sh --deploy -t kibana

  • Expected

    Successful playbook.

Deploy Kibana with SSL

  • Prerequisite

    Section for Kibana.

  • Test

    punchplatform-deployer.sh --deploy -t kibana --become -e  @/path/to/deployment_secrets.json

  • Expected

    Successful playbook.

Check Kibana configuration file

  • Prerequisite

    Deploy Kibana without SSL or Deploy Kibana with SSL

  • Test

    cat /data/kibana/admin/kibana-7.10.2-linux-x86_64/config/kibana.yml

  • Expected

    No syntax error.
    console.enabled: false.
    SSL parameters if activated.
    elasticsearch.username: punchkibanaserver.
    elasticsearch.password: punchkibanaserver.

Check Kibana service file

Check Kibana service status

Check Kibana logs

Push Kibana resources

  • Prerequisite

    Deploy Kibana without SSL or Deploy Kibana with SSL Deploy Operator.

  • Test

    Without Opendistro : 

    punchplatform-setup-kibana.sh --import -l http://demo1:5601/kibana
    With Opendistro : 
    punchplatform-setup-kibana.sh --import -l http://demo1:5601/kibana -u {{ opendistro.username }}:{{ opendistro.password }}
    With Opendistro and SSL : 
    punchplatform-setup-kibana.sh --import -l https://demo1:5601/kibana -u {{ opendistro.username }}:{{ opendistro.password }} --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt

  • Expected

    Logs in the targetted ES index.

Create index pattern

Gateway

Check Gateway punchplatform-deployment.settings

  • Prerequisite

    Section for Gateway.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Gateway without SSL

Deploy Gateway with SSL

Check Gateway configuration file

  • Prerequisite

    Deploy Gateway without SSL or Deploy Gateway with SSL

  • Test

    cat /data/opt/punch-gateway-6.3.0/conf/application-gateway.yml

  • Expected

    Gateway IP and port.
    SSL parameters if activated.
    Punchplatform tools activated or not.
    Elasticsearch cluster and credentials.
    Doc dir exists.
    Forwarding punchlet exists.

Check Gateway service file

Check Gateway service status

Check Gateway logs

  • Prerequisite

    Deploy Gateway without SSL or Deploy Gateway with SSL

  • Test

    sudo journalctl -lf -u gateway-mytenant.service

    cat /var/log/punchplatform/punch-gateway-6.3.0/gateway-mytenant.log

  • Expected

    No error logs.
    No deprecation logs.
    Code 200 for requests.

Check Elastic endpoint with SSL

  • Prerequisite

    Deploy Gateway with SSL
    Deploy Elasticsearch.
    Build your ES auth token with base64encode.org

  • Test

    curl -vk --key $PUNCHPLATFORM_SECRETS_DIR/server.pem --cert $PUNCHPLATFORM_SECRETS_DIR/server.crt --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt GET https://demo3:4242/v1/mytenant/es/es_search/_cat/indices -H "Authorization: Basic YOUR_TOKEN"

  • Expected

    List of ES indices.

Check channels endpoint with SSL

  • Prerequisite

    Deploy Gateway with SSL

  • Test

    curl -v http://demo3:4242/v1/mytenant/channels

    curl -vk --key $PUNCHPLATFORM_SECRETS_DIR/server.pem --cert $PUNCHPLATFORM_SECRETS_DIR/server.crt --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt GET https://demo3:4242/v1/mytenant/channels

  • Expected

    List of channels with their current status.

Check documentation on plugin

Create punchline on plugin

  • Prerequisite

    Deploy Gateway without SSL or Deploy Gateway with SSL
    Deploy Punch Plugin.

  • Test

    Go to Punch Plugin.
    Go to Administration pane. Create a new punchline resource.

  • Expected

    New resource is in Resource Manager.
    Resource is categorized as Punchline.
    Graphical editor updates code.

Launch extraction on plugin

  • Prerequisite

    Deploy Gateway without SSL or Deploy Gateway with SSL
    Deploy Punch Plugin.

  • Test

    Go to Punch Plugin.
    Go to Data Extraction.
    Create extraction job on platform-logs-*.

  • Expected

    Extraction is listed with right status.
    Output can be downloaded after success.
    All fields selected are present in the output.

Monitoring

Platform-monitoring

  • Prerequisite

    Deploy Operator.
    Deploy Shiva.
    Deploy Elastic.
    Deploy Kibana.
    Deploy all the components to be monitored.

  • Test

    Resolve file to check parameters : 

    punchlinectl -t platform resolve -f $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/monitoring/platform_health.hjson
    Launch platform health from operator : 
    export PUNCHPLATFORM_TENANT=platform
export PUNCHPLATFORM_CHANNEL=monitoring
platform-monitoring $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/monitoring/platform_health.hjson
    Launch platform health with Shiva : 
    channelctl -t platform start --channel monitoring --application monitoring/common/platform_health

  • Expected

    No error logs.
    Synthetic document in ES at platform-health-*.
    Health documents in ES at platform-monitoring-*.
    Monitoring Dashboards are green.

Channels-monitoring

  • Prerequisite

    Deploy Operator.
    Deploy Shiva.
    Deploy Elastic.
    Deploy Kibana.

  • Test

    Resolve file to check parameters : 

    punchlinectl -t platform resolve -f $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/monitoring/channels_monitoring.hjson
    Launch platform health from operator : 
    export PUNCHPLATFORM_TENANT=platform
export PUNCHPLATFORM_CHANNEL=monitoring
platform-monitoring $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/monitoring/channels_monitoring.hjson
    Launch platform health with Shiva : 
    channelctl -t platform start --channel monitoring --application monitoring/common/channels_monitoring

  • Expected

    No error logs.
    Synthetic document in ES at platform-health-*.
    Health documents in ES at platform-monitoring-*.
    Monitoring Dashboards are green.

Storm metrics

  • Prerequisite

    Deploy Storm or Shiva (if SSL enabled).
    Launch at least one channel.

  • Test

    Go to Kibana/Discover/mytenant-metrics-*.
    Filter on type:storm.

  • Expected

    All Storm metrics are in index.

Spark metrics

  • Prerequisite

    Deploy Spark.
    Launch aggregation channel.

  • Test

    Go to Kibana/Discover/mytenant-spark-metrics-*.

  • Expected

    All Spark metrics are in index.

Latency metrics

  • Prerequisite

    Deploy Storm or Shiva (if SSL enabled).
    Deploy Operator.
    Launch apache_httpd channel.

  • Test

    Go to Kibana/Discover/mytenant-metrics-*.
    Filter on name:storm.latency.

  • Expected

    Latency metric for each node in apache_httpd channel.

Housekeeping

Elasticsearch Housekeeping

  • Prerequisite

    Deploy Operator.
    Deploy Elasticsearch.

  • Test

    Resolve file : 

    punchlinectl -t platform resolve -f $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/housekeeping/elasticsearch-housekeeping.json
    Use resolved file as parameter : 
    elasticsearch-housekeeping --tenant-configuration-path $PUNCHPLATFORM_CONF_DIR/tenants/livedemo/channels/housekeeping/elasticsearch-housekeeping-resolved.json

  • Expected

    Successful actions on index.

Archive Housekeeping

  • Prerequisite

    Deploy Operator.
    Deploy Elasticsearch.
    Write a configuration file for archives-housekeeping.

  • Test

    Resolve file : 

    punchlinectl -t platform resolve -f $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/housekeeping/archives-housekeeping.json
    Launch archives-housekeeping : 
    archives-housekeeping $PUNCHPLATFORM_CONF_DIR/tenants/livedemo/channels/housekeeping/archives-housekeeping.json

  • Expected

    Successful actions on Archiving system.

Minio

Check Minio deployment.settings

  • Prerequisite

    Section for Metricbeat.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.
    All deployment parameters in $PUNCHPLATFORM_DEPLOYMENT_DIR/pp-deployment-conf/generated_inventory/group_vars/minio_servers.

Deploy Minio without SSL

Deploy Minio with SSL

Check Minio service file

  • Prerequisite

    Deploy Minio without SSL or Deploy Minio with SSL

  • Test

    cat /usr/lib/systemd/system/minio.service

  • Expected

    Description=Punchplatform Minio. Environment=MINIO_ACCESS_KEY.
    Environment=MINIO_SECRET_KEY.
    SyslogIdentifier=minio-{cluster}.
    Storage dir exists.

Curl Minio

Check Minio UI

  • Prerequisite

    Deploy Minio without SSL or Deploy Minio with SSL

  • Test

    Without SSL : 

    shuttle -r demo1 demo1:9000
    On local machine, go to http://demo1:9000 without SSL. On local machine, go to https://demo1:9000 with SSL.

  • Expected

    Login page.
    Able to create a bucket.
    Able to upload a file.
    The uploaded files are present on filesystem in /data/opt/minio-RELEASE.2020-08-26T00-00-49Z/{bucket}.

Check Archiving on Minio

  • Prerequisite

    Deploy Minio without SSL or Deploy Minio with SSL
    Deploy Operator. Change apache_httpd/archiving to archive on Minio.

  • Test

    Launch apache_httpd injector : 

    punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/resources/injectors/mytenant/apache_httpd_injector.json
    Launch apache_httpd channel : 
    channelctl -t mytenant start --channel apache_httpd
    Go to Minio UI.

  • Expected

    Bucket with pool name is created.
    Files are stored in Minio with directory and name.
    The uploaded files are present on filesystem in /data/opt/minio-RELEASE.2020-08-26T00-00-49Z/{bucket}.

Check Housekeeping on Minio

  • Prerequisite

    Deploy Minio without SSL or Deploy Minio with SSL
    Check Archiving on Minio
    Configure Archives Housekeeping for Minio.

  • Test

    Check resolved configuration file : 

    punchlinectl -t platform resolve -f $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/housekeeping/archives-housekeeping.json
    Launch application : 
    archives-housekeeping $PUNCHPLATFORM_CONF_DIR/tenants/platform/channels/housekeeping/archives-housekeeping.json
    Go to Minio UI.

  • Expected

    Files older than {retention} are deleted.
    Corresponding metadata documents in elasticsearch are deleted.

Metricbeat

Check Metricbeat deployment.settings

  • Prerequisite

    Section for Metricbeat.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Metricbeat without SSL

Deploy Metricbeat with SSL

Check Metricbeat configuration file - Kafka Output

Check Metricbeat configuration file - Elastic Output

Check Metricbeat service file

Check Metricbeat service status

Check Metricbeat Kafka Output

  • Prerequisite

    Deploy Metricbeat without SSL or Deploy Metricbeat with SSL
    Have Metricbeat Kafka Output.
    Deploy Kafka.
    Deploy Operator.
    Create Metricbeat Kafka Output topic.

  • Test

    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic metricbeat-topic -v

  • Expected

    Monitoring metrics in logs.

Check Metricbeat Elastic Output

  • Prerequisite

    Deploy Metricbeat without SSL or Deploy Metricbeat with SSL
    Have Metricbeat Elasticsearch Output.
    Deploy Elasticsearch.
    Deploy Operator.
    Create Metricbeat Elasticsearch Output Index Pattern.

  • Test

    Without Opendistro : 

    curl -v http://demo1:9200/_cat/indices?v
    With Opendistro : curl -v http://demo1:9200/_cat/indices?v -u {opendistro.user}:{opendistro.password} 
    With Opendistro and SSL :
curl -vk https://demo1:9200/_cat/indices?v -u {opendistro.user}:{opendistro.password} --key $PUNCHPLATFORM_SECRETS_DIR/server.pem --cert $PUNCHPLATFORM_SECRETS_DIR/server.crt --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt

  • Expected

    Metricbeat index exists.
    Documents in Metricbeat index.

Check Metricbeat Dashboards

  • Prerequisite

    Deploy Metricbeat without SSL or Deploy Metricbeat with SSL
    Deploy Elasticsearch.
    Deploy Kibana.
    Deploy Operator.
    Push pp_system_resources_monitoring.ndjson to Kibana.

  • Test

    Go to Kibana/Dashboards/[System Monitoring] Overview.

  • Expected

    All Metricbeat panels have data.

Packetbeat

Check Packetbeat deployment.settings

  • Prerequisite

    Section for Packetbeat.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Packetbeat without SSL

Deploy Packetbeat with SSL

Check Packetbeat configuration file - Kafka Output

Check Packetbeat configuration file - Elastic Output

Check Packetbeat service file

Check Packetbeat service status

Check Packetbeat Kafka Output

  • Prerequisite

    Deploy Packetbeat without SSL or Deploy Packetbeat with SSL
    Have Packetbeat Kafka Output.
    Deploy Kafka.
    Deploy Operator.
    Create Packetbeat Kafka Output topic.

  • Test

    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic packetbeat-topic -v

  • Expected

    Monitoring metrics in logs.

Check Packetbeat Elastic Output

  • Prerequisite

    Deploy Packetbeat without SSL or Deploy Packetbeat with SSL
    Have Packetbeat Elasticsearch Output.
    Deploy Elasticsearch.
    Deploy Operator.
    Create Packetbeat Elasticsearch Output Index Pattern.

  • Test

    Without Opendistro : 

    curl -v http://demo1:9200/_cat/indices?v
    With Opendistro : curl -v http://demo1:9200/_cat/indices?v -u {opendistro.user}:{opendistro.password} 
    With Opendistro and SSL :
curl -vk https://demo1:9200/_cat/indices?v -u {opendistro.user}:{opendistro.password} --key $PUNCHPLATFORM_SECRETS_DIR/server.pem --cert $PUNCHPLATFORM_SECRETS_DIR/server.crt --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt

  • Expected

    Packetbeat index exists.
    Documents in Packetbeat index.

Check Packetbeat Dashboards

  • Prerequisite

    Deploy Packetbeat without SSL or Deploy Packetbeat with SSL
    Deploy Elasticsearch.
    Deploy Kibana.
    Deploy Operator.
    Push pp_system_resources_monitoring.ndjson to Kibana.

  • Test

    Go to Kibana/Dashboards/[System Monitoring] Overview.

  • Expected

    All Packetbeat panels have data.

Filebeat

Check Filebeat deployment.settings

  • Prerequisite

    Section for Filebeat.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Filebeat without SSL

Deploy Filebeat with SSL

Check Filebeat configuration file - Kafka Output

Check Filebeat configuration file - Elastic Output

Check Filebeat service file

Check Filebeat service status

Check Filebeat Kafka Output

  • Prerequisite

    Deploy Filebeat without SSL or Deploy Filebeat with SSL
    Have Filebeat Kafka Output.
    Deploy Kafka.
    Deploy Operator.
    Create Filebeat Kafka Output topic.

  • Test

    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic filebeat-topic -v

  • Expected

    Monitoring metrics in logs.

Auditbeat

Check Auditbeat deployment.settings

  • Prerequisite

    Section for Auditbeat.

  • Test

    punchplatform-deployer.sh --generate-inventory

  • Expected

    Successful playbook.

Deploy Auditbeat without SSL

Deploy Auditbeat with SSL

Check Auditbeat configuration file - Kafka Output

Check Auditbeat configuration file - Elastic Output

Check Auditbeat service file

Check Auditbeat service status

Check Auditbeat Kafka Output

  • Prerequisite

    Deploy Auditbeat without SSL or Deploy Auditbeat with SSL
    Have Auditbeat Kafka Output.
    Deploy Kafka.
    Deploy Operator.
    Create Auditbeat Kafka Output topic.

  • Test

    punchplatform-log-injector.sh --kafka-consumer -brokers local -topic auditbeat-topic -v

  • Expected

    Monitoring metrics in logs.

Check Auditbeat Elastic Output

  • Prerequisite

    Deploy Auditbeat without SSL or Deploy Auditbeat with SSL
    Have Auditbeat Elasticsearch Output.
    Deploy Elasticsearch.
    Deploy Operator.
    Create Auditbeat Elasticsearch Output Index Pattern.

  • Test

    Without Opendistro : 

    curl -v http://demo1:9200/_cat/indices?v
    With Opendistro : curl -v http://demo1:9200/_cat/indices?v -u {opendistro.user}:{opendistro.password} 
    With Opendistro and SSL :
curl -vk https://demo1:9200/_cat/indices?v -u {opendistro.user}:{opendistro.password} --key $PUNCHPLATFORM_SECRETS_DIR/server.pem --cert $PUNCHPLATFORM_SECRETS_DIR/server.crt --cacert $PUNCHPLATFORM_SECRETS_DIR/fullchain.crt

  • Expected

    Auditbeat index exists.
    Documents in Auditbeat index.

Check Auditbeat Dashboards

  • Prerequisite

    Deploy Auditbeat without SSL or Deploy Auditbeat with SSL
    Deploy Elasticsearch.
    Deploy Kibana.
    Deploy Operator.
    Push pp_system_resources_monitoring.ndjson to Kibana.

  • Test

    Go to Kibana/Dashboards/[System Monitoring] Overview.

  • Expected

    All Auditbeat panels have data.