Skip to content

Elasticsearch and Kibana

You may not be familiar yet with Elasticsearch and Kibana. Before even trying out the punch features, it is a good idea to simply visit your local Kibana http://localhost:5601.

In this tour we visit some monitoring use case and we introduce some pre-defined punch dashboards for you to start quickly.

System Monitoring

Start exploring the monitoring dashboards. These come with a companion monitoring agent called Metricbeat. It is shipped with the standalone punchplatform and is already running. You can see it running by typing the following command:

punchplatform-standalone.sh --status

or even simpler:

punchplatform-metricbeat.sh --status

The metricbeat collects various system and monitoring metrics and forwards them to Elasticsearch. You then visualise these through a Kibana dashboard. Execute the following command to load the metricbeat dashboards.

cd $PUNCHPLATFORM_CONF_DIR/../external/metricbeat-*-x86_64/
./metricbeat setup -c metricbeat.yml --dashboards

Go back to Kibana. On the left-hand panel, select the Dashboard menu. You will see there a number of dashboards, ready to be visualised. Find and select the Metricbeat System Overview dashboard. You should see something like this:

image

The metricbeat dashboards let you visualise metrics of each of your computer hardware: cpu usage, disk usage, memory usage, etc. These metrics are generated by the Metricbeat.

Tip

The so-called Beats are the Elastic agents in charge of collecting various events (windows, network, host, files, audit). What you see here in action is the Metricbeat. Metricbeats are extensively used in the punch. They are deployed as part of the punchplatform setup and provide you with a complete view of your servers.

Audit Data

Let us now explore another beat: the Auditbeat. It monitors user activity and processes. Auditbeat communicates directly with the Linux audit framework and sends the events to the Elastic Stack in real time.

Because the auditbeat requires root privilege, it is not started automatically. Here is how you can start it:

cd $PUNCHPLATFORM_CONF_DIR/../external/auditbeat-*/
sudo chown root auditbeat.yml

# load the auditbeat dashboards (you can skip this step if you don't want the audit beat dashboard)
# this step may takes up to 1 minute
sudo ./auditbeat setup -c auditbeat.yml --dashboards

# Go for it !
sudo ./auditbeat -c auditbeat.yml -e

You can now visit the [Auditbeat] File Integrity dashboard. Have fun discovering what you can learn from such a tool.

image

Tip

When you look for a dashboard use the top level search box. Simply type 'Aud' and it will automatically list the available audit beat dashboards.

Elasticsearch Templates

The punch standalone comes with elasticsearch resources. Check out the $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch folder:

├ resources
├── elasticsearch
│   └── templates
│       ├── cyber
│       │   └── mapping_events.json
│       ├── other
│       │   ├── ecs-1.0.0-beta2-template.json
│       │   └── mapping_aggregations.json
│       ├── platform
│       │   ├── pp_mapping_applications.json
│       │   ├── pp_mapping_applicative_monitoring.json
│       │   ├── pp_mapping_archive.json
│       │   ├── pp_mapping_gateway.json
│       │   ├── pp_mapping_metadata.json
│       │   ├── pp_mapping_platform_health.json
│       │   ├── pp_mapping_platform_logs.json
│       │   ├── pp_mapping_platform_monitoring.json
│       │   ├── pp_mapping_topology_metrics.json
│       │   ├── pp_monitoring_default_refresh.json
│       │   └── README_pp_monitoring_mappings.md
│       └── standalone
│           ├── settings_global_standalone.json
│           └── settings_kibana.json

These templates are aotumatically loaded at startup. You can check it was successful by typing :

curl localhost:9200/_template | jq keys

This should print out all the loaded templates.

Important

The template mapping that you must load are the ones under the platform folder. They are needed to correctly insert the monitoring events generated by the PunchPlatform itself. Other mappings are examples and specific to the standalone demo channels, you must create your own mapping when you create a new channel

Demo Dashboards

The punch comes with predefined Kibana dashboards to easily start exploring your data. These dashboards are located under the conf/dashboard folder.

├── kibana
│   └── dashboards
│       ├── archiving_monitoring
│       │   ├── archive_monitoring.ndjson
│       │   └── archiving_monitoring.json
│       ├── cyber
│       │   ├── aggregation_mytenant_demo
│       │   │   └── aggregation.ndjson
│       │   ├── cybersecurity_mytenant_demo
│       │   │   └── cybersecurity_mytenant_demo.ndjson
│       │   └── elastic_common_schema
│       │       └── elastic_common_schema_demo.ndjson
│       ├── gateway_monitoring
│       │   └── gateway-monitoring.ndjson
│       ├── kafka_monitoring
│       │   └── kafka-monitoring.ndjson
│       ├── metrics_eps_standalone
│       │   ├── README.md
│       │   └── standalone_eps.ndjson
│       ├── platform_monitoring
│       │   ├── channels_applications_states.ndjson
│       │   ├── channels_monitoring.ndjson
│       │   ├── platform_monitoring.ndjson
│       │   ├── shiva_monitoring.ndjson
│       │   └── shiva_tasks_monitoring.ndjson
│       ├── README_monitoring_dashboards.md
│       ├── spark_monitoring
│       │   └── spark_monitoring_dashboard.ndjson
│       ├── system_monitoring
│       │   └── system-monitoring.ndjson
│       ├── tenants_monitoring
│       │   └── tenants_monitoring.ndjson
│       └── zookeeper_monitoring
│           └── zookeeper-monitoring.ndjson

To import these dashboards you can use this command-line:

punchplatform-setup-kibana.sh --import 

Or manually:

  1. Go to the Kibana UI
  2. On the left-side panel, go to the "Management > Saved Objects > Import"
  3. Drag-n-drop or select the NDJSON dashboard
  4. Go to the "Dashboard" tab and start exploring your dashboards.

Info

Dashboards from the "*_demo" folders are examples for standalone channels. Others are representative of monitoring dashboards used on production platforms.

Visit the punch dashboards documentation.