Elasticsearch and Kibana¶
You may not be familiar yet with Elasticsearch and Kibana. Before even trying out the punch features, it is a good idea to simply visit your local Kibana http://localhost:5601.
In this tour we visit some monitoring use case and we introduce some pre-defined punch dashboards for you to start quickly.
Start exploring the monitoring dashboards. These come with a companion monitoring agent called Metricbeat. It is shipped with the standalone punchplatform and is already running. You can see it running by typing the following command:
or even simpler:
The metricbeat collects various system and monitoring metrics and forwards them to Elasticsearch. You then visualise these through a Kibana dashboard. Execute the following command to load the metricbeat dashboards.
cd $PUNCHPLATFORM_CONF_DIR/../external/metricbeat-*-x86_64/ ./metricbeat setup -c metricbeat.yml --dashboards
Go back to Kibana. On the left-hand panel, select the Dashboard menu. You will see there a number of dashboards, ready to be visualized. Find and select the [Metricbeat System] Host Overview ECS dashboard. You should see something like this:
The metricbeat dashboards let you visualise metrics of each of your computer hardware: cpu usage, disk usage, memory usage, etc. These metrics are generated by the Metricbeat.
The so-called Beats are the Elastic agents in charge of collecting various events (windows, network, host, files, audit). What you see here in action is the Metricbeat. Metricbeats are extensively used in the punch. They are deployed as part of the punchplatform setup and provide you with a complete view of your servers.
Let us now explore another beat: the Auditbeat. It monitors user activity and processes. Auditbeat communicates directly with the Linux audit framework and sends the events to the Elastic Stack in real time.
Because the auditbeat requires root privilege, it is not started automatically. Here is how you can start it:
cd $PUNCHPLATFORM_CONF_DIR/../external/auditbeat-*/ sudo chown root auditbeat.yml # load the auditbeat dashboards (you can skip this step if you don't want the audit beat dashboard) # this step may takes up to 1 minute sudo ./auditbeat setup -c auditbeat.yml --dashboards # Go for it ! sudo ./auditbeat -c auditbeat.yml -e
You can now visit the
[Auditbeat] File Integrity dashboard. Have fun discovering
what you can learn from such a tool.
When you look for a dashboard use the top level search box. Simply type 'Aud' and it will automatically list the available audit beat dashboards.
The punch standalone comes with elasticsearch resources. Check out the
├ resources ├── elasticsearch │ └── templates │ ├── cyber │ │ └── mapping_events.json │ ├── other │ │ ├── ecs-1.0.0-beta2-template.json │ │ └── mapping_aggregations.json │ ├── platform │ │ ├── pp_mapping_applications.json │ │ ├── pp_mapping_applicative_monitoring.json │ │ ├── pp_mapping_archive.json │ │ ├── pp_mapping_gateway.json │ │ ├── pp_mapping_metadata.json │ │ ├── pp_mapping_platform_health.json │ │ ├── pp_mapping_platform_logs.json │ │ ├── pp_mapping_platform_monitoring.json │ │ ├── pp_mapping_topology_metrics.json │ │ ├── pp_monitoring_default_refresh.json │ │ └── README_pp_monitoring_mappings.md │ └── standalone │ ├── settings_global_standalone.json │ └── settings_kibana.json
These templates are aotumatically loaded at startup. You can check it was successful by typing :
curl localhost:9200/_template | jq keys
This should print out all the loaded templates.
The template mapping that you must load are the ones under the
They are needed to correctly insert the monitoring events generated by the PunchPlatform
itself. Other mappings are examples and specific to the standalone demo channels, you must create your
own mapping when you create a new channel
The punch comes with predefined Kibana dashboards to easily start exploring your data.
These dashboards are located under the
├── kibana │ └── dashboards │ ├── archiving_monitoring │ │ └── archiving_monitoring.ndjson │ ├── cyber │ │ ├── aggregation_mytenant_demo │ │ │ └── aggregation.ndjson │ │ ├── cybersecurity_mytenant_demo │ │ │ └── cybersecurity_mytenant_demo.ndjson │ │ └── elastic_common_schema │ │ └── elastic_common_schema_demo.ndjson │ ├── gateway_monitoring │ │ └── gateway-monitoring.ndjson │ ├── kafka_monitoring │ │ └── kafka-monitoring.ndjson │ ├── metrics_eps_standalone │ │ ├── README.md │ │ └── standalone_eps.ndjson │ ├── platform_monitoring │ │ ├── channels_applications_states.ndjson │ │ ├── channels_monitoring.ndjson │ │ ├── platform_monitoring.ndjson │ │ ├── shiva_monitoring.ndjson │ │ └── shiva_tasks_monitoring.ndjson │ ├── README_monitoring_dashboards.md │ ├── spark_monitoring │ │ └── spark_monitoring_dashboard.ndjson │ ├── system_monitoring │ │ └── system-monitoring.ndjson │ ├── tenants_monitoring │ │ └── tenants_monitoring.ndjson │ └── zookeeper_monitoring │ └── zookeeper-monitoring.ndjson
To import these dashboards you can use this command-line:
- Go to the Kibana UI
- On the left-side panel, go to the "Management > Saved Objects > Import"
- Drag-n-drop or select the NDJSON dashboard
- Go to the "Dashboard" tab and start exploring your dashboards.
Dashboards from the "*_demo" folders are examples for standalone channels. Others are representative of monitoring dashboards used on production platforms.
Visit the punch dashboards documentation.