Elasticsearch and Kibana¶
You may not be familiar yet with Elasticsearch and Kibana. Before even trying out the punch features, it is a good idea to simply visit your local Kibana http://localhost:5601.
In this tour we visit some monitoring use case and we introduce some pre-defined punch dashboards for you to start quickly.
Start exploring the monitoring dashboards. These come with a companion monitoring agent called Metricbeat. It is shipped with the standalone punchplatform and is already running. You can see it running by typing the following command:
or even simpler:
The metricbeat collects various system and monitoring metrics and forwards them to Elasticsearch. You then visualise these through a Kibana dashboard. Execute the following command to load the metricbeat dashboards.
cd $PUNCHPLATFORM_CONF_DIR/../external/metricbeat-*-x86_64/ ./metricbeat setup -c metricbeat.yml --dashboards
Go back to Kibana. On the left-hand panel, select the Dashboard menu. You will see there a number of dashboards, ready to be visualised. Find and select the Metricbeat System Overview dashboard. You should see something like this:
The metricbeat dashboards let you visualise metrics of each of your computer hardware: cpu usage, disk usage, memory usage, etc. These metrics are generated by the Metricbeat.
The so-called Beats are the Elastic agents in charge of collecting various events (windows, network, host, files, audit). What you see here in action is the Metricbeat. Metricbeats are extensively used in the punch. They are deployed as part of the punchplatform setup and provide you with a complete view of your servers.
Let us now explore another beat: the Auditbeat. It monitors user activity and processes. Auditbeat communicates directly with the Linux audit framework and sends the events to the Elastic Stack in real time.
Because the auditbeat requires root privilege, it is not started automatically. Here is how you can start it:
cd $PUNCHPLATFORM_CONF_DIR/../external/auditbeat-*/ sudo chown root auditbeat.yml # load the auditbeat dashboards (you can skip this step if you don't want the audit beat dashboard) # this step may takes up to 1 minute sudo ./auditbeat setup -c auditbeat.yml --dashboards # On Linux, there is an extra step: you must chose your architecture # For example, on a 64 bits computer, delete any unecessary 32-bits configuration files # Otherwise, delete the 64-bits files. rm audit.rules.d/*-32bit.conf # Go for it ! sudo ./auditbeat -c auditbeat.yml -e
You can now visit the "[Auditbeat] File Integrity" dashboard. Have fun discovering what you can learn from such a tool.
When you look for a dashboard use the top level search box. Simply type 'Aud' and it will automatically list the available audit beat dashboards.
The punch standalone comes with elasticsearch resources. Check out the
resources/elasticsearch/ └── templates ├── cyber │ └── mapping_events.json ├── other │ ├── ecs-1.0.0-beta2-template.json │ └── mapping_aggregations.json ├── platform │ ├── mapping_archive.json │ ├── mapping_jobs.json │ ├── mapping_platform_health.json │ ├── mapping_platform_logs.json │ ├── mapping_platform_monitoring.json │ ├── mapping_platform_refresh_interval.json │ └── mapping_topology_metrics.json ├── README.txt └── standalone ├── settings_global_standalone.json └── settings_kibana.json
Import all these mappings to elasticsearch. You can do that for each mapping as follows:
curl -H "Content-Type: application/json" -XPUT localhost:9200/_template/mapping_metrics -d @mapping_metrics.json
Or you can load them all at once using the
$ punchplatform-push-es-templates.sh --directory $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/resources/elasticsearch/templates --url http://localhost:9200 --verbose
The template mapping that you must load are the ones under the
They are needed to correctly insert the monitoring events generated by the PunchPlatform
itself. Other mappings are examples and specific to the standalone demo channels, you must create your
own mapping when you create a new channel
The punch comes with predefined Kibana dashboards to easily start exploring your data.
These dashboards are located under the
kibana/ └── dashboard ├── aggregation_mytenant_demo │ └── aggregation.ndjson ├── archiving_monitoring │ ├── archive_monitoring.ndjson ├── cybersecurity_mytenant_demo │ └── cybersecurity_mytenant_demo.ndjson ├── elastic_common_schema │ ├── elastic_common_schema_demo.ndjson │ └── README.md ├── platform_monitoring │ ├── platform_monitoring.ndjson ├── README.md ├── spark_monitoring │ ├── spark_monitoring.ndjson └── tenants_monitoring ├── tenants_monitoring.ndjson
To import these dashboards you can use this command-line:
- Go to the Kibana UI
- On the left-side panel, go to the "Management > Saved Objects > Import"
- Drag-n-drop or select the NDJSON dashboard
- Go to the "Dashboard" tab and start exploring your dashboards.
Dashboards from the "*_demo" folders are examples for standalone channels. Others are representative of monitoring dashboards used on production platforms.
Visit the punch dashboards documentation.