The PunchPlatform metrics system collects multiples families of metrics :
- The channel-level applicative metrics : computed by Storm or Spark components
- Java Virtual Machine metrics : computed by a generic plugin. The same metrics will be reported by each worker and provide information about garbage collection, cpu usage, etc.
- Platform components level : collected by a dedicated service that check the platform services health (such as kafka, storm, elasticsearch, ...)
- Operating System-level metrics : collected by agents (such as metricbeat, packetbeat, ...) installed on PunchPlatform application servers (cpu, disk usage, network usage, ...)
Only the channel-level metrics can be activated/configured by user configuration in the channel configuration files.
Channel-level applicative metrics¶
All these metrics are storm-component-related.
Please refer to the spouts and bolts documentation for a detailed list of published metrics and associated names/context tags.
PunchPlatform Storm Java VM metrics¶
All specific PunchPlatform topologies components run inside storm 'workers' Java virtual machines or in "local mode" JVM on dedicated servers.
The following metrics groups are published for each of these JVM :
- memory (Specially interesting is the ratio memory.total.used/memory.total.max)
These groups are published under the following metrics path :
topology_idis the Storm topology id for jvms running tasks belonging to a Storm topology. The topology id begins with
jvm_process_idis the linux process id of the JVM
jvm_hostis the linux hostname on which the JVM runs
The PunchPlatform comes with in-built application and system level monitoring. The collected metrics are stored in an Elasticsearch backend, on top of which you can design Kibana dashboards.
With Elasticsearch, it is not the most efficient to search/retrieve metrics based on name-pattern. Instead each metric has a (short) name, a value, and a number of tags : tenant, cluster, topology id, etc. Metrics can be requested using filters or exact values of some of these tags, which is both efficient and flexible.
With that design the name of a metric identifies a category, grouping all metrics sharing the same nature. Tags provide additional contextual information, to characterize typically the point of measure.
Elasticsearch back-end has been selected as the main metrics backend because :
- It is resilient and scalable
- It allows evolutive additive tagging of metrics value, that can ease dashboarding development without negative impact on existing ones
- It reduces the number of COTS needed for a PunchPlatform deployment (already included)
Naming convention for metrics context tags¶
All PunchPlatform metrics context tags are stored as subfields of the
"tags" field of metrics documents. When providing the tag subfield
names in the PunchPlatform documentation, the
. character identifies
The Elasticsearch mapping template for PunchPlatform metrics is
@timestamp as timestamp field.
The PunchPlatform Storm components (bolts and spouts), as well as system-level metrics collector publish various metrics in real time. These metrics are of several types:
- Histograms : measures the statistical distribution of values in a stream of data. In addition to minimum, maximum, mean, etc., it also measures median, 75th, 90th, 95th, 98th, 99th, and 99.9th percentiles.
- Meters: measures the rate of events over time (e.g., "requests per second"). In addition to the mean rate, meters also track 1-, 5-, and 15-minute moving averages.
- Counters: A counter is just a gauge for an AtomicLong instance.
These metrics are published only periodically to a metrics storage backend, either a loggers or an elasticsearch cluster. You can activate them with no impact on the platform performance, they are designed to cope with high traffic processing. All statistical calculations are performed in real time in storm workers, only the consolidated result is sent periodically to the back-end.
Metrics are not stored/indexed in Elasticsearch the way logs are indexed.
- Periodic (~10s, customizable) grouped sending of metrics values to the metrics backend(s).
- "metric name/metric value" type backends :
- "tagged values" type backends (i.e. additional information can accompany a metric value, instead of being inside the metric name):
- Sending is non-blocking ; in case on unreachable back end, metrics values will be lost and functional behaviour of the software will continue ; when back end is available again, sending of new metrics values will succeed.
- Functional parts of the software can provide instantaneous measures, or counter metrics increase.
- Metrics layer takes care of associated computed statistics (counter aggregation, long term average, 1 minute mean, ) and of providing a single value by sending period ("bucketing")
- "uptime" metrics are generic automatically-increased time counter metrics that indicate the process is still alive
- Metrics can be sent either directly by a process to the backend(s) AND can also be forwarded through the PunchPlatform channel, and sent by another process of the channel (acting as a "metrics proxy").
- Direct sending to the various backends, forwarding are settings that can be overridden at a topology setting level
- Metrics proxy activation can be activated at a spout setting level
Common variations about PunchPlatform Metrics¶
For all metrics indicating event rates and latency measures, the metric is automatically declined in multiple data subseries :
- m1_rate : the Exponentially Weighted Moving Average (EWMA) on a 1 minute period (period of time in minutes over which the reading is said to be averaged (the mean lifetime of each reading in the average), 60% of the value in computation).
- mean_rate : a moving long-term average with an exponential temporal attenuation
- count : the cumulated values that have been registered (this value is often meaningless, except for message counts, where it is actually representative of the number of messages cumulated over time)
For latency and size measures, the additional data series are provided, derived from the base measure :
- stddev, the standard deviation
- max : maximal value
- min : minimal value
When using elasticsearch backend, these values are grouped as a single
metric record, as subfields in the metric object. Therefore the series
value field is the name of the metric, then a
. (dot), then the subseries
identifier. For example:
To know the the available subseries, refer to the following table, and to the metrics types as listed in the Spouts and Bolts documents.
|Histogram||count, max, min, mean, stddev|
|Timer||count, m1_rate, max, min, mean, stddev|
|TimeValue||"max", "ts_diff", ... Depends on the metric|
This section describe the content of the metrics common fields. Some are mandatory and will be found in every metrics, other are only set when useful.
This fields normalization is essential, it is the key to create powerful Kibana dashboard based on metrics coming from various sources. Thanks to that, you will be able to apply the same filters on all the visualizations.
Document fields shared across all metrics¶
The platform unique ID when the metric has been generated. Useful when working with multiple LMR/LTR on the same project.
Designate the metric source type. For example, its value metrics coming from Storm topologies will be "storm" or "platform" for the ones coming from the platform monitoring.
Commons fields (when relevant)¶
Contrary to the event field 'rep.host.name', for metrics we decided to follow the ECS convention. This fields contains the hostname from which the measurement was taken.
The associated tenant
The associated channel
- module: system
- module: kafka
- module: zookeeper
See the official Metricbeat documentation for further information.