Skip to content

HOWTO configure EPS reporting functionality

Why do that

Billing and Capacity Management need accurate information on the platform. One of the most important information is about the EPS (Events Per Second) circulating inside the system. We will focus on this particular type of metric and see how to put an accurate, valuable and long term EPS reporting functionality.

We can do this in three steps :

  • Choose the components to observe in your channels
  • Configure Pygregator which will aggregate the metrics of the components
  • Visualize and export the results

Context

Note

If you are familiar with PunchPlatform then you can skip this section.

Each component of the PunchPlatform sends metrics. A metric is structured as follow :

  • name : nature of the metric, by example "storm.tuple.ack"
  • values :

    • count
    • m1_rate (Exponentially-weighted moving average)
    • ...
    • max

    The different type of the same metric

  • tags :

    • tags.pp.tenant
    • tags.pp.channel
    • tags.storm.topology_name
    • tags.storm.component_id
    • etc.

    Tags provide additionnal contextual information, to characterize typically the point of measure.

  • @timestamp : the generation timestamp of the metric

Activate metric reporting

Note that Metrics are activated by default. How does it works ? Configure an Elasticsearch metric reporter in your topology :

To do this, add a new section in the "metrics" property in your topology file :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
"metrics" : {
      "reporters" : [
            {              
                "type" : "elasticsearch",
                "reporting_interval": 30,
                "cluster_name" : "es_search",
                "transport" : "localhost:9300",
                "max_results_size" : 10000,
                "native_es_settings" : {
                  "transport.netty.workerCount" : "1"
                }
            }
        ]
}

Adapt the configuration of reporter to your platform ! In case LTR without Elasticsearch access, configure a "metric" topology to propagate the metric to the back-end. See also See Metrics for more information.

Storm component name

To retrieve the name of the component, look at your topology file and find the "component" property :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
 "spouts" : [
   {
         "type" : "syslog_spout",
         "spout_settings" : { 
              ...
         },
         "storm_settings" : {
           "executors": 1,
           "component" : "syslog_spout_apache_httpd",
           ...
       }
   },

Here the name of the component is "syslog_spout_apache_httpd".

What to do

  1. Identify the strategic Storm components of your platform, corresponding to your needs :
    1. Input : syslog spout or lumberjack spout or more generally all spout components
    2. During the processing : kafka spout component
    3. Third party external link : syslog bolt or filter bolt components
  2. Configure Pygregator Pygregator aggregates the metrics and send them in a new Elasticsearch index. Pygregator has already models to aggregate the metrics and so the configuration is really simple.

    By example, if we want to aggregate metrics of a syslog spout component then we apply the "InputEps" model.

    Example of Pygregator configuration :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
    aggregations:
    - name: "InputEps"
      period: "3d"
      model: "InputEps"
      indices:
        input:
         - "metrics-*"
        output: "agg-mytenant-eps"
      filters:
        - tags.pp.tenant: "mytenant"
        - name: "storm.tuple.ack"
      fields:
        interval: "1h"
        interval_in_seconds: 3600
        metric_name: "storm.tuple.ack"
        metric_field: "count"    
      date_query:
        field: "ts"
        duration: "3d"
        frame: "1d"
        range_lower: "gte"
        range_upper: "lte"
      buckets_path: "agg_histo>agg_channel>agg_topology>agg_component>eps.value"

What have we configured ?

  • Pygregator sends a request each day, on a period of 3 days and with a granularity of 1 hour.

More information on Pygregator : See Pygregator guide.

3 . Visualize and export information in Kibana :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
The final step is to use Kibana to do the extraction.

To do this execute the following tasks :

-   Define a new index "agg-mytenant-eps*"
-   Create a new Visualization on this index
-   Add "meta.name:InputEps" on the search bar.
-   On Y-Axis, set Aggregation "Max" for "metric"
-   On X-Axis, set Aggregation "Histogram" for
    "aggregation.agg_histo_val", with an interval of 1 hour
-   Add sub-buckets aggregation "split Area" of "Terms" type
    and select "aggregation.agg_component_val"

Example of EPS dashboard :

![image](../../../_static/InputEps.png)

You can export your data with 'Export:Raw Formatted' link in
Kibana on a custom period.

![image](../../../_static/InputEpsExport.png)

Other models

  • "ProcessingEps" gives the EPS average base on the topic/partition offset.
  • "MaxEps" gives the EPS max base on the tuple rate.

Additional configuration

  1. Optional. Configure the metric reporters of your topologies. This step is not strictly necessary but you can optimize your platform by reducing the number of metric in your index.

    See Metrics

    • apply a regex on metrics to keep only useful metrics
1
"regex" : ["kafka.spout.offset.ack.rate", "storm.tuple.ack", "storm.tuple.eps"]
1
2
-   apply a 'sampling' strategy in order to reduce the number of
    metrics
1
2
3
"sampling" : {
    "nb_skipped_cycles" : 4
 }
  1. make your aggregations properly scheduled using Shiva