Skip to content

HOWTO keep business metrics for long time

Why do that

For billing, it can be mandatory to keep specific metrics for long time (e.g. for a few months). Because this technology is storage-consuming (but pretty good to visualize and search data), we usually store these metrics for a shorter period (typically a few days). That\'s why PunchPlatform developed Pygregator guide

Prerequisites

  • Pygregator binaries installed on nodes running aggregations (typically Punchplatform Admin nodes in this example)
  • An Elasticsearch cluster reachable from these nodes
  • A Shiva cluster deployed (at least) one these nodes

These pre-requisites are a deployment problematic, please read the Administration Guide for any question.

What to do

Discover your data

Through Kibana, analyze data you want to keep or aggregate.

Warning

By default, the _source field is disabled for metrics. So you can't see the content of messages. You can enable it by changing the events-metrics.json in $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/templates.

Create the Pygregator configuration file

Read the Pygregator Guide.

In this HowTo example, we would like to keep:

  • the average of received logs per technology and per 10 minutes on a LTR
  • the maximum of received logs per technology and per 10 minutes on a LTR

So the adequate Pygregator configuration file will be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# Replace 'localhost:9200' by your Elasticsearch cluster endpoint
input: "http://localhost:9200"
output: "http://localhost:9200"

# Let's define our 2 aggregations, matching our needs to keep average
# rate and maximum rate of received logs.
aggregations:

 - name: "AverageRate"
   period: "10m"
   model: "AverageByTermAgg"
   indices:
     input:
      - "metrics-*"
     output: "agg-mytenant-rate"
   date_query:
     field: "_timestamp"
     duration: "10m"
   fields:
     term_field: "tags.storm.component_id"
     avg_field: "storm.tuple.ack.m1_rate"
   filters:
     - tags.pp.tenant: "mytenant"

 - name: "MaxRate"
   period: "10m"
   model: "MaxByTermAgg"
   indices:
     input:
      - "metrics-*"
     output: "agg-mytenant-rate"
   date_query:
     field: "_timestamp"
     duration: "10m"
   fields:
     term_field: "tags.storm.component_id"
     max_field: "storm.tuple.ack.m1_rate"
   filters:
     - tags.pp.tenant: "mytenant"  

logPath: "./pygregator-MaxAverageRate.log"
logLevel: "INFO"
# The keyword 'additionalArgDir' is standard. It's a directory
# created by Shiva (see further) where all needed resources
# are stored.
resourcesDirectory: "additionalArgDir/models"

You're free to copy and adapt this content on a new file named my-pygregator-config.yml, located on $PUNCHPLATFORM_CONF_DIR/resources/pygregator/ (on a punchplatform operator node / account).

Run Pygregator as a Shiva task

The PunchPlatform provides a distributed task scheduler to run arbitrary tasks, refer to Shiva Task Manager. It offers you:

  • Resiliency
  • Monitoring
  • Configuration management (git)

Shiva tasks can be embedded into channels (you're supposed to be familiar with them). So the only thing to do to run a Shiva task is to configure a channel (the procedure is described into another HowTo : HOWTO start pygregator as part of a channel.

In a few words you only have to create a channel your_channel in your tenant your_tenant with following channel_structure.json content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "topologies" : [],
  "shiva_pygregator_services": [
    {
      "name": "MyPygregatorService",
      "cluster": "MyShivaTag",
      "configuration": "my-pygregator-config.yml"
    }
  ]
}

Warning

Please replace MyShivaTag by your Shiva tag, as it has been specified in your Punchplatform deployment files.

Then, check your Shiva cluster status:

1
pp-operator@pp-admin-node$ punchplatform-shiva.sh health

Expected output is a green status.

Start your channel:

1
pp-operator@pp-admin-node$ punchplatform-channel.sh --start your_tenant/your_channel

Check your task status:

1
pp-operator@pp-admin-node$ punchplatform-shiva.sh list-tasks

Finally check on Kibana you're able to see your aggregated data.