HOWTO keep business metrics for long time

Why do that

For billing, it can be mandatory to keep specific metrics for long time (e.g. for a few months). Because this technology is storage-consuming (but pretty good to visualize and search data), we usually store these metrics for a shorter period (typically a few days). That\'s why PunchPlatform developed Pygregator guide

Prerequisites

• Pygregator binaries installed on nodes running aggregations (typically Punchplatform Admin nodes in this example)
• An Elasticsearch cluster reachable from these nodes
• A Shiva cluster deployed (at least) one these nodes

These pre-requisites are a deployment problematic, please read the Administration Guide for any question.

What to do

Discover your data

Through Kibana, analyze data you want to keep or aggregate.

Warning

By default, the _source field is disabled for metrics. So you can't see the content of messages. You can enable it by changing the events-metrics.json in $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/templates.

Create the Pygregator configuration file

Read the Pygregator Guide.

In this HowTo example, we would like to keep:

• the average of received logs per technology and per 10 minutes on a LTR
• the maximum of received logs per technology and per 10 minutes on a LTR

So the adequate Pygregator configuration file will be:

# Replace 'localhost:9200' by your Elasticsearch cluster endpoint
input: "http://localhost:9200"
output: "http://localhost:9200"

# Let's define our 2 aggregations, matching our needs to keep average
# rate and maximum rate of received logs.
aggregations:

 - name: "AverageRate"
   period: "10m"
   model: "AverageByTermAgg"
   indices:
     input:
      - "metrics-*"
     output: "agg-mytenant-rate"
   date_query:
     field: "_timestamp"
     duration: "10m"
   fields:
     term_field: "tags.storm.component_id"
     avg_field: "storm.tuple.ack.m1_rate"
   filters:
     - tags.pp.tenant: "mytenant"

 - name: "MaxRate"
   period: "10m"
   model: "MaxByTermAgg"
   indices:
     input:
      - "metrics-*"
     output: "agg-mytenant-rate"
   date_query:
     field: "_timestamp"
     duration: "10m"
   fields:
     term_field: "tags.storm.component_id"
     max_field: "storm.tuple.ack.m1_rate"
   filters:
     - tags.pp.tenant: "mytenant"  

logPath: "./pygregator-MaxAverageRate.log"
logLevel: "INFO"
# The keyword 'additionalArgDir' is standard. It's a directory
# created by Shiva (see further) where all needed resources
# are stored.
resourcesDirectory: "additionalArgDir/models"

You're free to copy and adapt this content on a new file named my-pygregator-config.yml, located on $PUNCHPLATFORM_CONF_DIR/resources/pygregator/ (on a punchplatform operator node / account).

Run Pygregator as a Shiva task

The PunchPlatform provides a distributed task scheduler to run arbitrary tasks, refer to Shiva Task Manager. It offers you:

• Resiliency
• Monitoring
• Configuration management (git)

Shiva tasks can be embedded into channels (you're supposed to be familiar with them). So the only thing to do to run a Shiva task is to configure a channel (the procedure is described into another HowTo : HOWTO start pygregator as part of a channel.

In a few words you only have to create a channel your_channel in your tenant your_tenant with following channel_structure.json content:

{
  "topologies" : [],
  "shiva_pygregator_services": [
    {
      "name": "MyPygregatorService",
      "cluster": "MyShivaTag",
      "configuration": "my-pygregator-config.yml"
    }
  ]
}

Warning

Please replace MyShivaTag by your Shiva tag, as it has been specified in your Punchplatform deployment files.

Then, check your Shiva cluster status:

pp-operator@pp-admin-node$ punchplatform-shiva.sh health

Expected output is a green status.

Start your channel:

pp-operator@pp-admin-node$ punchplatform-channel.sh --start your_tenant/your_channel

Check your task status:

pp-operator@pp-admin-node$ punchplatform-shiva.sh list-tasks

Finally check on Kibana you're able to see your aggregated data.