Skip to content

Metrics Reporters

Introduction

The PunchPlatform processing pipelines can easily publish metrics to several backend by using simple configuration parameters.

These metrics are generated by Storm topologies, Spark jobs or even by the platform itself. In any case, the metrics configuration stay exactly the same with a list of "reporters". There is no limit to the number of reporters, you can also use the same reporter more than once.

Let's take a quick example for each use case.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# storm_topology_example.json
{
  "spouts": [
    ...
  ],
  "bolts": [
    ...
  ],
  "metrics": {
    "reporters": [
      {
        "type": "elasticsearch",
        ...
      },
      {
        "type": "logger",
        ...
      }
    ]
  },
  "storm_settings": {
    ...
  }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# spark_job_example.hjson
{
  runtime_id: my-job
  tenant: job_tenant
  job: [
    {
        ...
    },
  ],
  metrics: {
    reporters: [
      {
        type: elasticsearch
        ...
      }
      {
        type: console
        ...
      }
    ]
  }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# punchplatform.properties
{
  "platform": {
    "platform_id": "my-platform",
    "reporters": [
      {
        "type": "elasticsearch",
        ...
      },
      {
        "type": "kafka",
        ...
      }
    ]
  },
  ...
}

Common Parameters

The parameters described in this section are common to all the reporter's configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metrics": {
  "reporters": [
    {
      "type": "elasticsearch",
      "reporting_interval": 10,
      "regex": [
        "netty\\..*",
        "storm\\.worker\\.uptime"
      ],
      ...
    },
    {
      "type": "kafka",
      ...
    }
  ]
}
  • type (string, mandatory)

    Define the reporter type.

    values: "elasticsearch", "kafka", "lumberjack", "logger", "tcp", "udp".

  • reporting_interval (integer: 10)

    The period of metrics publication in seconds

  • regex (array)

    Array of string that filters the published metrics. For each metric event received, at least one of this regumar expression pattern must match its metric "name" (i.e. "netty.app.recv"). All other metrics will be dropped. This is espacially useful to forward only a subset of metrics event to a specific reporter. The expression must be compliant with the java standard "java.util.regex.Pattern" library.
    Note: The regular expression must match the entire metric name, not only a substring. For example, the regexp "netty" will not match "netty.app.recv". To do so, you can use this expression "netty.*".
    Also note that the character "\" is a special escaping character in JSON so you need to double it (like in the example above).

Reporters Configurations

Elasticsearch Reporter

In order to make topologies send their metrics to Elasticsearch, add an "elasticsearch" reporter configuration to the topology file, as illustrated next:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metrics": {
  "reporters": [
    {
      "type": "elasticsearch",
      "cluster_name": "es_search"
    },
    {
      "type": "elasticsearch",
      "http_hosts": [
        {
          "host": "localhost",
          "port": 9200
        }
      ]
    }
  ]
}

If your Elasticsearch cluster is configured to use authentication, the Elasticsearch reporter can be configured with the "credentials" field. Here is an example using a basic auth a "user" and a "password" fields:

1
2
3
4
5
6
7
8
{
  "type": "elasticsearch",
  "cluster_name": "es_search",
  "credentials": {
    "user": "sam",
    "password": "wize"
  }
}

If needed, you can also use a token as authentication with the "token" and "token_type" fields:

1
2
3
4
5
6
7
8
{
  "type": "elasticsearch",
  "cluster_name": "es_search",
  "credentials": {
    "token": "dXNlcjpwYXNzd29yZA=",
    "token_type": "Basic"
  }
}

In this case, the "token_type" depends on the security layer protecting the Elasticsearch cluster, adjust it to fit your needs.

  • cluster_name (string)

    Elasticsearch target cluster name, this name must be defined in the properties file. Mandatory if "http_hosts" is not set.

  • http_hosts (array of map)

    An array of maps containing the Elasticsearch cluster host and port information. Mandatory if "cluster_name" is not set.

    example: [{ "host": "<your_host>", "port": 9200 }]

  • index_name (string: "<tenant>-metrics")

    The final Elasticseach index is composed by <index_name>-<date>. This parameter allows you to set a custom index name to this reporter. We advise you to use the keyword "metrics" in the index name to rely on provided mapping templates (i.e. 'mytenant-special-metrics').

  • index_type (string: "_doc")

    The index document type

  • index_suffix_date_pattern (string: "yyyy.MM.dd")

    The index name suffix formatted with year, month and day

  • credentials (map)

    Use this section to set an authentication to Elasticsearch. It can rely on both user/password or token strategies.

Here is a sample of an inserted Elasticsearch document:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
  "_index": "mytenant-metrics-2019.07.16",
  "_type": "_doc",
  "_id": "iKLb-WsBGH4zfCiSNX9f",
  "_score": 1.0,
  "_source": {
    "@timestamp": "2019-07-16T08:17:00.769Z",
    "storm": {
      "worker": {
        "start_date": "2019-07-16T08:16:50.764Z",
        "uptime": {
          "count": 9
        }
      }
    },
    "name": "storm.worker.uptime",
    "rep": {
      "host": {
        "name": "punch-elitebook"
      }
    },
    "type": "storm",
    "platform": {
      "storm": {
        "topology": "tcp_in_to_stdout",
        "container_id": "punch-elitebook"
      },
      "channel": "apache-httpd",
      "id": "punchplatform-primary",
      "tenant": "mytenant"
    }
  }
}

Log Reporter

To make topologies log their metrics in a standard software logger (for exemple for appending to a file), add a logger section to the topology file, as illustrated next:

1
2
3
4
5
6
7
8
9
"metrics": {
  "reporters": [
    {
      "type": "logger",
      "reporting_interval": 5,
      "format": "kv"
    }
  ]
}
  • format (string: "json")

    Select the log output format. "kv" is more log-parsing friendly, but "json" shows full metric document as would be recorded into Elasticsearch.

    values: "kv", "json"

You can then configure the Storm logger to either log the metrics to a file or to a centralized logging service (if available on your platform). In the $STORM_INSTALL_DIR/log4j2/worker.xml of each worker machine, include the following logger:

1
<logger name="org.thales.punch.libraries.reporters.console.LogReporter" level="INFO"/>

Note

On a standalone installation, this file is located in the $PUNCHPLATFORM_CONF_DIR/../external/apache-storm-*/log4j2 folder. Also note that whenever you start a topology using the punchplatform-topology.sh shell, the logger configuration file used is different and is located at $PUNCHPLATFORM_CONF_DIR/../external/punchplatform-operator-environment-*/bin/log4j2-topology.xml

Here are 2 output examples with both "json" and "kv" format:

1
2
# JSON
13:40:56 o.t.p.l.r.c.LogReporter [INFO] {"@timestamp":"2019-07-16T11:40:56.747Z","storm":{"tuple":{"pending":{"count":0}}},"name":"storm.tuple.pending","rep":{"host":{"name":"punch-elitebook"}},"type":"storm","platform":{"storm":{"component":{"name":"tcp_spout_apache_httpd","task_id":1,"type":"syslog_spout"},"topology":"tcp_in_to_stdout","container_id":"punch-elitebook"},"channel":"default","id":"punchplatform-primary","tenant":"mytenant"}}
1
2
# KV
13:43:15 o.t.p.l.r.c.LogReporter [INFO] storm.tuple.pending.count=0 platform.id=punchplatform-primary type=storm platform.storm.component.name=tcp_spout_apache_httpd @timestamp=2019-07-16T11:43:15.792Z platform.storm.component.task_id=1 name=storm.tuple.pending platform.storm.component.type=syslog_spout platform.storm.container_id=punch-elitebook rep.host.name=punch-elitebook platform.channel=default platform.storm.topology=tcp_in_to_stdout platform.tenant=mytenant 

Kafka Reporter

In order to make topologies send their metrics to a Kafka topic, add a "kafka" reporter to the topology file, as illustrated next:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
"metrics": {
  "reporters": [
    {
      "type": "kafka",
      "bootstrap.servers": "host1:9092,host2:9092",
      "topic": "<topic_name>",
      "metric_document_field_name": "log",
      "reporting_interval": 30,
      "encoding": "lumberjack"
    }
  ]
}
  • topic (string, mandatory)

    The Kafka topic where the metrics tuples will be written

  • bootstrap.servers (list)

    A comma-separated list of Kafka bootstrap servers name. Override brokers parameter

  • brokers (string)

    Kafka brokers cluster name, matching a kafka cluster declared in punchplatform.properties.

  • encoding (string: "lumberjack")

    Specify kafka encoding, "json" or "lumberjack"

  • metric_document_field_name (string: "log")

    Set the name of the JSON field that will contain the log document

The resulting metric's document send to the Kafka topic looks the following one. Note that the document will be Lumberjack encoded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "log": {
    "@timestamp": "2019-06-20T06:56:18.543Z",
    "storm": {...},
    "name": "storm.tuple.ack",
    "rep": {...},
    "type": "storm",
    "platform": {...}
  }
}

Lumberjack Reporter

In order to make topologies forward their metrics through Lumberjack, add a "lumberjack" reporter to the topology file, as illustrated next:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
"metrics": {
  "reporters": [
    {
      "type": "lumberjack",
      "metric_document_field_name": "log",
      "reporting_interval": 30,
      "destination": [
        {
          "compression": false,
          "host": "<target_address>",
          "port": 9999
        }
      ]
    }
  ]
}
  • metric_document_field_name (string: "log")

    The JSON root field name where the metric document is sent.

  • startup_connection_timeout (integer: 1000)

    In milliseconds, the time waited at initial startup phase, before decision is made to target the group that has reached the maximum weight (no effect when only one group is targetted)

  • group_connection_timeout (integer: 1000)

    In milliseconds, the time waited at initial startup phase before it is considered an error to have no available target group

  • destination (array, mandatory)

    A list of JSON destination. To see how to declare a destination, please refer the Lumberjack bolt documentation.

Here is a result sample:

1
2
3
{
  "log": "{\"@timestamp\":\"2019-06-20T07:28:22.407Z\",\"storm\":{\"tuple\":{\"ack\":{\"count\":0,\"m1_rate\":0.0}}},\"name\":\"storm.tuple.ack\",\"rep\":{\"host\":{\"name\":\"punch-elitebook\"}},\"type\":\"storm\",\"platform\":{\"storm\":{\"component\":{\"name\":\"tcp_spout_sourcefire\",\"task_id\":1,\"type\":\"syslog_spout\"},\"topology\":\"single\",\"container_id\":\"local_punch-elitebook\"},\"channel\":\"sourcefire\",\"id\":\"punchplatform-primary\",\"tenant\":\"mytenant\"}}"
}

TCP Socket Reporter

In order to make topologies forward their metrics through TCP connexion, add a TCP socket reporter to the topology file, as illustrated next:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
"metrics": {
  "reporters": [
    {
      "type": "tcp",
      "reporting_interval": 30,
      "destination": [
        {
          "host": "<target_address>",
          "port": 9999
        }
      ]
    }
  ]
}
  • startup_connection_timeout (integer: 1000)

    time waited at initial startup phase, before decision is made to target the group that has reached the maximum weight (no effect when only one group is targetted)

  • group_connection_timeout (integer: 1000)

    time waited at initial startup phase before it is considered an error to have no available target group

  • destination (map, mandatory)

    A list of JSON destination. To see how to declare a destination, please refer the Syslog bolt documentation.

Here is a result sample:

1
{"@timestamp":"2019-06-20T08:49:22.299Z","storm":{"tuple":{"rtt":{"min":0,"max":0,"mean":0.0,"count":0,"stddev":0.0}}},"name":"storm.tuple.rtt","rep":{"host":{"name":"punch-elitebook"}},"type":"storm","platform":{"storm":{"component":{"name":"tcp_spout_sourcefire","task_id":1,"type":"syslog_spout"},"topology":"single","container_id":"local_punch-elitebook"},"channel":"sourcefire","id":"punchplatform-primary","tenant":"mytenant"}}

UDP Socket Reporter

In order to make topologies forward their metrics over UDP protocol, add an UDP socket reporter to the topology file, as illustrated next:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
"metrics": {
  "reporters": [
    {
      "type": "udp",
      "reporting_interval": 30,
      "destination": [
        {
          "host": "<target_address>",
          "port": 9999
        }
      ]
    }
  ]
}
  • destination (map, mandatory)

    A list of JSON destination. To see how to declare a destination, please refer the Syslog bolt documentation.

Here is a result sample:

1
{"@timestamp":"2019-06-20T08:58:32.216Z","name":"netty.raw.recv","type":"storm","rep":{"host":{"name":"punch-elitebook"}},"platform":{"storm":{"component":{"name":"tcp_spout_sourcefire","task_id":1,"type":"syslog_spout"},"topology":"single","container_id":"local_punch-elitebook"},"channel":"sourcefire","id":"punchplatform-primary","tenant":"mytenant"},"netty":{"raw":{"recv":{"count":0}},"target":{"protocol":"tcp","port":9902,"host":"0.0.0.0"}}}

Console Reporter

This reporter is not designed to be used in production and must be used with caution: for debugging only. It is a simple wrapper around the Java System.out.println(...) method.

Because it will output the metrics to the current terminal/console, it is the quickest way to test the metric filtering when using the "regex" option.

1
2
3
4
5
6
7
8
"metrics": {
  "reporters": [
    {
      "type": "console",
      "reporting_interval": 10
    }
  ]
}