Skip to content

Executing Plans

This section explains how to launch analytics plans in practice and which are the useful commands to keep in mind.

It is important to have a minimal understanding of Spark concepts. If you haven't already, please go through the Plan concepts section first.

File Configuration

In addition to jobs, the PunchPlatform analytics client lets you also execute Plans.

In short, a Plan provides a way of executing jobs periodically with templating support. For example, to run a job every 10 minutes that fetch data from Elasticsearch only on these last 10 minutes.

To do so, you need to use 2 files: a plan and a job template. In one hand, the plan keeps the configuration and variables (such as cluster names, dates or scheduling information), in the other hand, the job template uses information coming from the plan to update its own variables.

Note

Jobs and Plans are compatible with both JSON and HJSON

Let's use some file examples to be clearer:

plan.hjson:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  configurations:{
    cluster_name: es_search
    index_input: metricbeat-6.5.4
    nodes: ["localhost"]
  }
  cron: "*/1 * * * *"
  dates: {
    day: {
      duration: -PT1m
      format: yyyy.MM.dd
    }
    from: {
      duration: -PT1m
      format: yyyy-MM-dd'T'HH:mmZ
    }
    timezone: {
      format: XXX
    }
    to: {
      format: yyyy-MM-dd'T'HH:mmZ
    }
  }
  delay: PT10s
}

job-template.hjson:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
{
  job: [
    {
      component: input
      publish: [
        {
          stream: data
        }
      ]
      settings: {
        cluster_name: "{{cluster_name}}"
        source_column: source
        index: "{{index_input}}-{{day}}"
        nodes: {{ nodes | json_encode() }}
        query: {
          range: {
            @timestamp: {
              gte: "{{from}}"
              lt: "{{to}}"
              time_zone: "{{timezone}}"
            }
          }
        }
      }
      type: elastic_batch_input
    }
    {
      component: print_count
      settings: {
        truncate: false
      }
      subscribe: [
        {
          component: input
          stream: data
        }
      ]
      type: show
    }
  ]
}

How to launch a Plan

Using a file path

To execute a plan, you can run these command from your terminal:

1
2
3
4
5
# foreground/local
$ punchplatform-analytics.sh --plan <plan_path> --template <template_path>

# one a cluster
$ punchplatform-analytics.sh --plan <plan_path> --template <template_path> --deploy-mode cluster --spark-master <master_url>