Skip to content

Plans

Abstract

A plan is a periodic scheduler of punchlines or other applications. This chapter provides the information to configure and setup a plan.

Two files are at play :

  • a punchline template that will serve to generate each punchline to be executed.
  • a plan configuration file that defines what variables you want to dynamically inject into your punchline template

Plan configuration file

The layout of a plan configuration file is defined as follows:

tenant: mytenant
channel: mychannel
version: '6.0'
name: myplan
model:
  # You can define here arbitrary variables to be used
  # in your punchline template using {{var1}}
  var1: a string
  var2: false

    # the special dates dictionary contains your date variables. These
    # follow a special format using standard ISO patters to represent
    # time intervals.

  dates:
    from:
      # this means "now minus one day"
      offset: -PT1d
      format: yyyy.MM.dd
    to:
      # this means now
      offset: -PT0
      format: yyyy.MM.dd

  settings:
    # Most often a plan runs forever, you may provide optional start and end times 
    # if you want to make it work on ly between two time ranges. For example: 
    #   start: 2020-01-01T10:15:30+01:00
    #   stop: 2059-01-14T13:00:18+01:00
    #   delay: PT10s

    # The required cron expression, here it says every minute.
    cron: '*/1 * * * *'

    # If you launch a plan with the --last-committed option,
    # you can provide the persistence backend configuration.
    # A default can be defined in your punchplatform.properties file.
    persistence:
    - type: elasticsearch
      index_name: platform-plan-cursor
      es_cluster: es_search
      nodes:
      - host: localhost
        port: 9200
      ssl: false
      credentials:
        user: admin
        password: admin

The mandatory properties are : * tenant: the name of the tenant in which your plan is defined and executed. * version: the plan format version number. * channel: the name of the channel in which the plan is executed * name: (mandatory) a short name specified in your channel structure * model: key-values where you set the variables to be substituted in your punchline.hjson configuration file. * model.dates: a reserved dictionary that contains the dates to be substituted in your punchline.hjson configuration file. Each date is itself a dictionary containing two properties, a format and an offset to represent a time duration.

Reporting

You can add an optional reporting section:

metrics:
  reporters:
  - type: elasticsearch
    cluster_name: es_search
    index_name: example_plan_metrics

Time Duration Format

Durations define the amount of intervening time in a time interval and are represented by the format P[n]Y[n]M[n]DT[n]H[n]M[n]S or P[n]W. In these representations, the [n] is replaced by the value for each of the date and time elements that follow the [n].

from:
    # the format for generating the substitued date
    format: yyyy-MM-dd'T'HH:mmZ
    # a time offset.
    offset: -PT2m

Leading zeros are not required, but the maximum number of digits for each element should be agreed to by the communicating parties. The capital letters P, Y, M, W, D, T, H, M, and S are designators for each of the date and time elements and are not replaced.

  • P is the duration designator (for period) placed at the start of the duration representation.
  • Y is the year designator that follows the value for the number of years.
  • M is the month designator that follows the value for the number of months.
  • W is the week designator that follows the value for the number of weeks.
  • D is the day designator that follows the value for the number of days.
  • T is the time designator that precedes the time components of the representation.
  • H is the hour designator that follows the value for the number of hours.
  • M is the minute designator that follows the value for the number of minutes.
  • S is the second designator that follows the value for the number of seconds.

For example, P3Y6M4DT12H30M5S represents a duration of "three years, six months, four days, twelve hours, thirty minutes, and five seconds".

Date and time elements including their designator may be omitted if their value is zero, and lower-order elements may also be omitted for reduced precision. For example, "P23DT23H" and "P4Y" are both acceptable duration representations. However, at least one element must be present, thus "P" is not a valid representation for a duration of 0 seconds. "PT0S" or "P0D", however, are both valid and represent the same duration.

To resolve ambiguity, "P1M" is a one-month duration and "PT1M" is a one-minute duration (note the time designator, T, that precedes the time value). The smallest value used may also have a decimal fraction, as in "P0.5Y" to indicate half a year. This decimal fraction may be specified with either a comma or a full stop, as in "P0,5Y" or "P0.5Y". The standard does not prohibit date and time values in a duration representation from exceeding their "carry over points" except as noted below. Thus, "PT36H" could be used as well as "P1DT12H" for representing the same duration. But keep in mind that "PT36H" is not the same as "P1DT12H" when switching from or to Daylight saving time.

Alternatively, a format for duration based on combined date and time representations may be used by agreement between the communicating parties either in the basic format PYYYYMMDDThhmmss or in the extended format P[YYYY]-[MM]-[DD]T[hh]:[mm]:[ss]. For example, the first duration shown above would be "P0003-06-04T12:30:05". However, individual date and time values cannot exceed their moduli (e.g. a value of 13 for the month or 25 for the hour would not be permissible).[34]

Note

Although the standard describes a duration as part of time intervals, the duration format is widely used independent of time intervals, as with the Java 8 Duration class.

Punchline Template Example

The punchline template is an arbitrary punchline file that contains one or several variables using the {{var}} notation. These will be substituted from the plan model file.

Here is a typical punchline block example containing two duration variables.

dag:
- type: elastic_batch_input
  settings:
    port: { port }}
    index:  {{ index_name }}
    query:
      query:
        bool:
          must:
          - range:
              '@timestamp':
                gte: {{ from }}
                lt: {{ to }}

Executing Arbitrary Applications

Depending on your use-case, argument: --user-defined can be specified as many times as needed and is non mandatory.

You must use --runtime user_defined to be able to launch planctl in this mode

planctl start \
    --runtime user_defined \
    --plan plan.hjson \
    --template template.hjson \
    --user-defined argument1 \
    --user-defined argument2

# note: interpreter that will be used if not specified will be -> /bin/sh
# for various reason, in case you wish to modify the interpreter that will be used for executing your
# templated file, you may add the --intepreter $ABSOLUTE_PATH_TO_EXECUTABLE or make use of SHEBANG header

planctl start \
    --runtime user_defined \
    --plan plan.hjson \
    --template template.hjson \
    --user-defined argument1 \
    --user-defined argument2 \
  --interpreter /bin/sh

With:

plan.hjson

{
  version: "6.0"
  name: basic_user
  model:{
    metric_index: mytenant-metrics
    plan_logs_index: platform-plan-logs*
    input_index: mytenant-events-*
    dates: {
       day: {
         offset: -PT1m
         format: yyyy.MM.dd
       }
       from: {
         offset: -PT1m
         format: yyyy-MM-dd'T'HH:mmZ
       }
       to: {
         format: yyyy-MM-dd'T'HH:mmZ
       }
    }
  }
  settings: {
    cron: "*/1 * * * *"
    persistence: [
      {
        type: elasticsearch
        index_name: mytenant-plan-cursor
      }
    ]
  }
  metrics: {
    reporters: [
      {
        type: kafka
      }
    ]
  }
}

template.hjson

#!/bin/sh -

echo "HELLO WORLD FROM={{from}} TO={{to}} $@"

Note

The plan application leverages the Jtwig library, a template engine similar to Jinja2 and its Json extension.

Refer to the planctl manual page for information about executing plans.