Plans¶

Abstract

A plan is a periodic scheduler of punchlines or other applications. This chapter provides the detailed information to configure and setup a plan.

Two files are at play :

a punchline template that will serve to generated each punchline to be executed.
a plan configuration file that defines what variables you want to dynamically inject into your punchline template

Plan configuration file¶

The layout of a plan configuration file is defined as follows:

tenant: mytenant
channel: mychannel
version: '6.0'
name: myplan
model:
  # You can define here arbitrary variables to be used
  # in your punchline template using {{var1}}
  var1: a string
  var2: false

    # the special dates dictionary contains your date variables. These
    # follow a special format using standard ISO patters to represent
    # time intervals.

  dates:
    from:
      # this means "now minus one day"
      offset: -PT1d
      format: yyyy.MM.dd
    to:
      # this means now
      offset: -PT0
      format: yyyy.MM.dd
settings:
    # Most often a plan runs forever, you may provide optional start and end times 
    # if you want to make it work on ly between two time ranges. For example: 
    #   start: 2020-01-01T10:15:30+01:00
    #   stop: 2059-01-14T13:00:18+01:00
    #   delay: PT10s

    # The required cron expression, here it says every minute.
    cron: '*/1 * * * *'

    # If you launch a plan with the --last-committed option,
    # you can provide the persistence backend configuration.
    # A default can be defined in your punchplatform.properties file.
    persistence:
    - type: elasticsearch
      index_name: platform-plan-cursor
      es_cluster: es_search
      nodes:
      - host: localhost
        port: 9200
      ssl: false
      credentials:
        user: admin
        password: admin

The mandatory properties are :

tenant: the name of the tenant in which your plan is defined and executed.
version: the plan format version number.
channel: the name of the channel in which the plan is executed
name: (mandatory) a short name specified in your channel structure
model: key-values where you set the variables to be substituted in your punchline.hjson configuration file.
model.dates: a reserved dictionary that contains the dates to be substituted in your punchline.hjson configuration file. Each date is itself a dictionary containing two properties, a format and an offset to represent a time duration.
settings.cron : the cron expression used to trigger the execution of the punchline

The optional properties are :

settings.persistence : a list of only one backend configuration used to persist the cursor of the plan. If multiple persistence backends are configured, the plan use only the first item.
settings.persistence[0].type : string only "elasticsearch" available
settings.persistence[0].nodes[].host : string the fqdn of the elasticsearch node
settings.persistence[0].nodes[].port : integer the port to access to the elasticsearch node
settings.persistence[0].nodes[].index_name : string the name of the index used to store the persistence information
settings.persistence[0].nodes[].ssl : boolean (false) enable or disable ssl to access to elasticsearch
settings.persistence[0].nodes[].ssl_certificate : string the path to the certificate used to perform client authentication
settings.persistence[0].nodes[].ssl_private_key : string the path to the key used to perform client authentication
settings.persistence[0].nodes[].ssl_private_key_pass : string the password to open the ssl_private_key
settings.persistence[0].nodes[].ssl_trusted_certificate : string the certificate used to trust the elasticsearch certificate
settings.persistence[0].nodes[].credtentials.users : string the user used to authenticate to elasticsearch
settings.persistence[0].nodes[].credtentials.password : string the password used to authenticate to elasticsearch

Reporting¶

You can add an optional reporting section:

metrics:
  reporters:
  - type: elasticsearch
    cluster_name: es_search
    index_name: example_plan_metrics

Time Duration Format¶

from:
    # the format for generating the substitued date
    format: yyyy-MM-dd'T'HH:mmZ
    # a time offset.
    offset: -PT2m

Durations define the amount of intervening time in a time interval and are represented by the format P[n]Y[n]M[n]DT[n]H[n]M[n]S or P[n]W. In these representations, the [n] is replaced by the value for each of the date and time elements that follow the [n].

Leading zeros are not required, but the maximum number of digits for each element should be agreed to by the communicating parties. The capital letters P, Y, M, W, D, T, H, M, and S are designators for each of the date and time elements and are not replaced.

P is the duration designator (for period) placed at the start of the duration representation.
Y is the year designator that follows the value for the number of years.
M is the month designator that follows the value for the number of months.
W is the week designator that follows the value for the number of weeks.
D is the day designator that follows the value for the number of days.
T is the time designator that precedes the time components of the representation.
H is the hour designator that follows the value for the number of hours.
M is the minute designator that follows the value for the number of minutes.
S is the second designator that follows the value for the number of seconds.

For example, `P3Y6M4DT12H30M5S represents a duration of "three years, six months, four days, twelve hours, thirty minutes, and five seconds".

Date and time elements including their designator may be omitted if their value is zero, and lower-order elements may also be omitted for reduced precision. For example, "P23DT23H" and "P4Y" are both acceptable duration representations. However, at least one element must be present, thus "P" is not a valid representation for a duration of 0 seconds. "PT0S" or "P0D", however, are both valid and represent the same duration.

To resolve ambiguity, "P1M" is a one-month duration and "PT1M" is a one-minute duration (note the time designator, T, that precedes the time value). The smallest value used may also have a decimal fraction, as in "P0.5Y" to indicate half a year. This decimal fraction may be specified with either a comma or a full stop, as in "P0,5Y" or "P0.5Y". The standard does not prohibit date and time values in a duration representation from exceeding their "carry over points" except as noted below. Thus, "PT36H" could be used as well as "P1DT12H" for representing the same duration. But keep in mind that "PT36H" is not the same as "P1DT12H" when switching from or to Daylight saving time.

Alternatively, a format for duration based on combined date and time representations may be used by agreement between the communicating parties either in the basic format PYYYYMMDDThhmmss or in the extended format P[YYYY]-[MM]-[DD]T[hh]:[mm]:[ss]. For example, the first duration shown above would be "P0003-06-04T12:30:05". However, individual date and time values cannot exceed their moduli (e.g. a value of 13 for the month or 25 for the hour would not be permissible).[34]

Note

Although the standard describes a duration as part of time intervals, the duration format is widely used independent of time intervals, as with the Java 8 Duration class.

Punchline Template Example¶

The punchline template is an arbitrary punchline file that contains one or several variables using the {{var}} notation. These will be substituted from the plan model file.

Here is a typical punchline block example containing two duration variables.

dag:
- type: elastic_input
  settings:
    port: { port }}
    index:  {{ index_name }}
    query:
      query:
        bool:
          must:
          - range:
              '@timestamp':
                gte: {{ from }}
                lt: {{ to }}

Note

The plan application leverages the Jtwig library, a template engine similar to Jinja2 and its Json extension.

Refer to the planctl manual page for information about executing plans.