Skip to content

Plans

Concepts

Plans are punch applications that provides you with periodic scheduling of punchlines. It is similar to a regular cron-like tool, and enables you to trigger PunchLines jon a regular intervals. In contrast to crons, it provides you with:

  • resiliency: you can restart a plan from another server, it will properly resume processing from a last saved cursor.
  • logging: state of your plan application, uptime, etc...
  • configuration: a plan is easily defined as (yet another) hjson or json file.
  • templating: every punchline executed by a plan is (typically) generated first from a template in a way to make it act on the right input datasets.

Most often templating is used to generate dates. Why dates ? because the essential parameters for retrieving input data is based on time.

The following diagram illustrates the way it works. Your punchline configuration file is actually a template file. The plan will be in charge of periodically defining date values, then generating actual job files from the templates, which in turn is scheduled for execution.

image

Configuration

The schema of a plan configuration file is defined as follows:

{
  tenant: mytenant
  channel: atestchannel
  version: "6.0"
  name: configuration.plan
  model:{

    // You can define here arbitrary variables to be used
    // in your punchline template.
    your_variable_1: "a string"
    your_variable_2: false
    ...

    // The dates dictionary contains your date variables. These
    // follow a special format usign standard ISO pattenrs to represent
    // time intervals.
    dates: {
       your_date_variable_1: {
         offset: -PT1m
         format: yyyy.MM.dd
       }
        ...
    }
  }
  settings: {
    # start not required
    start: 2019-11-14T13:00:18.940Z
    # stop not required
    stop: 2059-11-14T13:00:18.940Z
    # delay not required 
    delay: PT10s
    # cron required
    cron: "*/1 * * * *"
    # required with --last-committed
    # default to information based on your punchplatform.properties
    persistence: [
        {
            type: elasticsearch
            index_name: platform-plan-cursor
            es_cluster: es_search
            # Nodes parameter takes priority over es_cluster
            nodes: [
                {
                    host: localhost
                    port: 9200
                }
            ]
            ssl: false
            credentials: {
                token: mytoken
                token_type: ApiKey
            }
        }
    ]
  }
  # not required
  # generates logging and metrics information in an index...
  metrics: {
    reporters: [
      {
        type: elasticsearch
        cluster_name: es_search
        index_name: example_plan_metrics
      }
    ]
  }
}
- tenant: (mandatory) name of the tenant that will be executing the plan - version: (mandatory) version of your Platform - channel: (mandatory) name of the channel in which the plan will be executed - name: (mandatory) a short name specified in your channel structure - model: key-value where you will be setting variables to be substituted in your job.pml configuration file. - model.dates: reserved dictionary containing your dates to be substituted in your job.pml configuration file. > each date is itself a dictionary containing two key-value
my_date: {
    format: yyyy-MM-dd'T'HH:mmZ
    offset: -PT2m
}

Punchline Template Example

{
    jobs: [
        {
            type: elastic_batch_input
            settings: {
                port: {{ port }}
                index: {{ index_name }}
                query: {
                    query: {
                        bool: {
                            must: [
                                {
                                    range: {
                                        @timestamp: {
                                            gte: {{ mydate1 }}
                                            lt: {{ mydate2 }}
                                        }
                                    }
                                }
                            ]
                        }
                    }
                }
            }
            ...
        }
        ...
    ]
}

Note

The plan application leverages the Jtwig library, a template engine similar to Jinja2 and its Json extension.

How to launch/manage a Plan

Please refer to the dedicated Plans Executions operations guide.