Books

Abstract

This chapter explains how you assemble various applications into a book. A book is a set of ephemeral applications that will execute following a given order. Once all applications are done, the book will stop as well.

Difference with Channels

In contrast to channels, books have a different lifecyle. Channels are eternal. That simply means that once you start your channel, it will run forever.

In contrast a book is ephemeral. I.e. it starts, launches its inner (ephemeral) applications in some order, then stops.

A channel only contains eternal applications. An eternal application is:

• one that funs forever, such as a continuously streaming ingestion application
• one that is periodically executed, such as an aggregation application thatt processes finite datasets.

An book runs once, and submit in a defined order its inner (ephemeral) applications. Examples of such ephemeral applications are :

• an extraction application
• a data replay application

Book Structure

Layout

The book_structure.json file has the following structure:

{
    // version is usually the one of the main punch release.
    // This is to guarantee backward compatibility. 
    version: "6.0"

    // A channel can include several processing applications.
    applications: [
        { 
            application
        }
    ]
    // an potentially some shared resources such as kafka topics
    resources: [
        { resource part }
    ]

    settings: {
        // pause time between checking for submitted applications status
        inter_stage_delay: 10s

        // timeout when getting exited applications list
        exit_application_timeout: 3s
    }
}

Example

Here is a simple book that launches two punchlines:

{
  version: "6.0"
  resources: [
    {
      partitions: 1
      cluster: local
      replication_factor: 1
      name: a_book_topic 
      type: kafka_topic
    }
  ]
  applications: [
     {
      name: first
      runtime : shiva
      command : punchlinectl
      args: [ 
        start 
        --punchline 
        first.json 
        ]
        shiva_runner_tags: [ 
          local 
        ]
        cluster: common
    }
    {
      name: second
      runtime : shiva
      command : punchlinectl
      args: [ 
        start 
        --punchline 
        second.json 
        ]
        shiva_runner_tags: [ 
          local 
        ]
        cluster: common
        after_success : [ 
            first 
        ]
    }
  ]
}

Notice the after_success property to specify that the second punchline must be executed only after the first succeeded.

Tip

Books and channels structure files are very similar. Only the additional application ordering is new.