Books

Warning

books features is in alpha status.

Abstract

A book is a grouping of workflow of several applications. In contrast to channels, books launch punchlines that eventually terminates. Books lets you order an arbitrary chaining of application, and define when your application should terminate.

Overview

An example will best illustrate a book. Just like channels books are described using a book_structure.yml file. Here is an example:

version: '6.0'
applications:
    - name: first
      runtime: shiva
      cluster: common
      command: punchlinectl
      args:
        - start
        - --punchline
        - first.yml
    - name: second
      runtime: shiva
      cluster: common
      after_success:
        - first
      command: punchlinectl
      args:
        - start
        - --punchline
        - second.yml

This book will launch a first punchline first configured using a first.yml file. Then if it succeeds, it will launch the secondpunchline. As simple as that.

Exit Conditions

Books launch punchlines that eventually stop. Example are spark or pyspark punchlines that execute some batch processing then terminates.

Stream punchline can also be used in that mode, for example to transfer some data from a backend (say elasticsearch) to a file. True you can do that also with a spark punchline but stream punchlines are much more lightweight and will consume far less memory to do the job.

Streaming punchline must however be told when they should exit; i.e. what criteria must be used to know everything has been processed. You specify that use an exit configuration that includes a condition. Here is an example:

version: '6.0'
applications:
    - name: mypunchline
      runtime: shiva
      cluster: common
      command: punchlinectl
      args:
        - start
        - --punchline
        - mypunchline.yml
      exit:
        condition:
            success:
                acks_greater_or_equal_to: 100
            failure:
                fails_greater_or_equal_to: 1

The condition here tells the punchline to exit after 100 data items have been successfully processed and acknowledged. It will fail if some item failed to be processed.

It is often useful to request a streaming punchline to exit whenever new data does not arrive after some time. Here is how you specify that:

version: '6.0'
applications:
    - name: mypunchline
      runtime: shiva
      cluster: common
      command: punchlinectl
      args:
        - start
        - --punchline
        - mypunchline.yml
      exit:
        condition:
            success:
                acks_greater_or_equal_to: 1
                inactive_for_more_than: 20s
            failure:
                fails_greater_or_equal_to: 1

The condition here tells the punchline to exit 20 seconds after the last data item was successfully processed.

Tip

The exit condition can be defined either in the book_structure.yml file, or directly inside the punchline configuration file.