Skip to content

Flinkline

Flinkline CRD instances are managed by the Flinkline Operator

Flinklines are the next generation streaming pipelines. They are meant at some point in time to replace the Stormlines pipelines.

Note

Flinkline are relatively new and are considered beta.

Flinkline Operator lifecycle

Note

Only the core loop is described below and this is not the complete lifecycle of a flinkline instance

A flinkline instance can go through five different phases, similarly to Pod Phases: Pending, Running, Succeeded, Failed and Unknown.

  • When an instance of a Flinkline CRD is submitted to kubernetes ApiServer
  • flinkline instance status should be empty
  • Flinkline Operator will catch the submitted event and update the flinkline instance status to Pending
  • During Pending phase, needed kubernetes sub-resources (pods and configmaps) will be created by the Flinkline Operator
  • OwnerReferences are also set to all created sub-resources to the flinkline instance
  • When all sub-resources are created successfully, the flinkline instance status will be updated to Running
  • While in Running phase, the Flinkline Operator will aggregate all sub-resources owned by flinkline instance and update the flinkline instance status based on the aggregated result

Mutating/Validating webhooks

Using webhooks with Flinkline instances

Any Flinkline CRD instances defining the .metadata.annotations.platform.gitlab.thalesdigital.io/platform: <PLATFORM_CRD_INSTANCE_NAME> will have its fields updated and validated based on the <PLATFORM_CRD_INSTANCE_NAME> resource.

apiVersion: punchline.gitlab.thalesdigital.io/v1
kind: Flinkline
metadata:
  name: flinkline-sample
  annotations:
    platform.gitlab.thalesdigital.io/platform: "<PLATFORM_CRD_INSTANCE_NAME>"
...

Configuration

Native kubernetes fields

Fields such as:

  • .apiVersion
  • .kind
  • .metadata

...are common fields, part of kubernetes terminology.

apiVersion: punchline.gitlab.thalesdigital.io/v1
kind: Flinkline
metadata:
  name: flinkline-sample
...

.metadata field is propagated to all the flinkline instance sub-resources.

Customizing an instance based on .spec field

spec:
  # defaults to true if not set
  # state which context should be used to execute the flinkline instance
  # Pod in case oneshot: true
  # Deployment in case oneshot: false / or oneshot is not set
  oneshot: false
  # define an image name
  # should be one of our published flinkline image tag
  # see: ghcr.io/punchplatform
  image: flinkline:7.0.1
  # In general, this field is taken care by our webhooks
  # Define a SA in case additional rbac or imagePullSecrets are needed during runtime
  serviceAccount: admin-user
  # In general, this field is taken care by our webhooks
  # can be any initcontainer image as long as it follows our operator defined interface
  # we do provide one in our private repository
  # ghcr.io/punchplatform
  initContainerImage: resourcectl:7.0.1
  imagePullPolicy: IfNotPresent
  # setting this to true will result in the submitted instance to be garbage upon Succeeded Phase
  # to be used only when oneshot: true
  garbageCollect: false
  # In general, this field is taken care by our webhooks
  # This field enables you to mount secret resources belonging in the same namespace as the flinkline instance
  # so as your program can consume them for various purpose: e.g. fetching data from an elasticsearch cluster.
  secretRefs:
    - name: "resourcectl-tls"
      MountPath: "/var/run/kubernetes/platform/secrets/resourcectl/resourcectl-tls"
  # define a list of dependencies this flinkline depends on
  dependencies:
    - punch-parsers:org.thales.punch:punch-websense-parsers:1.0.0
    - punch-parsers:org.thales.punch:common-punchlets:4.0.2
    - file:org.thales.punch:geoip-resources:1.0.1
  # dag definition of a flinkline punchline
  # watchout: flinkline .spec.punchline definition differs to sparkline and stormline
  punchline:
    dag:
      - id: syslog
        type: tcp-source
        host: 0.0.0.0
        port: 9902
        delimiter: "end_of_line"
        columns:
          - log
        out:
          - print
      - id: print
        type: print
  # define additional files you want to be mounted on the container filesystem during runtime
  # key: file_name
  # value: file_content
  configs:
    # this will create a file 'myCustomConfMountedOnPod'
    # with content: <value of the key>
    myCustomConfMountedOnPod: |
      # this content will be mounted on
      # the pod container local filesystem at
      # /data/myCustomConfMountedOnPod
      test: hello world

Example(s)

Standard / Syslog to Stdout

---
apiVersion: punchline.gitlab.thalesdigital.io/v1
kind: Flinkline
metadata:
  name: flinkline-sample
spec:
  image: ghcr.io/punchplatform/flinkline:7.0.1-SNAPSHOT
  serviceAccount: admin-user
  imagePullPolicy: IfNotPresent
  initContainerImage: ghcr.io/punchplatform/resourcectl:7.0.1-SNAPSHOT
  # flinkline metrics reporters are yet to be implemented
  punchline:
    dag:
    - id: syslog
      type: tcp-source
      host: 0.0.0.0
      port: 9902
      delimiter: "end_of_line"
      columns:
      - log
      out:
      - print
    - id: print
      type: print