Skip to content

Streams

Abstract

Streams are next generation data processing pipelines powered by apache flink. Streams are alpha release and provided by a kast data processing pack. This chapter explains how to run a stream on top of kast.

Warning

This section describes the punch helm streams usage only for debugging. His administration (start/status/stop) is wrapped in punch commands so you do not need to use it directly.

Deploy Streams

Warning

Requirements :

  • An access to a Kast cluster

Command example :

helm install my-topology kast-stream-fn \
    --set-file topology=topology.json \
    -f config.yml
    --namespace kast

Configuration

Files

Files :

File Purpose
stream.yaml the stream configuration
resources.yaml Resource loader configuration

Platform configuration

Example :

image:
  repository: gitlab.thalesdigital.io:5005/sixdt/kast-stream
  pullPolicy: Always
  tag: "custom-nodes"

service:
  type: ClusterIP
  port: 8081
  nodePorts:
    - port: 9901
      type: tcp

Resource loader configuration

Example:

resourcesLoader: true

resourcesLoaderConf:
  user: username
  pass: password
  endpoint: <S3_HOST>
  resources: bucket_name/path/to/folder