Skip to content

Configure Fluentbit with Strimzi Kafka Bridge

In Kubernetes, Kafka Strimzi Operator provides a HTTP Bridge to Kafka. This makes it possible to interact with your Kafka outside the Kubernetes cluster through HTTP protocol.

Prerequisites

  • Strimzi Kafka Cluster.
  • Ingress controller.

Install Kafka Bridge

Following instructions are based on this documentation.

Kafka Bridge

  • Apply this file in the same namespace as your Kafka.
    apiVersion: kafka.strimzi.io/v1alpha1
    kind: KafkaBridge
    metadata:
      name: my-bridge
    spec:
      replicas: 1
      bootstrapServers: my-cluster-kafka-bootstrap:9092
      http:
        port: 8080
    
  • For instance :
    kubectl apply -f /path/to/kafka-bridge.yaml -n processing
    

Ingress Bridge

  • Apply this file in the same namespace as your Kafka.
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: my-bridge-ingress
    spec:
      rules:
      - host: my-bridge.io
        http:
          paths:
          - path: /
            backend:
              serviceName: my-bridge-bridge-service
              servicePort: 8080
    
  • For instance :
    kubectl apply -f /path/to/ingress-bridge.yaml -n processing
    

DNS

  • Add my-bridge.io to your /etc/hosts, or in your DNS configuration file.
  • For instance :
    10.170.4.51    kubeworker0  my-bridge.io
    

Check

  • Check that your receive a 200 Code with following command :
    curl -v GET http://my-bridge.io/healthy
    
  • Try to POST message to my-topic :
    curl -X POST \
      http://my-bridge.io/topics/my-topic \
      -H 'content-type: application/vnd.kafka.json.v2+json' \
      -d '{
        "records": [
            {
                "key": "key-1",
                "value": "value-1"
            },
            {
                "key": "key-2",
                "value": "value-2"
            }
        ]
    }'
    

Configure Fluentbit

Following instructions are based on this documentation.

  • Add following configuration in your /etc/td-agent-bit/os_log.conf :
[INPUT]
    Name              tail
    Tag               records
    Path              /data/aircraft_data.log
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  5
    Read_from_Head    True

[OUTPUT]
    Name        kafka-rest
    Match       *
    Host        my-bridge.io
    Port        80
    Topic       source-aircraft-report
  • This will read the lines in /data/aircraft_data.log and POST them to Kafka bridge using following URL : http://my-bridge.io:80/topics/source-aircraft-report