Kafka on Kubernetes

 

Deploying Kafka on Kubernetes(GKE) using Strimzi Operator:

Kafka:

Apache Kafka is an event streaming platform. 

Kafka combines three key capabilities so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:

To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.
To store streams of events durably and reliably for as long as you want.
To process streams of events as they occur or retrospectively.

Kafka Producers

In Kafka, the producers send data directly to the broker that plays the role of leader for a given partition.

Kafka Brokers

In Kafka, the cluster usually contains multiple nodes, that are known as brokers, to maintain the load balance. The brokers are stateless, and hence their cluster state is maintained by the ZooKeeper.

Kafka Consumers

In Kafka, the consumer has to issue requests to the brokers indicating the partitions it wants to consume. The consumer is required to specify its offset in the request and receives a chunk of log beginning from the offset position from the broker. 

Strimzi Operators:

Operators are a method of packaging, deploying, and managing Kubernetes applications. 

They provide a way to extend the Kubernetes API and simplify the administration tasks associated with specific applications.

Strimzi operators support tasks related to a Kafka deployment. Strimzi custom resources provide the deployment configuration. 

Operators within the Strimzi architecture

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka | grep operator

my-cluster-entity-operator-59dd54b78-l5hx7   2/2     Running   0          2d

strimzi-cluster-operator-65dff57876-p28mx    1/1     Running   0          2d

ketan_patel@cloudshell:~ (timebase-ts)$ 

This includes configuration for Kafka clusters, topics, users, and other components. 

Leveraging custom resource configuration, Strimzi operators create, configure, and manage Kafka components within a Kubernetes environment. 

Using operators reduces the need for manual intervention and streamlines the process of managing Kafka in a Kubernetes cluster.

Example architecture for the Cluster Operator:

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl create namespace kafka

namespace/kafka created

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created
customresourcedefinition.apiextensions.k8s.io/strimzipodsets.core.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkanodepools.kafka.strimzi.io created
configmap/strimzi-cluster-operator created
Warning: autopilot-default-resources-mutator:Autopilot updated Deployment kafka/strimzi-cluster-operator: adjusted resources to meet requirements for containers [strimzi-cluster-operator] (see http://g.co/gke/autopilot-resources)
deployment.apps/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created
serviceaccount/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io created

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka | grep operator
my-cluster-entity-operator-59dd54b78-l5hx7   2/2     Running   0          2d
strimzi-cluster-operator-65dff57876-p28mx    1/1     Running   0          2d
ketan_patel@cloudshell:~ (timebase-ts)$ 

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get crds | grep kafka

kafkabridges.kafka.strimzi.io                      2024-01-05T22:18:42Z
kafkaconnectors.kafka.strimzi.io                   2024-01-05T22:18:40Z
kafkaconnects.kafka.strimzi.io                     2024-01-05T22:18:40Z
kafkamirrormaker2s.kafka.strimzi.io                2024-01-05T22:18:41Z
kafkamirrormakers.kafka.strimzi.io                 2024-01-05T22:18:42Z
kafkanodepools.kafka.strimzi.io                    2024-01-05T22:18:43Z
kafkarebalances.kafka.strimzi.io                   2024-01-05T22:18:46Z
kafkas.kafka.strimzi.io                            2024-01-05T22:18:41Z
kafkatopics.kafka.strimzi.io                       2024-01-05T22:18:41Z
kafkausers.kafka.strimzi.io                        2024-01-05T22:18:40Z

Create an Apache Kafka cluster

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-ephemeral.yaml -n kafka
kafka.kafka.strimzi.io/my-cluster created

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka

kafka.kafka.strimzi.io/my-cluster condition met
ketan_patel@cloudshell:~ (timebase-ts)$ 

                     

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get kafka/my-cluster -n kafka

NAME         DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS
my-cluster   3                        3                     True    

WRITE MESSAGES: (PRODUCER)

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka

NAME                                         READY   STATUS    RESTARTS   AGE
my-cluster-entity-operator-59dd54b78-cwk8b   2/2     Running   0          22m
my-cluster-kafka-0                           1/1     Running   0          31m
my-cluster-kafka-1                           1/1     Running   0          31m
my-cluster-kafka-2                           1/1     Running   0          31m
my-cluster-zookeeper-0                       1/1     Running   0          37m
my-cluster-zookeeper-1                       1/1     Running   0          37m
my-cluster-zookeeper-2                       1/1     Running   0          23m
strimzi-cluster-operator-65dff57876-hmvwj    1/1     Running   0          29m

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic

Warning: autopilot-default-resources-mutator:Autopilot updated Pod kafka/kafka-producer: defaulted unspecified resources for containers [kafka-producer] (see http://g.co/gke/autopilot-defaults)
If you don't see a command prompt, try pressing enter.
>

>>Hello ketan
>Supernal 
>Ketan Patel

READ MESSAGES: (CONSUMER)

ANOTHER WINDOW : 
Verify messages:

Kafka-Producer pod is created.

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka
NAME                                         READY   STATUS    RESTARTS   AGE
kafka-producer                               1/1     Running   0          91s
my-cluster-entity-operator-59dd54b78-cwk8b   2/2     Running   0          25m
my-cluster-kafka-0                           1/1     Running   0          35m
my-cluster-kafka-1                           1/1     Running   0          35m
my-cluster-kafka-2                           1/1     Running   0          35m
my-cluster-zookeeper-0                       1/1     Running   0          41m
my-cluster-zookeeper-1                       1/1     Running   0          41m
my-cluster-zookeeper-2                       1/1     Running   0          26m
strimzi-cluster-operator-65dff57876-hmvwj    1/1     Running   0          32m

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
Warning: autopilot-default-resources-mutator:Autopilot updated Pod kafka/kafka-consumer: defaulted unspecified resources for containers [kafka-consumer] (see http://g.co/gke/autopilot-defaults)

If you don't see a command prompt, try pressing enter.

Hello ketan
Supernal 
Ketan Patel

ANOTHER WINDOW:

Kafka-Consumer Pod is created.

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka

NAME                                         READY   STATUS    RESTARTS   AGE
kafka-consumer                               1/1     Running   0          34s
kafka-producer                               1/1     Running   0          2m13s
my-cluster-entity-operator-59dd54b78-cwk8b   2/2     Running   0          26m
my-cluster-kafka-0                           1/1     Running   0          36m
my-cluster-kafka-1                           1/1     Running   0          36m
my-cluster-kafka-2                           1/1     Running   0          36m
my-cluster-zookeeper-0                       1/1     Running   0          41m
my-cluster-zookeeper-1                       1/1     Running   0          41m
my-cluster-zookeeper-2                       1/1     Running   0          27m
strimzi-cluster-operator-65dff57876-hmvwj    1/1     Running   0          33m
ketan_patel@cloudshell:~ (timebase-ts)$

============== ======================

Another Producer & Consumer pod(0110) is created with new topic(1010)

ketan [ ~ ]$  kubectl  run kafka-producer-0110 -ti --image=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic-1010

If you don't see a command prompt, try pressing enter.

>011024

>ketan

>patel

>123

>hello011024

>

ketan [ ~ ]$ kubectl run kafka-consumer-0110 -ti --image=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic-1010 --from-beginning

If you don't see a command prompt, try pressing enter.

011024

ketan

patel

123

hello011024

ketan [ ~ ]$ kubectl get pods | grep kafka

kafka-consumer                                1/1     Running                  0             4d22h

kafka-consumer-0110                           1/1     Running                  0             87s

kafka-producer                                1/1     Running                  0             4d22h

kafka-producer-0110                           1/1     Running                  0             3m46s

my-cluster-kafka-0                            1/1     Running                  0             4d22h

my-cluster-kafka-1                            1/1     Running                  0             4d21h

my-cluster-kafka-2                            1/1     Running                  0             4d7h

ketan [ ~ ]$