Kafka on Kubernetes

 n kafkas/my-cluster

ketan_patel@cloudshell:~ (timebase-ts)$ 

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl create namespace kafka

namespace/kafka created

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io created

clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created

customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created

rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created

customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io created

rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created

customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created

rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created

customresourcedefinition.apiextensions.k8s.io/strimzipodsets.core.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created

customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkanodepools.kafka.strimzi.io created

configmap/strimzi-cluster-operator created

Warning: autopilot-default-resources-mutator:Autopilot updated Deployment kafka/strimzi-cluster-operator: adjusted resources to meet requirements for containers [strimzi-cluster-operator] (see http://g.co/gke/autopilot-resources)

deployment.apps/strimzi-cluster-operator created

clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created

rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created

serviceaccount/strimzi-cluster-operator created

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created

clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created

customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io created

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka

NAME                                        READY   STATUS    RESTARTS   AGE

strimzi-cluster-operator-65dff57876-djnq7   1/1     Running   0          4m14s

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get crds | grep kafka

kafkabridges.kafka.strimzi.io                      2024-01-05T22:18:42Z

kafkaconnectors.kafka.strimzi.io                   2024-01-05T22:18:40Z

kafkaconnects.kafka.strimzi.io                     2024-01-05T22:18:40Z

kafkamirrormaker2s.kafka.strimzi.io                2024-01-05T22:18:41Z

kafkamirrormakers.kafka.strimzi.io                 2024-01-05T22:18:42Z

kafkanodepools.kafka.strimzi.io                    2024-01-05T22:18:43Z

kafkarebalances.kafka.strimzi.io                   2024-01-05T22:18:46Z

kafkas.kafka.strimzi.io                            2024-01-05T22:18:41Z

kafkatopics.kafka.strimzi.io                       2024-01-05T22:18:41Z

kafkausers.kafka.strimzi.io                        2024-01-05T22:18:40Z

CREATE KAFKA CLUSTER:

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-ephemeral.yaml -n kafka

kafka.kafka.strimzi.io/my-cluster created

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka

kafka.kafka.strimzi.io/my-cluster condition met

ketan_patel@cloudshell:~ (timebase-ts)$ 

VERIFY KAFKA CLUSTER (with 3 kafka and 3 zookeeper pods are Ready(True))                   

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get kafka/my-cluster -n kafka

NAME         DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS

my-cluster   3                                                     3                                             True    

WRITE MESSAGES:(PRODUCER)

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka

NAME                                         READY   STATUS    RESTARTS   AGE

my-cluster-entity-operator-59dd54b78-cwk8b   2/2     Running   0          22m

my-cluster-kafka-0                           1/1     Running   0          31m

my-cluster-kafka-1                           1/1     Running   0          31m

my-cluster-kafka-2                           1/1     Running   0          31m

my-cluster-zookeeper-0                       1/1     Running   0          37m

my-cluster-zookeeper-1                       1/1     Running   0          37m

my-cluster-zookeeper-2                       1/1     Running   0          23m

strimzi-cluster-operator-65dff57876-hmvwj    1/1     Running   0          29m

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic

Warning: autopilot-default-resources-mutator:Autopilot updated Pod kafka/kafka-producer: defaulted unspecified resources for containers [kafka-producer] (see http://g.co/gke/autopilot-defaults)

If you don't see a command prompt, try pressing enter.

>

>>Hello ketan
>Supernal 
>Ketan Patel

READ MESSAGES: (CONSUMER)

ANOTHER WINDOW : 
Verify messages:

Kafka-Producer pod is created.

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka

NAME                                         READY   STATUS    RESTARTS   AGE

kafka-producer                               1/1     Running   0          91s

my-cluster-entity-operator-59dd54b78-cwk8b   2/2     Running   0          25m

my-cluster-kafka-0                           1/1     Running   0          35m

my-cluster-kafka-1                           1/1     Running   0          35m

my-cluster-kafka-2                           1/1     Running   0          35m

my-cluster-zookeeper-0                       1/1     Running   0          41m

my-cluster-zookeeper-1                       1/1     Running   0          41m

my-cluster-zookeeper-2                       1/1     Running   0          26m

strimzi-cluster-operator-65dff57876-hmvwj    1/1     Running   0          32m

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning

Warning: autopilot-default-resources-mutator:Autopilot updated Pod kafka/kafka-consumer: defaulted unspecified resources for containers [kafka-consumer] (see http://g.co/gke/autopilot-defaults)

If you don't see a command prompt, try pressing enter.

Hello ketan

Supernal 

Ketan Patel

ANOTHER WINDOW:

Kafka-Consumer Pod is created.

ketan_patel@cloudshell:~ (timebase-ts)$ kubectl get pods -n kafka

NAME                                         READY   STATUS    RESTARTS   AGE

kafka-consumer                               1/1     Running   0          34s

kafka-producer                               1/1     Running   0          2m13s

my-cluster-entity-operator-59dd54b78-cwk8b   2/2     Running   0          26m

my-cluster-kafka-0                           1/1     Running   0          36m

my-cluster-kafka-1                           1/1     Running   0          36m

my-cluster-kafka-2                           1/1     Running   0          36m

my-cluster-zookeeper-0                       1/1     Running   0          41m

my-cluster-zookeeper-1                       1/1     Running   0          41m

my-cluster-zookeeper-2                       1/1     Running   0          27m

strimzi-cluster-operator-65dff57876-hmvwj    1/1     Running   0          33m

ketan_patel@cloudshell:~ (timebase-ts)$