Overview
Set up a Google Kubernetes Engine cluster, then deploy the Managed Service for Prometheus to ingest metrics from a simple application.
Managed Service for Prometheus is Google Cloud's fully managed storage and query service for Prometheus metrics. This service is built on top of Monarch, the same globally scalable data store as Cloud Monitoring.
A thin fork of Prometheus replaces existing Prometheus deployments and sends data to the managed service with no user intervention. This data can then be queried by using PromQL through the Prometheus Query API supported by the managed service and by using the existing Cloud Monitoring query mechanisms.
Objectives
Deploy the Managed Service for Prometheus to a GKE cluster
Deploy a Python application to monitor
Create a Cloud Monitoring dashboard to view metrics collected
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ gcloud auth list
Credentialed Accounts
ACTIVE: *
ACCOUNT: student-04-61f5f7cf52ff@qwiklabs.net
To set the active account, run:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ gcloud config list project
[core]
project = qwiklabs-gcp-00-a051edc2be39
Your active configuration is: [cloudshell-8839]
Run the following command to deploy a standard GKE cluster, which will prompt you to authorize and enable the GKE API:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ gcloud beta container clusters create gmp-cluster --num-nodes=1 --zone us-east1-b --enable-managed-prometheus
Default change: VPC-native is the default mode during cluster creation for versions greater than 1.21.0-gke.1500. To create advanced routes based clusters, please pass the `--no-enable-ip-alias` flag
Default change: During creation of nodepools or autoscaling configuration changes for cluster versions greater than 1.24.1-gke.800 a default location policy is applied. For Spot and PVM it defaults to ANY, and for all other VM kinds a BALANCED policy is used. To change the default values use the `--location-policy` flag.Note: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
Creating cluster gmp-cluster in us-east1-b... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1beta1/projects/qwiklabs-gcp-00-a051edc2be39/zones/us-east1-b/clusters/gmp-cluster].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east1-b/gmp-cluster?project=qwiklabs-gcp-00-a051edc2be39
kubeconfig entry generated for gmp-cluster.
NAME: gmp-cluster
LOCATION: us-east1-b
MASTER_VERSION: 1.27.3-gke.100
MASTER_IP: 35.196.141.45
MACHINE_TYPE: e2-medium
NODE_VERSION: 1.27.3-gke.100
NUM_NODES: 1
STATUS: RUNNING
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$
Run the following command to authenticate to the cluster:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ gcloud container clusters get-credentials gmp-cluster --zone us-east1-b
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gmp-cluster.
Task 2. Deploy the Prometheus service
Run the following command to create a namespace to do the work in:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl create ns gmp-test
namespace/gmp-test created
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl get ns
NAME STATUS AGE
default Active 5m36s
gmp-public Active 4m31s
gmp-system Active 4m31s
gmp-test Active 8s
kube-node-lease Active 5m36s
kube-public Active 5m36s
kube-system Active 5m37s
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$
Task 3. Deploy the application
Deploy a simple application which emits metrics at the /metrics endpoint:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/flask_deployment.yaml
deployment.apps/helloworld-gke created
--------------------- ----------------------------------------
# This file configures the hello-world app which serves public web traffic.
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-gke
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello-app
image: gcr.io/ops-demo-330920/flask_telemetry:61a2a7aabc7077ef474eb24f4b69faeab47deed9
# This app listens on port 4000 for web traffic by default.
ports:
- containerPort: 4000
name: flaskport
env:
- name: PORT
value: "4000"
----------------------------- ----------------------------------------
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/flask_service.yaml
service/hello created
--------------------- --------------------------------------
# The hello service provides a load-balancing proxy over the hello-app
# pods. By specifying the type as a 'LoadBalancer', Kubernetes Engine will
# create an external HTTP load balancer.
apiVersion: v1
kind: Service
metadata:
name: hello
spec:
type: LoadBalancer
selector:
app: hello
ports:
- port: 80
targetPort: 4000
------------------------- ---------------------------------------------
Verify that this simple Python Flask app is serving metrics with the following command:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ url=$(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}')
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ echo $url
34.148.86.126
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ curl $url/metrics
# HELP flask_exporter_info Multiprocess metric
# TYPE flask_exporter_info gauge
flask_exporter_info{version="0.18.5"} 1.0
Tell Prometheus where to begin scraping the metrics from by applying the PodMonitoring file:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/prom_deploy.yaml
podmonitoring.monitoring.googleapis.com/prom-example created
Before finishing up here, generate some load on the application with a really simple interaction with the app:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ timeout 120 bash -c -- 'while true; do curl $(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}'); sleep $((RANDOM % 4)) ; done'
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 17:35:19 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 17:35:20 GMT"}
<TRUNCATED...>
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 17:37:14 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 17:37:17 GMT"}
Task 4. Observing the app via metrics
In this last section, quickly use gcloud to deploy a custom monitoring dashboard that shows the metrics from this application in a line chart.
Be sure to copy the entirety of this code block:
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ gcloud monitoring dashboards create --config='''
{
"category": "CUSTOM",
"displayName": "Prometheus Dashboard Example",
"mosaicLayout": {
"columns": 12,
"tiles": [
{
"height": 4,
"widget": {
"title": "prometheus/flask_http_request_total/counter [MEAN]",
"xyChart": {
"chartOptions": {
"mode": "COLOR"
},
"dataSets": [
{
"minAlignmentPeriod": "60s",
"plotType": "LINE",
"targetAxis": "Y1",
"timeSeriesQuery": {
"apiSource": "DEFAULT_CLOUD",
"timeSeriesFilter": {
"aggregation": {
"alignmentPeriod": "60s",
"crossSeriesReducer": "REDUCE_NONE",
"perSeriesAligner": "ALIGN_RATE"
},
"filter": "metric.type=\"prometheus.googleapis.com/flask_http_request_total/counter\" resource.type=\"prometheus_target\"",
"secondaryAggregation": {
"alignmentPeriod": "60s",
''' ] } "yPos": 0,,e": "LINEAR", "0s",r": "ALIGN_MEAN"AN",
Created [a2cc4385-f866-4095-b946-361a9f7fb883].
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$
\student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl get deployment -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
gmp-system gmp-operator 1/1 1 1 46m
gmp-system rule-evaluator 1/1 1 1 46m
gmp-test helloworld-gke 1/1 1 1 40m
kube-system event-exporter-gke 1/1 1 1 46m
kube-system konnectivity-agent 1/1 1 1 46m
kube-system konnectivity-agent-autoscaler 1/1 1 1 46m
kube-system kube-dns 1/1 1 1 46m
kube-system kube-dns-autoscaler 1/1 1 1 46m
kube-system l7-default-backend 1/1 1 1 46m
kube-system metrics-server-v0.5.2 1/1 1 1 46m
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-gmp-cluster-default-pool-59dbcd3c-ldx9 Ready <none> 45m v1.27.3-gke.100
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
gmp-system alertmanager-0 2/2 Running 0 46m
gmp-system collector-95mp5 2/2 Running 0 45m
gmp-system gmp-operator-5f89fc5d7c-9zhkx 1/1 Running 0 46m
gmp-system rule-evaluator-f578d69c7-jpxjq 2/2 Running 1 (45m ago) 45m
gmp-test helloworld-gke-5f574446d7-rqhc8 1/1 Running 0 41m
kube-system event-exporter-gke-7bf6c99dcb-5rhwn 2/2 Running 0 46m
kube-system fluentbit-gke-zjp76 2/2 Running 0 45m
kube-system gke-metrics-agent-8hfcd 2/2 Running 0 45m
kube-system konnectivity-agent-5fc7ff9689-zrmj9 1/1 Running 0 46m
kube-system konnectivity-agent-autoscaler-5d9dbcc6d8-tt7h2 1/1 Running 0 46m
kube-system kube-dns-5bfd847c64-wb726 4/4 Running 0 46m
kube-system kube-dns-autoscaler-84b8db4dc7-c42rz 1/1 Running 0 46m
kube-system kube-proxy-gke-gmp-cluster-default-pool-59dbcd3c-ldx9 1/1 Running 0 44m
kube-system l7-default-backend-d86c96845-9h7p9 1/1 Running 0 46m
kube-system metrics-server-v0.5.2-6bf74b5d5f-tppp8 2/2 Running 0 45m
kube-system pdcsi-node-nb5dd 2/2 Running 0 45m
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$ kubectl get services -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 47m
gmp-system alertmanager ClusterIP None <none> 9093/TCP 46m
gmp-system gmp-operator ClusterIP 10.0.13.221 <none> 8443/TCP,443/TCP 46m
gmp-test hello LoadBalancer 10.0.8.51 34.148.86.126 80:30976/TCP 40m
kube-system default-http-backend NodePort 10.0.5.244 <none> 80:30154/TCP 46m
kube-system kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 47m
kube-system metrics-server ClusterIP 10.0.3.242 <none> 443/TCP 46m
student_04_61f5f7cf52ff@cloudshell:~ (qwiklabs-gcp-00-a051edc2be39)$
===================== ===================== =====================
ketan_patel@cloudshell:~ (new-user-learning)$ gcloud beta container clusters create gmp-cluster --num-nodes=1 --enable-managed-prometheus
Default change: VPC-native is the default mode during cluster creation for versions greater than 1.21.0-gke.1500. To create advanced routes based clusters, please pass the `--no-enable-ip-alias` flag
Default change: During creation of nodepools or autoscaling configuration changes for cluster versions greater than 1.24.1-gke.800 a default location policy is applied. For Spot and PVM it defaults to ANY, and for all other VM kinds a BALANCED policy is used. To change the default values use the `--location-policy` flag.
Note: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
Creating cluster gmp-cluster in us-west1-b... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1beta1/projects/new-user-learning/zones/us-west1-b/clusters/gmp-cluster].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1-b/gmp-cluster?project=new-user-learning
kubeconfig entry generated for gmp-cluster.
NAME: gmp-cluster
LOCATION: us-west1-b
MASTER_VERSION: 1.27.3-gke.100
MASTER_IP: X.X.X.X
MACHINE_TYPE: e2-medium
NODE_VERSION: 1.27.3-gke.100
NUM_NODES: 1
STATUS: RUNNING
ketan_patel@cloudshell:~ (new-user-learning)$ gcloud container clusters get-credentials gmp-cluster
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gmp-cluster.
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl config current-context
gke_new-user-learning_us-west1-b_gmp-cluster
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get deployment
No resources found in default namespace.
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get deployment -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
gmp-system gmp-operator 1/1 1 1 29m
gmp-system rule-evaluator 1/1 1 1 29m
kube-system event-exporter-gke 1/1 1 1 30m
kube-system konnectivity-agent 1/1 1 1 29m
kube-system konnectivity-agent-autoscaler 1/1 1 1 29m
kube-system kube-dns 1/1 1 1 30m
kube-system kube-dns-autoscaler 1/1 1 1 30m
kube-system l7-default-backend 1/1 1 1 29m
kube-system metrics-server-v0.5.2 1/1 1 1 29m
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-gmp-cluster-default-pool-4defca36-jkbh Ready <none> 29m v1.27.3-gke.100
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get services -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 30m
gmp-system alertmanager ClusterIP None <none> 9093/TCP 29m
gmp-system gmp-operator ClusterIP 10.0.9.243 <none> 8443/TCP,443/TCP 29m
kube-system default-http-backend NodePort 10.0.6.173 <none> 80:31366/TCP 30m
kube-system kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 30m
kube-system metrics-server ClusterIP 10.0.10.0 <none> 443/TCP 29m
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
gmp-system alertmanager-0 2/2 Running 0 29m
gmp-system collector-gr6zl 2/2 Running 0 28m
gmp-system gmp-operator-6888d59866-8982x 1/1 Running 0 30m
gmp-system rule-evaluator-77976dd4d9-qsqmx 2/2 Running 2 (28m ago) 28m
kube-system event-exporter-gke-7bf6c99dcb-cksqf 2/2 Running 0 30m
kube-system fluentbit-gke-bmcr6 2/2 Running 0 29m
kube-system gke-metrics-agent-2cwx7 2/2 Running 0 29m
kube-system konnectivity-agent-758789cc74-xwk4w 1/1 Running 0 30m
kube-system konnectivity-agent-autoscaler-5d9dbcc6d8-9qltt 1/1 Running 0 30m
kube-system kube-dns-5bfd847c64-jjr4j 4/4 Running 0 30m
kube-system kube-dns-autoscaler-84b8db4dc7-wp45d 1/1 Running 0 30m
kube-system kube-proxy-gke-gmp-cluster-default-pool-4defca36-jkbh 1/1 Running 0 28m
kube-system l7-default-backend-d86c96845-rgqwf 1/1 Running 0 30m
kube-system metrics-server-v0.5.2-6bf74b5d5f-wpp9p 2/2 Running 0 28m
kube-system pdcsi-node-h95cv 2/2 Running 0 29m
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl create ns gmp-test
namespace/gmp-test created
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get ns
NAME STATUS AGE
default Active 31m
gmp-public Active 30m
gmp-system Active 30m
gmp-test Active 24s
kube-node-lease Active 31m
kube-public Active 31m
kube-system Active 31m
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/flask_deployment.yaml
deployment.apps/helloworld-gke created
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/flask_service.yaml
service/hello created
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get pods -n gmp-test
NAME READY STATUS RESTARTS AGE
helloworld-gke-5f574446d7-97gx7 1/1 Running 0 21s
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get services -n gmp-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello LoadBalancer 10.0.7.206 <pending> 80:30322/TCP 18s
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get services -n gmp-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello LoadBalancer 10.0.7.206 <pending> 80:30322/TCP 31s
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ url=$(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}')
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ echo $url
ketan_patel@cloudshell:~ (new-user-learning)$ echo $url
ketan_patel@cloudshell:~ (new-user-learning)$ url=$(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}')
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ echo $url
35.199.168.92
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ curl $url/metrics
# HELP flask_exporter_info Multiprocess metric
# TYPE flask_exporter_info gauge
flask_exporter_info{version="0.18.5"} 1.0
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/prom_deploy.yaml
podmonitoring.monitoring.googleapis.com/prom-example created
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ timeout 120 bash -c -- 'while true; do curl $(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}'); sleep $((RANDOM % 4)) ; done'
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:20 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:23 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:24 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:26 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:27 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:29 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:31 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:34 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:35 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:38 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:41 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:42 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:43 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:46 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:48 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:50 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:50 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:54 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:54 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:56 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:57 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:59 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:28:59 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:02 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:05 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:05 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:08 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:09 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:12 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:15 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:18 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:21 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:22 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:25 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:28 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:29 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:31 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:31 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:33 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:33 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:36 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:36 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:38 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:40 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:41 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:43 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:46 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:47 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:49 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:52 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:54 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:55 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:57 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:57 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:29:58 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:01 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:01 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:02 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:03 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:04 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:06 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:07 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:10 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:10 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:12 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:15 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:15 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:16 GMT"}
{"message":"Hello World!","severity":"info","timestamp":"Thu, 26 Oct 2023 19:30:19 GMT"}
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ gcloud monitoring dashboards create --config='''
{
"category": "CUSTOM",
"displayName": "Prometheus Dashboard Example",
"mosaicLayout": {
"columns": 12,
"tiles": [
{
"height": 4,
"widget": {
"title": "prometheus/flask_http_request_total/counter [MEAN]",
"xyChart": {
"chartOptions": {
"mode": "COLOR"
},
"dataSets": [
{
"minAlignmentPeriod": "60s",
"plotType": "LINE",
"targetAxis": "Y1",
"timeSeriesQuery": {
"apiSource": "DEFAULT_CLOUD",
"timeSeriesFilter": {
"aggregation": {
"alignmentPeriod": "60s",
"crossSeriesReducer": "REDUCE_NONE",
"perSeriesAligner": "ALIGN_RATE"
},
"filter": "metric.type=\"prometheus.googleapis.com/flask_http_request_total/counter\" resource.type=\"pro''' ] } "yPos": 0,,e": "LINEAR", "0s",r": "ALIGN_MEAN"AN",
Created [7ac033b1-4bd1-4116-b236-397d55fb783c].
ketan_patel@cloudshell:~ (new-user-learning)$
523 gcloud beta container clusters create gmp-cluster --num-nodes=1 --enable-managed-prometheus
524 gcloud container clusters get-credentials gmp-cluster
525 kubectl config current-context
526 kubectl get deployment
527 kubectl get deployment -A
528 kubectl get nodes
529 kubectl get services -A
530 kubectl get pods -A
531 kubectl create ns gmp-test
532 kubectl get ns
533 kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/flask_deployment.yaml
534 kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/flask_service.yaml
535 kubectl get pods -n gmp-test
536 kubectl get services -n gmp-test
537 url=$(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}')
538 echo $url
539 url=$(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}')
540 echo $url
541 curl $url/metrics
542 kubectl -n gmp-test apply -f https://raw.githubusercontent.com/kyleabenson/flask_telemetry/master/gmp_prom_setup/prom_deploy.yaml
543 timeout 120 bash -c -- 'while true; do curl $(kubectl get services -n gmp-test -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}'); sleep $((RANDOM % 4)) ; done'
544 gcloud monitoring dashboards create --config='''
{
"category": "CUSTOM",
"displayName": "Prometheus Dashboard Example",
"mosaicLayout": {
"columns": 12,
"tiles": [
{
"height": 4,
"widget": {
"title": "prometheus/flask_http_request_total/counter [MEAN]",
"xyChart": {
"chartOptions": {
"mode": "COLOR"
},
"dataSets": [
{
"minAlignmentPeriod": "60s",
"plotType": "LINE",
"targetAxis": "Y1",
"timeSeriesQuery": {
"apiSource": "DEFAULT_CLOUD",
"timeSeriesFilter": {
"aggregation": {
"alignmentPeriod": "60s",
"crossSeriesReducer": "REDUCE_NONE",
"perSeriesAligner": "ALIGN_RATE"
},
"filter": "metric.type=\"prometheus.googleapis.com/flask_http_request_total/counter\" resource.type=\"prometheus_target\"",
"secondaryAggregation": {
"alignmentPeriod": "60s",
"crossSeriesReducer": "REDUCE_MEAN",
"groupByFields": [
"metric.label.\"status\""
],
"perSeriesAligner": "ALIGN_MEAN"
}
}
}
}
],
"thresholds": [],
"timeshiftDuration": "0s",
"yAxis": {
"label": "y1Axis",
"scale": "LINEAR"
}
}
},
"width": 6,
"xPos": 0,
"yPos": 0
}
]
}
}
'''
545 history
ketan_patel@cloudshell:~ (new-user-learning)
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl get pods -n gmp-test
NAME READY STATUS RESTARTS AGE
helloworld-gke-5f574446d7-97gx7 1/1 Running 0 68m
ketan_patel@cloudshell:~ (new-user-learning)$ kubectl describe pod -n gmp-test
Name: helloworld-gke-5f574446d7-97gx7
Namespace: gmp-test
Priority: 0
Service Account: default
Node: gke-gmp-cluster-default-pool-4defca36-jkbh/10.138.0.32
Start Time: Thu, 26 Oct 2023 19:26:29 +0000
Labels: app=hello
pod-template-hash=5f574446d7
Annotations: <none>
Status: Running
IP: 10.124.0.16
IPs:
IP: 10.124.0.16
Controlled By: ReplicaSet/helloworld-gke-5f574446d7
Containers:
hello-app:
Container ID: containerd://292d9ebcbda1b1ad6438ba5e28b56a4dc22c63ed9442c3b34bd88314866c5e02
Image: gcr.io/ops-demo-330920/flask_telemetry:61a2a7aabc7077ef474eb24f4b69faeab47deed9
Image ID: gcr.io/ops-demo-330920/flask_telemetry@sha256:ff5bc984a8aecc0a5d3d32b2ad5ea81ff16574dc71351f23fb79c45489a29f8c
Port: 4000/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 26 Oct 2023 19:26:47 +0000
Ready: True
Restart Count: 0
Environment:
PORT: 4000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c7snr (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-c7snr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
ketan_patel@cloudshell:~ (new-user-learning)$
No comments:
Post a Comment