In today’s fast-paced world, there is emphasis on getting instant insights. Typical use-cases involve SaaS operators providing real-time metrics for their KPIs or marketeers' need for quick insights on performance of their offers or experiments on the website.
This solution will demonstrate how to build a real-time website analytics dashboard on GCP.
Components
User events / Message bus provides system decoupling, Pub/Sub is a fully managed message/event bus and provides an easy way to handle the fast click-stream generated by typical websites. The click-stream contains signals which can be processed to derive insights in real time.
Metrics processing pipeline is required to process the click-stream from Pub/Sub into the metrics database. Dataflow will be used, which is a serverless, fully managed processing service supporting real-time streaming jobs.
Metrics Database, needs to be an in-memory database to support real-time use-cases. Some common web analytic metrics are unique visitors, number of active experiments, conversion rate of each experiment, etc. The common theme is to calculate uniques, i.e. Cardinality counting, although from a marketeer's standpoint a good estimation is sufficient, the HyperLogLog algorithm is an efficient solution to the count-unique problem by trading off some accuracy.
Cloud Memorystore (Redis) provides a slew of in-built functions for sets and cardinality measurement, alleviating the need to perform them in code.
The analytics reporting and visualization makes the reports available to the marketeer easily. A Spring dashboard application is used for demo purposes only. The application uses Jedis client to access metrics from Redis using scard and sinterstore commands for identifying user overlap and other cardinality values. It then uses Javascript based web-ui to render graphs using Google Charts library.
1) Create PubSub:
Name - web-events
2) Download Application Code:
ketan_patel@cloudshell:~ (new-user-learning)$ git clone https://github.com/GoogleCloudPlatform/redis-dataflow-realtime-analytics
Cloning into 'redis-dataflow-realtime-analytics'...
remote: Enumerating objects: 80, done.
remote: Counting objects: 100% (80/80), done.
remote: Compressing objects: 100% (55/55), done.
Receiving objects: 100% (80/80), 134.45 KiB | 6.72 MiB/s, done.
remote: Total 80 (delta 26), reused 47 (delta 8), pack-reused 0
Resolving deltas: 100% (26/26), done.
ketan_patel@cloudshell:~ (new-user-learning)$ cd redis-dataflow-realtime-analytics/
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics (new-user-learning)$ ls -l
total 260
-rw-r--r-- 1 ketan_patel ketan_patel 351 Aug 21 00:11 AUTHORS
-rw-r--r-- 1 ketan_patel ketan_patel 1101 Aug 21 00:11 CONTRIBUTING.md
drwxr-xr-x 3 ketan_patel ketan_patel 4096 Aug 21 00:11 dashboard
-rw-r--r-- 1 ketan_patel ketan_patel 22715 Aug 21 00:11 LICENSE
drwxr-xr-x 2 ketan_patel ketan_patel 4096 Aug 21 00:11 loggen
drwxr-xr-x 3 ketan_patel ketan_patel 4096 Aug 21 00:11 processor
-rw-r--r-- 1 ketan_patel ketan_patel 8492 Aug 21 00:11 README.md
-rwxr-xr-x 1 ketan_patel ketan_patel 203949 Aug 21 00:11 realtime-analytics-architecture.svg
-rw-r--r-- 1 ketan_patel ketan_patel 882 Aug 21 00:11 set_variables.sh
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics (new-user-learning)$
Now open CLOUD SHELL and verify 3 applications. (dashboard, loggen, processor)
Loggen:
- simple python application
- what it does, it takes parameters from you as a CLI in terms of
- how many events we want to do here it's an event count (let say 100)
- Name of the topic onto which you want to actually Publish the events
- Name of the project
- After that it creates a pub/sub API and call and sends that particular dummy event to pub/sub and keeps printing it
3) Create Python Virtual Enviornment:
To ensure that my python dependencies don't mix with system python and my application remains seperate.
ketan_patel@cloudshell:~ (new-user-learning)$ python3 -m venv venv
ketan_patel@cloudshell:~ (new-user-learning)$ source venv/bin/activate
(venv) ketan_patel@cloudshell:~ (new-user-learning)$
4) Install all requirments for the application:
venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$ ls -l
total 12
-rw-r--r-- 1 ketan_patel ketan_patel 4616 Aug 21 00:11 message_generator.py
-rw-r--r-- 1 ketan_patel ketan_patel 27 Aug 21 00:11 requirements.txt
(venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$ cat requirements.txt
google
google-cloud-pubsub
(venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$ pip install -r requirements.txt
Collecting google
Downloading google-3.0.0-py2.py3-none-any.whl (45 kB)
|████████████████████████████████| 45 kB 2.8 MB/s
Collecting google-cloud-pubsub
Downloading google_cloud_pubsub-2.18.3-py2.py3-none-any.whl (265 kB)
|████████████████████████████████| 265 kB 13.3 MB/s
Collecting beautifulsoup4
Downloading beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)
|████████████████████████████████| 142 kB 33.9 MB/s
Collecting grpcio<2.0dev,>=1.51.3
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
|████████████████████████████████| 158 kB 8.8 MB/s
Collecting soupsieve>1.2
Downloading soupsieve-2.4.1-py3-none-any.whl (36 kB)
Installing collected packages: pyasn1, urllib3, six, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, requests, grpcio, googleapis-common-protos, google-auth, soupsieve, grpcio-status, google-api-core, proto-plus, grpc-google-iam-v1, beautifulsoup4, google-cloud-pubsub, google
Successfully installed beautifulsoup4-4.12.2 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 google-3.0.0 google-api-core-2.11.1 google-auth-2.22.0 google-cloud-pubsub-2.18.3 googleapis-common-protos-1.60.0 grpc-google-iam-v1-0.12.6 grpcio-1.57.0 grpcio-status-1.57.0 idna-3.4 proto-plus-1.22.3 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 requests-2.31.0 rsa-4.9 six-1.16.0 soupsieve-2.4.1 urllib3-1.26.16
(venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$
5) Start application so it's start generating logs:
(venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$ python message_generator.py --topic web-events --project-id new-user-learning
ProjectId: new-user-learning
Pub/Sub Topic: projects/new-user-learning/topics/web-events
Sending events in background.
Press Ctrl+C to exit/stop.
6) Create Database - MemoryStore (InMemory database)
Will take 5-7 minutes.
Equivalent gcloud cli code
gcloud redis instances create --project=new-user-learning redis-1 --tier=standard --size=5 --region=us-west1 --redis-version=redis_5_0 --network=projects/new-user-learning/global/networks/default --read-replicas-mode=READ_REPLICAS_ENABLED --replica-count=2 --connect-mode=DIRECT_PEERING
OR
gcloud redis instances create --project=new-user-learning redis-2 --tier=basic --size=1 --region=us-west1 --redis-version=redis_5_0 --network=projects/new-user-learning/global/networks/default --connect-mode=PRIVATE_SERVICE_ACCESS
7) Spin up Spring Application to create that graphs by reading from Redis
TimeSeriesMetric.java is a Controller which is going to build all the metrics which UI is going to rread from Redis and put that into an API with a front-end is then going to read.
Front End Application:
"metrics_ui.js" is FE is a simple Java script which reads or has configuraiton of all the metrics which needs to be shown. It use Google Charts to actually show things
"index.html" which has divs to put all those graphs.
8) Create Bastion Host (VM)
9) SSH TUNNEL:
Use “gcloud config set project [PROJECT_ID]” to change to a different project.
ketan_patel@cloudshell:~ (new-user-learning)$ gcloud compute ssh instance-1 --zone us-west1 -- -N -L 6379:10.151.182.86:6379 -4
WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
This tool needs to create the directory [/home/ketan_patel/.ssh] before being able to generate SSH keys.
Do you want to continue (Y/n)? Y
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ketan_patel/.ssh/google_compute_engine
Your public key has been saved in /home/ketan_patel/.ssh/google_compute_engine.pub
The key fingerprint is:
SHA256:ike5b7EkSlLyKn1JBSSAEA/ggTfZyustrNoMJ7KREWw ketan_patel@cs-863288215820-default
The key's randomart image is:
+---[RSA 3072]----+
|@+.+. |
|=o=.o |
|.E.o . |
|..+ . .. |
|. = .o S |
| oo +o.oo |
|*+.*.o+o o |
|oO* =. .o |
|=ooo .. |
+----[SHA256]-----+
ERROR: (gcloud.compute.ssh) Could not fetch resource:
- Invalid value for field 'zone': 'us-west1'. Unknown zone.
ketan_patel@cloudshell:~ (new-user-learning)$ gcloud compute ssh instance-1 --zone us-west1-b -- -N -L 6379:10.151.182.86:6379 -4
Updating project ssh metadata...working..Updated [https://www.googleapis.com/compute/v1/projects/new-user-learning].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.1890254370391392431' (ECDSA) to the list of know hosts.
10) Configure Redis IP address. Use private IP address and use SSH Tunneling. Also applicaiton is going to run on AppEngine or VM which has access to Redis without exposing the Redis system.
application.properties
10) Deploy spring application (open another shell)
Run Maven command to compile and deploy spring applicaitn which is going to show al the Graphs
ketan_patel@cloudshell:~ (new-user-learning)$ cd redis-dataflow-realtime-analytics/dashboard/
Will download all dependencies and compile everything it runs application on port 8080
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/dashboard (new-user-learning)$ mvn spring-boot:run
[INFO] Scanning for projects...
[INFO]
[INFO] ----------< com.google.cloud.example.realtimedash:dashboard >-----------
[INFO] Building dashboard 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] >>> spring-boot-maven-plugin:2.1.9.RELEASE:run (default-cli) > test-compile @ dashboard >>>
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ dashboard ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ dashboard ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 6 source files to /home/ketan_patel/redis-dataflow-realtime-analytics/dashboard/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ dashboard ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/ketan_patel/redis-dataflow-realtime-analytics/dashboard/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ dashboard ---
[INFO] No sources to compile
[INFO]
[INFO] <<< spring-boot-maven-plugin:2.1.9.RELEASE:run (default-cli) < test-compile @ dashboard <<<
[INFO]
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.9.RELEASE:run (default-cli) @ dashboard ---
Downloading from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-loader-tools/2.1.9.RELEASE/spring-boot-loader-tools-2.1.9.RELEASE.pom
<Truncated>
Downloaded from central: https://repo.maven.apache.org/maven2/com/google/guava/guava/19.0/guava-19.0.jar (2.3 MB at 882 kB/s)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.9.RELEASE)
2023-08-21 01:39:24.205 INFO 8234 --- [ main] c.g.c.s.r.d.DashboardApplication : Starting DashboardApplication on cs-863288215820-default with PID 8234 (/home/ketan_patel/redis-dataflow-realtime-analytics/dashboard/target/classes started by ketan_patel in /home/ketan_patel/redis-dataflow-realtime-analytics/dashboard)
2023-08-21 01:39:24.212 INFO 8234 --- [ main] c.g.c.s.r.d.DashboardApplication : No active profile set, falling back to default profiles: default
2023-08-21 01:39:26.722 INFO 8234 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode!
2023-08-21 01:39:26.727 INFO 8234 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
2023-08-21 01:39:26.799 INFO 8234 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 34ms. Found 0 repository interfaces.
2023-08-21 01:39:28.224 INFO 8234 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2023-08-21 01:39:28.312 INFO 8234 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-08-21 01:39:28.315 INFO 8234 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.26]
2023-08-21 01:39:28.557 INFO 8234 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-08-21 01:39:28.558 INFO 8234 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 4164 ms
2023-08-21 01:39:29.603 INFO 8234 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2023-08-21 01:39:29.797 INFO 8234 --- [ main] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html]
2023-08-21 01:39:31.368 INFO 8234 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2023-08-21 01:39:31.376 INFO 8234 --- [ main] c.g.c.s.r.d.DashboardApplication : Started DashboardApplication in 8.759 seconds (JVM running for 30.759)
12) View running application by clicking WEB REVIEW on cloud console:(on port 8080)
You will see empty dashboard (as no ETL pipeline run from Dataflow, which connects to pub/sub events to redis database so it reads the messages from the events process them generates the metrics and put them into database for the dashboard to actually show them up)
13) In "processor" applicaiton -- "MetricsPipelineOptions.java"
Option to begin with which are
Redis host port #
Redis host IP address
Pub/Sub Topic from which the data flow needs to actually read and then it deploys this java applicaiton.
Create Simple shell script to deploy the dataflow applicaiton by first compling it and then calling the dataflow api to actually put it onto dataflow engine. To run shell script it need to look at options to provide such as
================= deploy_dataflow.sh ===================
#!/usr/bin/env bash
mvn clean compile exec:java \
-Dexec.mainClass=com.google.cloud.solutions.realtimedash.pipeline.MetricsCalculationPipeline \
-Dexec.cleanupDaemonThreads=false \
-Dmaven.test.skip=true \
-Dexec.args=" \
--project=$PROJECT_ID \
--runner=DataflowRunner \
--stagingLocation=gs://$TEMP_GCS_BUCKET/stage/ \
--tempLocation=gs://$TEMP_GCS_BUCKET/temp/ \
--inputTopic=projects/$PROJECT_ID/topics/$APP_EVENTS_TOPIC \
--workerMachineType=n1-standard-4 \
--region=$REGION_ID \
--subnetwork=regions/$REGION_ID/subnetworks/$VPC_NETWORK_NAME \
--redisHost=$REDIS_IP \
--redisPort=6379 \
--streaming\
"
=================== END ====================================
- Create bucket to store temporary files for dataflow
ketan_patel@cloudshell:~ (new-user-learning)$ gsutil mb -p new-user-learning -l us-west1 -b on gs://realtime-gs-back-0820
Creating gs://realtime-gs-back-0820/...
ketan_patel@cloudshell:~ (new-user-learning)$
Modify above shell script with parameters by changing project, staginglocation, templocation, inputTopic, region, subnetwork,redisHost
================== Modify script =======================
#!/usr/bin/env bash
mvn clean compile exec:java \
-Dexec.mainClass=com.google.cloud.solutions.realtimedash.pipeline.MetricsCalculationPipeline \
-Dexec.cleanupDaemonThreads=false \
-Dmaven.test.skip=true \
-Dexec.args=" \
--project=new-user-learning \
--runner=DataflowRunner \
--stagingLocation=gs://realtime-gs-back-0820/stage/ \
--tempLocation=gs://realtime-gs-back-0820/temp/ \
--inputTopic=projects/new-user-learning/topics/web-events \
--workerMachineType=n1-standard-4 \
--region=us-west1 \
--subnetwork=regions/us-west1/subnetworks/default \
--redisHost=10.151.182.86 \
--redisPort=6379 \
--streaming\
"
========================== END ==========================
ketan_patel@cloudshell:~ (new-user-learning)$ gsutil mb -p new-user-learning -1 us-west1 -b on gs://realtime-gs-back-0820
ketan_patel@cloudshell:~ (new-user-learning)$ cd redis-dataflow-realtime-analytics/processor/
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/processor (new-user-learning)$ sh deploy_dataflow.sh
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.21.0/maven-surefire-plugin-2.21.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.21.0/maven-surefire-plugin-2.21.0.pom (5.2 kB at 9.9 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.21.0/surefire-2.21.0.pom
/repository/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar to gs://realtime-gs-back-0820/stage/jsr305-3.0.2-3YOsy4mTY8MrB9ehsuTOQA.jar
Aug 21, 2023 2:14:26 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
IN
21, 2023 2:14:36 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.20.0
Aug 21, 2023 2:14:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-west1/2023-08-20_19_14_36-9722414146411591874?project=new-user-learning
Aug 21, 2023 2:14:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2023-08-20_19_14_36-9722414146411591874
Aug 21, 2023 2:14:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=new-user-learning cancel --region=us-west1 2023-08-20_19_14_36-9722414146411591874
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.095 s
[INFO] Finished at: 2023-08-21T02:14:37Z
[INFO] ------------------------------------------------------------------------
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/processor (new-user-learning)$
Open DataFlow and verify job is running and you can see DAG (Direct Acyclic Graph) collecting Tasks together, organized with dependencies and relationships to say how they should run.
Now go back to Dashboard Application and verify that you can see realtime website statistics
Visits/min
Unique Users/min
STOP DATAFLOW: (WILL SEE FOLLOWING OUTPUT)
ketan_patel@cloudshell:~ (new-user-learning)$ cd redis-dataflow-realtime-analytics/processor/
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/processor (new-user-learning)$ sh deploy_dataflow.sh
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.21.0/maven-surefire-plugin-2.21.0.pom
<TRUNCATED>
Aug 21, 2023 2:14:36 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.20.0
Aug 21, 2023 2:14:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-west1/2023-08-20_19_14_36-9722414146411591874?project=new-user-learning
Aug 21, 2023 2:14:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2023-08-20_19_14_36-9722414146411591874
Aug 21, 2023 2:14:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=new-user-learning cancel --region=us-west1 2023-08-20_19_14_36-9722414146411591874
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.095 s
[INFO] Finished at: 2023-08-21T02:14:37Z
[INFO] ------------------------------------------------------------------------
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/processor (new-user-learning)$
ketan_patel@cloudshell:~ (new-user-learning)$ gcloud compute ssh instance-1 --zone us-west1-b -- -N -L 6379:10.151.182.86:6379 -4
Updating project ssh metadata...working..Updated [https://www.googleapis.com/compute/v1/projects/new-user-learning].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.1890254370391392431' (ECDSA) to the list of known hosts.
channel 1015: open failed: connect failed: open failed
channel 1015: open failed: connect failed: open failed
channel 1017: open failed: connect failed: open failed
channel 1018: open failed: connect failed: open failed
channel 1019: open failed: connect failed: open failed
ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/dashboard (new-user-learning)$ mvn spring-boot:run
<TRUNCATED>
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
2023-08-21 02:25:55.768 ERROR 8234 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Connection reset] with root cause
java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) ~[na:na]
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) ~[na:na]
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:126) ~[na:na]
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196) ~[jedis-2.10.2.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.10.2.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:153) ~[jedis-2.10.2.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:218) ~[jedis-2.10.2.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:341) ~[jedis-2.10.2.jar:na]
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:266) ~[jedis-2.10.2.jar:na]
at redis.clients.jedis.Jedis.pfcount(Jedis.java:3659) ~[jedis-2.10.2.jar:na]
at com.google.cloud.solutions.realtimedash.dashboard.TimeseriesMetricsController.users(TimeseriesMetricsController.java:81) ~[classes/:na]
at jdk.internal.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:893) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.26.jar:9.0.26]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.26.jar:9
(venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$ python message_generator.py --topic web-events --project-id new-user-learning
ProjectId: new-user-learning
Pub/Sub Topic: projects/new-user-learning/topics/web-events
Sending events in background.
Press Ctrl+C to exit/stop.
^C
Sent 1580 messages.
Exiting
(venv) ketan_patel@cloudshell:~/redis-dataflow-realtime-analytics/loggen (new-user-learning)$
PUBSUB DETAILS WHILE RUNNING THIS
No comments:
Post a Comment