Build a one-to-many Pub/Sub system

In this tutorial, you start a publisher application that sends a "Hello, World!" message to two subscribers using one-to-many communication, as illustrated in the following diagram:

The two subscriber applications use the same code, but you start them at different times. This process demonstrates how Pub/Sub enables asynchronous communication. To build this system, you complete the following steps:

Create an IAM service account that the applications use for authentication.

Set up IAM permissions.

Create a Pub/Sub topic and a subscription.

Start three independent applications: one publisher and two subscribers.

ketan_patel@ketanvm:~$ sudo apt-get update

ketan_patel@ketanvm:~$ sudo apt-get install apt-transport-https ca-certificates gnupg curl sudo

ketan_patel@ketanvm:~$ echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main

ketan_patel@ketanvm:~$ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

ketan_patel@ketanvm:~$ sudo apt-get update && sudo apt-get install google-cloud-cli

ketan_patel@ketanvm:~$ gcloud init

Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:

  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.

Checking network connection...done.                                                                           

Reachability Check passed.

Network diagnostic passed (1/1 checks passed).

Choose the account you would like to use to perform operations for this configuration:

 [1] 180636258465-compute@developer.gserviceaccount.com

 [2] Log in with a new account

Please enter your numeric choice:  2

You are running on a Google Compute Engine virtual machine.

It is recommended that you use service accounts for authentication.

You can run:

  $ gcloud config set account `ACCOUNT`

to switch accounts if necessary.

Your credentials may be visible to others with access to this

virtual machine. Are you sure you want to authenticate with

your personal account?

Do you want to continue (Y/n)?  Y

Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fauthcode.html&scope=openid+https%XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&code_challenge_method=S256

Enter authorization code: 4/0AZEOvhWXXXXXXXXXXXXXXXXXXXXXXXX

You are logged in as: [ketan.patel@supernal.aero].

Pick cloud project to use: 

 [1] c-35

 [2] gcp-trial

 [3] mvp

 [4] Enter a project ID

 [5] Create a new project

Please enter numeric choice or text value (must exactly match list item):  3

Your current project has been set to: [mvp].

Do you want to configure a default Compute Region and Zone? (Y/n)?  Y

Which Google Compute Engine zone would you like to use as project default?

If you do not specify a zone via a command line flag while working with Compute Engine resources, the default 

is assumed.

 [1] us-east1-b

 [2] us-east1-c

 [10] us-central1-b

 [11] us-west1-b

 [12] us-west1-c

Please enter numeric choice or text value (must exactly match list item):  11

Your project default Compute Engine zone has been set to [us-west1-b].

You can change it by running [gcloud config set compute/zone NAME].

Your project default Compute Engine region has been set to [us-west1].

You can change it by running [gcloud config set compute/region NAME].

Created a default .boto configuration file at [/home/ketan_patel/.boto]. See this file and

[https://cloud.google.com/storage/docs/gsutil/commands/config] for more

information about configuring Google Cloud Storage.

Your Google Cloud SDK is configured and ready to use!

* Commands that require authentication will use ketan.patel@supernal.aero by default

* Commands will reference project `svo-mvp` by default

* Compute Engine commands will use region `us-west1` by default

* Compute Engine commands will use zone `us-west1-b` by default

Run `gcloud help config` to learn how to change individual settings

This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.

Run `gcloud topic configurations` to learn more.

Some things to try next:

* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.

* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting

* Run `gcloud cheat-sheet` to see a roster of go-to `gcloud` commands.

ketan_patel@ketanvm:~$ gcloud config  list

[compute]

region = us-west1

zone = us-west1-b

[core]

account = ketan.patel@supernal.aero

disable_usage_reporting = True

project = svo-mvp

Your active configuration is: [default]

ketan_patel@ketanvm:~$ gcloud config get-value project

svo-mvp

Enable the Pub/Sub API:

ketan_patel@ketanvm:~$ gcloud services enable pubsub.googleapis.com

Create local authentication credentials for your Google Account:

ketan_patel@ketanvm:~$ gcloud auth application-default login

You are running on a Google Compute Engine virtual machine.

The service credentials associated with this virtual machine

will automatically be used by Application Default

Credentials, so it is not necessary to use this command.

If you decide to proceed anyway, your user credentials may be visible

to others with access to this virtual machine. Are you sure you want

to authenticate with your personal account?

Do you want to continue (Y/n)?  Y

Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&redirect_uri=https%XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&code_challenge_method=S256

Enter authorization code: 4/0XXXXPeP_ITnqIAFXXXXXXXXXXXXXXXXXXXXX

Credentials saved to file: [/home/ketan_patel/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "svo-mvp" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

Grant roles to your Google Account. Run the following command once for each of the following IAM roles: roles/pubsub.publisher, roles/pubsub.subscriber(IF PROVIDED EARLIER THEN SKIP IT)

gcloud projects add-iam-policy-binding PROJECT_ID --member="user:EMAIL_ADDRESS" --role=ROLE

Set up your Pub/Sub project

To manage message flow between publishing and subscribing applications, you create a topic and two different subscriptions.

Create a Pub/Sub topic

ketan_patel@ketanvm:~$ gcloud pubsub topics create hello_topic

Created topic [projects/svo-mvp/topics/hello_topic].

ketan_patel@ketanvm:~$ gcloud pubsub topics list

---

name: projects/svo-mvp/topics/ketan_order_topic

---

name: projects/svo-mvp/topics/hello_topic

Create Pub/Sub subscriptions

ketan_patel@ketanvm:~$ gcloud pubsub subscriptions create sub_one --topic=hello_topic

Created subscription [projects/svo-mvp/subscriptions/sub_one].

ketan_patel@ketanvm:~$ gcloud pubsub subscriptions create sub_two --topic=hello_topic

Created subscription [projects/svo-mvp/subscriptions/sub_two].

ketan_patel@ketanvm:~$  git clone https://github.com/googleapis/python-pubsub.git

Cloning into 'python-pubsub'...

remote: Enumerating objects: 10179, done.

remote: Counting objects: 100% (396/396), done.

remote: Compressing objects: 100% (186/186), done.

remote: Total 10179 (delta 242), reused 314 (delta 199), pack-reused 9783

Receiving objects: 100% (10179/10179), 4.67 MiB | 14.93 MiB/s, done.

Resolving deltas: 100% (6584/6584), done.

ketan_patel@ketanvm:~$ type python python2 python3

-bash: type: python: not found

-bash: type: python2: not found

python3 is hashed (/usr/bin/python3)

ketan_patel@ketanvm:~$ alias python='python3'

ketan_patel@ketanvm:~$ type python python2 python3

python is aliased to `python3'

-bash: type: python2: not found

python3 is hashed (/usr/bin/python3)

ketan_patel@ketanvm:~$ sudo apt install python3.9-venv

ketanvm:~$ sudo apt install python3.9-venv

ketan_patel@ketanvm:~$ sudo apt install python3-virtualenv

Reading package lists... Done

Building dependency tree       

Reading state information... Done

ketan_patel@ketanvm:~$ sudo apt install python3-pip

Reading package lists... Done

ketan_patel@ketanvm:~$ sudo apt install python-is-python3

ketan_patel@ketanvm:~$ ls -l /usr/bin/python

lrwxrwxrwx 1 root root 7 Apr 15  2020 /usr/bin/python -> python3

ketan_patel@ketanvm:~$ sudo apt install python3.8-venv

Reading package lists... Done

\

ketan_patel@ketanvm:~$ python -m venv pyenv-qs

ketan_patel@ketanvm:~$ source pyenv-qs/bin/activate

(pyenv-qs) ketan_patel@ketanvm:~$ 

(pyenv-qs) ketan_patel@ketanvm:~$ python -m pip install --upgrade google-cloud-pubsub

Collecting google-cloud-pubsub

  Downloading google_cloud_pubsub-2.18.0-py2.py3-none-any.whl (265 kB)

     |████████████████████████████████| 265 kB 9.3 MB/s 

Collecting google-api-

(pyenv-qs) ketan_patel@ketanvm:~$ export PROJECT=`gcloud config get-value project`

(pyenv-qs) ketan_patel@ketanvm:~$ echo $PROJECT

svo-mvp

Start the apps and observe message flow

(pyenv-qs) ketan_patel@ketanvm:~/python-pubsub/samples/snippets/quickstart$ python pub.py $PROJECT hello_topic

Published Hello, World! to projects/svo-mvp/topics/hello_topic: 8704737987800576

(pyenv-qs) ketan_patel@ketanvm:~/python-pubsub/samples/snippets/quickstart$ 

TERMINAL 2:

(pyenv-qs) ketan_patel@ketanvm:~$ export PROJECT=`gcloud config get-value project`

(pyenv-qs) ketan_patel@ketanvm:~$ cd python-pubsub/samples/snippets/quickstart/

(pyenv-qs) ketan_patel@ketanvm:~/python-pubsub/samples/snippets/quickstart$ python sub.py $PROJECT sub_one

Listening for messages on projects/svo-mvp/subscriptions/sub_one..

Received Message {

  data: b'Hello, World!'

  ordering_key: ''

  attributes: {}

}.

Acknowledged 8704737987800576.

TERMINAL 3:

ketan_patel@ketanvm:~$ source pyenv-qs/bin/activate

(pyenv-qs) ketan_patel@ketanvm:~$ export PROJECT=`gcloud config get-value project`

(pyenv-qs) ketan_patel@ketanvm:~$ cd python-pubsub/samples/snippets/quickstart/

(pyenv-qs) ketan_patel@ketanvm:~/python-pubsub/samples/snippets/quickstart$ python sub.py $PROJECT sub_two

Listening for messages on projects/svo-mvp/subscriptions/sub_two..

Received Message {

  data: b'Hello, World!'

  ordering_key: ''

  attributes: {}

}.

Acknowledged 8704737987800576.

(pyenv-qs) ketan_patel@ketanvm:~/python-pubsub/samples/snippets/quickstart$ history

    1  df -h .

    2  id

    3  uname -a

    4  uname -n

    5  uname -k

    6  uname -i

    7  unmae -h

    8  uname -l

    9  uname --help

   10  uname -o

   11  uname -s

   12  uname -v

   13  uname -m

   14  uname -p

   15  uname -s

   16  uname -i

   17  terraform

   18  snap install terraform

   19  sudo snap install terraform

   20  uname -a

   21  apt policy terraform

   22  sudo apt-get update

   23  sudo apt-get install apt-transport-https ca-certificates gnupg curl sudo

   24  echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

   25  ls -l /etc/apt/sources.list/

   26  curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

   27  sudo apt-get update && sudo apt-get install google-cloud-cli

   28  history

   29  gcloud init

   30  gcloud cheat-sheet

   31  gcloud

   32  gcloud config list

   33  gcloud  config get-value project

   34  gcloud config 

   35  gcloud config  list

   36  gcloud config get-value project

   37  gcloud services enable pubsub.googleapis.com

   38  gcloud auth application-default login

   39  gcloud projects list-iam-policy-binding

   40  gcloud projects

   41  gcloud projects get-iam-policy

   42  gcloud projects get-iam-policy svo-mvp --member="ketan.patel@supernal.aero" 

   43  python -v

   44  python 

   45  python3

   46  python

   47  gcloud pubsub topics create hello_topic

   48  gcloud pubsub topics list

   49  gcloud pubsub subscriptions create sub_one --topic=hello_topic

   50  gcloud pubsub subscriptions create sub_two --topic=hello_topic

   51  pwd

   52  python -m venv pyenv-qs

   53  type python python2 python3

   54  alias python='python3'

   55  type python python2 python3

   56  python -m venv pyenv-qs

   57  python3 -m venv pyenv-qs

   58  sudo apt install python3.9-venv

   59  python -m venv pyenv-qs

   60  sudo python -m venv pyenv-qs

   61  sudo

   62  sudo apt install python3.9-venv

   63  sudo apt install python3-virtualenv

   64  pip3 install venv

   65  sudo apt install python3-pip

   66  pip3 install venv

   67  sudo pip3 install venv

   68  sudo apt install python-is-python3

   69  ls -l /usr/bin/pyton

   70  ls -l /usr/bin/python

   71  python -m venv pyenv-qs

   72  sudo apt install python3.8-venv

   73  python -m venv pyenv-qs

   74  source pyenv-qs/bin/activate

   75  bin/activate

   76  (pyenv-qs) ketan_patel@ketanvm:~$ 

   77  python -m pip install --upgrade google-cloud-pubsub

   78  export PROJECT=`gcloud config get-value project`

   79  echo $PROJECT

   80  cd python-pubsub/samples/snippets/quickstart/

   81  python pub.py $PROJECT hello_topic

   82  history

(pyenv-qs) ketan_patel@ketanvm:~/python-pubsub/samples/snippets/quickstart$