Showing posts with label Security. Show all posts
Showing posts with label Security. Show all posts

Google Cloud Packet Mirroring with OpenSource IDS

Traffic Mirroring is a key feature in Google Cloud networking for security and network analysis. Its functionality is similar to that of a network tap or a span session in traditional networking. In short, Packet Mirroring captures network traffic (ingress and egress) from select "mirrored sources", copies the traffic, and forwards the copy to "collectors".

It is important to note that Packet Mirroring captures the full payload of each packet and thus consumes additional bandwidth. Because Packet Mirroring is not based on any sampling period, it is able to be used for better troubleshooting, security solutions, and higher layer application based analysis.

Packet Mirroring is founded on a "Packet Mirroring Policy", which contains the following attributes:

  • Region
  • VPC Network(s)
  • Mirrored Source(s)
  • Collector (destination)
  • Mirrored traffic (filter)

Here are a some key points that also need to be considered:

  • Only TCP, UDP and ICMP traffic may be mirrored. This, however, should satisfy the majority of use cases.
  • "Mirrored Sources" and "Collectors" must be in the SAME Region, but can be in different zones and even different VPCs, as long as those VPCs are properly Peered.
  • Additional bandwidth charges apply, especially between zones. To limit the traffic being mirrored, filters can be used.
One prime use case for "Packet Mirroring" is to use it in an Intrusion Detection System (IDS) solution. Some cloud-based IDS solutions require a special service to run on each source VM, or to put an IDS virtual appliance in-line between the network source and destination. Both of these have significant implications. For example, the service based solution, though fully distributed, requires that the guest operating system supports the software. The "in-line" solution can create a network bottleneck as all traffic must be funneled through the IDS appliance. The in-line solution will also not be able to capture "east-west" traffic within VMs in the same VPC.

Google Cloud Packet Mirroring does not require any additional software on the VMs and it is fully distributed across each of the mirrored virtual machines. The "Collector" IDS is placed out-of-path using an Internal Network Load Balancer (ILB) and will receive both "north-south" traffic and "east-west" traffic.

Packet Mirroring lab description

To demonstrate how Packet Mirroring can be used with an IDS consider this example using OpenSource IDS Suricata.
  • A single VPC with 2 subnets, one for mirrored sources and one for the collector
  • 2 Web servers created with a public IP address
  • 1 Collector server (IDS) created with NO public IP for security reasons
  • CloudNAT enabled for Internet access as needed
  • All VMs created in the same region and zone, for simplicity and cost reasons

In this lab you will create a Google Cloud environment, configure the "Collector" ILB, configure the Packet Mirror Policy, as well as install and configure [Suricata] (https://suricata-ids.org/) on a virtual instance to act as an IDS. Once complete, network tests will be performed to validate the configuration and use of Packet Mirroring with the Open Source IDS. A very stripped down rule-set and Suricata configuration is used to simplify the demonstration.








Google Cloud environment diagram

Objectives:

Build out a Google Cloud Networking environment as shown in the diagram above

Create 2 virtual machines with gcloud commands to act as WEB SERVERS

Create a single virtual machine with gcloud commands to act as IDS

Create an Internal LoadBalancer (ILB) to act as a "collector" for Packet Mirroring

Install and configure an Open Source IDS (Suricata) on the IDS VM

Review some basic IDS alert rules

Create a Packet Mirror Policy

Test Packet Mirroring by generating network traffic to the "mirrored" subnet

Test Suricata IDS by generating network traffic to simulate an IDS event and review IDS logging





BUILD LAB:


student_04_6d7b0b6748d9@cloudshell:~ (qwiklabs-gcp-03-983f2e6b9894)$ history
    1  gcloud compute networks create dm-stamford --subnet-mode=custom
    2  gcloud compute networks subnets create dm-stamford-uswest4 --range=172.21.0.0/24 --network=dm-stamford --region=us-west4
    3  gcloud compute networks subnets create dm-stamford-uswest4-ids --range=172.21.1.0/24 --network=dm-stamford --region=us-west4
    4  gcloud compute firewall-rules create fw-dm-stamford-allow-any-web --direction=INGRESS --priority=1000 --network=dm-stamford --action=ALLOW --rules=tcp:80,icmp --source-ranges=0.0.0.0/0
    5  gcloud compute firewall-rules create fw-dm-stamford-ids-any-any --direction=INGRESS --priority=1000 --network=dm-stamford --action=ALLOW --rules=all --source-ranges=0.0.0.0/0 --target-tags=ids
    6  gcloud compute firewall-rules create fw-dm-stamford-iapproxy --direction=INGRESS --priority=1000 --network=dm-stamford --action=ALLOW --rules=tcp:22,icmp --source-ranges=35.235.240.0/20
    7  gcloud compute routers create router-stamford-nat-west4 --region=us-west4 --network=dm-stamford
    8  gcloud compute routers nats create nat-gw-dm-stamford-west4 --router=router-stamford-nat-west4 --router-region=us-west4 --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges
    9  gcloud compute instance-templates create template-dm-stamford-web-us-west4 --region=us-west4 --network=dm-stamford --subnet=dm-stamford-uswest4 --machine-type=g1-small --image=ubuntu-1604-xenial-v20200807 --image-project=ubuntu-os-cloud --tags=webserver --metadata=startup-script='#! /bin/bash
  apt-get update
  apt-get install apache2 -y
  vm_hostname="$(curl -H "Metadata-Flavor:Google" \
  http://169.254.169.254/computeMetadata/v1/instance/name)"
  echo "Page served from: $vm_hostname" | \
  tee /var/www/html/index.html
  systemctl restart apache2'
   10  gcloud compute instance-groups managed create mig-dm-stamford-web-uswest4     --template=template-dm-stamford-web-us-west4     --size=2     --zone=us-west4-a
   11  gcloud compute instance-templates create template-dm-stamford-ids-us-west4 --region=us-west4 --network=dm-stamford --no-address --subnet=dm-stamford-uswest4-ids --image=ubuntu-1604-xenial-v20200807 --image-project=ubuntu-os-cloud --tags=ids,webserver --metadata=startup-script='#! /bin/bash
  apt-get update
  apt-get install apache2 -y
  vm_hostname="$(curl -H "Metadata-Flavor:Google" \
  http://169.254.169.254/computeMetadata/v1/instance/name)"
  echo "Page served from: $vm_hostname" | \
  tee /var/www/html/index.html
  systemctl restart apache2'
   12  gcloud compute instance-groups managed create mig-dm-stamford-ids-uswest4     --template=template-dm-stamford-ids-us-west4     --size=1     --zone=us-west4-a
   13  gcloud compute health-checks create tcp hc-tcp-80 --port 80
   14  gcloud compute backend-services create be-dm-stamford-suricata-us-west4 --load-balancing-scheme=INTERNAL --health-checks=hc-tcp-80 --network=dm-stamford --protocol=TCP --region=us-west4
   15  gcloud compute backend-services add-backend be-dm-stamford-suricata-us-west4 --instance-group=mig-dm-stamford-ids-uswest4 --instance-group-zone=us-west4-a --region=us-west4
   16  gcloud compute packet-mirrorings create mirror-dm-stamford-web --collector-ilb=ilb-dm-stamford-suricata-ilb-us-west4 --network=dm-stamford --mirrored-subnets=dm-stamford-uswest4 --region=us-west4
   17  gcloud compute instances list
   18  sudo tcpdump -i ens4 -nn -n "(icmp or port 80) and net 172.21.0.0/24"
   19  sudo apt install iputils-ping
   20  ping -c 4 34.125.170.248
   21  ping -c 4 34.125.91.185
   22  history
student_04_6d7b0b6748d9@cloudshell:~ (qwiklabs-gcp-03-983f2e6b9894)$ 









INSTALL IDS software and test lab.


student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ history
    1  sudo apt-get update -y
    2  sudo apt-get install libpcre3-dbg libpcre3-dev autoconf automake libtool libpcap-dev libnet1-dev libyaml-dev zlib1g-dev libcap-ng-dev libmagic-dev libjansson-dev libjansson4 -y
    3  sudo apt-get install libnspr4-dev -y
    4  sudo apt-get install libnss3-dev -y
    5  sudo apt-get install liblz4-dev -y
    6  sudo apt install rustc cargo -y
    8  sudo add-apt-repository ppa:oisf/suricata-stable -y
    9  sudo apt-get update -y
   10  sudo apt-get install suricata -y
   11  suricata -V
   12  sudo systemctl stop suricata
   13  sudo cp /etc/suricata/suricata.yaml /etc/suricata/suricata.backup
   14  wget https://storage.googleapis.com/tech-academy-enablement/GCP-Packet-Mirroring-with-OpenSource-IDS/suricata.yaml
   15  wget https://storage.googleapis.com/tech-academy-enablement/GCP-Packet-Mirroring-with-OpenSource-IDS/my.rules
   16  sudo mkdir /etc/suricata/poc-rules
   17  sudo cp my.rules /etc/suricata/poc-rules/my.rules
   18  /etc/suricata/poc-rules/my.rules
   19  sudo cp suricata.yaml /etc/suricata/suricata.yaml
   20  sudo systemctl start suricata
   21  sudo systemctl restart suricata
   22  cat /etc/suricata/poc-rules/my.rules
   23  sudo tcpdump -i ens4 -nn -n "(icmp or port 80) and net 172.21.0.0/24"

  
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo add-apt-repository ppa:oisf/suricata-stable -ygpg: keyring `/tmp/tmpsvkw80sy/secring.gpg' created
gpg: keyring `/tmp/tmpsvkw80sy/pubring.gpg' created
gpg: requesting key 66EB736F from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpsvkw80sy/trustdb.gpg: trustdb created
gpg: key 66EB736F: public key "Launchpad PPA for Peter Manev" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo apt-get update -y
Hit:1 http://us-west4.gce.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://us-west4.gce.archive.ubuntu.com/ubuntu xenial-updates InRelease                                  
Hit:3 http://us-west4.gce.archive.ubuntu.com/ubuntu xenial-backports InRelease                                
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [99.8 kB]                                   
Hit:5 http://archive.canonical.com/ubuntu xenial InRelease                                                 
Get:6 http://ppa.launchpad.net/oisf/suricata-stable/ubuntu xenial InRelease [17.5 kB]                      
Get:7 http://ppa.launchpad.net/oisf/suricata-stable/ubuntu xenial/main amd64 Packages [1,396 B]  
Get:8 http://ppa.launchpad.net/oisf/suricata-stable/ubuntu xenial/main Translation-en [1,204 B]
Fetched 120 kB in 1s (109 kB/s)                       
Reading package lists... Done

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo apt-get install suricata -y
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ suricata -V
This is Suricata version 6.0.3 RELEASE
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo systemctl stop suricata

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo cp /etc/suricata/suricata.yaml /etc/suricata/suricata.backup

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ wget https://storage.googleapis.com/tech-academy-enablement/GCP-Packet-Mirroring-with-OpenSource-IDS/suricata.yaml

--2023-08-05 22:08:51--  https://storage.googleapis.com/tech-academy-enablement/GCP-Packet-Mirroring-with-OpenSource-IDS/suricata.yaml
Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.128, 74.125.137.128, 142.250.101.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70565 (69K) [application/x-yaml]
Saving to: ‘suricata.yaml’

suricata.yaml               100%[==========================================>]  68.91K  --.-KB/s    in 0.001s  

2023-08-05 22:08:51 (96.8 MB/s) - ‘suricata.yaml’ saved [70565/70565]

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ wget https://storage.googleapis.com/tech-academy-enablement/GCP-Packet-Mirroring-with-OpenSource-IDS/my.rules

--2023-08-05 22:09:05--  https://storage.googleapis.com/tech-academy-enablement/GCP-Packet-Mirroring-with-OpenSource-IDS/my.rules
Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.128, 74.125.137.128, 142.250.101.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 422 [application/octet-stream]
Saving to: ‘my.rules’

my.rules                    100%[==========================================>]     422  --.-KB/s    in 0s      

2023-08-05 22:09:05 (49.1 MB/s) - ‘my.rules’ saved [422/422]


student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo mkdir /etc/suricata/poc-rules

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo cp my.rules /etc/suricata/poc-rules/my.rules

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo cp suricata.yaml /etc/suricata/suricata.yaml

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo systemctl start suricata

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo systemctl restart suricata

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ cat /etc/suricata/poc-rules/my.rules

####RULES#####
#UDP ALERTS
alert udp $HOME_NET any -> 8.8.8.8 53 (msg:"BAD UDP DNS REQUEST"; sid:99996; rev:1;)

#HTTP ALERTS
alert http any any -> $HOME_NET 80 (msg:"BAD HTTP PHP REQUEST"; http.uri; content:"index.php"; sid:99997; rev:1;)

#ICMP ALERTS
alert icmp any any -> $HOME_NET any (msg:"BAD ICMP"; sid:99998; rev:1;)

#TCP ALERTS
alert tcp $HOME_NET any -> any 6667 (msg:"BAD TCP 6667 REQUEST"; sid:99999; rev:1;)student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
 

student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ sudo tcpdump -i ens4 -nn -n "(icmp or port 80) and net 172.21.0.0/24"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), capture size 262144 bytes
22:13:25.844115 IP 35.230.28.46 > 172.21.0.3: ICMP echo request, id 65442, seq 1, length 64
22:13:25.845292 IP 172.21.0.3 > 35.230.28.46: ICMP echo reply, id 65442, seq 1, length 64
22:13:26.845271 IP 35.230.28.46 > 172.21.0.3: ICMP echo request, id 65442, seq 2, length 64
22:13:26.845374 IP 172.21.0.3 > 35.230.28.46: ICMP echo reply, id 65442, seq 2, length 64
22:13:27.846825 IP 35.230.28.46 > 172.21.0.3: ICMP echo request, id 65442, seq 3, length 64
22:13:27.846917 IP 172.21.0.3 > 35.230.28.46: ICMP echo reply, id 65442, seq 3, length 64
22:13:28.848784 IP 35.230.28.46 > 172.21.0.3: ICMP echo request, id 65442, seq 4, length 64
22:13:28.848899 IP 172.21.0.3 > 35.230.28.46: ICMP echo reply, id 65442, seq 4, length 64
22:13:43.538816 IP 35.230.28.46 > 172.21.0.2: ICMP echo request, id 296, seq 1, length 64
22:13:43.540182 IP 172.21.0.2 > 35.230.28.46: ICMP echo reply, id 296, seq 1, length 64
22:13:44.539496 IP 35.230.28.46 > 172.21.0.2: ICMP echo request, id 296, seq 2, length 64
22:13:44.539648 IP 172.21.0.2 > 35.230.28.46: ICMP echo reply, id 296, seq 2, length 64
22:13:45.541106 IP 35.230.28.46 > 172.21.0.2: ICMP echo request, id 296, seq 3, length 64
22:13:45.541236 IP 172.21.0.2 > 35.230.28.46: ICMP echo reply, id 296, seq 3, length 64
22:13:46.542448 IP 35.230.28.46 > 172.21.0.2: ICMP echo request, id 296, seq 4, length 64
22:13:46.542565 IP 172.21.0.2 > 35.230.28.46: ICMP echo reply, id 296, seq 4, length 64
22:14:13.612270 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [S], seq 177583878, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 17006898 ecr 0], length 0
22:14:13.613553 IP 172.21.0.3.80 > 136.226.79.33.29119: Flags [S.], seq 3771805674, ack 177583879, win 64768, options [mss 1420,sackOK,TS val 1341901328 ecr 17006898,nop,wscale 7], length 0
22:14:13.629772 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [.], ack 1, win 2068, options [nop,nop,TS val 17006901 ecr 1341901328], length 0
22:14:13.629802 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [P.], seq 1:470, ack 1, win 2068, options [nop,nop,TS val 17006901 ecr 1341901328], length 469: HTTP: GET / HTTP/1.1
22:14:13.629890 IP 172.21.0.3.80 > 136.226.79.33.29119: Flags [.], ack 470, win 503, options [nop,nop,TS val 1341901345 ecr 17006901], length 0
22:14:13.630326 IP 172.21.0.3.80 > 136.226.79.33.29119: Flags [P.], seq 1:335, ack 470, win 503, options [nop,nop,TS val 1341901346 ecr 17006901], length 334: HTTP: HTTP/1.1 200 OK
22:14:13.731802 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [.], ack 335, win 2068, options [nop,nop,TS val 17006913 ecr 1341901346], length 0
22:14:13.927417 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [P.], seq 470:882, ack 335, win 2068, options [nop,nop,TS val 17006934 ecr 1341901346], length 412: HTTP: GET /favicon.ico HTTP/1.1
22:14:13.927735 IP 172.21.0.3.80 > 136.226.79.33.29119: Flags [P.], seq 335:827, ack 882, win 501, options [nop,nop,TS val 1341901643 ecr 17006934], length 492: HTTP: HTTP/1.1 404 Not Found
22:14:14.028150 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [.], ack 827, win 2068, options [nop,nop,TS val 17006946 ecr 1341901643], length 0
22:14:14.439507 IP 64.227.150.86.55489 > 172.21.0.2.80: Flags [S], seq 2437880717, win 65535, length 0
22:14:14.441258 IP 172.21.0.2.80 > 64.227.150.86.55489: Flags [S.], seq 235362896, ack 2437880718, win 65320, options [mss 1420], length 0
22:14:14.642951 IP 64.227.150.86.55489 > 172.21.0.2.80: Flags [R], seq 2437880718, win 0, length 0
22:14:18.834628 IP 172.21.0.3.80 > 136.226.79.33.29119: Flags [F.], seq 827, ack 882, win 501, options [nop,nop,TS val 1341906549 ecr 17006946], length 0
22:14:18.850822 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [.], ack 828, win 2068, options [nop,nop,TS val 17007481 ecr 1341906549], length 0
22:14:18.850834 IP 136.226.79.33.29119 > 172.21.0.3.80: Flags [R.], seq 882, ack 828, win 2068, options [nop,nop,TS val 17007481 ecr 1341906549], length 0
22:14:30.471656 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [S], seq 2919054858, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 17008772 ecr 0], length 0
22:14:30.473083 IP 172.21.0.2.80 > 136.226.79.33.31101: Flags [S.], seq 2504697951, ack 2919054859, win 64768, options [mss 1420,sackOK,TS val 1834532093 ecr 17008772,nop,wscale 7], length 0
22:14:30.488585 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [.], ack 1, win 2068, options [nop,nop,TS val 17008774 ecr 1834532093], length 0
22:14:30.488591 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [P.], seq 1:469, ack 1, win 2068, options [nop,nop,TS val 17008774 ecr 1834532093], length 468: HTTP: GET / HTTP/1.1
22:14:30.488710 IP 172.21.0.2.80 > 136.226.79.33.31101: Flags [.], ack 469, win 503, options [nop,nop,TS val 1834532110 ecr 17008774], length 0
22:14:30.489427 IP 172.21.0.2.80 > 136.226.79.33.31101: Flags [P.], seq 1:335, ack 469, win 503, options [nop,nop,TS val 1834532110 ecr 17008774], length 334: HTTP: HTTP/1.1 200 OK
22:14:30.588957 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [.], ack 335, win 2068, options [nop,nop,TS val 17008786 ecr 1834532110], length 0
22:14:30.804022 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [P.], seq 469:879, ack 335, win 2068, options [nop,nop,TS val 17008809 ecr 1834532110], length 410: HTTP: GET /favicon.ico HTTP/1.1
22:14:30.804431 IP 172.21.0.2.80 > 136.226.79.33.31101: Flags [P.], seq 335:826, ack 879, win 501, options [nop,nop,TS val 1834532425 ecr 17008809], length 491: HTTP: HTTP/1.1 404 Not Found
22:14:30.903743 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [.], ack 826, win 2068, options [nop,nop,TS val 17008821 ecr 1834532425], length 0
22:14:35.713172 IP 172.21.0.2.80 > 136.226.79.33.31101: Flags [F.], seq 826, ack 879, win 501, options [nop,nop,TS val 1834537334 ecr 17008821], length 0
22:14:35.728581 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [.], ack 827, win 2068, options [nop,nop,TS val 17009357 ecr 1834537334], length 0
22:14:35.728590 IP 136.226.79.33.31101 > 172.21.0.2.80: Flags [R.], seq 879, ack 827, win 2068, options [nop,nop,TS val 17009357 ecr 1834537334], length 0


^C
45 packets captured
45 packets received by filter
0 packets dropped by kernel
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$














GENERATE LOG

7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ dig @8.8.8.8 example.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42717
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;example.com.                   IN      A

;; ANSWER SECTION:
example.com.            5421    IN      A       93.184.216.34

;; Query time: 7 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Aug 05 22:22:48 UTC 2023
;; MSG SIZE  rcvd: 56

student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ telnet 100.64.1.1 6667
Trying 100.64.1.1...




^C
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ telnet 100.64.1.1 6667
Trying 100.64.1.1...



^C
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ ping -c 3 34.125.170.248
PING 34.125.170.248 (34.125.170.248) 56(84) bytes of data.
64 bytes from 34.125.170.248: icmp_seq=1 ttl=61 time=1.91 ms
64 bytes from 34.125.170.248: icmp_seq=2 ttl=61 time=0.360 ms
64 bytes from 34.125.170.248: icmp_seq=3 ttl=61 time=0.379 ms

--- 34.125.170.248 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2031ms
rtt min/avg/max/mdev = 0.360/0.886/1.919/0.730 ms
student-04-6d7b0b6748d9@mig-dm-stamford-web-uswest4-6hsd:~$ 



LOG in SURICATA:




student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD UDP DNS" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:22:48.698673+0000","flow_id":1651899379525937,"in_iface":"ens4","event_type":"alert","src_ip":"172.21.0.3","src_port":52318,"dest_ip":"8.8.8.8","dest_port":53,"proto":"UDP","alert":{"action":"allowed","gid":1,"signature_id":99996,"rev":1,"signature":"BAD UDP DNS REQUEST","category":"","severity":3},"dns":{"query":[{"type":"query","id":42717,"rrname":"example.com","rrtype":"A","tx_id":0}]},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":82,"bytes_toclient":0,"start":"2023-08-05T22:22:48.698673+0000"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ date
Sat Aug  5 22:23:10 UTC 2023
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:13:25.844115+0000","flow_id":1096727574995283,"in_iface":"ens4","event_type":"alert","src_ip":"35.230.28.46","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":98,"bytes_toclient":0,"start":"2023-08-05T22:13:25.844115+0000"}}
@GCP: {"timestamp":"2023-08-05T22:13:43.538816+0000","flow_id":778801211979968,"in_iface":"ens4","event_type":"alert","src_ip":"35.230.28.46","src_port":0,"dest_ip":"172.21.0.2","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":98,"bytes_toclient":0,"start":"2023-08-05T22:13:43.538816+0000"}}
@GCP: {"timestamp":"2023-08-05T22:16:32.406987+0000","flow_id":1852762090386891,"in_iface":"ens4","event_type":"alert","src_ip":"185.234.213.135","src_port":0,"dest_ip":"172.21.0.2","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:16:32.406987+0000"}}
@GCP: {"timestamp":"2023-08-05T22:16:46.871117+0000","flow_id":1322578443389645,"in_iface":"ens4","event_type":"alert","src_ip":"185.234.213.135","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:16:46.871117+0000"}}
@GCP: {"timestamp":"2023-08-05T22:17:38.543815+0000","flow_id":1597984634719303,"in_iface":"ens4","event_type":"alert","src_ip":"123.184.59.70","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:17:38.543815+0000"}}
@GCP: {"timestamp":"2023-08-05T22:17:40.163398+0000","flow_id":1638417456987718,"in_iface":"ens4","event_type":"alert","src_ip":"123.184.59.70","src_port":0,"dest_ip":"172.21.0.2","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:17:40.163398+0000"}}
@GCP: {"timestamp":"2023-08-05T22:20:52.873300+0000","flow_id":1247034279744340,"in_iface":"ens4","event_type":"alert","src_ip":"195.123.211.81","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":50,"bytes_toclient":0,"start":"2023-08-05T22:20:52.873300+0000"}}
@GCP: {"timestamp":"2023-08-05T22:22:48.698673+0000","flow_id":1651899379525937,"in_iface":"ens4","event_type":"alert","src_ip":"172.21.0.3","src_port":52318,"dest_ip":"8.8.8.8","dest_port":53,"proto":"UDP","alert":{"action":"allowed","gid":1,"signature_id":99996,"rev":1,"signature":"BAD UDP DNS REQUEST","category":"","severity":3},"dns":{"query":[{"type":"query","id":42717,"rrname":"example.com","rrtype":"A","tx_id":0}]},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":82,"bytes_toclient":0,"start":"2023-08-05T22:22:48.698673+0000"}}
@GCP: {"timestamp":"2023-08-05T22:26:01.289420+0000","flow_id":796466460846732,"in_iface":"ens4","event_type":"alert","src_ip":"34.125.170.248","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":98,"bytes_toclient":0,"start":"2023-08-05T22:26:01.289420+0000"}}
@GCP: {"timestamp":"2023-08-05T22:26:01.289502+0000","flow_id":796466460846505,"in_iface":"ens4","event_type":"alert","src_ip":"34.125.170.248","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":2,"pkts_toclient":1,"bytes_toserver":196,"bytes_toclient":98,"start":"2023-08-05T22:26:01.289193+0000"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD ICMP" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:13:25.844115+0000","flow_id":1096727574995283,"in_iface":"ens4","event_type":"alert","src_ip":"35.230.28.46","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":98,"bytes_toclient":0,"start":"2023-08-05T22:13:25.844115+0000"}}
@GCP: {"timestamp":"2023-08-05T22:13:43.538816+0000","flow_id":778801211979968,"in_iface":"ens4","event_type":"alert","src_ip":"35.230.28.46","src_port":0,"dest_ip":"172.21.0.2","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":98,"bytes_toclient":0,"start":"2023-08-05T22:13:43.538816+0000"}}
@GCP: {"timestamp":"2023-08-05T22:16:32.406987+0000","flow_id":1852762090386891,"in_iface":"ens4","event_type":"alert","src_ip":"185.234.213.135","src_port":0,"dest_ip":"172.21.0.2","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:16:32.406987+0000"}}
@GCP: {"timestamp":"2023-08-05T22:16:46.871117+0000","flow_id":1322578443389645,"in_iface":"ens4","event_type":"alert","src_ip":"185.234.213.135","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:16:46.871117+0000"}}
@GCP: {"timestamp":"2023-08-05T22:17:38.543815+0000","flow_id":1597984634719303,"in_iface":"ens4","event_type":"alert","src_ip":"123.184.59.70","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:17:38.543815+0000"}}
@GCP: {"timestamp":"2023-08-05T22:17:40.163398+0000","flow_id":1638417456987718,"in_iface":"ens4","event_type":"alert","src_ip":"123.184.59.70","src_port":0,"dest_ip":"172.21.0.2","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":48,"bytes_toclient":0,"start":"2023-08-05T22:17:40.163398+0000"}}
@GCP: {"timestamp":"2023-08-05T22:20:52.873300+0000","flow_id":1247034279744340,"in_iface":"ens4","event_type":"alert","src_ip":"195.123.211.81","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":50,"bytes_toclient":0,"start":"2023-08-05T22:20:52.873300+0000"}}
@GCP: {"timestamp":"2023-08-05T22:26:01.289420+0000","flow_id":796466460846732,"in_iface":"ens4","event_type":"alert","src_ip":"34.125.170.248","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":8,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":98,"bytes_toclient":0,"start":"2023-08-05T22:26:01.289420+0000"}}
@GCP: {"timestamp":"2023-08-05T22:26:01.289502+0000","flow_id":796466460846505,"in_iface":"ens4","event_type":"alert","src_ip":"34.125.170.248","src_port":0,"dest_ip":"172.21.0.3","dest_port":0,"proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","gid":1,"signature_id":99998,"rev":1,"signature":"BAD ICMP","category":"","severity":3},"flow":{"pkts_toserver":2,"pkts_toclient":1,"bytes_toserver":196,"bytes_toclient":98,"start":"2023-08-05T22:26:01.289193+0000"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "TCP" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:20:16.794615+0000","flow_id":2103081389051793,"in_iface":"ens4","event_type":"anomaly","src_ip":"66.240.192.82","src_port":33844,"dest_ip":"172.21.0.3","dest_port":80,"proto":"TCP","anomaly":{"app_proto":"tls","type":"applayer","event":"APPLAYER_MISMATCH_PROTOCOL_BOTH_DIRECTIONS","layer":"proto_detect"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD HTTP" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:27:20.133109+0000","flow_id":1983990563618130,"in_iface":"ens4","event_type":"alert","src_ip":"136.226.79.33","src_port":1663,"dest_ip":"172.21.0.2","dest_port":80,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":99997,"rev":1,"signature":"BAD HTTP PHP REQUEST","category":"","severity":3},"http":{"hostname":"34.125.91.185","url":"/index.php","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36","xff":"24.23.174.37","http_content_type":"text/html","http_method":"GET","protocol":"HTTP/1.1","status":404,"length":275},"app_proto":"http","flow":{"pkts_toserver":7,"pkts_toclient":6,"bytes_toserver":1432,"bytes_toclient":1396,"start":"2023-08-05T22:27:20.015698+0000"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "TCP" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:20:16.794615+0000","flow_id":2103081389051793,"in_iface":"ens4","event_type":"anomaly","src_ip":"66.240.192.82","src_port":33844,"dest_ip":"172.21.0.3","dest_port":80,"proto":"TCP","anomaly":{"app_proto":"tls","type":"applayer","event":"APPLAYER_MISMATCH_PROTOCOL_BOTH_DIRECTIONS","layer":"proto_detect"}}
@GCP: {"timestamp":"2023-08-05T22:27:20.133109+0000","flow_id":1983990563618130,"in_iface":"ens4","event_type":"alert","src_ip":"136.226.79.33","src_port":1663,"dest_ip":"172.21.0.2","dest_port":80,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":99997,"rev":1,"signature":"BAD HTTP PHP REQUEST","category":"","severity":3},"http":{"hostname":"34.125.91.185","url":"/index.php","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36","xff":"24.23.174.37","http_content_type":"text/html","http_method":"GET","protocol":"HTTP/1.1","status":404,"length":275},"app_proto":"http","flow":{"pkts_toserver":7,"pkts_toclient":6,"bytes_toserver":1432,"bytes_toclient":1396,"start":"2023-08-05T22:27:20.015698+0000"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "BAD TCP" /var/log/suricata/eve.json
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ egrep "TCP" /var/log/suricata/eve.json
@GCP: {"timestamp":"2023-08-05T22:20:16.794615+0000","flow_id":2103081389051793,"in_iface":"ens4","event_type":"anomaly","src_ip":"66.240.192.82","src_port":33844,"dest_ip":"172.21.0.3","dest_port":80,"proto":"TCP","anomaly":{"app_proto":"tls","type":"applayer","event":"APPLAYER_MISMATCH_PROTOCOL_BOTH_DIRECTIONS","layer":"proto_detect"}}
@GCP: {"timestamp":"2023-08-05T22:27:20.133109+0000","flow_id":1983990563618130,"in_iface":"ens4","event_type":"alert","src_ip":"136.226.79.33","src_port":1663,"dest_ip":"172.21.0.2","dest_port":80,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":99997,"rev":1,"signature":"BAD HTTP PHP REQUEST","category":"","severity":3},"http":{"hostname":"34.125.91.185","url":"/index.php","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36","xff":"24.23.174.37","http_content_type":"text/html","http_method":"GET","protocol":"HTTP/1.1","status":404,"length":275},"app_proto":"http","flow":{"pkts_toserver":7,"pkts_toclient":6,"bytes_toserver":1432,"bytes_toclient":1396,"start":"2023-08-05T22:27:20.015698+0000"}}
student-04-6d7b0b6748d9@mig-dm-stamford-ids-uswest4-hr0r:~$ 





Service Accounts and Roles: Fundamentals

Service accounts are a special type of Google account that grant permissions to virtual machines instead of end users. Service accounts are primarily used to ensure safe, managed connections to APIs and Google Cloud services. Granting access to trusted connections and rejecting malicious ones is a must-have security feature for any Google Cloud project. In this lab, you get hands-on practice with the ins and outs of service accounts.



Create and manage service accounts.

Create a virtual machine and associate it with a service account.

Use client libraries to access BigQuery from a service account.

Run a query on a BigQuery public dataset from a Compute Engine instance.





What are service accounts?
A service account is a special Google account that belongs to your application or a virtual machine (VM) instead of an individual end user. Your application uses the service account to call the Google API of a service, so that the users aren't directly involved.

For example, a Compute Engine VM may run as a service account, and that account can be given permissions to access the resources it needs. This way the service account is the identity of the service, and the service account's permissions control which resources the service can access.

A service account is identified by its email address, which is unique to the account.

Types of service accounts
User-managed service accounts
When you create a new Cloud project using Google Cloud console and if Compute Engine API is enabled for your project, a Compute Engine Service account is created for you by default. It is identifiable using the email:

PROJECT_NUMBER-compute@developer.gserviceaccount.com
If your project contains an App Engine application, the default App Engine service account is created in your project by default. It is identifiable using the email:

PROJECT_ID@appspot.gserviceaccount.com
Google-managed service accounts
In addition to the user-managed service accounts, you might see some additional service accounts in your project’s IAM policy or in the console. These service accounts are created and owned by Google. These accounts represent different Google services and each account is automatically granted IAM roles to access your Google Cloud project.

Google APIs service account
An example of a Google-managed service account is a Google API service account identifiable using the email:

PROJECT_NUMBER@cloudservices.gserviceaccount.com
This service account is designed specifically to run internal Google processes on your behalf and is not listed in the Service Accounts section of the console. By default, the account is automatically granted the project editor role on the project and is listed in the IAM section of the console. This service account is deleted only when the project is deleted.

Note: Google services rely on the account having access to your project, so you should not remove or change the service account’s role on your project.
Task 1. Creating and managing service accounts
When you create a new Cloud project, Google Cloud automatically creates one Compute Engine service account and one App Engine service account under that project. You can create up to 98 additional service accounts to your project to control access to your resources.

Creating a service account
Creating a service account is similar to adding a member to your project, but the service account belongs to your applications rather than an individual end user.



















student_04_717d1aa28d01@cloudshell:~ (qwiklabs-gcp-03-f3d72bb5968f)$ history
    1  gcloud compute ssh source-instance --zone=$ZONE
    2  gcloud auth list
    3  gauth config list
    4  gcloud config list
    5  PROJECT_NUMBER-compute@developer.gserviceaccount.com
    6   
    7   
    8  gcloud config set compute/region us-west4
    9  gcloud config list project
   10  gcloud iam service-accounts create my-sa-123 --display-name "my service account"
   11  gcloud projects add-iam-policy-binding $DEVSHELL_PROJECT_ID     --member serviceAccount:my-sa-123@$DEVSHELL_PROJECT_ID.iam.gserviceaccount.com --role roles/editor


Linux bigquery-instance 5.10.0-23-cloud-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Creating directory '/home/student-04-717d1aa28d01'.
student-04-717d1aa28d01@bigquery-instance:~$ sudo apt-get update
Hit:1 https://deb.debian.org/debian bullseye InRelease
Get:2 https://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 https://packages.cloud.google.com/apt google-compute-engine-bullseye-stable InRelease [5146 B]
Get:4 https://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:5 https://deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Get:6 https://packages.cloud.google.com/apt cloud-sdk-bullseye InRelease [6406 B]
Get:7 https://deb.debian.org/debian-security bullseye-security/main Sources [204 kB]
Get:8 https://deb.debian.org/debian-security bullseye-security/main amd64 Packages [252 kB]
Get:9 https://deb.debian.org/debian-security bullseye-security/main Translation-en [164 kB]
Get:10 https://packages.cloud.google.com/apt google-compute-engine-bullseye-stable/main amd64 Packages [1942 B]
Get:11 https://deb.debian.org/debian bullseye-updates/main Sources.diff/Index [19.6 kB]
Get:12 https://deb.debian.org/debian bullseye-updates/main amd64 Packages.diff/Index [19.6 kB]
Get:13 https://deb.debian.org/debian bullseye-updates/main Translation-en.diff/Index [8361 B]
Get:14 https://deb.debian.org/debian bullseye-updates/main Sources T-2023-07-31-2005.11-F-2023-07-31-2005.11.pdiff [1673 B]
Get:15 https://deb.debian.org/debian bullseye-updates/main amd64 Packages T-2023-07-31-2005.11-F-2023-07-31-2005.11.pdiff [3382 B]
Get:14 https://deb.debian.org/debian bullseye-updates/main Sources T-2023-07-31-2005.11-F-2023-07-31-2005.11.pdiff [1673 B]
Get:15 https://deb.debian.org/debian bullseye-updates/main amd64 Packages T-2023-07-31-2005.11-F-2023-07-31-2005.11.pdiff [3382 B]
Get:16 https://deb.debian.org/debian bullseye-updates/main Translation-en T-2023-07-31-2005.11-F-2023-07-31-2005.11.pdiff [2273 B]
Get:16 https://deb.debian.org/debian bullseye-updates/main Translation-en T-2023-07-31-2005.11-F-2023-07-31-2005.11.pdiff [2273 B]
Get:17 https://deb.debian.org/debian bullseye-backports/main Sources.diff/Index [63.3 kB]
Get:18 https://deb.debian.org/debian bullseye-backports/main amd64 Packages.diff/Index [63.3 kB]
Get:19 https://deb.debian.org/debian bullseye-backports/main Translation-en.diff/Index [63.3 kB]
Get:20 https://deb.debian.org/debian bullseye-backports/main Sources T-2023-07-18-0816.49-F-2023-07-18-0209.04.pdiff [28.3 kB]
Get:20 https://deb.debian.org/debian bullseye-backports/main Sources T-2023-07-18-0816.49-F-2023-07-18-0209.04.pdiff [28.3 kB]
Get:21 https://deb.debian.org/debian bullseye-backports/main amd64 Packages T-2023-07-18-1410.09-F-2023-07-18-0209.04.pdiff [20.1 kB]
Get:21 https://deb.debian.org/debian bullseye-backports/main amd64 Packages T-2023-07-18-1410.09-F-2023-07-18-0209.04.pdiff [20.1 kB]
Get:22 https://deb.debian.org/debian bullseye-backports/main Translation-en T-2023-07-18-2011.14-F-2023-07-18-0209.04.pdiff [5503 B]
Get:23 https://packages.cloud.google.com/apt cloud-sdk-bullseye/main amd64 Packages [339 kB]
Get:22 https://deb.debian.org/debian bullseye-backports/main Translation-en T-2023-07-18-2011.14-F-2023-07-18-0209.04.pdiff [5503 B]
Fetched 1413 kB in 1s (1222 kB/s)                                                                          
Reading package lists... Done
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ sudo apt-get install -y git python3-pip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential bzip2 cpp cpp-10 dpkg-dev fakeroot
  fontconfig-config fonts-dejavu-core g++ g++-10 gcc gcc-10 git-man javascript-common libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libbinutils libc-dev-bin
  libc-devtools libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdeflate0 libdpkg-perl liberror-perl
  libexpat1-dev libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-10-dev libgd3 libgdbm-compat4
  libgomp1 libisl23 libitm1 libjbig0 libjpeg62-turbo libjs-jquery libjs-sphinxdoc libjs-underscore
  liblocale-gettext-perl liblsan0 libmpc3 libmpfr6 libnsl-dev libperl5.32 libpython3-dev libpython3.9-dev
  libquadmath0 libstdc++-10-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp6 libx11-6 libx11-data
  libxau6 libxcb1 libxdmcp6 libxpm4 linux-libc-dev make manpages-dev patch perl perl-modules-5.32
  python-pip-whl python3-dev python3-distutils python3-lib2to3 python3-setuptools python3-wheel python3.9-dev
  zlib1g-dev
Suggested packages:
  binutils-doc bzip2-doc cpp-doc gcc-10-locales debian-keyring g++-multilib g++-10-multilib gcc-10-doc
  gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-10-multilib git-daemon-run
  | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn apache2
  | lighttpd | httpd glibc-doc bzr libgd-tools libstdc++-10-doc make-doc ed diffutils-doc perl-doc
  libterm-readline-gnu-perl | libterm-readline-perl-perl libtap-harness-archive-perl python-setuptools-doc
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential bzip2 cpp cpp-10 dpkg-dev fakeroot
  fontconfig-config fonts-dejavu-core g++ g++-10 gcc gcc-10 git git-man javascript-common
  libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libbinutils
  libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdeflate0 libdpkg-perl
  liberror-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-10-dev libgd3
  libgdbm-compat4 libgomp1 libisl23 libitm1 libjbig0 libjpeg62-turbo libjs-jquery libjs-sphinxdoc
  libjs-underscore liblocale-gettext-perl liblsan0 libmpc3 libmpfr6 libnsl-dev libperl5.32 libpython3-dev
  libpython3.9-dev libquadmath0 libstdc++-10-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp6 libx11-6
  libx11-data libxau6 libxcb1 libxdmcp6 libxpm4 linux-libc-dev make manpages-dev patch perl perl-modules-5.32
  python-pip-whl python3-dev python3-distutils python3-lib2to3 python3-pip python3-setuptools python3-wheel
  python3.9-dev zlib1g-dev
0 upgraded, 85 newly installed, 0 to remove and 11 not upgraded.
Need to get 91.4 MB of archives.
After this operation, 358 MB of additional disk space will be used.
Get:1 https://deb.debian.org/debian bullseye/main amd64 perl-modules-5.32 all 5.32.1-4+deb11u2 [2823 kB]
Get:2 https://deb.debian.org/debian bullseye/main amd64 libgdbm-compat4 amd64 1.19-2 [44.7 kB]
Get:3 https://deb.debian.org/debian bullseye/main amd64 libperl5.32 amd64 5.32.1-4+deb11u2 [4106 kB]
Get:4 https://deb.debian.org/debian bullseye/main amd64 perl amd64 5.32.1-4+deb11u2 [293 kB]
Get:5 https://deb.debian.org/debian bullseye/main amd64 liblocale-gettext-perl amd64 1.07-4+b1 [19.0 kB]
Get:6 https://deb.debian.org/debian bullseye/main amd64 bzip2 amd64 1.0.8-4 [49.3 kB]
Get:7 https://deb.debian.org/debian bullseye/main amd64 binutils-common amd64 2.35.2-2 [2220 kB]
Get:8 https://deb.debian.org/debian bullseye/main amd64 libbinutils amd64 2.35.2-2 [570 kB]
Get:9 https://deb.debian.org/debian bullseye/main amd64 libctf-nobfd0 amd64 2.35.2-2 [110 kB]
Get:10 https://deb.debian.org/debian bullseye/main amd64 libctf0 amd64 2.35.2-2 [53.2 kB]
Get:11 https://deb.debian.org/debian bullseye/main amd64 binutils-x86-64-linux-gnu amd64 2.35.2-2 [1809 kB]
Get:12 https://deb.debian.org/debian bullseye/main amd64 binutils amd64 2.35.2-2 [61.2 kB]
Get:13 https://deb.debian.org/debian bullseye/main amd64 libc-dev-bin amd64 2.31-13+deb11u6 [276 kB]
Get:14 https://deb.debian.org/debian-security bullseye-security/main amd64 linux-libc-dev amd64 5.10.179-3 [1619 kB]
Get:15 https://deb.debian.org/debian bullseye/main amd64 libcrypt-dev amd64 1:4.4.18-4 [104 kB]
Get:16 https://deb.debian.org/debian bullseye/main amd64 libtirpc-dev amd64 1.3.1-1+deb11u1 [191 kB]
Get:17 https://deb.debian.org/debian bullseye/main amd64 libnsl-dev amd64 1.3.0-2 [66.4 kB]
Get:18 https://deb.debian.org/debian bullseye/main amd64 libc6-dev amd64 2.31-13+deb11u6 [2360 kB]
Get:19 https://deb.debian.org/debian bullseye/main amd64 libisl23 amd64 0.23-1 [676 kB]
Get:20 https://deb.debian.org/debian bullseye/main amd64 libmpfr6 amd64 4.1.0-3 [2012 kB]
Get:21 https://deb.debian.org/debian bullseye/main amd64 libmpc3 amd64 1.2.0-1 [45.0 kB]
Get:22 https://deb.debian.org/debian bullseye/main amd64 cpp-10 amd64 10.2.1-6 [8528 kB]
Get:23 https://deb.debian.org/debian bullseye/main amd64 cpp amd64 4:10.2.1-1 [19.7 kB]
Get:24 https://deb.debian.org/debian bullseye/main amd64 libcc1-0 amd64 10.2.1-6 [47.0 kB]
Get:25 https://deb.debian.org/debian bullseye/main amd64 libgomp1 amd64 10.2.1-6 [99.9 kB]
Get:26 https://deb.debian.org/debian bullseye/main amd64 libitm1 amd64 10.2.1-6 [25.8 kB]
Get:27 https://deb.debian.org/debian bullseye/main amd64 libatomic1 amd64 10.2.1-6 [9008 B]
Get:28 https://deb.debian.org/debian bullseye/main amd64 libasan6 amd64 10.2.1-6 [2065 kB]
Get:29 https://deb.debian.org/debian bullseye/main amd64 liblsan0 amd64 10.2.1-6 [828 kB]
Get:30 https://deb.debian.org/debian bullseye/main amd64 libtsan0 amd64 10.2.1-6 [2000 kB]
Get:31 https://deb.debian.org/debian bullseye/main amd64 libubsan1 amd64 10.2.1-6 [777 kB]
Get:32 https://deb.debian.org/debian bullseye/main amd64 libquadmath0 amd64 10.2.1-6 [145 kB]
Get:33 https://deb.debian.org/debian bullseye/main amd64 libgcc-10-dev amd64 10.2.1-6 [2328 kB]
Get:34 https://deb.debian.org/debian bullseye/main amd64 gcc-10 amd64 10.2.1-6 [17.0 MB]
Get:35 https://deb.debian.org/debian bullseye/main amd64 gcc amd64 4:10.2.1-1 [5192 B]
Get:36 https://deb.debian.org/debian bullseye/main amd64 libstdc++-10-dev amd64 10.2.1-6 [1741 kB]
Get:37 https://deb.debian.org/debian bullseye/main amd64 g++-10 amd64 10.2.1-6 [9380 kB]
Get:38 https://deb.debian.org/debian bullseye/main amd64 g++ amd64 4:10.2.1-1 [1644 B]
Get:39 https://deb.debian.org/debian bullseye/main amd64 make amd64 4.3-4.1 [396 kB]
Get:40 https://deb.debian.org/debian bullseye/main amd64 libdpkg-perl all 1.20.12 [1551 kB]
Get:41 https://deb.debian.org/debian bullseye/main amd64 patch amd64 2.7.6-7 [128 kB]
Get:42 https://deb.debian.org/debian bullseye/main amd64 dpkg-dev all 1.20.12 [2312 kB]
Get:43 https://deb.debian.org/debian bullseye/main amd64 build-essential amd64 12.9 [7704 B]
Get:44 https://deb.debian.org/debian bullseye/main amd64 libfakeroot amd64 1.25.3-1.1 [47.0 kB]
Get:45 https://deb.debian.org/debian bullseye/main amd64 fakeroot amd64 1.25.3-1.1 [87.0 kB]
Get:46 https://deb.debian.org/debian bullseye/main amd64 fonts-dejavu-core all 2.37-2 [1069 kB]
Get:47 https://deb.debian.org/debian bullseye/main amd64 fontconfig-config all 2.13.1-4.2 [281 kB]
Get:48 https://deb.debian.org/debian bullseye/main amd64 liberror-perl all 0.17029-1 [31.0 kB]
Get:49 https://deb.debian.org/debian bullseye/main amd64 git-man all 1:2.30.2-1+deb11u2 [1828 kB]
Get:50 https://deb.debian.org/debian bullseye/main amd64 git amd64 1:2.30.2-1+deb11u2 [5518 kB]
Get:51 https://deb.debian.org/debian bullseye/main amd64 javascript-common all 11+nmu1 [6260 B]
Get:52 https://deb.debian.org/debian bullseye/main amd64 libalgorithm-diff-perl all 1.201-1 [43.3 kB]
Get:53 https://deb.debian.org/debian bullseye/main amd64 libalgorithm-diff-xs-perl amd64 0.04-6+b1 [12.0 kB]
Get:54 https://deb.debian.org/debian bullseye/main amd64 libalgorithm-merge-perl all 0.08-3 [12.7 kB]
Get:55 https://deb.debian.org/debian bullseye/main amd64 libfontconfig1 amd64 2.13.1-4.2 [347 kB]
Get:56 https://deb.debian.org/debian bullseye/main amd64 libjpeg62-turbo amd64 1:2.0.6-4 [151 kB]
Get:57 https://deb.debian.org/debian bullseye/main amd64 libdeflate0 amd64 1.7-1 [53.1 kB]
Get:58 https://deb.debian.org/debian bullseye/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]
Get:59 https://deb.debian.org/debian-security bullseye-security/main amd64 libwebp6 amd64 0.6.1-2.1+deb11u1 [258 kB]
Get:60 https://deb.debian.org/debian bullseye/main amd64 libtiff5 amd64 4.2.0-1+deb11u4 [290 kB]
Get:61 https://deb.debian.org/debian bullseye/main amd64 libxau6 amd64 1:1.0.9-1 [19.7 kB]
Get:62 https://deb.debian.org/debian bullseye/main amd64 libxdmcp6 amd64 1:1.1.2-3 [26.3 kB]
Get:63 https://deb.debian.org/debian bullseye/main amd64 libxcb1 amd64 1.14-3 [140 kB]
Get:64 https://deb.debian.org/debian-security bullseye-security/main amd64 libx11-data all 2:1.7.2-1+deb11u1 [311 kB]
Get:65 https://deb.debian.org/debian-security bullseye-security/main amd64 libx11-6 amd64 2:1.7.2-1+deb11u1 [772 kB]
Get:66 https://deb.debian.org/debian bullseye/main amd64 libxpm4 amd64 1:3.5.12-1.1~deb11u1 [49.6 kB]
Get:67 https://deb.debian.org/debian bullseye/main amd64 libgd3 amd64 2.3.0-2 [137 kB]
Get:68 https://deb.debian.org/debian bullseye/main amd64 libc-devtools amd64 2.31-13+deb11u6 [246 kB]
Get:69 https://deb.debian.org/debian bullseye/main amd64 libexpat1-dev amd64 2.2.10-2+deb11u5 [141 kB]
Get:70 https://deb.debian.org/debian bullseye/main amd64 libfile-fcntllock-perl amd64 0.22-3+b7 [35.5 kB]
Get:71 https://deb.debian.org/debian bullseye/main amd64 libjs-jquery all 3.5.1+dfsg+~3.5.5-7 [315 kB]
Get:72 https://deb.debian.org/debian bullseye/main amd64 libjs-underscore all 1.9.1~dfsg-3 [100 kB]
Get:73 https://deb.debian.org/debian bullseye/main amd64 libjs-sphinxdoc all 3.4.3-2 [127 kB]
Get:74 https://deb.debian.org/debian bullseye/main amd64 libpython3.9-dev amd64 3.9.2-1 [4028 kB]
Get:75 https://deb.debian.org/debian bullseye/main amd64 libpython3-dev amd64 3.9.2-3 [21.7 kB]
Get:76 https://deb.debian.org/debian bullseye/main amd64 manpages-dev all 5.10-1 [2309 kB]
Get:77 https://deb.debian.org/debian bullseye/main amd64 python-pip-whl all 20.3.4-4+deb11u1 [1948 kB]
Get:78 https://deb.debian.org/debian bullseye/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-2+deb11u2 [191 kB]
Get:79 https://deb.debian.org/debian bullseye/main amd64 python3.9-dev amd64 3.9.2-1 [515 kB]
Get:80 https://deb.debian.org/debian bullseye/main amd64 python3-lib2to3 all 3.9.2-1 [77.8 kB]
Get:81 https://deb.debian.org/debian bullseye/main amd64 python3-distutils all 3.9.2-1 [143 kB]
Get:82 https://deb.debian.org/debian bullseye/main amd64 python3-dev amd64 3.9.2-3 [24.8 kB]
Get:83 https://deb.debian.org/debian bullseye/main amd64 python3-setuptools all 52.0.0-4 [366 kB]
Get:84 https://deb.debian.org/debian bullseye/main amd64 python3-wheel all 0.34.2-1 [24.0 kB]
Get:85 https://deb.debian.org/debian bullseye/main amd64 python3-pip all 20.3.4-4+deb11u1 [337 kB]
Fetched 91.4 MB in 1s (97.3 MB/s)       
Extracting templates from packages: 100%
Preconfiguring packages ...
Selecting previously unselected package perl-modules-5.32.
(Reading database ... 57457 files and directories currently installed.)
Preparing to unpack .../00-perl-modules-5.32_5.32.1-4+deb11u2_all.deb ...
Unpacking perl-modules-5.32 (5.32.1-4+deb11u2) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../01-libgdbm-compat4_1.19-2_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.19-2) ...
Selecting previously unselected package libperl5.32:amd64.
Preparing to unpack .../02-libperl5.32_5.32.1-4+deb11u2_amd64.deb ...
Unpacking libperl5.32:amd64 (5.32.1-4+deb11u2) ...
Selecting previously unselected package perl.
Preparing to unpack .../03-perl_5.32.1-4+deb11u2_amd64.deb ...
Unpacking perl (5.32.1-4+deb11u2) ...
Selecting previously unselected package liblocale-gettext-perl.
Preparing to unpack .../04-liblocale-gettext-perl_1.07-4+b1_amd64.deb ...
Unpacking liblocale-gettext-perl (1.07-4+b1) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../05-bzip2_1.0.8-4_amd64.deb ...
Unpacking bzip2 (1.0.8-4) ...
Selecting previously unselected package binutils-common:amd64.
Preparing to unpack .../06-binutils-common_2.35.2-2_amd64.deb ...
Unpacking binutils-common:amd64 (2.35.2-2) ...
Selecting previously unselected package libbinutils:amd64.
Preparing to unpack .../07-libbinutils_2.35.2-2_amd64.deb ...
Unpacking libbinutils:amd64 (2.35.2-2) ...
Selecting previously unselected package libctf-nobfd0:amd64.
Preparing to unpack .../08-libctf-nobfd0_2.35.2-2_amd64.deb ...
Unpacking libctf-nobfd0:amd64 (2.35.2-2) ...
Selecting previously unselected package libctf0:amd64.
Preparing to unpack .../09-libctf0_2.35.2-2_amd64.deb ...
Unpacking libctf0:amd64 (2.35.2-2) ...
Selecting previously unselected package binutils-x86-64-linux-gnu.
Preparing to unpack .../10-binutils-x86-64-linux-gnu_2.35.2-2_amd64.deb ...
Unpacking binutils-x86-64-linux-gnu (2.35.2-2) ...
Selecting previously unselected package binutils.
Preparing to unpack .../11-binutils_2.35.2-2_amd64.deb ...
Unpacking binutils (2.35.2-2) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../12-libc-dev-bin_2.31-13+deb11u6_amd64.deb ...
Unpacking libc-dev-bin (2.31-13+deb11u6) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../13-linux-libc-dev_5.10.179-3_amd64.deb ...
Unpacking linux-libc-dev:amd64 (5.10.179-3) ...
Selecting previously unselected package libcrypt-dev:amd64.
Preparing to unpack .../14-libcrypt-dev_1%3a4.4.18-4_amd64.deb ...
Unpacking libcrypt-dev:amd64 (1:4.4.18-4) ...
Selecting previously unselected package libtirpc-dev:amd64.
Preparing to unpack .../15-libtirpc-dev_1.3.1-1+deb11u1_amd64.deb ...
Unpacking libtirpc-dev:amd64 (1.3.1-1+deb11u1) ...
Selecting previously unselected package libnsl-dev:amd64.
Preparing to unpack .../16-libnsl-dev_1.3.0-2_amd64.deb ...
Unpacking libnsl-dev:amd64 (1.3.0-2) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../17-libc6-dev_2.31-13+deb11u6_amd64.deb ...
Unpacking libc6-dev:amd64 (2.31-13+deb11u6) ...
Selecting previously unselected package libisl23:amd64.
Preparing to unpack .../18-libisl23_0.23-1_amd64.deb ...
Unpacking libisl23:amd64 (0.23-1) ...
Selecting previously unselected package libmpfr6:amd64.
Preparing to unpack .../19-libmpfr6_4.1.0-3_amd64.deb ...
Unpacking libmpfr6:amd64 (4.1.0-3) ...
Selecting previously unselected package libmpc3:amd64.
Preparing to unpack .../20-libmpc3_1.2.0-1_amd64.deb ...
Unpacking libmpc3:amd64 (1.2.0-1) ...
Selecting previously unselected package cpp-10.
Preparing to unpack .../21-cpp-10_10.2.1-6_amd64.deb ...
Unpacking cpp-10 (10.2.1-6) ...
Selecting previously unselected package cpp.
Preparing to unpack .../22-cpp_4%3a10.2.1-1_amd64.deb ...
Unpacking cpp (4:10.2.1-1) ...
Selecting previously unselected package libcc1-0:amd64.
Preparing to unpack .../23-libcc1-0_10.2.1-6_amd64.deb ...
Unpacking libcc1-0:amd64 (10.2.1-6) ...
Selecting previously unselected package libgomp1:amd64.
Preparing to unpack .../24-libgomp1_10.2.1-6_amd64.deb ...
Unpacking libgomp1:amd64 (10.2.1-6) ...
Selecting previously unselected package libitm1:amd64.
Preparing to unpack .../25-libitm1_10.2.1-6_amd64.deb ...
Unpacking libitm1:amd64 (10.2.1-6) ...
Selecting previously unselected package libatomic1:amd64.
Preparing to unpack .../26-libatomic1_10.2.1-6_amd64.deb ...
Unpacking libatomic1:amd64 (10.2.1-6) ...
Selecting previously unselected package libasan6:amd64.
Preparing to unpack .../27-libasan6_10.2.1-6_amd64.deb ...
Unpacking libasan6:amd64 (10.2.1-6) ...
Selecting previously unselected package liblsan0:amd64.
Preparing to unpack .../28-liblsan0_10.2.1-6_amd64.deb ...
Unpacking liblsan0:amd64 (10.2.1-6) ...
Selecting previously unselected package libtsan0:amd64.
Preparing to unpack .../29-libtsan0_10.2.1-6_amd64.deb ...
Unpacking libtsan0:amd64 (10.2.1-6) ...
Selecting previously unselected package libubsan1:amd64.
Preparing to unpack .../30-libubsan1_10.2.1-6_amd64.deb ...
Unpacking libubsan1:amd64 (10.2.1-6) ...
Selecting previously unselected package libquadmath0:amd64.
Preparing to unpack .../31-libquadmath0_10.2.1-6_amd64.deb ...
Unpacking libquadmath0:amd64 (10.2.1-6) ...
Selecting previously unselected package libgcc-10-dev:amd64.
Preparing to unpack .../32-libgcc-10-dev_10.2.1-6_amd64.deb ...
Unpacking libgcc-10-dev:amd64 (10.2.1-6) ...
Selecting previously unselected package gcc-10.
Preparing to unpack .../33-gcc-10_10.2.1-6_amd64.deb ...
Unpacking gcc-10 (10.2.1-6) ...
Selecting previously unselected package gcc.
Preparing to unpack .../34-gcc_4%3a10.2.1-1_amd64.deb ...
Unpacking gcc (4:10.2.1-1) ...
Selecting previously unselected package libstdc++-10-dev:amd64.
Preparing to unpack .../35-libstdc++-10-dev_10.2.1-6_amd64.deb ...
Unpacking libstdc++-10-dev:amd64 (10.2.1-6) ...
Selecting previously unselected package g++-10.
Preparing to unpack .../36-g++-10_10.2.1-6_amd64.deb ...
Unpacking g++-10 (10.2.1-6) ...
Selecting previously unselected package g++.
Preparing to unpack .../37-g++_4%3a10.2.1-1_amd64.deb ...
Unpacking g++ (4:10.2.1-1) ...
Selecting previously unselected package make.
Preparing to unpack .../38-make_4.3-4.1_amd64.deb ...
Unpacking make (4.3-4.1) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../39-libdpkg-perl_1.20.12_all.deb ...
Unpacking libdpkg-perl (1.20.12) ...
Selecting previously unselected package patch.
Preparing to unpack .../40-patch_2.7.6-7_amd64.deb ...
Unpacking patch (2.7.6-7) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../41-dpkg-dev_1.20.12_all.deb ...
Unpacking dpkg-dev (1.20.12) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../42-build-essential_12.9_amd64.deb ...
Unpacking build-essential (12.9) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../43-libfakeroot_1.25.3-1.1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.25.3-1.1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../44-fakeroot_1.25.3-1.1_amd64.deb ...
Unpacking fakeroot (1.25.3-1.1) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../45-fonts-dejavu-core_2.37-2_all.deb ...
Unpacking fonts-dejavu-core (2.37-2) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../46-fontconfig-config_2.13.1-4.2_all.deb ...
Unpacking fontconfig-config (2.13.1-4.2) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../47-liberror-perl_0.17029-1_all.deb ...
Unpacking liberror-perl (0.17029-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../48-git-man_1%3a2.30.2-1+deb11u2_all.deb ...
Unpacking git-man (1:2.30.2-1+deb11u2) ...
Selecting previously unselected package git.
Preparing to unpack .../49-git_1%3a2.30.2-1+deb11u2_amd64.deb ...
Unpacking git (1:2.30.2-1+deb11u2) ...
Selecting previously unselected package javascript-common.
Preparing to unpack .../50-javascript-common_11+nmu1_all.deb ...
Unpacking javascript-common (11+nmu1) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../51-libalgorithm-diff-perl_1.201-1_all.deb ...
Unpacking libalgorithm-diff-perl (1.201-1) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../52-libalgorithm-diff-xs-perl_0.04-6+b1_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-6+b1) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../53-libalgorithm-merge-perl_0.08-3_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-3) ...
Selecting previously unselected package libfontconfig1:amd64.
Preparing to unpack .../54-libfontconfig1_2.13.1-4.2_amd64.deb ...
Unpacking libfontconfig1:amd64 (2.13.1-4.2) ...
Selecting previously unselected package libjpeg62-turbo:amd64.
Preparing to unpack .../55-libjpeg62-turbo_1%3a2.0.6-4_amd64.deb ...
Unpacking libjpeg62-turbo:amd64 (1:2.0.6-4) ...
Selecting previously unselected package libdeflate0:amd64.
Preparing to unpack .../56-libdeflate0_1.7-1_amd64.deb ...
Unpacking libdeflate0:amd64 (1.7-1) ...
Selecting previously unselected package libjbig0:amd64.
Preparing to unpack .../57-libjbig0_2.1-3.1+b2_amd64.deb ...
Unpacking libjbig0:amd64 (2.1-3.1+b2) ...
Selecting previously unselected package libwebp6:amd64.
Preparing to unpack .../58-libwebp6_0.6.1-2.1+deb11u1_amd64.deb ...
Unpacking libwebp6:amd64 (0.6.1-2.1+deb11u1) ...
Selecting previously unselected package libtiff5:amd64.
Preparing to unpack .../59-libtiff5_4.2.0-1+deb11u4_amd64.deb ...
Unpacking libtiff5:amd64 (4.2.0-1+deb11u4) ...
Selecting previously unselected package libxau6:amd64.
Preparing to unpack .../60-libxau6_1%3a1.0.9-1_amd64.deb ...
Unpacking libxau6:amd64 (1:1.0.9-1) ...
Selecting previously unselected package libxdmcp6:amd64.
Preparing to unpack .../61-libxdmcp6_1%3a1.1.2-3_amd64.deb ...
Unpacking libxdmcp6:amd64 (1:1.1.2-3) ...
Selecting previously unselected package libxcb1:amd64.
Preparing to unpack .../62-libxcb1_1.14-3_amd64.deb ...
Unpacking libxcb1:amd64 (1.14-3) ...
Selecting previously unselected package libx11-data.
Preparing to unpack .../63-libx11-data_2%3a1.7.2-1+deb11u1_all.deb ...
Unpacking libx11-data (2:1.7.2-1+deb11u1) ...
Selecting previously unselected package libx11-6:amd64.
Preparing to unpack .../64-libx11-6_2%3a1.7.2-1+deb11u1_amd64.deb ...
Unpacking libx11-6:amd64 (2:1.7.2-1+deb11u1) ...
Selecting previously unselected package libxpm4:amd64.
Preparing to unpack .../65-libxpm4_1%3a3.5.12-1.1~deb11u1_amd64.deb ...
Unpacking libxpm4:amd64 (1:3.5.12-1.1~deb11u1) ...
Selecting previously unselected package libgd3:amd64.
Preparing to unpack .../66-libgd3_2.3.0-2_amd64.deb ...
Unpacking libgd3:amd64 (2.3.0-2) ...
Selecting previously unselected package libc-devtools.
Preparing to unpack .../67-libc-devtools_2.31-13+deb11u6_amd64.deb ...
Unpacking libc-devtools (2.31-13+deb11u6) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../68-libexpat1-dev_2.2.10-2+deb11u5_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.2.10-2+deb11u5) ...
Selecting previously unselected package libfile-fcntllock-perl.
Preparing to unpack .../69-libfile-fcntllock-perl_0.22-3+b7_amd64.deb ...
Unpacking libfile-fcntllock-perl (0.22-3+b7) ...
Selecting previously unselected package libjs-jquery.
Preparing to unpack .../70-libjs-jquery_3.5.1+dfsg+~3.5.5-7_all.deb ...
Unpacking libjs-jquery (3.5.1+dfsg+~3.5.5-7) ...
Selecting previously unselected package libjs-underscore.
Preparing to unpack .../71-libjs-underscore_1.9.1~dfsg-3_all.deb ...
Unpacking libjs-underscore (1.9.1~dfsg-3) ...
Selecting previously unselected package libjs-sphinxdoc.
Preparing to unpack .../72-libjs-sphinxdoc_3.4.3-2_all.deb ...
Unpacking libjs-sphinxdoc (3.4.3-2) ...
Selecting previously unselected package libpython3.9-dev:amd64.
Preparing to unpack .../73-libpython3.9-dev_3.9.2-1_amd64.deb ...
Unpacking libpython3.9-dev:amd64 (3.9.2-1) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../74-libpython3-dev_3.9.2-3_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.9.2-3) ...
Selecting previously unselected package manpages-dev.
Preparing to unpack .../75-manpages-dev_5.10-1_all.deb ...
Unpacking manpages-dev (5.10-1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../76-python-pip-whl_20.3.4-4+deb11u1_all.deb ...
Unpacking python-pip-whl (20.3.4-4+deb11u1) ...
Selecting previously unselected package zlib1g-dev:amd64.
Preparing to unpack .../77-zlib1g-dev_1%3a1.2.11.dfsg-2+deb11u2_amd64.deb ...
Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-2+deb11u2) ...
Selecting previously unselected package python3.9-dev.
Preparing to unpack .../78-python3.9-dev_3.9.2-1_amd64.deb ...
Unpacking python3.9-dev (3.9.2-1) ...
Selecting previously unselected package python3-lib2to3.
Preparing to unpack .../79-python3-lib2to3_3.9.2-1_all.deb ...
Unpacking python3-lib2to3 (3.9.2-1) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../80-python3-distutils_3.9.2-1_all.deb ...
Unpacking python3-distutils (3.9.2-1) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../81-python3-dev_3.9.2-3_amd64.deb ...
Unpacking python3-dev (3.9.2-3) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../82-python3-setuptools_52.0.0-4_all.deb ...
Unpacking python3-setuptools (52.0.0-4) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../83-python3-wheel_0.34.2-1_all.deb ...
Unpacking python3-wheel (0.34.2-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../84-python3-pip_20.3.4-4+deb11u1_all.deb ...
Unpacking python3-pip (20.3.4-4+deb11u1) ...
Setting up javascript-common (11+nmu1) ...
Setting up manpages-dev (5.10-1) ...
Setting up libxau6:amd64 (1:1.0.9-1) ...
Setting up libxdmcp6:amd64 (1:1.1.2-3) ...
Setting up libxcb1:amd64 (1.14-3) ...
Setting up perl-modules-5.32 (5.32.1-4+deb11u2) ...
Setting up binutils-common:amd64 (2.35.2-2) ...
Setting up libdeflate0:amd64 (1.7-1) ...
Setting up linux-libc-dev:amd64 (5.10.179-3) ...
Setting up libctf-nobfd0:amd64 (2.35.2-2) ...
Setting up libgomp1:amd64 (10.2.1-6) ...
Setting up bzip2 (1.0.8-4) ...
Setting up python3-wheel (0.34.2-1) ...
Setting up libjbig0:amd64 (2.1-3.1+b2) ...
Setting up libfakeroot:amd64 (1.25.3-1.1) ...
Setting up libasan6:amd64 (10.2.1-6) ...
Setting up fakeroot (1.25.3-1.1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
Setting up libtirpc-dev:amd64 (1.3.1-1+deb11u1) ...
Setting up libjpeg62-turbo:amd64 (1:2.0.6-4) ...
Setting up libx11-data (2:1.7.2-1+deb11u1) ...
Setting up make (4.3-4.1) ...
Setting up libmpfr6:amd64 (4.1.0-3) ...
Setting up libquadmath0:amd64 (10.2.1-6) ...
Setting up libmpc3:amd64 (1.2.0-1) ...
Setting up libatomic1:amd64 (10.2.1-6) ...
Setting up patch (2.7.6-7) ...
Setting up libwebp6:amd64 (0.6.1-2.1+deb11u1) ...
Setting up fonts-dejavu-core (2.37-2) ...
Setting up libgdbm-compat4:amd64 (1.19-2) ...
Setting up libperl5.32:amd64 (5.32.1-4+deb11u2) ...
Setting up libubsan1:amd64 (10.2.1-6) ...
Setting up libnsl-dev:amd64 (1.3.0-2) ...
Setting up libcrypt-dev:amd64 (1:4.4.18-4) ...
Setting up git-man (1:2.30.2-1+deb11u2) ...
Setting up libx11-6:amd64 (2:1.7.2-1+deb11u1) ...
Setting up python-pip-whl (20.3.4-4+deb11u1) ...
Setting up libtiff5:amd64 (4.2.0-1+deb11u4) ...
Setting up libjs-jquery (3.5.1+dfsg+~3.5.5-7) ...
Setting up libbinutils:amd64 (2.35.2-2) ...
Setting up libisl23:amd64 (0.23-1) ...
Setting up libc-dev-bin (2.31-13+deb11u6) ...
Setting up python3-lib2to3 (3.9.2-1) ...
Setting up libcc1-0:amd64 (10.2.1-6) ...
Setting up liblocale-gettext-perl (1.07-4+b1) ...
Setting up liblsan0:amd64 (10.2.1-6) ...
Setting up cpp-10 (10.2.1-6) ...
Setting up libitm1:amd64 (10.2.1-6) ...
Setting up libjs-underscore (1.9.1~dfsg-3) ...
Setting up libtsan0:amd64 (10.2.1-6) ...
Setting up libctf0:amd64 (2.35.2-2) ...
Setting up python3-distutils (3.9.2-1) ...
Setting up python3-setuptools (52.0.0-4) ...
Setting up libxpm4:amd64 (1:3.5.12-1.1~deb11u1) ...
Setting up fontconfig-config (2.13.1-4.2) ...
Setting up libgcc-10-dev:amd64 (10.2.1-6) ...
Setting up perl (5.32.1-4+deb11u2) ...
Setting up python3-pip (20.3.4-4+deb11u1) ...
Setting up libjs-sphinxdoc (3.4.3-2) ...
Setting up libdpkg-perl (1.20.12) ...
Setting up cpp (4:10.2.1-1) ...
Setting up libc6-dev:amd64 (2.31-13+deb11u6) ...
Setting up libfontconfig1:amd64 (2.13.1-4.2) ...
Setting up binutils-x86-64-linux-gnu (2.35.2-2) ...
Setting up libstdc++-10-dev:amd64 (10.2.1-6) ...
Setting up libfile-fcntllock-perl (0.22-3+b7) ...
Setting up libalgorithm-diff-perl (1.201-1) ...
Setting up binutils (2.35.2-2) ...
Setting up dpkg-dev (1.20.12) ...
Setting up liberror-perl (0.17029-1) ...
Setting up libexpat1-dev:amd64 (2.2.10-2+deb11u5) ...
Setting up libgd3:amd64 (2.3.0-2) ...
Setting up git (1:2.30.2-1+deb11u2) ...
Setting up gcc-10 (10.2.1-6) ...
Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-2+deb11u2) ...
Setting up libalgorithm-diff-xs-perl (0.04-6+b1) ...
Setting up libc-devtools (2.31-13+deb11u6) ...
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up g++-10 (10.2.1-6) ...
Setting up libpython3.9-dev:amd64 (3.9.2-1) ...
Setting up gcc (4:10.2.1-1) ...
Setting up g++ (4:10.2.1-1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up python3.9-dev (3.9.2-1) ...
Setting up build-essential (12.9) ...
Setting up libpython3-dev:amd64 (3.9.2-3) ...
Setting up python3-dev (3.9.2-3) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+deb11u6) ...
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ pip3 install --upgrade pip
Requirement already satisfied: pip in /usr/lib/python3/dist-packages (20.3.4)
Collecting pip
  Downloading pip-23.2.1-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 7.3 MB/s 
Installing collected packages: pip
  WARNING: The scripts pip, pip3, pip3.11 and pip3.9 are installed in '/home/student-04-717d1aa28d01/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-23.2.1
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ pip3 install google-cloud-bigquery
Defaulting to user installation because normal site-packages is not writeable
Collecting google-cloud-bigquery
  Obtaining dependency information for google-cloud-bigquery from https://files.pythonhosted.org/packages/cc/6a/d0ef792288f2fa2cfea80899a82de302b3332dfda41984fe114e2cfbf700/google_cloud_bigquery-3.11.4-py2.py3-none-any.whl.metadata
  Downloading google_cloud_bigquery-3.11.4-py2.py3-none-any.whl.metadata (8.5 kB)
Collecting grpcio<2.0dev,>=1.47.0 (from google-cloud-bigquery)
  Obtaining dependency information for grpcio<2.0dev,>=1.47.0 from https://files.pythonhosted.org/packages/f5/f6/57fbd39af17aaae321109411ef2faf121768473ebc1bbf3694b06d3282c8/grpcio-1.56.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading grpcio-1.56.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 (from google-cloud-bigquery)
  Obtaining dependency information for google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 from https://files.pythonhosted.org/packages/6e/c4/c3cd048b6cbeba8d9ae50dd7643ac065b85237338aa7501b0efae91eb4d9/google_api_core-2.11.1-py3-none-any.whl.metadata
  Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Collecting proto-plus<2.0.0dev,>=1.15.0 (from google-cloud-bigquery)
  Obtaining dependency information for proto-plus<2.0.0dev,>=1.15.0 from https://files.pythonhosted.org/packages/36/5b/e02636d221917d6fa2a61289b3f16002eb4c93d51c0191ac8e896d527182/proto_plus-1.22.3-py3-none-any.whl.metadata
  Downloading proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting google-cloud-core<3.0.0dev,>=1.6.0 (from google-cloud-bigquery)
  Obtaining dependency information for google-cloud-core<3.0.0dev,>=1.6.0 from https://files.pythonhosted.org/packages/a2/40/02045f776fdb6e44194f34b6375a26ce8a61bd9bd03cd8930ed91cf51a62/google_cloud_core-2.3.3-py2.py3-none-any.whl.metadata
  Downloading google_cloud_core-2.3.3-py2.py3-none-any.whl.metadata (2.5 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0 (from google-cloud-bigquery)
  Downloading google_resumable_media-2.5.0-py2.py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.7/77.7 kB 5.4 MB/s eta 0:00:00
Collecting packaging>=20.0.0 (from google-cloud-bigquery)
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 9.9 MB/s eta 0:00:00
Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery)
  Obtaining dependency information for protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 from https://files.pythonhosted.org/packages/01/cb/445b3e465abdb8042a41957dc8f60c54620dc7540dbcf9b458a921531ca2/protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl.metadata
  Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting python-dateutil<3.0dev,>=2.7.2 (from google-cloud-bigquery)
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 19.0 MB/s eta 0:00:00
Requirement already satisfied: requests<3.0.0dev,>=2.21.0 in /usr/lib/python3/dist-packages (from google-cloud-bigquery) (2.25.1)
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Obtaining dependency information for googleapis-common-protos<2.0.dev0,>=1.56.2 from https://files.pythonhosted.org/packages/a7/bc/416a1ffeba4dcd072bc10523dac9ed97f2e7fc4b760580e2bdbdc1e2afdd/googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata
  Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting google-auth<3.0.dev0,>=2.14.1 (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Obtaining dependency information for google-auth<3.0.dev0,>=2.14.1 from https://files.pythonhosted.org/packages/9c/8d/bff87fc722553a5691d8514da5523c23547f3894189ba03b57592e37bdc2/google_auth-2.22.0-py2.py3-none-any.whl.metadata
  Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting grpcio-status<2.0.dev0,>=1.33.2 (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Obtaining dependency information for grpcio-status<2.0.dev0,>=1.33.2 from https://files.pythonhosted.org/packages/ef/16/3018689d96918e9c4c7407adf96b721df4d6748ba65db82c5eaa63564335/grpcio_status-1.56.2-py3-none-any.whl.metadata
  Downloading grpcio_status-1.56.2-py3-none-any.whl.metadata (1.3 kB)
Collecting google-crc32c<2.0dev,>=1.0 (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery)
  Downloading google_crc32c-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0dev,>=2.7.2->google-cloud-bigquery) (1.16.0)
Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Obtaining dependency information for cachetools<6.0,>=2.0.0 from https://files.pythonhosted.org/packages/a9/c9/c8a7710f2cedcb1db9224fdd4d8307c9e48cbddc46c18b515fefc0f1abbe/cachetools-5.3.1-py3-none-any.whl.metadata
  Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 35.5 MB/s eta 0:00:00
Collecting rsa<5,>=3.1.4 (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Downloading rsa-4.9-py3-none-any.whl (34 kB)
Requirement already satisfied: urllib3<2.0 in /usr/lib/python3/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery) (1.26.5)
Collecting pyasn1<0.6.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery)
  Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 18.3 MB/s eta 0:00:00
Downloading google_cloud_bigquery-3.11.4-py2.py3-none-any.whl (219 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 219.6/219.6 kB 38.2 MB/s eta 0:00:00
Downloading google_cloud_core-2.3.3-py2.py3-none-any.whl (29 kB)
Downloading grpcio-1.56.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 51.1 MB/s eta 0:00:00
Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 10.2 MB/s eta 0:00:00
Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl (304 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 304.5/304.5 kB 43.7 MB/s eta 0:00:00
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 23.7 MB/s eta 0:00:00
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 34.6 MB/s eta 0:00:00
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 38.1 MB/s eta 0:00:00
Downloading grpcio_status-1.56.2-py3-none-any.whl (5.1 kB)
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Installing collected packages: python-dateutil, pyasn1, protobuf, packaging, grpcio, google-crc32c, cachetools, rsa, pyasn1-modules, proto-plus, googleapis-common-protos, google-resumable-media, grpcio-status, google-auth, google-api-core, google-cloud-core, google-cloud-bigquery
  WARNING: The scripts pyrsa-decrypt, pyrsa-encrypt, pyrsa-keygen, pyrsa-priv2pub, pyrsa-sign and pyrsa-verify are installed in '/home/student-04-717d1aa28d01/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed cachetools-5.3.1 google-api-core-2.11.1 google-auth-2.22.0 google-cloud-bigquery-3.11.4 google-cloud-core-2.3.3 google-crc32c-1.5.0 google-resumable-media-2.5.0 googleapis-common-protos-1.60.0 grpcio-1.56.2 grpcio-status-1.56.2 packaging-23.1 proto-plus-1.22.3 protobuf-4.23.4 pyasn1-0.5.0 pyasn1-modules-0.3.0 python-dateutil-2.8.2 rsa-4.9
student-04-717d1aa28d01@bigquery-instance:~$ pip3 install pyarrow
Defaulting to user installation because normal site-packages is not writeable
Collecting pyarrow
  Obtaining dependency information for pyarrow from https://files.pythonhosted.org/packages/54/a2/5976df95323c4ca2b7baba31cb7a2a61a17461706043239d38a8e9dc281e/pyarrow-12.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading pyarrow-12.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting numpy>=1.16.6 (from pyarrow)
  Obtaining dependency information for numpy>=1.16.6 from https://files.pythonhosted.org/packages/69/1f/c95b1108a9972a52d7b1b63ed8ca70466b59b8c1811bd121f1e667cc45d8/numpy-1.25.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading numpy-1.25.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Downloading pyarrow-12.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (39.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.0/39.0 MB 24.2 MB/s eta 0:00:00
Downloading numpy-1.25.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 49.3 MB/s eta 0:00:00
Installing collected packages: numpy, pyarrow
  WARNING: The scripts f2py, f2py3 and f2py3.9 are installed in '/home/student-04-717d1aa28d01/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.25.2 pyarrow-12.0.1
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ pip3 install pandas
Defaulting to user installation because normal site-packages is not writeable
Collecting pandas
  Obtaining dependency information for pandas from https://files.pythonhosted.org/packages/9e/0d/91a9fd2c202f2b1d97a38ab591890f86480ecbb596cbc56d035f6f23fdcc/pandas-2.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading pandas-2.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Requirement already satisfied: python-dateutil>=2.8.2 in ./.local/lib/python3.9/site-packages (from pandas) (2.8.2)
Collecting pytz>=2020.1 (from pandas)
  Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.3/502.3 kB 13.8 MB/s eta 0:00:00
Collecting tzdata>=2022.1 (from pandas)
  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 kB 48.7 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.20.3 in ./.local/lib/python3.9/site-packages (from pandas) (1.25.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
Downloading pandas-2.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.4/12.4 MB 51.1 MB/s eta 0:00:00
Installing collected packages: pytz, tzdata, pandas
Successfully installed pandas-2.0.3 pytz-2023.3 tzdata-2023.3
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ pip3 install db-dtypes
Defaulting to user installation because normal site-packages is not writeable
Collecting db-dtypes
  Downloading db_dtypes-1.1.1-py2.py3-none-any.whl (14 kB)
Requirement already satisfied: packaging>=17.0 in ./.local/lib/python3.9/site-packages (from db-dtypes) (23.1)
Requirement already satisfied: pandas>=0.24.2 in ./.local/lib/python3.9/site-packages (from db-dtypes) (2.0.3)
Requirement already satisfied: pyarrow>=3.0.0 in ./.local/lib/python3.9/site-packages (from db-dtypes) (12.0.1)
Requirement already satisfied: numpy>=1.16.6 in ./.local/lib/python3.9/site-packages (from db-dtypes) (1.25.2)
Requirement already satisfied: python-dateutil>=2.8.2 in ./.local/lib/python3.9/site-packages (from pandas>=0.24.2->db-dtypes) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in ./.local/lib/python3.9/site-packages (from pandas>=0.24.2->db-dtypes) (2023.3)
Requirement already satisfied: tzdata>=2022.1 in ./.local/lib/python3.9/site-packages (from pandas>=0.24.2->db-dtypes) (2023.3)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas>=0.24.2->db-dtypes) (1.16.0)
Installing collected packages: db-dtypes
Successfully installed db-dtypes-1.1.1
student-04-717d1aa28d01@bigquery-instance:~$ vi query.py
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ sed -i -e "s/qwiklabs-gcp-03-f3d72bb5968f/$(gcloud config get-value project)/g" query.py
student-04-717d1aa28d01@bigquery-instance:~$ vi query.py
student-04-717d1aa28d01@bigquery-instance:~$ rm query.py 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ echo "
from google.auth import compute_engine
from google.cloud import bigquery
credentials = compute_engine.Credentials(
    service_account_email='YOUR_SERVICE_ACCOUNT')
query = '''
SELECT
  year,
  COUNT(1) as num_babies
FROM
  publicdata.samples.natality
WHERE
  year > 2000
GROUP BY
  year
'''
client = bigquery.Client(
    project='qwiklabs-gcp-03-f3d72bb5968f',
    credentials=credentials)
print(client.query(query).to_dataframe())
" > query.py
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ cat query.py 

from google.auth import compute_engine
from google.cloud import bigquery
credentials = compute_engine.Credentials(
    service_account_email='YOUR_SERVICE_ACCOUNT')
query = '''
SELECT
  year,
  COUNT(1) as num_babies
FROM
  publicdata.samples.natality
WHERE
  year > 2000
GROUP BY
  year
'''
client = bigquery.Client(
    project='qwiklabs-gcp-03-f3d72bb5968f',
    credentials=credentials)
print(client.query(query).to_dataframe())

student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ sed -i -e "s/qwiklabs-gcp-03-f3d72bb5968f/$(gcloud config get-value project)/g" query.py
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ cat query.py 

from google.auth import compute_engine
from google.cloud import bigquery
credentials = compute_engine.Credentials(
    service_account_email='YOUR_SERVICE_ACCOUNT')
query = '''
SELECT
  year,
  COUNT(1) as num_babies
FROM
  publicdata.samples.natality
WHERE
  year > 2000
GROUP BY
  year
'''
client = bigquery.Client(
    project='qwiklabs-gcp-03-f3d72bb5968f',
    credentials=credentials)
print(client.query(query).to_dataframe())

student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ sed -i -e "s/YOUR_SERVICE_ACCOUNT/bigquery-qwiklab@$(gcloud config get-value project).iam.gserviceaccount.com/g" query.py
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ cat query.py 

from google.auth import compute_engine
from google.cloud import bigquery
credentials = compute_engine.Credentials(
    service_account_email='bigquery-qwiklab@qwiklabs-gcp-03-f3d72bb5968f.iam.gserviceaccount.com')
query = '''
SELECT
  year,
  COUNT(1) as num_babies
FROM
  publicdata.samples.natality
WHERE
  year > 2000
GROUP BY
  year
'''
client = bigquery.Client(
    project='qwiklabs-gcp-03-f3d72bb5968f',
    credentials=credentials)
print(client.query(query).to_dataframe())

student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ 
student-04-717d1aa28d01@bigquery-instance:~$ python3 query.py
   year  num_babies
0  2006     4273225
1  2004     4118907
2  2005     4145619
3  2002     4027376
4  2007     4324008
5  2008     4255156
6  2001     4031531
7  2003     4096092
student-04-717d1aa28d01@bigquery-instance:~$ history
    1  sudo apt-get update
    2  sudo apt-get install -y git python3-pip
    3  pip3 install --upgrade pip
    4  pip3 install google-cloud-bigquery
    5  pip3 install pyarrow
    6  pip3 install pandas
    7  pip3 install db-dtypes
    8  vi query.py
    9  sed -i -e "s/qwiklabs-gcp-03-f3d72bb5968f/$(gcloud config get-value project)/g" query.py
   10  vi query.py
   11  rm query.py 
   12  echo "
from google.auth import compute_engine
from google.cloud import bigquery
credentials = compute_engine.Credentials(
    service_account_email='YOUR_SERVICE_ACCOUNT')
query = '''
SELECT
  year,
  COUNT(1) as num_babies
FROM
  publicdata.samples.natality
WHERE
  year > 2000
GROUP BY
  year
'''
client = bigquery.Client(
    project='qwiklabs-gcp-03-f3d72bb5968f',
    credentials=credentials)
print(client.query(query).to_dataframe())
" > query.py
   13  cat query.py 
   14  sed -i -e "s/qwiklabs-gcp-03-f3d72bb5968f/$(gcloud config get-value project)/g" query.py
   15  cat query.py 
   16  sed -i -e "s/YOUR_SERVICE_ACCOUNT/bigquery-qwiklab@$(gcloud config get-value project).iam.gserviceaccount.com/g" query.py
   17  cat query.py 
   18  python3 query.py
   19  history
student-04-717d1aa28d01@bigquery-instance:~$

create a Data Fusion instance and deploy a sample pipeline

Create a Data Fusion instance and deploy a sample pipeline that reads an input file from Cloud Storage, transforms and filters the data to o...