GCP Cloud Digital Leader
前言
官网: https://cloud.google.com/learn/certification/cloud-digital-leader
网课:https://www.udemy.com/course/google-cloud-digital-leader-certification/
题库: https://www.udemy.com/course/latest-gcp-cdl-google-cloud-digital-leader-practice-exams-tests/
题库: https://www.examtopics.com/exams/google/cloud-digital-leader/view/
知识点
1. Regions and Zones
Regions and Zones
- Google provide 20+ regions around the world
- High Availability, Low Latency, Global Footprint, Government Regulation
- Each Region has three or more Zones
- Advantage of Zones: Increase availability and fault tolerance
2. Compute
Compute Engine
Compute Engine (GCE)
: Provision & Manage Virtual Machines
- Create and manage lifecycle of VM instances
- Load balancing and auto scaling for VM instances
- Can attach storage, manage network connectivity and configuration
VM Setup
Startup Script
: For boostrapping- Install OS patches or software when an VM instance is launched
Instance Templates
: Specify VM instance details- Used to create VM instances and managed instance groups
- Cannot be updated, need to be deleted and modified
Custom Image
: Prefered way than Startup Script (推荐的做法)- Can have OS patches and software pre-installed
Use Discount
Sustained Use Discounts
- Automatic discounts for running VM instances for significant portion of the billing month (意思就是在一个月内使用这个 instance 到一定程度就可以获得 discount)
Committed Use Discounts
- 1 year or 3 year reservations for workloads with predictable resource needs
Preemptible VM
Preemptible VM
: Short-lived cheaper (up to 80%) compute instances for non time-critical, fault-tolerant workloads (只能存在 24 小时)
Spot VM
Spot VM
: Latest version of preemptible VM (这个就是 Spot Instance)- Does not have a maximum runtime, 没有 24 小时限制
Sole Tenant Nodes
Sole Tenant Nodes
: 类似 Dedicated Host 或者 Dedicated Instance- Virtualized instances on hardware dedicated to one customer
Custom Machine Types
Custom Machine Types
: 可以自定义 CPU, memory, GPU
VM costs
- 2 primary costs in running VMs using GCE
Infrastructure
cost (VM) &Licensing
cost (OS)
Instance Groups
Instance Group
: Group of VM instances managed as a single entity- Two Types of Instance Groups
Managed
: Identical VMs created using a templateUnmanaged
: Different configuration for VMs in same group
- Location can be Zonal or Regional (Regional gives higher availability)
Managed Instance Groups
Managed Instance Groups
: Maintain certain number of instances- Detect application failures using health check (self healing)
- Increase and decrease instances based on load (auto scaling)
- Add load balancer to distribute load
Cloud Load balancing
Cloud Load balancing
: Distribute traffic across VM instances in one or more regions
3. Managed Services
IAAS (Infrastructure as a Service)
- Use only infrastructure from cloud provider
- Using VM to deploy your applications or databases
PAAS (Platform as a Service)
- Use a platform provided by cloud
- You are responsible for Configuration and Application code
SAAS (Software as a Service)
- Centrally hosted software (mostly on the cloyd)
- Offered on a subscription basis (pay-as-you-go)
Containers
- Container ensure we have one way of deploying any microservices
- Create Docker images for each microservice, include
- Application Runtime
- Application code and Dependencies
- Runs the same way on any infrastructure
- Advantage: light weight (No Guest OS), isolation for containers, cloud neutral
- Container Orchestration (Many build upon Kubernetes)
- Feature: Auto Scaling, Load Balancing, Self Healing, Fast Deployment
Serverless
- Focus on code and the cloud managed service take cares of other stuffs
- Pay for use
Shared Responsibility Model
- Securiet in cloud is a Shared Responsibility
SaaS
: Content + Access Policies + UsagePaaS
: SaaS + Deployment + Web Application SecurityIaaS
: PaaS + Operations + Network Security + Guest OS- Google Cloud is always responsible for Hardware, Network, Audit Logging etc
GCP Service Category
- Compute Engine:
IAAS
(provide customize OS) - Google Kubernetes Engine:
CAAS
- App Engine:
PAAS
- Cloud Functions:
FAAS
- Cloud Run:
CAAS
(run one container quickly)
4. Managed Compute Service in GCP
App Engine
App Engine
: Simplest way to deploy and scale your applications in GCP- Automatic load balancing & auto scaling
- Managed platform update & health monitoring
- Application versioning
- Traffic splitting
- No usage charges -> pay for resources provisioned
Compute Engine
vs.App Engine
- Compute Engine is
IAAS
, App Engine isPAAS
- App Engine is
serverless
- Compute Engine you have more Responsibility than App Engine
- App Engine is lower flexibility
- Compute Engine is
App Engine Environments
Standard
: Applications run in language specific sandboxes- Run in language specific sandbox
- Supports scale down to Zero instances
Flexible
: Application instances run within Docker containers- Support ANY runtime
- CANNOT scale down to Zero instances
Google Kubernetes Engine (GKE)
Google Kubernetes Engine
: Managed Kubernetes service- Minimize operations with
auto-repair
(repair failed nodes) andauto-upgrade
(use latest version of K8S always) features - Provides Pod and Cluster Autoscaling
- Two Modes:
Standard
(You manage) &Autopilot
(GKE manage)
Cloud Functions (GCF)
Cloud Functions
: Excute some code when an event happen- Don’t worry about servers or scaling or availability (only worry about your code)
- Pay only for what you use
Time Bound
- Default 1 min and MAX 60 minutes(3600 seconds)
Cloud Run & Anthos
Cloud Run
: “Container to Production in Seconds”- Fully managed serverless platform for containerized applications
Cloud Run for Anthos
: Deploy your workloads toAnthos
clusters running on-premises or on Google Cloud
Compute Service in GCP
- Create Virtual Machines:
Compute Engine
- Create a group of similar VMs:
Managed Image Group
- Distribute load among VMs:
Cloud Load Balancing
- Simplify setting up web application:
App Engine
- Easiest way to run one container:
Google Cloud Run
- Orchestrate containers:
Google Kubernetes Engine
- Build serverless event driven functions:
Cloud Functions
- Manage multi-cloud and on-premise Kubernetes clusters:
Anthos
5. Storage
Block Storage & File Storage
Block Storage
- Persistent Disk: Network Block Storage (Zonal or Regional)
- Local SSDs: Local Block Storage
File Storage
- Filestore: High performance file storage
Cloud Storage
- Most popular, very flexible & inexpensive storage service
- Store large objects using a key-value approach (Object Storage)
- Provides REST API to access and modify objects
- Store all file types - text, binary, backup & archives
Storage Classes
- Different kinds of data can be stored in Cloud Storage
- Can I pay a cheaper price for objects I access less frequently
- Storage classes help to optimize your costs based on your access needs
Standard
: Frequently used data/Short period of timeNearline storage
: Read or modify once a month on average (30 天内读写一次)Coldline storage
: Read or modify at most once a quarter (90 天内读写一次)Archive storage
: Less than once a year (365 天内读写一次)
Object Lifecycle Management
How do you save costs by moving files automatically between storage classes
Object Lifecycle Management
Identify objects using conditions based on
- Age, CreatedBefore, IsLive, MatchesStorageClass, NumberOfNewerVersions
- Set multiple conditions: all conditions must be satisfied for action to happen
Two kinds of actions
SetStorageClass actions
(change from one storage class to another)Deletion actions
(delete objects)
Transferring data from On-Premises to Cloud
- Most popular data destination is Google Cloud Storage
Online Transfer
: Use gsutil or API to transfer data to Google Cloud Storage (data < 1 TB or from on-premise)Storage Transfer Service
: Recommended for large-scale (petabytes) online data transfers from your private data centers, AWS, Azure, and Google Cloud (data > 1 TB or from another cloud)Transfer Appliance
: Physical transfer using an appliance (data > 20 TB or time > 1 week)
Storage in GCP
- A shared space for collaborating on
media
projects that involve large files:Filestore
(File Storage) - A cost-effective solution to store and serve a large amount of
unstructured data
(Videos, Music, Files) globally:Cloud Storage
(Object Storage) - Data is automatically managed and transitioned between
storage classes
to reduce costs:Object Lifecycle Management
in Cloud Storage - A massive, one-time migration of data to the cloud, where online transfer is not feasible: Using
Transfer Appliance
for large- scale, physical data migration
6. Database Fundamentals
Cloud SQL
& Cloud Spanner
(OLTP)
Applications where large number of users make large number of small transactions
- Popular databases: MySQL, Oracle, SQL Server etc
Recommended Google Managed Services
Cloud SQL
: Supports PostgreSQL, MySQL, and SQL Server for regional relational databasesCloud Spanner
: Unlimited scale (multiple PBs) and 99.999% availability for global applications with horizontal scaling
BigQuery
(OLAP)
Applications allowing users to analyze petabytes of data
- Reporting applications, Data ware houses, Business intelligence applications, Analytics systems
Recommended Google Managed Services
BigQuery
: Petabyte-scale distributed data ware house
OLAP
vs OLTP
- OLAP and OLTP use similar data structures, but different in how data is stored
- OLTP databases use
row
storage- Efficient for processing small transactions
- OLAP databases use
columnar
storage- High compression, Distribute data, Execute single query across multiple nodes
Cloud Firestore
vs BigTable
(NOSQL)
- NoSQL databases trade-off “Strong consistency and SQL features” to achieve “scalability and high-performance”
- NoSQL = not only SQL
- Google Managed Services:
Cloud Firestore (Datastore)
&BigTable
Cloud Datastore
- Managed serverless NoSQL document database- Designed for
transactional
mobile and web applications
- Designed for
BigTable
- Managed, scalable NoSQL wide column database- Recommended for
large analytical
(> 10 TB) and operational workloads (not serverless)
- Recommended for
Memory Store
(In-memory Databases)
- Retrieving data from memory is much faster than retrieving data from disk (Redis)
- Recommended GCP Managed Service:
Memory Store
- Use cases: Caching, session management, gaming leader boards, geospatial applications
Database in GCP
Relational OLTP databases: Cloud SQL
, Cloud Spanner
- Have
predefined schema
and verystrong transcational
capabilities (Row storage
)
Relational OLAP databases: BigQuery
Columnar storage
with predefined schema.Datawarehouse
&BigData
workloads
NoSQL Databases: Cloud Firestore (Datastore)
, BigTable
- Apps that need quickly evolving structure (
schema-less
)
In memory databases/caches: Memory Store
- Applications needing microsecond responses
7. IAM
Cloud IAM
- How do you identify users in GCP?
- Identity and Access Management (
Cloud IAM
) provides this service
- Identity and Access Management (
IAM Example
- Provide access to manage a specific cloud storage bucket to a colleague
- Choose a Role with right permissions (Ex: Storage Object Admin)
- Create Policy binding member (your friend) with role (permissions)
- Roles: A set of permissions (to perform specific actions on specific resources)
IAM Roles
Roles are Permissions
: Perform some set of actions on some set of resourcesBasic Roles
- Owner / Editor / ViewerPredefined Roles
- Fine grained roles predefined and managed by GoogleCustom Roles
- When predefined roles are NOT sufficient, you can create your own custom roles
IAM Policy
- Roles are assigned to users through
IAM Policy documents
- Represented by a policy object
8. Encryption
Data Lifecycle States
- Data at rest: Stored on a device or a backup
- Data in motion: Being transferred across a network
- Data in use: Active data processed in a non-persistent state
Encryption
Symmetric
Key Encryption- Use the same key for encryption and decryption
Asymmetric
Key Encryption- Encrypt data with Public Key and decrypt with Private Key
Cloud KMS
Cloud KMS
: Create and manage cryptographic keys (symmetric and asymmetric)
9. Organizing GCP Resources
Resource Hierarchy
Organization > Folder > Project > Resources
- Resources are created in projects
- A Folder can contain multiple projects
- Organization can contain multiple Folders
Billing Accounts
- Billing Account is mandatory for creating resources in a project
- Setup a
Cloud Billing Budget
to avoid surprises - Alerts
IAM Best Practices
Principle of Least Privilege
: Give least possible privilege needed for a roleSeparation of Duties
: Involve atleast 2 people in sensitive tasksConstant Monitoring
: Review Cloud Audit Logs to audit changes to IAM policies and access to Service Account keys
Public, Private, Hybrid Cloud
Public Cloud
: You host everything in the cloud- DO NOT need a data center,NO Capital Expenditure needed
- Hardware resources are owned by Google Cloud
Private Cloud
: You host everything in your own data center- Needs Capital Expenditure
Hybrid Cloud
: Combination of both (Public & Private)- Use Public Cloud for some workloads and Private cloud for others
Multi Cloud
: Using Multiple Cloud Platforms with/without onpremise infrastructure
Cloud VPN
Cloud VPN
: Connect on-premise network to the GCP network- Encrypted
- For low bandwidth, Cloud VPN is recommended
Cloud Interconnect
Cloud Interconnect
: High speed physical connection between on-premise and VPC networks- Highly available with high throughput
- Use only for high bandwidth needs
Organization Policy Service
- How to enable centralized constraints on all resources created in an Organization?
- Configure Organization Policy
- Needs a Role - Organization Policy Administrator
- IAM focuses on WHO, Organization Policy focuses on WHAT
Corporate Directory Federation
- Federate Cloud Identity or Google Workspace with your external identity provider (IdP) such as Active Directory or Azure Active Directory
Identity Platform
Identity Platform
: Customer identity and access management- Difference between
Cloud IAM
andIdentity Platform
Cloud IAM
: Employees and Partners AuthorizationIdentity Platform
: Customer identity and access management
10. DevOps
CI, CD Tools
Cloud Source Repositories
: Fully-featured, private Git repositoryContainer Registry
: Store your Docker imagesCloud Build
: Build deployable artifacts from your source code and configuration
Container Registry and Artifact Registry
Container Registry
: Uses GCS bucket to store images, supports Container images onlyArtifact Registry
: Evolution of Container Registry, manage BOTH container images and non-container artifacts
Infrastructure as code
- Treat infrastructure the same way as application code
- Bring repeatability into your infrastructure
- GCP service:
Google Cloud Deployment Manager
Cloud Operations & Insights
- Monitoring - Metrics and Alerts:
Cloud Monitoring
- Centralized Logging:
Cloud Logging
- Audit Logging:
Cloud Audit Logs
- Real-time exception monitoring:
Error Reporting
- Live Debugging:
Cloud Debugger
- Distributed tracing:
Cloud Trace
- Statistical, low-overhead profiler:
Cloud Profiler
Site Reliability Engineering (SRE)
- SRE teams focus on every aspect of an application
- Manage by Service Level Objectives (SLOs)
- Minimize Toil
- Move Fast by Reducing Cost of Failure
- Share Ownership with Developers
SRE - Key Metrics
- Service Level Indicator(SLI): Quantitative measure of an aspect of a service
- Categories: availability, latency, throughput, durability, correctness (error rate)
- Service Level Objective (SLO) - SLI + target
- 99.99% Availability, 99.999999999% Durability
- Service Level Agreement (SLA): SLO + consequences (contract)
- What is the consequence of NOT meeting an SLO? (Defined in a contract)
- Error budgets: (100% – SLO)
- How well is a team meeting their reliability objectives?
SRE - Best Practices
Handling Excess Loads
: Load Shedding, Reduced Quality of ServiceAvoiding Cascading Failures
: Plan to avoid thrashing
Penetration Testing
(Ethical Hacking)Load Testing
(JMeter, LoadRunner, Locust, Gatling etc)
Resilience Testing
- “How does an application behaves under stress?”
11. Pub/Sub
Synchronous
Communication: Applications makes synchronous calls to the logging service (What if the logging service goes down?)
Asynchronous
Communication: Create a topic and have applications put log messages on the topic. Logging service picks them up for processing when ready
Pub/Sub
Pub/Sub
: Reliable, scalable, fully-managedasynchronous messaging service
- Backbone for Highly Available and Highly Scalable Solutions
- Event ingestion and delivery for streaming analytics pipelines
Publisher
- Sender of a messageSubscriber
- Receiver of the messagePull
- Subscriber pulls messages when readyPush
- Messages are sent to subscribers
Cloud Dataflow
Cloud Dataflow
is a difficult service to describe- Pub/Sub > Dataflow > BigQuery (Streaming)
- Pub/Sub > Dataflow > Cloud Storage (Streaming - files)
- Cloud Storage > Dataflow > Bigtable/CloudSpanner/Datastore/BigQuery (Batch - Load data into databases)
12. Data Architectures in GCP
Loose Coupling with Pub/Sub
- Whenever you want to decouple a publisher from a subscriber, consider
Pub/Sub
Date Formats
- Structured: Tables, Rows and Columns (Relational)
Cloud SQL
,Cloud Spanner
,BigQuery
- Semi Structured: Flexible Schema
Cloud Firestore/Datastore
- Unstructured: Video, Audio, Image, Text, Binary files
Cloud Storage
Cloud Dataproc
Cloud Dataproc
: Managed Spark and Hadoop service
Big Data Flow - Batch Ingest
- Use extract, transform, and load (ETL) to load data into BigQuery
Dataprep
: Clean and prepare dataDataflow
: Create data pipelines (and ETL)Dataproc
: Complex processing using Spark and Hadoop
Steaming Data
Pub/Sub
: Receive messagesDataflow
: Analyze, aggregate andfilter data- For pre-defined time series analytics, storing data in
Bigtable
- For ad hoc complex analysis, prefer
BigQuery
IOT
IoT Core
: Manage IoT (registration, authentication, and authorization) devicesPub/Sub
: Durable message ingestion service (allows buffering)Dataflow
: Processing data (ETL & more..)- Data Storage and Analytics:
- Make IOT data available to mobile or web apps =>
Datastore
- Execute pre-defined time series queries =>
Bigtable
- More complex or ad hoc analytics/analysis =>
BigQuery
- Make IOT data available to mobile or web apps =>
Data Lake
- Single platform with combination of solutions for data storage, data management and data analytics
- Storage
Cloud Storage
(low cost + durability + performance + flexible processing)
- Data Ingestion
- Streaming data -
Cloud Pub/Sub
+Cloud Dataflow
- Batch - Transfer Service + Transfer Appliance + gsutil
- Streaming data -
- Processing and analytics
- Run in-place querying using SQL queries using
BigQuery
or (Hive on Dataproc)
- Run in-place querying using SQL queries using
- Data Mining and Exploration
- Clean and transform raw data with
Dataprep
- Use
Cloud Datalab
(data science libraries such as TensorFlow and NumPy) for exploring
- Clean and transform raw data with
Data Governance
- Bad data: Bad data leads to poor business decisions
- Data leaks: Data leaks can lead to a reputation loss
Dataplex
Dataplex
is a Data Mesh: Unified dashboard with visibility into all data assets (data lakes, data warehouses, ..)
13. API Management in GCP
API Management
Apigee API Management
: Comprehensive API management platformCloud Endpoints
: Basic API Management for Google Cloud backendsAPI gateway
: Newer, Simpler API Management for Google Cloud backends
14. Trust and Security with GCP
Cloud Security
Control
: Decide who gets accessCompliance
: Follows legal rulesConfidentiality
: Keeps information secretIntegrity
: Ensures data stays accurateAvailability
: Ensure apps & data are available always
Enhanced Security with 2SV
- 2 Step Verification (2SV): Add a 2nd step to verify user
- Make 2SV Mandatory: For Google Cloud accounts
GCP Security Offerings
KMS
: Create and manage cryptographic keys (symmetric and asymmetric). Control their use
in your applications and GCP ServicesSecret Manager
: Manage your database passwords, your API keys securelyCloud Data Loss Prevention
: Discover, classify, & mask sensitive dataCloud Armor
: Protect your production apps (at run time) from denial of service and common web attacks
Web Security Scanner
: Identify vulnerabilities by running security testsBinary Authorization
: Ensure that only trusted container images are deployed to Google CloudContainer Threat Detection
: Detects container runtime attacksSecurity Command Center
: Get a consolidated picture of security in Google Cloud
Zero Trust Security Model
- Zero Trust - “No person or device should be trusted by default, even if they are already inside an organization’s network”
15. ML in GCP
ML in GCP - Pre-Trained
Speech-to-Text API
: convert speech into textText-to-Speech API
: convert text into speechTranslation API
: Translate texts into more than one hundred languagesNatural Language API
: Derive insights from unstructured textCloud Vision API
: Recommended for generic usecases
ML in GCP - Custom Models
AutoML
: Build custom models with minimum ML expertise and effortAutoML Vision
: Build custom models based on ImagesAutoML Video Intelligence
: Add labels to VideoAutoML Tables
: Automatically build models on structured data
BigQuery ML
: Build ML models using QueriesVertex AI
: Build & deploy ML models faster
16. Cloud Native
Cloud Native Pillars
Microservices
: Fix issues and deliver new features quicklyContainers
: Portable & LightweightContainer Orchestration
: Kubernetes (GKE) - Auto Scaling, Load Balancing, Self Healing, Zero Downtime Deployment etcDevOps
(Dev + Ops, CI/CD, IaC): Increased automation of processes
Container Compute Examples
Cloud Run
: Develop and deploy highly scalable containerized applicationsGoogle Kubernetes Engine
: Orchestrate containerized microservices on KubernetesAnthos
: Manage Kubernetes Clusters in Multi-cloud and On-premises
Serverless Examples
Cloud Functions
: Serverless compute for event-driven appsCloud Run
: Run isolated containers, without orchestration (Serverless)Cloud Firestore
: Apps needing quickly evolving structure (schema-less)Cloud Dataflow
: Serverless Stream and Batch processing using Apache BeamCloud Pub/Sub
: Realtime Messaging in the cloud. Pay for number of messagesBigQuery
: Relational OLAP, Data warehousing & BigData workloads
17. Cost Management in GCP
CapEx vs OpEx
- Capital Expenditure (
CapEx
): Money spent to buy infrastructure - Operational Expenditure (
OpEx
): Money spent to use a service or a product
Pricing Calculator
Pricing Calculator
: Estimating the cost of a Google Cloud solution
GCP Cost Management
Cost Management
: Tools for monitoring, controlling, and optimizing your costs
18. GCP review
Basic Compute Services
Compute Engine
: Use VMs when you need control over OS OR you want to run custom softwarePreemptible VMs
: Short lived VMs for non time-critical workloadsSole-tenant Nodes
: Dedicated physical serversVMware Engine
: Run VMware workloads in Google CloudManaged Instance Groups
: Create multiple Compute Engine VMsCloud Load Balancing
: Balance load to multiple instances of an application or a service
Managed Compute Services
App Engine
: PaaS. Deploy web apps and RESTful APIs quicklyCloud Run
: Run isolated containers, without orchestration (Serverless)Kubernetes Engine
: Managed Kubernetes Service. Provides container orchestrationCloud Functions
: Serverless compute for event-driven appsAnthos
: Manage Kubernetes Clusters in Multi-cloud and On-premisesFirebase
: Google’s mobile platform. Build Apps for iOS, Android, the web, C++, and Unity
Storage
Persistent Disk
: Block Storage for your VMsLocal SSD
: Local ephemeral block storage for your VMsCloud Filestore
: File shares in the cloudCloud Storage
: Object storage in the cloud
Databases
Cloud SQL
: Regional Relational OLTP database (MySQL, PostgreSQL, SQL server)Cloud Spanner
: Global Relational OLTP database. Unlimited scale and 99.999% availability for global applications with horizontal scalingCloud Firestore
: Apps needing quickly evolving structure (schema-less)Cloud BigTable
: Large databases(10 TB - PBs). Streaming (IOT), analytical & operational workloads. NOT serverlessCloud Memorystore
: In memory databases/cache. Applications needing microsecond responses
Streams, Analytics, Big Data
Cloud Pub/Sub
: Realtime Messaging in the cloudBigQuery
: Relational OLAP databases. Datawarehousing & BigData workloadsBigQuery ML
: Simplified Machine Learning using data in BigQueryCloud Dataflow
: Serverless Stream and Batch processing using Apache Beam (open-source)Cloud Dataproc
: Managed Service for Spark and Hadoop. Not serverlessCloud Data Fusion
: Visually manage your data pipelinesData Studio
: Visualize dataLooker
: Enterprise Business Intelligence
Migration
Database Migration Service
: Migrate to Cloud SQLStorage Transfer Service
: Online Transfer to Cloud StorageTransfer Appliance
: Physical transfer using an applianceMigrate for Compute Engine
: Migrate VMs and VM storage to GCEMigrate for Anthos
: Migrate VMs to GKE containersBigQuery Data Transfer Service
: Migrate your analytics data