Overview
Serving and scaling models has two official objectives. Task 4.1 covers serving: deploying models for batch and online inference on Agent Platform, Model Garden, Cloud Run and GKE, packaging PyTorch, XGBoost and other models in prebuilt or custom containers, organizing versions in Model Registry, comparing versions with A/B tests and canary rollouts, and building pre- and post-processing logic. Task 4.2 covers scaling online serving: serving features from Feature Store, choosing public or private endpoints, picking CPU, GPU, TPU or edge hardware, scaling on throughput, and tuning models for production.
At about 20% of the exam, this is the second-heaviest section. Questions focus on meeting latency, cost and reliability targets once a model leaves the training environment.
How it's tested
Domain 4 questions state how predictions are consumed, how fast they must return, how traffic behaves and what network rules apply. You choose the inference mode, the serving platform, the rollout method, the endpoint type or the scaling setting. Read the timing and network clues before anything else.
-
Batch or online
Prediction timing and volume are given and you choose batch prediction or an online endpoint.
-
Safe rollout
A new model version must be introduced with limited risk, and you choose traffic splitting, canary or A/B testing.
-
Meet the latency target
An endpoint is too slow or costly under load, and you adjust hardware, replicas, model size or feature lookup.
A bank must score every account for credit risk once a night, writing results to BigQuery before 6 a.m. An online endpoint could loop over accounts but keeps nodes running and adds request overhead. Streaming each account through Dataflow to an endpoint is complex. Batch prediction reads from BigQuery, scales out temporarily and writes results back, so batch prediction is the expected answer.
Objectives 2
Official objectives from the Google Cloud outline, in outline order.
4.1 Serving modelsBatch and online inference, prebuilt and custom containers, Model Registry, rollout strategies and inference pre- and post-processing.
- What it means
- This task covers getting a trained model to answer requests. You deploy models for batch and online inference using Agent Platform, Model Garden, Cloud Run or GKE, package models from frameworks such as PyTorch and XGBoost in prebuilt or custom containers, organize and version models in Model Registry, compare versions with A/B tests and canary rollouts, and implement processing before and after the model call.
- What to understand
- Batch prediction suits large, scheduled jobs with no user waiting; online endpoints suit real-time requests. Prebuilt serving containers cover TensorFlow, PyTorch, scikit-learn and XGBoost; custom containers are needed for other runtimes or extra logic, and must expose health and predict routes. Model Registry tracks versions, aliases such as default, and evaluation results. An endpoint can host several deployed models with a traffic split, which supports canary releases and A/B comparisons. Cloud Run fits lightweight models with bursty traffic; GKE fits teams needing full control, such as serving open LLMs with vLLM.
- How it shows up
- A consumption pattern, framework or release risk is described, and you choose the inference mode, container type, platform or rollout method.
- Common confusion
- Thinking a new model version needs a new endpoint; versions can share one endpoint with a traffic split.
4.2 Scaling online model servingFeature Store serving, public and private endpoints, serving hardware, throughput-based scaling and production tuning.
- What it means
- This task covers keeping online predictions fast and reliable as traffic grows. You manage and serve features with Feature Store, deploy to public or private endpoints, choose CPU, GPU, TPU or edge hardware for serving, scale backends based on throughput, and tune models for production serving.
- What to understand
- Online serving needs features at low latency, so read precomputed values from Feature Store online serving rather than querying BigQuery per request. Private endpoints through Private Service Connect keep traffic off the public internet for regulated workloads. Endpoints autoscale replicas between a minimum and maximum based on CPU or accelerator use; a minimum above zero avoids cold starts. Edge deployment suits offline or ultra-low-latency devices. Production tuning includes quantization, distillation, compiling models and batching requests to raise throughput per accelerator.
- How it shows up
- Latency, traffic, network or device constraints are described and you choose the feature lookup, endpoint type, hardware or scaling setting.
- Common confusion
- Querying BigQuery at request time for features, which adds seconds of latency to each prediction.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Batch prediction
Asynchronous scoring of large inputs from BigQuery or Cloud Storage.
- Remember
- No endpoint needed; results written back.
- Tested as
- Scheduled scoring scenarios.
- Trap
- Using an online endpoint for nightly jobs.
-
Online endpoint
Deployed model that answers requests in real time.
- Remember
- Billed while nodes run.
- Tested as
- Real-time prediction scenarios.
- Trap
- Leaving it running for rare batch use.
-
Prebuilt serving container
Google-provided images for common frameworks.
- Remember
- Covers TensorFlow, PyTorch, scikit-learn and XGBoost.
- Tested as
- Packaging questions.
- Trap
- Building a custom image without need.
-
Custom serving container
Your own image with any runtime and logic.
- Remember
- Must implement health and prediction routes.
- Tested as
- Unusual framework or extra logic scenarios.
- Trap
- Forgetting the required routes.
-
Model Registry
Central store of model versions with aliases and evaluations.
- Remember
- Supports versioning and promotion.
- Tested as
- Governance and release questions.
- Trap
- Tracking versions in file names instead.
-
Traffic split
Percentages of requests sent to each deployed model on an endpoint.
- Remember
- Enables canary and A/B testing.
- Tested as
- Rollout questions.
- Trap
- Creating separate endpoints for each version.
-
Canary rollout
Send a small share of traffic to a new version first.
- Remember
- Limits blast radius; increase gradually.
- Tested as
- Safe deployment questions.
- Trap
- Switching all traffic at once.
-
Private endpoint
Endpoint reachable only through private networking.
- Remember
- Uses Private Service Connect.
- Tested as
- Security and compliance scenarios.
- Trap
- Assuming IAM alone removes public exposure.
-
Autoscaling replicas
Endpoint nodes scale between minimum and maximum counts.
- Remember
- Minimum replicas avoid cold starts.
- Tested as
- Traffic spike questions.
- Trap
- Setting max equal to min and expecting scaling.
-
Online feature serving
Low-latency feature reads from Feature Store.
- Remember
- Same definitions as training features.
- Tested as
- Latency and skew questions.
- Trap
- Calling BigQuery per request.
-
Model optimization for serving
Quantization, distillation and compilation.
- Remember
- Lower latency and cost per prediction.
- Tested as
- Production tuning questions.
- Trap
- Only adding hardware.
-
Edge deployment
Running models on devices outside the cloud.
- Remember
- Works offline with lowest latency.
- Tested as
- Device scenarios.
- Trap
- Choosing cloud endpoints for offline devices.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Inference modes
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Batch prediction | Scores a data set as a job. | Cheap for large scheduled work; no instant answer. | Using it when a user waits. |
| Online prediction | Real-time responses from an endpoint. | Low latency but always-on cost. | Using it for overnight bulk scoring. |
Serving platforms
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Agent Platform endpoints | Managed model serving with autoscaling and traffic splits. | Least operations work with built-in monitoring. | Assuming it cannot serve custom containers. |
| Cloud Run | Serverless containers that scale to zero. | Good for light, bursty models. | Expecting large accelerator fleets. |
| GKE | Kubernetes clusters you manage. | Most control, such as LLM serving with vLLM. | Choosing it when the team wants no cluster management. |
Release strategies
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Canary | Small traffic share to the new version first. | Focus is risk reduction. | Treating it as a statistical test. |
| A/B test | Split traffic to compare versions on business metrics. | Focus is choosing the better model. | Ending it before enough traffic is collected. |
| Shadow deployment | New version gets copies of requests but its answers are not used. | Zero user impact while measuring behavior. | Thinking users see shadow predictions. |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Endpoint for nightly scoring
- What candidates pick
- Deploy an online endpoint and call it in a loop
- Why it looks right
- Endpoints are the familiar serving path.
- Why it's wrong
- Nightly bulk scoring has no user waiting, so batch prediction is cheaper and simpler.
Clue to watch for A schedule and a large table to score.
-
New endpoint per version
- What candidates pick
- Create a separate endpoint for the new model
- Why it looks right
- It keeps versions isolated.
- Why it's wrong
- Clients would need to change URLs; one endpoint with a traffic split supports gradual rollout.
Clue to watch for Gradual or reversible rollout is required.
-
Public endpoint with IAM
- What candidates pick
- Keep the public endpoint and rely on IAM
- Why it looks right
- IAM already blocks unauthorized callers.
- Why it's wrong
- The requirement is no traffic over the public internet, which needs a private endpoint.
Clue to watch for Data must stay on private networks.
-
GPU for a small model
- What candidates pick
- Serve a gradient-boosted tree model on GPUs
- Why it looks right
- GPUs sound faster.
- Why it's wrong
- Small tree models run efficiently on CPUs, so GPUs add cost without improving latency.
Clue to watch for Small tabular model with modest traffic.
-
Custom container by default
- What candidates pick
- Build a custom serving image for an XGBoost model
- Why it looks right
- It gives full control.
- Why it's wrong
- A prebuilt XGBoost container already serves it with less effort.
Clue to watch for Standard framework, no extra logic.
-
Scale-to-zero for steady traffic
- What candidates pick
- Set minimum replicas to zero to save cost
- Why it looks right
- It reduces idle spending.
- Why it's wrong
- Steady or latency-sensitive traffic then suffers cold starts; keep a minimum replica count.
Clue to watch for Strict latency target during business hours.
Answer framework
Use these steps to choose how a model should be served and scaled.
- Decide on timing first: if no user is waiting and inputs arrive as a data set, use batch prediction; otherwise deploy an online endpoint.
- Pick the platform: managed Agent Platform endpoints by default, Cloud Run for light bursty models, GKE when you need full control or open LLM serving.
- Package with a prebuilt container for standard frameworks and a custom container only for other runtimes or added processing logic.
- Register each version in Model Registry and release through a traffic split, starting with a small canary share before increasing.
- Meet latency and scale targets with online feature serving, private endpoints where required, right-sized hardware, minimum replicas and model optimization.
Practice question
Want more like this? The GCP PMLE practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Practice the full serving path once. Upload a small scikit-learn or XGBoost model to Model Registry, deploy it to an Agent Platform endpoint with a prebuilt container, and send test requests. Then deploy a second version to the same endpoint with a 10% traffic split, and run a batch prediction job from a BigQuery table. These three actions cover most of task 4.1.
For task 4.2, read the documentation on private endpoints, autoscaling settings and online feature serving, and write down the clue that points to each. Compare Agent Platform endpoints, Cloud Run and GKE in a short table covering operations effort, accelerator support and scale-to-zero.
Since this section is about 20% of the exam, revisit it twice: once after studying training hardware in Section 3, and again after pipelines in Section 5, because deployment steps reappear inside CI/CD questions. Use timed practice to get fast at spotting batch versus online clues.
FAQ
How much of the GCP PMLE exam is Serving and scaling models?
Serving and scaling models is about 20% of the exam, the second-largest section, or roughly 10 to 12 questions.
When should I use batch prediction?
When predictions are needed for a whole data set on a schedule and no user waits for an immediate answer.
Can two model versions share one endpoint?
Yes. An Agent Platform endpoint can host several deployed models and split traffic between them by percentage.
What is the difference between canary and A/B testing?
A canary limits risk by exposing a small share of traffic first; an A/B test compares versions on metrics to pick a winner.
When do I need a custom serving container?
When the framework is not covered by prebuilt containers or you need extra processing logic inside the serving image.
How do I avoid cold starts?
Keep a minimum number of replicas above zero on the endpoint, sized for baseline traffic.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 4 knowledge
You have covered inference modes, containers, Model Registry, rollouts and endpoint scaling. Try exam-style scenarios to confirm you can meet latency and risk targets.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the GCP PMLE AI tutor Get any concept explained your way