MLA-C02 · Domain 4 of 4

MLA-C02 Domain 4: Operating, Monitoring, and Securing ML and AI Solutions

Domain 4 covers running ML and AI systems after launch: monitoring models, FMs and agents for drift and failures, controlling infrastructure and token costs, and securing pipelines, endpoints, data and credentials with IAM, VPCs, logging and Bedrock Guardrails.

Exam weight
24%
Official task statements
3
Estimated questions estimate
~12
Suggested study time estimate
12–16 hrs

About 12 min read Reviewed October 2026

Overview

Operating, Monitoring, and Securing ML and AI Solutions has three task statements. Task 4.1 covers monitoring inference and performance: CloudWatch generative AI observability, Bedrock evaluations, drift detection, workflow anomalies, data distribution changes, A/B testing and agent coordination failures such as tool errors or truncated streaming.

Task 4.2 covers cost and performance: inference instance families, troubleshooting with CloudWatch, AgentCore Observability and X-Ray, dashboards, capacity planning, cost quotas, purchasing options, FM inference cost, agent resource use and token or embedding costs. Task 4.3 covers security: scanning CI/CD code and images with CodeGuru and Inspector, least privilege, IAM roles, CloudTrail and Config auditing, VPC isolation, threat mitigation, Bedrock API keys versus IAM credentials and Guardrails for sensitive data.

How it's tested

Domain 4 questions describe a model, FM application or agent already in production and a symptom: falling accuracy, rising latency, a cost spike, an audit finding or a data exposure risk. You identify the cause or choose the monitoring, cost or security control. Expect AWS-native, automated answers with least privilege and logging built in.

  • Diagnose production drift

    Accuracy drops or input statistics shift, and you choose Model Monitor data quality, model quality, bias drift or feature attribution drift checks.

  • Control cost and capacity

    Bills or latency rise, and you change instance families, purchasing options, token use, caching or scaling rather than adding capacity blindly.

  • Lock down ML and AI assets

    An audit or threat scenario appears, and you apply least-privilege roles, VPC isolation, encryption, CloudTrail logging, scanning or Guardrails.

Reasoning it through

A churn model's precision falls over three months while the endpoint shows no errors. Ground truth arrives weekly. Endpoint health metrics in CloudWatch will not reveal this, because the service is up. A Model Monitor data quality schedule compares live inputs with the training baseline, and a model quality schedule merges ground truth labels to measure precision. An alarm on violations can trigger retraining.

Task statements 3

Official task statements from the Amazon Web Services outline, in outline order.

4.1 Monitor ML and AI model inference and performanceProduction performance, workflow anomalies, distribution changes, A/B tests, agent coordination and FM monitoring.
What it means
This task statement covers monitoring production performance with CloudWatch generative AI observability, Bedrock Model Evaluation and drift detection pipelines, detecting anomalies or errors in data processing and inference, spotting changes in data distribution, A/B testing in production, monitoring and automating agent performance and coordination including tool failures and truncated streaming, and configuring AI-specific monitoring for foundation models.
What to understand
SageMaker Model Monitor runs scheduled jobs against a baseline built from training data, with four monitor types: data quality, model quality, bias drift and feature attribution drift. Data capture on the endpoint stores requests and responses in S3 for these jobs. Model quality needs ground truth labels merged later. Production variants split traffic for A/B tests and report per-variant metrics. For GenAI, CloudWatch generative AI observability tracks invocations, latency, token counts and errors across models and agents, and Bedrock evaluations can be rerun on samples of production prompts. Agent traces reveal failed tool calls, loops and cut-off streams.
How it shows up
A production model, FM or agent shows a symptom, and you choose the monitor, metric or test that detects or explains it.
Common confusion
Data drift is a change in inputs; concept drift is a change in the input-to-label relationship that only ground truth reveals.
4.2 Optimize and manage ML and AI infrastructure costs and performanceInstance families, troubleshooting tools, dashboards, capacity, cost tools, purchasing options, FM and agent costs and token use.
What it means
This task statement covers choosing inference instance families, troubleshooting with CloudWatch, AgentCore Observability and X-Ray, building performance dashboards, optimizing capacity for cost and reliability, setting cost quotas with cost management tools, choosing purchasing options, evaluating FM inference cost in production, monitoring agent resource use, managing FM usage and tracking token, embedding and vector storage costs.
What to understand
Match hardware to the model: CPU instances for light models, GPU families such as G and P for heavy deep learning, and AWS Inferentia instances for cost-efficient high-throughput inference. Inference Recommender benchmarks instance types. Savings Plans for SageMaker AI lower cost for steady usage, and Spot capacity with checkpointing cuts training cost. AWS Budgets, Cost Explorer and cost allocation tags track and cap spend. For FMs, shorter prompts, prompt caching, smaller or distilled models, batch inference and intelligent prompt routing reduce token cost. Vector storage cost depends on dimensions, index type and the chosen store.
How it shows up
Cost or performance is off target, and you select the instance, purchasing option, FM usage change or tool that fixes it.
Common confusion
Spot capacity suits interruptible training with checkpoints, not latency-sensitive real-time endpoints.
4.3 Secure ML and AI workloads and model endpointsPipeline scanning, least privilege, IAM, auditing, troubleshooting, VPC isolation, threat mitigation, FM credentials and Guardrails.
What it means
This task statement covers scanning CI/CD code and container images with Amazon CodeGuru and Amazon Inspector, least-privilege access to ML and AI artifacts, IAM policies and roles, monitoring and compliance with CloudTrail and Config, troubleshooting security issues, VPCs, subnets and security groups for isolation, mitigating risks and vulnerabilities, choosing Bedrock API keys or IAM credentials, and protecting sensitive data with Bedrock Guardrails.
What to understand
SageMaker AI jobs and endpoints assume execution roles, so permissions should be scoped to specific S3 prefixes, ECR repositories and KMS keys. Encrypt data at rest with AWS KMS and in transit with TLS; enable inter-container traffic encryption for distributed training. Network isolation and VPC-only settings keep jobs off the internet, with interface endpoints for AWS APIs. CloudTrail records API calls, while AWS Config tracks resource settings against rules. Inspector scans ECR images. Bedrock API keys are convenient for short-term exploration; IAM roles with temporary credentials are preferred for production. Guardrails filter harmful content, deny topics and mask PII.
How it shows up
A security requirement or incident is described, and you choose the IAM, network, logging, scanning or guardrail control that meets it.
Common confusion
CloudTrail answers who called which API; CloudWatch collects metrics and logs; Config records resource configuration history.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • SageMaker Model Monitor

    Scheduled checks against a training baseline.

    Remember
    Data quality, model quality, bias drift, feature attribution drift.
    Tested as
    Drift questions.
    Trap
    Expecting it to work without data capture.

  • Data capture

    Endpoint option that logs requests and responses to S3.

    Remember
    Prerequisite for monitoring jobs.
    Tested as
    Monitoring setup questions.
    Trap
    Forgetting to enable it.

  • Production variants

    Multiple models behind one endpoint with traffic weights.

    Remember
    Supports A/B testing.
    Tested as
    Comparison questions.
    Trap
    Confusing it with shadow testing.

  • CloudWatch generative AI observability

    Metrics, traces and dashboards for model and agent invocations.

    Remember
    Tokens, latency and errors in one view.
    Tested as
    GenAI monitoring questions.
    Trap
    Building custom log parsing.

  • AgentCore Observability

    Traces and metrics for agent sessions and tool calls.

    Remember
    Finds tool failures and loops.
    Tested as
    Agent troubleshooting questions.
    Trap
    Relying only on endpoint metrics.

  • AWS X-Ray

    Distributed tracing across services.

    Remember
    Locates latency in multi-step calls.
    Tested as
    Troubleshooting questions.
    Trap
    Using it for model drift.

  • Instance family choice

    CPU, GPU (G, P) or Inferentia for inference.

    Remember
    Inference Recommender benchmarks options.
    Tested as
    Cost-performance questions.
    Trap
    Defaulting to the largest GPU.

  • Purchasing options

    On-demand, SageMaker Savings Plans and Spot training.

    Remember
    Spot needs checkpoints.
    Tested as
    Cost questions.
    Trap
    Running real-time endpoints on Spot.

  • FM token cost controls

    Shorter prompts, caching, smaller models, batch and routing.

    Remember
    Cost scales with input and output tokens.
    Tested as
    FM cost questions.
    Trap
    Adding capacity to fix a token bill.

  • Least-privilege execution roles

    Roles scoped to needed buckets, repos and keys.

    Remember
    One role per job type.
    Tested as
    IAM questions.
    Trap
    Attaching full-access managed policies.

  • CloudTrail vs. Config

    API call history vs. resource configuration history.

    Remember
    Both support audits.
    Tested as
    Compliance questions.
    Trap
    Mixing either with CloudWatch metrics.

  • Bedrock API keys vs. IAM credentials

    Simple bearer keys vs. role-based temporary credentials.

    Remember
    Prefer IAM roles in production.
    Tested as
    Credential choice questions.
    Trap
    Embedding long-term keys in apps.

  • Amazon Bedrock Guardrails

    Content filters, denied topics, PII masking and grounding checks.

    Remember
    Applies to inputs and outputs.
    Tested as
    Sensitive data protection questions.
    Trap
    Treating it as access control.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

Model Monitor types

ConceptMeaningKey differenceCommon trap
Data quality Input statistics vs. baseline Detects data drift Expecting it to measure accuracy
Model quality Prediction metrics vs. ground truth Needs labels Running it without ground truth
Bias drift Fairness metrics over time Uses Clarify Checking bias only before training
Feature attribution drift Change in feature importance Uses SHAP baselines Confusing it with data quality

Logging and audit services

ConceptMeaningKey differenceCommon trap
CloudWatch Metrics, logs and alarms Operational health Using it as the API audit trail
CloudTrail Record of API calls Who did what, when Expecting performance metrics
AWS Config Resource configuration history and rules Compliance drift Expecting request-level detail

FM access credentials

ConceptMeaningKey differenceCommon trap
Bedrock API keys Bearer keys for Bedrock calls Quick start and exploration Using long-term keys in production
IAM roles Temporary credentials by role Least privilege and rotation built in Sharing one role across all apps

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. CloudWatch shows drift

    What candidates pick
    Use CloudWatch endpoint metrics to detect a drop in prediction quality.
    Why it looks right
    CloudWatch is the default monitoring service.
    Why it's wrong
    Invocation and latency metrics show health, not correctness; Model Monitor data and model quality jobs detect drift and accuracy loss.

    Clue to watch for The endpoint has no errors but results worsen.

  2. Model quality without labels

    What candidates pick
    Schedule model quality monitoring when no ground truth is collected.
    Why it looks right
    It sounds like the direct accuracy check.
    Why it's wrong
    Model quality monitoring merges ground truth labels; without them, only data quality and drift monitors can run.

    Clue to watch for Labels arrive later or not at all.

  3. Spot for live inference

    What candidates pick
    Run a real-time endpoint on Spot capacity to save money.
    Why it looks right
    Spot is the cheapest option.
    Why it's wrong
    Spot capacity can be reclaimed, so it fits checkpointed training jobs; steady endpoints save with Savings Plans or right-sized instances.

    Clue to watch for Latency-sensitive production traffic.

  4. Scale out for token cost

    What candidates pick
    Add provisioned throughput because the Bedrock bill doubled.
    Why it looks right
    More capacity seems to fix demand.
    Why it's wrong
    On-demand cost follows tokens; trimming prompts, caching, routing to smaller models or batching cuts spend, while extra capacity adds cost.

    Clue to watch for Spend rose but request counts did not.

  5. Guardrails as access control

    What candidates pick
    Use Bedrock Guardrails to stop unauthorized users from calling a model.
    Why it looks right
    Guardrails is a security-sounding feature.
    Why it's wrong
    Guardrails filter content in prompts and responses; who may invoke a model is controlled by IAM policies and credentials.

    Clue to watch for The problem is about identity or permissions.

  6. API keys everywhere

    What candidates pick
    Embed a long-term Bedrock API key in a production application.
    Why it looks right
    Keys are simple to configure.
    Why it's wrong
    Long-term keys are hard to rotate and scope; production workloads should use IAM roles with temporary credentials.

    Clue to watch for Production, least privilege or rotation.

  7. CloudTrail for configuration drift

    What candidates pick
    Use CloudTrail to prove endpoints always ran in a VPC.
    Why it looks right
    CloudTrail records changes.
    Why it's wrong
    AWS Config evaluates and records resource configuration over time against rules, which is what continuous compliance evidence needs.

    Clue to watch for Continuous compliance or configuration rules.

Answer framework

Use these steps for Domain 4 questions about monitoring, cost and security in production.

  1. Classify the symptom first: quality decline, latency or errors, cost growth, an audit finding or a data exposure risk, because each one leads to a different toolset.
  2. For quality decline, enable data capture and choose the Model Monitor type, adding ground truth for model quality and Bedrock evaluations for FMs.
  3. For latency or failures, trace with CloudWatch, X-Ray and AgentCore Observability before changing capacity or instance families.
  4. For cost, check tokens, prompt length, model size, purchasing option and idle capacity, and set Budgets alerts and quotas.
  5. For security, scope execution roles, isolate with VPCs, encrypt with KMS, log with CloudTrail and Config, and add Guardrails for content.

Practice question

A company's Bedrock-based support agent doubled its monthly cost although the number of user conversations stayed flat. Traces show some sessions call the same search tool dozens of times before answering. What should the ML engineer do first?

Want more like this? The MLA-C02 practice test has questions for every domain.

Checklist

0 of 15

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

Treat this domain as three short courses. For monitoring, deploy a small endpoint with data capture, create a Model Monitor baseline, and schedule a data quality job. Read how model quality, bias drift and feature attribution drift differ, and write one scenario for each. Then open CloudWatch generative AI observability on a Bedrock application or agent and note which metrics appear for tokens, latency and errors.

For cost, build a list of levers in the order you would check them: idle endpoints, instance family, purchasing option, scaling policy, and for FMs, prompt length, caching, model size, batch inference and agent iteration limits. Run Inference Recommender once if you can, and set up an AWS Budgets alert with cost allocation tags.

For security, write least-privilege policies for a training role and an inference role, scoped to one bucket prefix and one KMS key. Configure a VPC-only training job. Compare CloudTrail, CloudWatch and Config until you can say which answers each audit question. Finish by creating a Bedrock guardrail with PII masking and a denied topic.

FAQ

How much of MLA-C02 is Domain 4?

Operating, Monitoring, and Securing ML and AI Solutions is 24% of scored content, about twelve of fifty scored questions, across three task statements, the same weight as in MLA-C01.

What are the four SageMaker Model Monitor types?

Data quality, model quality, bias drift and feature attribution drift. Each compares live data or predictions with a baseline on a schedule.

How do I monitor a foundation model in production?

Use CloudWatch generative AI observability for invocations, latency, tokens and errors, and run Bedrock evaluations on sampled production prompts to track output quality.

What agent problems does this domain mention?

Coordination failures, truncated streaming and tool failures, plus agent resource consumption. AgentCore Observability and CloudWatch traces help find them.

Should I use Bedrock API keys or IAM credentials?

API keys suit quick exploration. Production applications should use IAM roles that issue temporary, scoped credentials and support least privilege and rotation.

What security scanning does the exam guide mention?

Checking CI/CD pipelines for code and image vulnerabilities, with Amazon CodeGuru and Amazon Inspector given as examples.

What did MLA-C02 remove from Domain 4?

The comparison page lists infrastructure monitoring with EventBridge events and troubleshooting capacity concerns such as provisioned concurrency and service quotas as deletions.

Other Artificial Intelligence & Machine Learning certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 4 knowledge

You have covered production monitoring, drift, agent observability, cost optimization and ML security controls. Check whether you can find the cause behind each symptom.

Take the MLA-C02 practice test Check your MLA-C02 readiness

Keep going