MLA-C02 · Domain 2 of 4

MLA-C02 Domain 2: ML Model and Foundation Model (FM) Development

Domain 2 covers choosing between ML models, AWS AI services and Bedrock foundation models, then training, tuning, customizing and evaluating them with SageMaker AI, Bedrock evaluations and metrics such as ROUGE, BERTScore and LLM-as-a-judge.

Exam weight
24%
Official task statements
3
Estimated questions estimate
~12
Suggested study time estimate
14–18 hrs

About 12 min read Reviewed October 2026

Overview

ML Model and Foundation Model (FM) Development has three task statements. Task 2.1 is about choosing a modeling approach: selecting Bedrock FMs, fine-tuning strategies, built-in algorithms or AWS AI services such as Textract, Rekognition, Comprehend and Transcribe, RAG architecture patterns, and tradeoffs between performance, latency, training time and cost.

Task 2.2 covers training and customization: SageMaker AI built-in algorithms, script mode, automatic model tuning, early stopping, distributed training, overfitting and catastrophic forgetting, ensembles, hyperparameters, prompt engineering, fine-tuning and retrieval tuning. Task 2.3 covers evaluation: reproducible experiments with MLflow, baselines and drift, shadow variants, explainability, convergence debugging, NLP metrics, human review and RAG retrieval accuracy.

How it's tested

Domain 2 questions describe a business problem, a dataset and constraints such as interpretability, latency, budget or training time. You choose the model family, the customization method, the training configuration or the evaluation metric. Correct answers often pick the simplest managed option that meets the requirement before any custom training.

  • Service, model or FM

    A task such as invoice extraction or sentiment analysis is described, and you decide between an AWS AI service, a SageMaker algorithm and a Bedrock FM.

  • Customize the FM

    An FM gives generic or outdated answers, and you choose prompt engineering, RAG, fine-tuning or continued pre-training based on what is missing.

  • Read the training signal

    Loss curves, metrics or evaluation scores are shown, and you diagnose overfitting, poor convergence or a weak retrieval step.

Reasoning it through

A legal team wants an assistant that answers questions about contracts updated every week. Fine-tuning would bake in last month's text and need retraining after each change. RAG with a Bedrock knowledge base retrieves current clauses at query time and can cite sources. Fine-tuning becomes worth it only if the model also needs a specific style or task behavior that prompts cannot achieve.

Task statements 3

Official task statements from the Amazon Web Services outline, in outline order.

2.1 Choose appropriate modeling approaches for ML and AI solutionsSelecting FMs, fine-tuning strategies, algorithms, managed services, RAG patterns, AI services and cost tradeoffs.
What it means
This task statement covers evaluating Bedrock FMs against task and performance criteria, identifying fine-tuning strategies, comparing ML models, GenAI models, algorithms and solution templates, weighing custom builds against managed services, pre-trained models and FMs, selecting RAG patterns, balancing performance with training time, latency and cost, and applying services such as Textract, Rekognition, Comprehend and Transcribe.
What to understand
Start with the least effort that meets the need: an AI service such as Textract for forms or Comprehend for entities needs no training. For tabular prediction, XGBoost and Linear Learner are strong, explainable choices. FMs on Bedrock are compared by modality, context window, latency, price and quality on your own prompts through model evaluation. Fine-tuning adapts behavior or format with labeled data; parameter-efficient methods such as LoRA cost less than full fine-tuning on SageMaker AI. RAG suits private or changing knowledge. Smaller or distilled models lower latency and cost when quality holds.
How it shows up
A business need and constraints are given, and you select the AI service, algorithm, FM or customization strategy that fits with least effort.
Common confusion
RAG adds knowledge at query time; fine-tuning changes model behavior and does not keep facts current.
2.2 Train, fine-tune, and customize models for ML and AI solutionsBuilt-in algorithms, script mode, AMT, faster training, fit problems, ensembles, hyperparameters, prompt engineering and retrieval tuning.
What it means
This task statement covers SageMaker AI built-in algorithms and common libraries, script mode with frameworks such as PyTorch, TensorFlow and scikit-learn, automatic model tuning, early stopping and distributed training, preventing overfitting, underfitting and catastrophic forgetting, ensembles, epochs, steps and batch size, prompt engineering and fine-tuning for AI solutions, and optimizing retrieval components and embedding models.
What to understand
Script mode runs your own training script inside a prebuilt framework container, so you avoid building images. Automatic model tuning searches hyperparameter ranges with Bayesian, random, grid or Hyperband strategies against an objective metric. Early stopping ends weak trials or epochs; distributed data parallel splits batches across GPUs, while model parallel splits large models. Regularization, dropout and more data reduce overfitting; more features or capacity reduce underfitting. Catastrophic forgetting during fine-tuning is reduced by mixing general data, lower learning rates or parameter-efficient methods. Retrieval improves with better chunking, hybrid search, reranking and a suitable embedding model.
How it shows up
A training job is slow, unstable or poorly fitted, or an FM needs adapting, and you pick the training feature, setting or customization step.
Common confusion
Catastrophic forgetting is loss of earlier general ability after fine-tuning, not the same as overfitting to training data.
2.3 Analyze and evaluate the performance of ML and AI systemsExperiments, baselines and drift, shadow testing, explainability, convergence, evaluation metrics, human review and RAG monitoring.
What it means
This task statement covers reproducible experiments with MLflow on SageMaker AI, Bedrock evaluations and Prompt Management, performance baselines and drift detection, comparing shadow variants with production, explaining outputs, debugging convergence, evaluating traditional ML and GenAI models, human-in-the-loop evaluation, NLP metrics such as BLEU, ROUGE, BERTScore and semantic similarity, AI evaluation including LLM-as-a-judge, and RAG retrieval accuracy monitoring.
What to understand
Classification uses precision, recall, F1 and AUC; regression uses RMSE or MAE. ROUGE measures overlap for summaries, BLEU for translation, and BERTScore or embedding similarity compares meaning. Bedrock evaluations offer automatic metrics, human review teams and LLM-as-a-judge, plus RAG evaluation of retrieval and generation quality. MLflow tracks parameters, metrics and artifacts so runs can be reproduced. SageMaker Clarify gives SHAP-based feature attributions. SageMaker Debugger flags vanishing gradients and stalled loss. Shadow testing sends copies of live traffic to a new variant without affecting users.
How it shows up
Results from a model or GenAI system are presented, and you choose the metric, evaluation method or debugging tool that answers the question.
Common confusion
BLEU and ROUGE reward word overlap, so a correct paraphrase can score poorly; use semantic metrics or human review.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • SageMaker AI built-in algorithms

    Managed algorithms such as XGBoost, Linear Learner, K-Means and DeepAR.

    Remember
    Each expects specific input formats.
    Tested as
    Algorithm selection questions.
    Trap
    Writing custom code for a solved problem.

  • Script mode

    Your training script in a prebuilt framework container.

    Remember
    No custom image needed.
    Tested as
    Framework training questions.
    Trap
    Building a container when script mode suffices.

  • Automatic model tuning

    Managed hyperparameter search against an objective metric.

    Remember
    Bayesian and Hyperband use fewer trials than grid.
    Tested as
    Tuning questions.
    Trap
    Tuning without an objective metric.

  • Distributed training

    Data parallel splits data; model parallel splits the model.

    Remember
    Model parallel for models too big for one GPU.
    Tested as
    Training speed questions.
    Trap
    Choosing model parallel for a small model.

  • Overfitting vs. underfitting

    Memorizing training data vs. missing the pattern.

    Remember
    Gap between train and validation shows overfitting.
    Tested as
    Learning curve questions.
    Trap
    Adding layers to an overfit model.

  • Catastrophic forgetting

    Loss of earlier capability after fine-tuning.

    Remember
    Mitigate with mixed data, low learning rate or PEFT.
    Tested as
    FM fine-tuning questions.
    Trap
    Calling it underfitting.

  • RAG vs. fine-tuning

    Retrieve knowledge at runtime vs. change model weights.

    Remember
    RAG for freshness and citations.
    Tested as
    Customization choice questions.
    Trap
    Fine-tuning to add weekly-changing facts.

  • Model distillation

    A smaller student model learns from a larger teacher.

    Remember
    Cuts latency and cost.
    Tested as
    Cost tradeoff questions.
    Trap
    Expecting it to add new knowledge.

  • ROUGE, BLEU, BERTScore

    Overlap metrics for summaries and translation, plus a semantic metric.

    Remember
    Match the metric to the task.
    Tested as
    Evaluation metric questions.
    Trap
    Using accuracy for text generation.

  • LLM-as-a-judge

    One model scores another model's outputs against criteria.

    Remember
    Available in Bedrock evaluations.
    Tested as
    GenAI evaluation questions.
    Trap
    Treating it as a replacement for all human review.

  • MLflow on SageMaker AI

    Managed experiment tracking and model registry integration.

    Remember
    Makes runs reproducible.
    Tested as
    Experiment questions.
    Trap
    Tracking results in spreadsheets.

  • SageMaker Debugger

    Monitors tensors and rules during training.

    Remember
    Detects vanishing gradients and stalled loss.
    Tested as
    Convergence questions.
    Trap
    Waiting until training ends to inspect loss.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

FM customization options

ConceptMeaningKey differenceCommon trap
Prompt engineering Instructions and examples in the prompt No training, fastest to try Skipping it before fine-tuning
RAG Retrieve documents at query time Current, citable knowledge Expecting it to change tone or format
Fine-tuning Train on labeled prompt-completion pairs Changes behavior for a task Using it to keep facts current
Continued pre-training Train on unlabeled domain text Teaches domain language Supplying labeled pairs
Distillation Teacher model trains a smaller student Lower cost and latency Assuming quality always holds

Text evaluation metrics

ConceptMeaningKey differenceCommon trap
ROUGE Recall of overlapping n-grams Summarization Rewarding verbatim copying
BLEU Precision of overlapping n-grams Translation Using it for open-ended chat
BERTScore or semantic similarity Embedding-based meaning match Credits paraphrases Treating it as a factuality check
Human or LLM-as-a-judge review Scores against rubrics such as helpfulness Covers subjective quality Skipping a rubric

Training speedups

ConceptMeaningKey differenceCommon trap
Early stopping Stops when validation stops improving Saves time and reduces overfitting Stopping on training loss
Data parallel Same model on many GPUs, split batches Large datasets Using it for oversized models
Model parallel Model split across devices Models too big for one GPU Applying it to small models

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. Fine-tune for freshness

    What candidates pick
    Fine-tune a Bedrock model each week with new policy documents.
    Why it looks right
    Fine-tuning sounds like the strongest customization.
    Why it's wrong
    Weights capture a snapshot and retraining is slow and costly; RAG retrieves current documents at query time and can cite them.

    Clue to watch for Content changes often or answers need sources.

  2. Grid search everywhere

    What candidates pick
    Run grid search over many hyperparameters to find the best model.
    Why it looks right
    Grid search feels thorough.
    Why it's wrong
    Grid search grows exponentially with parameters; Bayesian or Hyperband tuning in automatic model tuning finds good values in fewer jobs.

    Clue to watch for Limited budget or many hyperparameters.

  3. Accuracy on skewed classes

    What candidates pick
    Report 98% accuracy as proof a rare-event model works.
    Why it looks right
    Accuracy is the most familiar metric.
    Why it's wrong
    With rare positives, a model predicting only the majority scores high accuracy; recall, precision, F1 or PR AUC show real performance.

    Clue to watch for Rare positives such as fraud or defects.

  4. Overfitting means more layers

    What candidates pick
    Add layers to a model whose validation loss rises while training loss falls.
    Why it looks right
    Bigger models often perform better.
    Why it's wrong
    That pattern is overfitting; the fix is regularization, dropout, more data or early stopping, not more capacity.

    Clue to watch for Training and validation curves diverge.

  5. Custom model before AI service

    What candidates pick
    Train a custom OCR model to read invoices.
    Why it looks right
    Custom models feel more precise.
    Why it's wrong
    Amazon Textract extracts text, forms and tables without training; a custom model adds cost and time unless Textract fails a stated need.

    Clue to watch for Least development effort.

  6. BLEU for summaries

    What candidates pick
    Use BLEU to evaluate a summarization model.
    Why it looks right
    BLEU is the best-known text metric.
    Why it's wrong
    BLEU is precision-oriented and designed for translation; ROUGE is standard for summaries, with semantic metrics or human review for meaning.

    Clue to watch for The task is summarization.

Answer framework

Use these steps for Domain 2 questions about model choice, training and evaluation.

  1. Check whether an AWS AI service such as Textract or Comprehend, or an existing FM with good prompts, already solves the problem before proposing any training.
  2. If knowledge is missing or changes often, choose RAG; if behavior or format is wrong, consider fine-tuning or continued pre-training.
  3. For custom training, pick a built-in algorithm or script mode, then tune with automatic model tuning against one clear objective metric.
  4. Read the learning curves: diverging validation loss means overfitting, flat high loss means underfitting or a convergence problem.
  5. Evaluate with a task-appropriate metric, adding human or LLM-as-a-judge review for generated text and retrieval checks for RAG.

Practice question

A team fine-tuned a foundation model on support transcripts. It now answers support questions well but fails basic general reasoning tasks it handled before training. What is the most likely cause and the best mitigation?

Want more like this? The MLA-C02 practice test has questions for every domain.

Checklist

0 of 14

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

Split your preparation into two tracks that meet at evaluation. For classic ML, review the main SageMaker AI built-in algorithms by problem type, then run one XGBoost job in script mode and wrap it in automatic model tuning. Watch the job metrics and practice reading learning curves for overfitting and underfitting. Know what early stopping, data parallel and model parallel each fix.

For GenAI, build a decision table for prompt engineering, RAG, fine-tuning, continued pre-training and distillation, with the missing ingredient each one supplies. In the Bedrock console, compare two models on the same prompts, then run a model evaluation job with automatic metrics and an LLM-as-a-judge evaluation. If you can, try a small fine-tuning job to see the JSONL format and the custom model workflow.

Finish with metrics drills. For each scenario, say whether the right measure is recall, F1, RMSE, ROUGE, BLEU, BERTScore or human review, and why. Add MLflow experiment tracking and Clarify explanations so you can answer reproducibility and explainability questions without guessing.

FAQ

How much of MLA-C02 is Domain 2?

ML Model and Foundation Model (FM) Development is 24% of scored content, about twelve of fifty scored questions, across three task statements. It was 26% in MLA-C01.

Do I still need classic ML algorithms for MLA-C02?

Yes. Built-in algorithms, script mode, hyperparameter tuning and fit problems remain, alongside new FM selection, customization and evaluation skills.

When should I choose RAG over fine-tuning?

Choose RAG when answers depend on private or frequently changing knowledge or need citations. Choose fine-tuning when the model must learn a task, tone or output format.

What is catastrophic forgetting?

It is the loss of previously learned general abilities after a model is fine-tuned narrowly. Mixed training data, lower learning rates and parameter-efficient methods reduce it.

Which metrics should I know for GenAI evaluation?

ROUGE for summarization, BLEU for translation, BERTScore and semantic similarity for meaning, plus human review and LLM-as-a-judge for subjective quality.

What does automatic model tuning do?

It launches many training jobs with different hyperparameter values and searches for the combination that optimizes an objective metric you define.

Were any MLA-C01 Domain 2 topics removed?

Yes. The comparison page lists two deletions here: the older skill on fine-tuning with custom datasets, now replaced by new FM customization skills, and model size reduction such as pruning and compression.

Other Artificial Intelligence & Machine Learning certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 2 knowledge

You have covered model and FM selection, training, tuning, customization and evaluation metrics. Check whether you can choose the right approach for each scenario.

Take the MLA-C02 practice test Check your MLA-C02 readiness

Keep going