Overview
Domain 2 follows a classical machine learning model from experiment to production and back again. Microsoft lists four functional groups. Orchestrating training covers MLflow experiment tracking, automated machine learning, notebooks, hyperparameter sweeps, training scripts, distributed training, pipelines and comparing runs. Registration and versioning covers MLflow models, packaging a feature retrieval specification, responsible AI evaluation and archiving.
The deployment group covers managed online and batch endpoints, testing and troubleshooting them, and progressive rollout with rollback. The monitoring group covers data drift, production performance metrics and alerts or retraining triggers when thresholds are crossed. Questions assume Azure Machine Learning with the CLI v2 and Python SDK v2, and they reward answers that keep runs reproducible, deployments reversible and production behaviour measured rather than assumed.
How it's tested
Expect scenarios built around a model's life: a data scientist needs to compare runs, a team must cut tuning cost, a new version has to go live without downtime, or accuracy is dropping in production. You choose the Azure Machine Learning feature, setting or sequence of steps that solves the problem with the least risk.
-
Tune for less
A sweep is too slow or expensive, and you pick the sampling method and early termination policy that fit.
-
Ship without downtime
A new model version must replace the old one gradually, and you choose traffic splits, mirrored traffic or a separate deployment.
-
Real time or batch
Latency, volume and schedule requirements decide between a managed online endpoint and a batch endpoint.
A fraud model scored by a checkout API must be replaced, but the business wants to see live results before any customer is affected. Adding a green deployment under the same managed online endpoint and mirroring a share of traffic to it lets you compare outputs while blue still answers every request. Only after metrics look healthy do you shift live traffic, and rollback means moving traffic back to blue.
Functional groups 4
Official functional groups from the Microsoft outline, in outline order.
2.1 Orchestrate model trainingMLflow tracking, AutoML, notebooks, sweeps, scripts, distributed training, pipelines and run comparison.
- What it means
- This group covers how training runs are executed, tracked and compared in Azure Machine Learning. It spans interactive notebook exploration, command jobs that run scripts, automated machine learning, hyperparameter sweeps, distributed training for large models and multi-step pipelines.
- What to understand
- Azure Machine Learning workspaces act as an MLflow tracking server, so mlflow.autolog or explicit log_metric and log_param calls record runs that you compare in studio or query with the MLflow client. AutoML needs a task type, primary metric and limits such as timeout and trial count. Sweep jobs use grid, random or Bayesian sampling; Bandit, median stopping and truncation selection policies end poor trials early, but Bayesian sampling does not support early termination. Distributed jobs set instance_count and a distribution type such as PyTorch or MPI. Pipelines chain components and can reuse outputs from unchanged steps.
- How it shows up
- Watch for cost and time limits in sweep questions, and for the step that compares runs, which is usually MLflow metrics rather than custom logging code.
- Common confusion
- AutoML finds a model and its settings automatically; a sweep tunes hyperparameters of a training script you wrote.
2.2 Implement model registration and versioningMLflow models, feature retrieval specifications, responsible AI evaluation and model lifecycle.
- What it means
- Registration turns a run output into a named, versioned model in the workspace or a registry. This group also covers packaging the feature retrieval specification with the model, evaluating models against responsible AI principles before release and managing older versions, including archiving.
- What to understand
- Registering in MLflow format stores an MLmodel file that describes the flavour, signature and dependencies, which enables no-code deployment to online and batch endpoints. When features come from the Azure Machine Learning managed feature store, a feature retrieval specification packaged with the model lets inference fetch the same features used in training. The Responsible AI dashboard combines error analysis, fairness assessment, interpretability, counterfactual and causal analysis, and its scorecard records findings for reviewers. Archiving hides a model version from default lists without breaking existing references.
- How it shows up
- Look for the requirement that inference uses the same features as training, or that a model needs fairness and error analysis evidence before approval.
- Common confusion
- Archiving a model version is not deletion; existing deployments and jobs can still use it.
2.3 Deploy machine learning models for production environmentsManaged online and batch endpoints, testing, troubleshooting, progressive rollout and rollback.
- What it means
- This group covers serving registered models. Managed online endpoints return predictions synchronously with Azure-managed infrastructure, while batch endpoints run asynchronous scoring jobs over large datasets on compute clusters. You also test endpoints, diagnose failures and release new versions safely.
- What to understand
- An endpoint is a stable URI and authentication boundary; deployments under it hold the model, environment, code and instance type. Traffic percentages split live requests between deployments, and mirrored traffic copies a share of requests to a deployment without returning its responses. Local deployment with Docker speeds debugging, and az ml online-deployment get-logs shows container output. Typical failures include image build errors, quota shortages and scoring scripts whose init function fails. Batch deployments set mini_batch_size, instance count and output action.
- How it shows up
- Questions often ask for the next step after a failed deployment, or which traffic setting tests a new version without affecting users.
- Common confusion
- Mirrored traffic never answers callers, so it measures behaviour but cannot be used to complete a gradual cutover.
2.4 Monitor and maintain machine learning models in productionData drift, production performance metrics and threshold-based retraining or alerts.
- What it means
- After deployment, models degrade as input data or relationships change. This group covers collecting production data, detecting data and prediction drift, tracking model performance metrics and acting when thresholds are exceeded, either by alerting people or starting retraining automatically.
- What to understand
- Azure Machine Learning model monitoring compares production data with a reference dataset, often training data, using signals such as data drift, prediction drift, data quality and feature attribution drift, plus model performance when ground truth arrives. Online endpoints can collect inputs and outputs with the data collector. Each signal has metrics and thresholds, and breaches surface as alerts through Azure Monitor. Event Grid events and alerts can call Logic Apps, Azure Functions or GitHub Actions to start a retraining pipeline, while scheduled pipelines handle predictable retraining.
- How it shows up
- Expect a scenario where accuracy drops or input distributions shift, and you pick the signal, reference data or trigger that responds correctly.
- Common confusion
- Data drift measures change in inputs and can occur even when labels are unavailable; performance monitoring needs ground truth.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
MLflow tracking
Logging parameters, metrics and artifacts for each run.
- Remember
- Azure Machine Learning workspaces are MLflow-compatible tracking servers.
- Tested as
- Run comparison and experiment tracking questions.
- Trap
- Writing custom logging to blob storage instead of using MLflow.
-
Sweep job
Hyperparameter tuning job across many trials.
- Remember
- Sampling options are grid, random and Bayesian.
- Tested as
- Cost and time optimisation questions.
- Trap
- Pairing Bayesian sampling with an early termination policy.
-
Early termination policy
Rule that stops poorly performing trials early.
- Remember
- Bandit uses slack, median stopping uses running averages, truncation drops a bottom percentage.
- Tested as
- Picking the policy that saves compute.
- Trap
- Expecting it to improve the best trial's score.
-
MLflow model format
Model saved with an MLmodel file describing flavour and signature.
- Remember
- Enables deployment without a scoring script or environment.
- Tested as
- Simplest deployment path questions.
- Trap
- Writing a scoring script when the model is already MLflow format.
-
Feature retrieval specification
Feature store lookup definition packaged with a model.
- Remember
- Keeps inference features consistent with training features.
- Tested as
- Training-serving skew scenarios.
- Trap
- Assuming the model artifact alone carries feature logic.
-
Responsible AI dashboard
Error analysis, fairness, interpretability, counterfactual and causal tools.
- Remember
- The scorecard exports a PDF summary for stakeholders.
- Tested as
- Pre-release model review questions.
- Trap
- Treating accuracy alone as evidence of responsible behaviour.
-
Managed online endpoint
Real-time scoring endpoint on Azure-managed compute.
- Remember
- Supports multiple deployments with traffic and mirror settings.
- Tested as
- Low-latency serving and rollout scenarios.
- Trap
- Choosing it for scheduled scoring of millions of rows.
-
Batch endpoint
Asynchronous scoring of large datasets on compute clusters.
- Remember
- Tune mini_batch_size and instance count for throughput.
- Tested as
- Overnight or bulk scoring scenarios.
- Trap
- Expecting synchronous responses from a batch endpoint.
-
Model monitoring signals
Data drift, prediction drift, data quality, feature attribution drift and performance.
- Remember
- Each signal compares production data with a reference dataset against thresholds.
- Tested as
- Production degradation scenarios.
- Trap
- Using performance monitoring when no labels are collected.
-
Retraining trigger
Automated action when a monitor or schedule fires.
- Remember
- Alerts or Event Grid events can start pipelines through Logic Apps, Functions or GitHub Actions.
- Tested as
- Closing the MLOps loop.
- Trap
- Retraining on every alert without validating the new model.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Online versus batch endpoints
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Managed online endpoint | Synchronous request-response scoring. | Low latency, always-on instances, traffic splitting. | Paying for idle instances when scoring is nightly. |
| Kubernetes online endpoint | Online scoring on an attached AKS or Arc cluster. | You operate the cluster. | Selecting it when no cluster management is wanted. |
| Batch endpoint | Asynchronous job over files or tables. | Runs on clusters that scale to zero between jobs. | Expecting per-request responses for interactive apps. |
Safe rollout techniques
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Traffic split | Percentage of live requests sent to each deployment. | New deployment answers real callers. | Moving 100% at once and calling it gradual. |
| Mirrored traffic | Copy of a share of requests sent to a deployment. | Responses are discarded; callers only see the live deployment. | Thinking mirroring completes a cutover. |
| Rollback | Returning traffic to the previous deployment. | Fast because the old deployment still exists. | Deleting blue before green is proven. |
Training automation options
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| AutoML | Service searches algorithms and settings for a task. | You set task, primary metric and limits. | Using it to tune a custom deep learning script. |
| Sweep job | Tunes hyperparameters of your own script. | You define the search space and sampling. | Expecting it to try different algorithms automatically. |
| Pipeline job | Chains components into a repeatable workflow. | Can reuse cached step outputs. | Confusing pipeline steps with sweep trials. |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Mirroring as rollout
- What candidates pick
- Mirror 100% of traffic to the new deployment
- Why it looks right
- Mirroring sounds like a safe way to move users.
- Why it's wrong
- Mirrored requests return nothing to callers, so traffic percentages must still be changed to complete a rollout.
Clue to watch for The question asks for users to be served by the new version.
-
Bayesian plus early stop
- What candidates pick
- Bayesian sampling with a Bandit policy
- Why it looks right
- Both sound like smart cost savers.
- Why it's wrong
- Azure Machine Learning sweep jobs do not allow early termination with Bayesian sampling.
Clue to watch for A sweep needs to be cheaper and the options pair these two.
-
Online for bulk
- What candidates pick
- A managed online endpoint for nightly scoring of a large file
- Why it looks right
- Online endpoints are familiar and fast.
- Why it's wrong
- Batch endpoints process large inputs asynchronously on clusters that scale to zero, which suits scheduled bulk work.
Clue to watch for Words like nightly, millions of records or no latency requirement.
-
Archive equals delete
- What candidates pick
- Archive a model to stop its production deployment
- Why it looks right
- Archiving sounds like retirement.
- Why it's wrong
- Archived versions are hidden from lists but remain usable; endpoints keep serving them until changed.
Clue to watch for The scenario needs to stop serving a model, not tidy the registry.
-
Drift means retrain now
- What candidates pick
- Automatically deploy a retrained model on every drift alert
- Why it looks right
- It closes the loop quickly.
- Why it's wrong
- Retraining should be followed by evaluation and approval before deployment, otherwise a worse model can ship.
Clue to watch for Requirements mention quality gates or approval.
-
Custom logging for comparison
- What candidates pick
- Write metrics to a CSV and compare manually
- Why it looks right
- It looks flexible.
- Why it's wrong
- MLflow tracking already records and compares metrics across jobs in the workspace with less effort.
Clue to watch for The question asks for minimal code to compare runs.
Answer framework
Work through these questions when a scenario follows a model from training to production.
- Decide whether the task needs AutoML, a sweep over your script or a pipeline, based on who wrote the model code and what is being searched.
- Check sweep settings for compatible sampling and early termination, and confirm MLflow logs the metric being optimised.
- Register the chosen model in MLflow format, attach feature retrieval details if a feature store is involved, and review it with responsible AI tools.
- Pick a managed online or batch endpoint from latency and volume needs, then release new versions with mirrored traffic and gradual splits.
- Configure monitoring signals against a reference dataset, set thresholds, and route alerts to a retraining pipeline with an approval step.
Practice question
Want more like this? The AI-300 practice test has questions for every domain.
Checklist
0 of 13
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Plan the most time for this domain because it carries the highest weight. Work through one end-to-end project in Azure Machine Learning: track a training script with MLflow, run an AutoML job on the same data, then tune your script with a sweep job using random sampling and a Bandit policy. Compare all runs in studio and with the MLflow client so you know where metrics live.
Register the best model in MLflow format and generate a Responsible AI dashboard. Deploy it to a managed online endpoint, then add a second deployment and practise mirrored traffic, a 90/10 split, a full cutover and a rollback. Break a scoring script on purpose and use local deployment and get-logs to find the cause. Create a batch endpoint for the same model and compare cost and behaviour. Finally, enable data collection and a monitor with a drift signal, then wire an alert to a pipeline or GitHub Actions workflow. Use practice questions to test sweep rules and rollout settings, which are frequent sources of lost marks.
FAQ
How much of AI-300 is Domain 2, Implement machine learning model lifecycle and operations?
Implement machine learning model lifecycle and operations is weighted 25–30%, the largest domain on AI-300. On an assumed 50 scored items that is roughly 12 to 15 questions.
Is AI-300 Domain 2 the same as DP-100 content?
It overlaps with retired DP-100 topics such as MLflow, AutoML and sweeps, but AI-300 adds operations: safe rollout, troubleshooting, monitoring and automated retraining triggers.
Do I need to know distributed training code?
You should know how to configure distributed jobs, including instance count, processes per instance and distribution type. Deep framework internals are less likely than configuration choices.
When should I choose a batch endpoint?
Choose one for large or scheduled scoring where callers do not wait for an immediate response. Batch endpoints run jobs on compute clusters that can scale down to zero between runs.
What does a feature retrieval specification do?
It records which feature store features a model needs, so inference retrieves the same features used in training and avoids training-serving skew.
Can drift monitoring work without labels?
Yes. Data drift and prediction drift compare distributions and need no ground truth. Model performance monitoring needs actual outcomes collected from production.
Which tool should I use to compare runs?
Use MLflow tracking in the workspace. Metrics, parameters and artifacts are logged per job and can be charted in studio or queried programmatically.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 2 knowledge
See whether you can choose the right training, registration, rollout and monitoring option in the heaviest AI-300 domain.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the AI-300 AI tutor Get any concept explained your way