AI-300 · Domain 4 of 5

AI-300 Domain 4: Implement generative AI quality assurance and observability

Proving generative AI applications and agents work and keep working: building test datasets and column mappings, quality and safety evaluators, automated evaluation in pipelines, and monitoring latency, tokens, cost and traces in Microsoft Foundry.

Exam weight
10–15%
Official functional groups
2
Estimated questions estimate
~6
Suggested study time estimate
8–10 hrs

About 10 min read Reviewed October 2026

Overview

Domain 4 is about evidence. Before release you need to show that a generative AI application or agent answers well and safely; after release you need to see how it behaves under real load. Microsoft lists two functional groups. Evaluation and validation covers test datasets and data mapping, quality metrics such as groundedness, relevance, coherence and fluency, risk and safety evaluations, and automated workflows that combine built-in and custom evaluators.

Observability covers continuous monitoring in Microsoft Foundry, performance metrics including latency, throughput and response times, cost metrics driven by token consumption, and logging and tracing detailed enough to debug production problems. The tools are the Azure AI Evaluation SDK and Foundry portal evaluations, Application Insights connected to a project, OpenTelemetry-based tracing and Azure Monitor metrics.

How it's tested

Scenarios describe an assistant, a RAG app or an agent along with a concern such as invented answers, unsafe output, slow responses or growing token spend. You choose the evaluator, dataset mapping, automation step or monitoring signal that detects or explains the problem.

  • Match the metric

    A quality problem is described, such as unsupported claims or off-topic answers, and you pick groundedness, relevance, coherence or fluency.

  • Gate the release

    A team wants every prompt or model change checked, and you design automated evaluation in CI with thresholds.

  • Find the bottleneck

    Users report slow or costly responses, and you use traces, latency metrics and token counts to locate the cause.

Reasoning it through

A support assistant built on retrieval sometimes states refund rules that do not appear in the policy documents it retrieved. Fluency and coherence scores look fine because the text reads well. The groundedness evaluator compares each response with the supplied context, so it flags these unsupported claims. Your dataset must therefore map the retrieved context column, not only the query and response.

Functional groups 2

Official functional groups from the Microsoft outline, in outline order.

4.1 Configure evaluation and validation for generative AI applications and agentsTest datasets, data mapping, quality metrics, safety evaluations and automated evaluation workflows.
What it means
This group covers measuring the quality and safety of generative AI output before and after release. It includes preparing evaluation data, choosing and configuring evaluators, and running evaluations automatically as part of development and deployment workflows.
What to understand
Evaluation datasets are usually JSONL with fields such as query, response, context and ground_truth; column mapping tells each evaluator which field to read. AI-assisted quality evaluators use a judge model deployment and typically score 1 to 5: groundedness checks claims against context, relevance checks the answer addresses the query, coherence checks logical flow and fluency checks language quality. Risk and safety evaluators detect hateful and unfair, sexual, violent and self-harm content, protected material and jailbreak attempts, reporting severity. Agent evaluators such as intent resolution, tool call accuracy and task adherence assess agent behaviour. Custom code-based or prompt-based evaluators cover domain rules, and the evaluate function or a GitHub Actions step automates runs.
How it shows up
Expect questions asking which evaluator detects a described failure, or which input a given evaluator requires.
Common confusion
Relevance asks whether the answer addresses the question; groundedness asks whether its claims are supported by the provided context.
4.2 Implement observability for generative AI applications and agentsContinuous monitoring, latency and throughput, token cost tracking, logging and tracing.
What it means
This group covers watching generative AI applications and agents in production. It includes continuous monitoring and evaluation in Microsoft Foundry, performance and cost metrics, and detailed tracing and logs that let engineers reproduce and fix problems.
What to understand
Connecting Application Insights to a Foundry project enables tracing based on OpenTelemetry, which records spans for model calls, retrieval steps and agent tool calls with inputs, outputs, durations and token counts. Azure Monitor metrics for model deployments include request counts, latency, time to first token, token usage and provisioned utilization. Cost tracking combines token metrics with Cost Management, and tags or separate deployments help attribute spend. Diagnostic settings send logs to a Log Analytics workspace for queries and alerts. Continuous evaluation samples production traffic and scores it with evaluators so quality trends appear on dashboards.
How it shows up
Look for symptoms such as slow first tokens, throttling or rising bills, then pick the trace, metric or log that pinpoints the cause.
Common confusion
Tracing shows the path and timing of one request; aggregated metrics show trends but not which step caused a slow call.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • Groundedness evaluator

    Scores whether response claims are supported by supplied context.

    Remember
    Needs the context field mapped in the dataset.
    Tested as
    Hallucination detection in RAG apps.
    Trap
    Running it without retrieved context.

  • Relevance evaluator

    Scores whether the response addresses the query.

    Remember
    A grounded answer can still be irrelevant.
    Tested as
    Off-topic answer scenarios.
    Trap
    Using it to detect unsupported facts.

  • Coherence and fluency

    Coherence checks logical flow; fluency checks grammar and readability.

    Remember
    Both say nothing about factual support.
    Tested as
    Writing quality scenarios.
    Trap
    Assuming fluent text is correct.

  • AI-assisted evaluator

    Evaluator that uses a judge model to score output.

    Remember
    Requires a model deployment configuration and consumes tokens.
    Tested as
    Setting up quality evaluation.
    Trap
    Forgetting the judge model adds cost.

  • Risk and safety evaluators

    Detect harmful content, protected material and jailbreak attempts.

    Remember
    Report severity levels for each category.
    Tested as
    Pre-release safety checks.
    Trap
    Relying on content filters alone for evaluation evidence.

  • Column mapping

    Links dataset fields to evaluator inputs.

    Remember
    Fields such as query, response, context and ground_truth must match expectations.
    Tested as
    Evaluation setup errors.
    Trap
    Mismatched names causing empty or failed scores.

  • Custom evaluator

    Code-based or prompt-based evaluator for domain rules.

    Remember
    Runs alongside built-in evaluators in the same workflow.
    Tested as
    Business-specific quality checks.
    Trap
    Trying to force a built-in metric to measure a custom rule.

  • Tracing

    Span-level record of each request through models, retrieval and tools.

    Remember
    Uses OpenTelemetry and Application Insights in Foundry.
    Tested as
    Debugging slow or wrong answers.
    Trap
    Expecting metrics alone to show which step failed.

  • Token usage metrics

    Prompt and completion token counts per deployment.

    Remember
    Drive both cost and throughput limits.
    Tested as
    Cost optimisation questions.
    Trap
    Tracking only request counts.

  • Continuous evaluation

    Ongoing scoring of sampled production traffic.

    Remember
    Reveals quality drift after release.
    Tested as
    Post-deployment monitoring scenarios.
    Trap
    Assuming pre-release evaluation is sufficient forever.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

Quality evaluators

ConceptMeaningKey differenceCommon trap
Groundedness Claims supported by provided context. Needs context; targets hallucination. Confusing it with relevance.
Relevance Answer addresses the user's query. Needs query and response. Expecting it to catch invented facts.
Coherence Logical, well-organised response. About structure, not facts. Using it as an accuracy measure.
Fluency Grammatical, natural language. About language quality only. Treating high fluency as safe output.

Observability signals

ConceptMeaningKey differenceCommon trap
Traces Per-request spans with timing and payloads. Pinpoints the slow or failing step. Not enabling Application Insights on the project.
Metrics Aggregated latency, requests and tokens over time. Shows trends and drives alerts. Expecting metrics to explain a single bad answer.
Logs Diagnostic records in Log Analytics. Queryable history for audits and alerts. Forgetting diagnostic settings are needed.

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. Relevance catches hallucination

    What candidates pick
    Use the relevance evaluator to detect unsupported claims
    Why it looks right
    Off-target answers and invented answers feel similar.
    Why it's wrong
    Relevance only judges whether the answer addresses the query; unsupported claims need groundedness against context.

    Clue to watch for Answers mention facts missing from the source documents.

  2. Content filters are evaluation

    What candidates pick
    Rely on content filtering instead of safety evaluation
    Why it looks right
    Filters already block harmful output.
    Why it's wrong
    Filters act at runtime; risk and safety evaluations measure how often and how severely harmful content appears before release.

    Clue to watch for The scenario asks for evidence or a release gate.

  3. Metrics explain one failure

    What candidates pick
    Check average latency to debug one slow conversation
    Why it looks right
    Metrics dashboards are easy to open.
    Why it's wrong
    Averages hide individual requests; a trace shows which retrieval, tool or model step took the time.

    Clue to watch for A specific request or user session is slow.

  4. Manual evaluation in portal

    What candidates pick
    Run evaluations by hand before each release
    Why it looks right
    The portal makes ad hoc runs simple.
    Why it's wrong
    The objective asks for automated workflows, so evaluation should run in CI or on a schedule with thresholds.

    Clue to watch for Every change or every pull request must be checked.

  5. Ground truth always required

    What candidates pick
    Collect ground truth before any evaluation
    Why it looks right
    Classical ML evaluation needs labels.
    Why it's wrong
    Many AI-assisted evaluators, such as groundedness and coherence, score without ground truth; similarity-style metrics need it.

    Clue to watch for No labelled answers are available.

Answer framework

Follow these steps when asked to evaluate or monitor a generative AI application or agent.

  1. Start by naming the specific failure you need to detect, such as invented facts, off-topic answers, unsafe content, slow responses or rising cost.
  2. Choose matching evaluators or signals, for example groundedness for unsupported claims, safety evaluators for harm, traces for latency, token metrics for cost.
  3. Build or sample a dataset with the fields those evaluators need and map columns such as query, response and context correctly.
  4. Automate runs in a CI workflow or schedule with thresholds that block a release or raise an alert when scores fall.
  5. Connect Application Insights and diagnostic logs so production traces, metrics and continuous evaluation results can be investigated.

Practice question

A RAG-based HR assistant sometimes quotes leave policies that are not in the documents it retrieved, although its answers read naturally and stay on topic. You need an automated check before each release that detects this problem. Which evaluator should you add to the evaluation workflow?

Want more like this? The AI-300 practice test has questions for every domain.

Checklist

0 of 12

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

This domain is small but precise, so focus on matching metrics to problems. Install the Azure AI Evaluation SDK, create a ten-row JSONL dataset for a simple RAG app, and run groundedness, relevance, coherence and fluency evaluators. Deliberately remove the context column and see how the results change, then add a safety evaluator and a small custom evaluator for a business rule.

Move the evaluation into a GitHub Actions workflow that fails when a score drops below a threshold, which mirrors the automated workflow objective. Then turn to observability: connect Application Insights to a Foundry project, enable tracing in a sample app or agent, and inspect spans for model and tool calls. Open the deployment metrics in Azure Monitor and find latency, time to first token and token counts, then estimate cost from token usage. Finish by making a two-column sheet listing each evaluator or signal and the symptom it detects. Expect roughly half a dozen questions here, so concentrate on precise distinctions rather than broad reading.

FAQ

How much of AI-300 is Domain 4, Implement generative AI quality assurance and observability?

Implement generative AI quality assurance and observability is weighted 10–15% of AI-300, roughly 5 to 7 questions if the exam has about 50 scored items.

Which SDK is used for evaluation?

The Azure AI Evaluation SDK provides built-in quality, safety and agent evaluators plus an evaluate function, and Foundry portal evaluations use the same evaluators.

Do quality evaluators need ground truth?

Not all of them. Groundedness, relevance, coherence and fluency use AI-assisted judging without labels, while similarity and F1-style metrics compare against ground truth.

What does a risk and safety evaluation report?

It reports how often harmful content such as hate, sexual, violent or self-harm material, protected material or jailbreak success appears, with severity levels.

How do I enable tracing in Foundry?

Connect an Application Insights resource to the project and instrument the app with the tracing libraries, which use OpenTelemetry to capture spans.

How is token cost monitored?

Use deployment token metrics in Azure Monitor together with Cost Management. Splitting apps across deployments or using tags helps attribute spend.

Other Artificial Intelligence & Machine Learning certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 4 knowledge

Check whether you can match evaluators and monitoring signals to generative AI quality, safety, latency and cost problems.

Take the AI-300 practice test Check your AI-300 readiness

Keep going