Overview
Evaluation and Monitoring carries 12% of the exam, about five or six questions, across ten official objectives. On the evaluation side you choose an LLM size and architecture from quantitative metrics, evaluate agents with MLflow scoring and tracing, identify which evaluation judges require ground truth, write custom scorers for agents and LLMs, and fold subject matter expert feedback back into agent quality.
On the monitoring side you select the metrics that matter for a deployment, use inference logging to assess a live RAG application, track an endpoint with inference tables and Agent Monitoring, use AI Gateway (inference tables, usage tables and rate limiting) for agents deployed with Agent Framework, and apply Databricks features that control LLM costs. The common thread is MLflow 3: the same scorers run offline with mlflow.genai.evaluate and on sampled production traces, so quality is measured the same way before and after release.
How it's tested
Questions pair a measurement goal with Databricks tooling. Some show an evaluation dataset and ask which judges can run on it; others describe a production concern such as rising cost or falling quality and ask where to look or what to configure. Note whether expected answers exist and whether the system is in development or production.
-
Which judge works here
An evaluation dataset is described, with or without expected answers, and you choose the judges or scorers that can run on it.
-
Where is the data
A production question about payloads, token usage or quality is asked and you choose inference tables, usage tables, traces or Agent Monitoring.
-
Close the loop
Quality issues or costs need action and you choose SME review, a custom scorer, a smaller model or AI Gateway limits.
A team has 200 test questions but no reference answers and wants to check that responses stay on topic and are grounded in retrieved context. A correctness judge seems natural, but it compares answers against expected facts that do not exist. Relevance and retrieval groundedness judges work without ground truth, so those are the scorers to run now.
Objectives 10
Official objectives from the Databricks outline, in outline order.
6.1 Select an LLM choice (size and architecture) based on a set of quantitative evaluation metricsComparing model sizes and architectures with measured results.
- What it means
- Given metrics such as judge scores, accuracy, latency, throughput and cost for several models, this objective asks you to choose the size and architecture, for example a small dense model or a larger mixture-of-experts model, that best fits the requirements.
- What to understand
- Set the minimum acceptable quality first, then prefer the cheapest and fastest model that meets it. Larger models usually improve reasoning but raise latency and cost. Mixture-of-experts models activate only part of their parameters per token, which can offer strong quality at lower inference cost. Compare models on the same evaluation set in MLflow.
- How it shows up
- A table of metrics for candidate models is given and you choose the one that meets the stated thresholds.
- Common confusion
- Picking the model with the highest quality score when a smaller one passes the bar with better latency.
6.2 Select key metrics to monitor for a specific LLM deployment scenarioChoosing quality, safety, operational and cost metrics for production.
- What it means
- Different deployments need different signals. This objective asks which metrics to track for a scenario: a customer chatbot may need safety, relevance and latency; a batch summarizer may need throughput and token cost; a RAG assistant may need groundedness and retrieval relevance.
- What to understand
- Group metrics into quality (relevance, groundedness, guideline adherence), safety (toxicity, PII), operations (latency, error rate, throughput) and cost (tokens, requests). Pick those tied to the scenario's risks and goals. Track trends over time and set alerts on thresholds rather than only reviewing averages.
- How it shows up
- A deployment is described and you choose the most relevant metrics to monitor.
- Common confusion
- Monitoring only latency and errors for a customer-facing app where quality and safety are the main risks.
6.3 Evaluate agent performance using MLflow scoring and tracingRunning mlflow.genai.evaluate with scorers and reading traces.
- What it means
- MLflow evaluates agents by running them over an evaluation dataset, recording a trace for each request and applying scorers to those traces. This objective covers setting up that run and using traces to understand why scores are low.
- What to understand
- mlflow.genai.evaluate takes data, a predict function or existing traces, and a list of scorers. Each result links to its trace, showing retrieval, tool calls and model outputs. Low groundedness with poor retrieved context points to retrieval; good context with wrong answers points to the prompt or model. Results are stored in an MLflow experiment for comparison across versions.
- How it shows up
- An agent evaluation setup is described and you choose the call, input or trace evidence needed.
- Common confusion
- Reading only aggregate scores without opening traces to locate the failing step.
6.4 Use inference logging to assess deployed RAG application performanceAnalyzing logged production requests and responses.
- What it means
- Once deployed, requests and responses can be logged to Delta tables. This objective covers using those logs to judge how the RAG application performs on real traffic: which questions fail, where latency spikes and which answers users dislike.
- What to understand
- Inference logs capture inputs, outputs, timestamps and status. Query them in SQL to find common questions, error patterns and slow requests, and join with feedback. Sample logged requests for review or scoring. Failing real queries make strong additions to the evaluation dataset for the next version.
- How it shows up
- A team wants to understand production behavior and you choose to analyze logged inference data.
- Common confusion
- Assessing a deployed app only with the original offline evaluation set, which may not reflect real traffic.
6.5 Use Databricks features to control LLM costsModel choice, serving mode, batching, rate limits and usage tracking.
- What it means
- LLM costs grow with tokens and capacity. This objective covers Databricks levers: choosing smaller or task-specific models, picking pay-per-token or provisioned throughput appropriately, batching with ai_query(), scale-to-zero for idle custom endpoints, AI Gateway rate limits, and usage tracking through system tables.
- What to understand
- Match serving mode to traffic: pay-per-token for variable or low traffic, provisioned throughput for steady high volume. Shorter prompts and fewer retrieved chunks cut tokens. Rate limits cap runaway usage per user or endpoint. Usage system tables show who consumes what, supporting chargeback and budget alerts.
- How it shows up
- Costs are rising and you choose the Databricks feature that controls them without breaking requirements.
- Common confusion
- Reaching for a larger provisioned endpoint to fix cost when the issue is excessive tokens per request.
6.6 Use inference tables and Agent Monitoring to track a live LLM endpointPayload logging plus continuous quality scoring in production.
- What it means
- Inference tables log endpoint requests and responses into a Unity Catalog Delta table. Agent Monitoring builds on production traces by running registered scorers on a sample of live traffic so quality metrics appear over time without manual review.
- What to understand
- Enable inference tables on the serving endpoint or through AI Gateway. Register scorers such as safety, relevance or guidelines for production monitoring and set a sampling rate to manage cost. Use label-free scorers in production. Review dashboards and drill into traces when scores drop.
- How it shows up
- A live endpoint needs ongoing quality tracking and you choose inference tables and Agent Monitoring settings.
- Common confusion
- Expecting inference tables alone to compute quality scores; they store payloads, and scorers produce quality metrics.
6.7 Identify evaluation judges that require ground truthKnowing which MLflow judges need expected answers or facts.
- What it means
- MLflow's built-in LLM judges differ in their inputs. Some compare the response with expected answers or facts and therefore need ground truth; others judge the response or retrieved context on their own.
- What to understand
- Correctness needs expected facts or an expected response. Retrieval sufficiency needs expected facts to judge whether context covered them. Expectations-based guideline checks need per-row expectations. Relevance to query, safety, retrieval relevance, retrieval groundedness and global guidelines judges run without ground truth, so they suit production monitoring.
- How it shows up
- An evaluation dataset is described and you choose which judges can or cannot run on it.
- Common confusion
- Assuming groundedness needs ground truth; it compares the answer with retrieved context, not with expected answers.
6.8 Use AI Gateway (Inference Tables, Usage Tables, and rate limiting) to track an LLM or agent deployed via Agent FrameworkGovernance and tracking features on serving endpoints.
- What it means
- Mosaic AI Gateway adds governance to model serving endpoints. This objective covers its tracking features for agents deployed with Agent Framework: inference tables for payloads, usage tracking in system tables for tokens and requests, and rate limits per user, group or endpoint.
- What to understand
- Inference tables answer what was asked and answered. Usage tables answer who used how much and when, which supports cost attribution. Rate limits prevent a single user or app from overloading the endpoint or budget. Gateway guardrails and fallbacks complement these features. Settings are configured per endpoint.
- How it shows up
- A tracking or control need is described and you choose the AI Gateway feature that meets it.
- Common confusion
- Looking for token consumption in inference tables when usage tracking tables are designed for it.
6.9 Use Databricks custom Scorers for evaluating agents and LLMsCode-based and LLM-based custom scoring in MLflow.
- What it means
- Built-in judges do not cover every requirement. This objective covers writing custom scorers in MLflow: Python functions decorated as scorers that check something deterministic, or custom LLM judges written with your own criteria, then using them in evaluation and monitoring.
- What to understand
- Code scorers suit precise checks such as valid JSON, required citations, response length or tool call counts. Custom LLM judges suit subjective criteria such as brand tone or policy adherence. Scorers can read inputs, outputs, expectations and traces. The same scorer can run offline and in production for consistency.
- How it shows up
- A domain-specific quality requirement is described and you choose a custom scorer approach.
- Common confusion
- Using an LLM judge for a deterministic check that simple Python code can verify exactly.
6.10 Incorporate SME feedback to improve agent performanceCollecting expert reviews and turning them into evaluations and fixes.
- What it means
- Subject matter experts know what good answers look like. This objective covers gathering their feedback, for example through the Review App and labeling sessions, attaching it to traces, and using it to build evaluation datasets, align judges and guide changes to prompts, retrieval or tools.
- What to understand
- Experts label responses and supply corrected answers or expected facts, which become ground truth for later evaluations. Feedback attached to traces shows which steps caused problems. Aligning LLM judges with expert labels makes automated scoring more trustworthy. Agent Bricks also uses expert feedback to tune its agents.
- How it shows up
- Experts must review agent output and you choose how to collect and use their feedback.
- Common confusion
- Collecting feedback in spreadsheets disconnected from traces, so it cannot drive evaluation or debugging.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
mlflow.genai.evaluate
Runs an agent or traces through scorers on a dataset.
- Remember
- Results link back to traces.
- Tested as
- Agent evaluation questions.
- Trap
- Ignoring traces when scores drop.
-
Judges needing ground truth
Correctness and retrieval sufficiency need expected facts or answers.
- Remember
- Check the dataset for expectations first.
- Tested as
- Judge selection questions.
- Trap
- Running correctness without references.
-
Label-free judges
Relevance, safety, groundedness and retrieval relevance.
- Remember
- Suitable for production monitoring.
- Tested as
- Monitoring questions.
- Trap
- Thinking groundedness needs labels.
-
Custom code scorer
Python function scoring deterministic properties.
- Remember
- Exact and cheap for format or rule checks.
- Tested as
- Custom evaluation questions.
- Trap
- Using an LLM for exact checks.
-
Custom LLM judge
Judge prompt with your own criteria.
- Remember
- Use for subjective, domain-specific quality.
- Tested as
- Custom evaluation questions.
- Trap
- Leaving it unaligned with expert labels.
-
Inference tables
Delta tables of endpoint requests and responses.
- Remember
- Payloads, not quality scores.
- Tested as
- Production analysis questions.
- Trap
- Looking there for token billing.
-
Usage tracking tables
System tables of token and request consumption.
- Remember
- Support cost attribution.
- Tested as
- Cost questions.
- Trap
- Confusing them with inference tables.
-
AI Gateway rate limits
Caps on queries or tokens per user or endpoint.
- Remember
- Control abuse and spend.
- Tested as
- Cost and safety questions.
- Trap
- Ignoring them for internal apps.
-
Agent Monitoring
Scheduled scoring of sampled production traces.
- Remember
- Set a sampling rate to manage cost.
- Tested as
- Live endpoint questions.
- Trap
- Scoring every request at full cost.
-
Review App
Interface for experts to chat with and label agent responses.
- Remember
- Feedback attaches to traces.
- Tested as
- SME feedback questions.
- Trap
- Collecting feedback outside MLflow.
-
Model size vs quality threshold
Smallest model meeting the quality bar.
- Remember
- Then compare latency and cost.
- Tested as
- LLM choice questions.
- Trap
- Choosing the top scorer by default.
-
Cost levers
Smaller models, fewer tokens, batch inference, right serving mode.
- Remember
- Measure usage before changing capacity.
- Tested as
- Cost control questions.
- Trap
- Scaling capacity to fix token waste.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
MLflow judges and ground truth
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Correctness | Response matches expected facts | Needs ground truth | Running it without references |
| Retrieval sufficiency | Context covers expected facts | Needs ground truth | Confusing it with relevance |
| Retrieval groundedness | Response supported by context | No ground truth needed | Assuming it needs labels |
| Relevance to query and safety | On topic and free of harm | No ground truth needed | Skipping them in production |
Production tracking data
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Inference tables | Request and response payloads | What was asked and answered | Expecting quality scores |
| Usage tracking tables | Tokens and requests by caller | Who used how much | Looking for payloads there |
| Agent Monitoring scores | Scorer results on sampled traces | Quality over time | Scoring 100% of traffic needlessly |
| MLflow traces | Step-level execution records | Why a request failed | Using them for billing |
Custom scorer types
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Code-based scorer | Python logic returning a value | Deterministic and cheap | Overusing it for subjective criteria |
| Custom LLM judge | Prompted judge with own guidelines | Handles subjective criteria | Not aligning with experts |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Correctness without labels
- What candidates pick
- Run the correctness judge on live production traffic.
- Why it looks right
- Correctness is the most intuitive quality measure.
- Why it's wrong
- Correctness compares responses with expected facts, which live traffic lacks; label-free judges are needed in production.
Clue to watch for The data has no reference answers.
-
Groundedness needs labels
- What candidates pick
- Postpone groundedness checks until experts write expected answers.
- Why it looks right
- Grounding sounds like comparing with truth.
- Why it's wrong
- Groundedness compares the response with retrieved context, so it runs without ground truth.
Clue to watch for Retrieved context is available but expected answers are not.
-
Inference tables for billing
- What candidates pick
- Query inference tables to attribute token costs to teams.
- Why it looks right
- They log every request.
- Why it's wrong
- Usage tracking system tables are built for token and request consumption by caller; inference tables hold payloads.
Clue to watch for The question is about consumption and chargeback.
-
Bigger capacity for cost
- What candidates pick
- Add provisioned throughput to bring costs down.
- Why it looks right
- Reserved capacity can be cheaper per token.
- Why it's wrong
- If spend comes from bloated prompts or excessive retrieval, reducing tokens or using a smaller model cuts cost more directly.
Clue to watch for Token counts per request are unusually high.
-
LLM judge for format
- What candidates pick
- Write an LLM judge to verify outputs are valid JSON.
- Why it looks right
- LLM judges are flexible.
- Why it's wrong
- A code-based scorer checks JSON validity exactly and cheaply; LLM judges suit subjective criteria.
Clue to watch for The criterion is deterministic and machine-checkable.
-
Averages only
- What candidates pick
- Watch a single weekly average quality score.
- Why it looks right
- Averages are simple to report.
- Why it's wrong
- Averages hide drops on specific question types; traces and segmented metrics reveal where quality fell.
Clue to watch for Complaints rise while the average looks stable.
Answer framework
For an Evaluation and Monitoring scenario, decide the phase, the data available and the signal needed.
- Decide whether the scenario is offline evaluation before release or monitoring of a live endpoint, since that determines available data and scorers.
- Check whether expected answers or facts exist; use correctness and sufficiency only when they do, and label-free judges otherwise.
- Add custom scorers for requirements built-in judges miss, using code for deterministic checks and custom LLM judges for subjective ones.
- In production, enable inference tables, Agent Monitoring with sampling and AI Gateway usage tracking and rate limits, and inspect traces when metrics move.
- Close the loop by turning SME feedback and failing production queries into evaluation data, then compare model, prompt and cost options on it.
Practice question
Want more like this? The Databricks GenAI Assoc practice test has questions for every domain.
Checklist
0 of 11
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Study this section with MLflow open. Build a small evaluation dataset of fifteen questions, add expected facts to half of them, and run mlflow.genai.evaluate against a simple agent with a mix of built-in judges. Notice which judges fail or skip rows without expectations; that single exercise answers the ground-truth objective.
Next, write two custom scorers: a Python function that checks the response contains a citation, and a custom LLM judge for tone. Add both to the evaluation and open the traces for the lowest-scoring rows.
Then move to production features. Read the AI Gateway documentation and list exactly what inference tables, usage tracking and rate limits each provide. Review the Agent Monitoring setup, including registering scorers and choosing a sampling rate. Write down five cost levers with the situation each suits.
Finally, try the Review App or labeling sessions with a colleague acting as the expert, and note how their feedback lands on traces. Practice questions that mix evaluation and monitoring will confirm you can tell the phases apart under time pressure.
FAQ
How much of the Databricks GenAI Associate exam is Evaluation and Monitoring?
Evaluation and Monitoring is 12% of the exam, about five or six scored questions across ten official objectives.
Which MLflow judges need ground truth?
Correctness and retrieval sufficiency need expected facts or answers, as do expectation-based guideline checks. Relevance, safety, retrieval relevance and groundedness do not.
What do inference tables contain?
Requests and responses sent to a serving endpoint, stored as a Delta table in Unity Catalog for analysis and evaluation.
How do I track token usage by team?
Use AI Gateway usage tracking, which records token and request consumption in system tables that can be grouped by user or group.
What is Agent Monitoring?
Production monitoring that runs registered MLflow scorers on a sample of live traces so quality and safety are tracked over time.
When should I write a custom scorer?
When built-in judges do not cover a requirement, such as a citation format, brand tone or a domain rule.
How does SME feedback improve an agent?
Expert labels and corrected answers become ground truth for evaluation, help align LLM judges and point to which prompts, retrieval steps or tools need changes.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 6 knowledge
You have covered model choice from metrics, MLflow scoring and tracing, judges, custom scorers, inference tables, AI Gateway, cost, SME feedback. Test whether you can match tools to measurement needs.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks GenAI Assoc AI tutor Get any concept explained your way