Overview
Application Development is the heaviest section of the Generative AI Engineer Associate exam at 30%, roughly 13 or 14 questions, and it has 13 official objectives. They span framework choice (LangChain, LangGraph and similar libraries), qualitative review of responses for quality and safety problems, choosing a chunking strategy from evaluation evidence, augmenting prompts with key fields and intents from the user's input, and rewriting a prompt to move a response from a baseline to a target.
Model selection gets four objectives: picking an LLM from application attributes, sizing an embedding model's context length, reading model cards in a hub or Databricks Marketplace, and choosing a model from experiment metrics. The rest cover implementing guardrails, using MLflow with Mosaic AI Agent Framework to build agentic systems, contrasting evaluation with monitoring in the lifecycle, and letting multi-agent systems query data through Genie spaces or the Genie Conversation API.
How it's tested
Expect the widest variety of question styles here: code snippets with a missing line, comparisons of model cards or experiment results, prompts to improve, and agent architectures to complete. Most questions name an attribute such as latency, cost, context length, safety or data source, and that attribute decides the answer.
-
Pick the model
Requirements on context, latency, cost, license or language are listed and you select the LLM or embedding model that meets them.
-
Fix the prompt
A baseline response is shown alongside the desired one and you choose the prompt change or augmentation that closes the gap.
-
Complete the agent
An agentic system is partly built and you choose the MLflow, Agent Framework, guardrail or Genie component that is missing.
A sales assistant must answer questions about pipeline numbers stored in Unity Catalog tables as well as product documents. Adding the tables to the vector index is tempting, but embeddings handle aggregate numeric questions poorly. A Genie space translates natural language into SQL over governed tables, so the agent should call Genie for figures and the retriever for documents.
Objectives 13
Official objectives from the Databricks outline, in outline order.
3.1 Select Langchain/similar tools for use in a Generative AI applicationLangChain, LangGraph, LlamaIndex, DSPy and Databricks integrations.
- What it means
- This objective asks which library or component suits a task. LangChain offers prompt templates, retrievers and chains; LangGraph models stateful agent graphs; LlamaIndex focuses on indexing and retrieval; DSPy optimizes prompts programmatically. The databricks-langchain package connects these to Databricks models and Vector Search.
- What to understand
- Know the role of each building block: a chat model wrapper for a serving endpoint, a retriever wrapping a Vector Search index, an output parser, and a graph for loops and branching. LangGraph suits agents needing state and cycles. MLflow autologging traces these frameworks. Agent Framework is framework-agnostic, so the choice depends on the application's needs.
- How it shows up
- A capability is needed, such as stateful tool loops or a retriever over Vector Search, and you choose the library or class.
- Common confusion
- Thinking Databricks requires one specific framework; Agent Framework accepts agents written with many libraries.
3.2 Qualitatively assess responses to identify common issues such as quality and safetySpotting hallucination, irrelevance, incompleteness, toxicity and leakage.
- What it means
- Before metrics, an engineer reads outputs. This objective asks you to recognize common problems in sample responses: hallucinated facts, ignoring retrieved context, answering a different question, missing parts of the request, unsafe or toxic content, and exposing personal data.
- What to understand
- Compare the answer to the question and the retrieved context. Claims absent from context suggest hallucination. Correct but off-topic text is a relevance issue. Partial answers are completeness issues. Harmful or biased content is a safety issue, and names or account numbers that should not appear are a privacy issue. Each points to a different fix.
- How it shows up
- A response is shown with its context and you identify the quality or safety issue it displays.
- Common confusion
- Labeling every wrong answer a hallucination when the context itself was missing or wrong.
3.3 Select chunking strategy based on model & retrieval evaluationUsing evaluation results to choose between chunking options.
- What it means
- Data Preparation introduced chunking methods; this objective uses evidence to choose. You compare chunking variants by retrieval metrics and answer quality, along with the constraints of the embedding model and LLM, then pick the one that performs best for the application.
- What to understand
- Run the same evaluation set against each variant and compare recall, precision and judge scores such as groundedness. If answers lack context, larger or parent-child chunks may win; if retrieval is noisy, smaller chunks may win. Track runs in MLflow so the comparison is reproducible. The best option balances quality with token cost.
- How it shows up
- A table of evaluation results for several chunking setups is given and you select the best one for the stated goal.
- Common confusion
- Picking the variant with the highest single metric while ignoring the requirement the question emphasizes.
3.4 Augment a prompt with additional context from a user's input based on key fields, terms, and intentsEnriching prompts and retrieval with extracted entities and intent.
- What it means
- User messages often hold key details: a product name, an account type, a date range or an intent such as 'cancel'. This objective covers extracting those signals and using them to enrich the prompt or filter retrieval, so the model receives focused context.
- What to understand
- Extract entities and intent with a lightweight model call or rules, then use them as metadata filters on Vector Search, as variables in the prompt template, or to route to a specialist chain. Query rewriting can expand terse questions. Structured fields improve precision more than adding raw conversation history.
- How it shows up
- A user query contains specific fields and you choose how to use them to improve the prompt or retrieval.
- Common confusion
- Passing the whole conversation verbatim instead of the specific fields that sharpen retrieval.
3.5 Create a prompt that adjusts an LLM's response from a baseline to a desired outputIterating prompt instructions, examples and constraints toward a target.
- What it means
- Given a baseline response and the desired one, this objective asks what prompt change bridges the gap: adding constraints on length or tone, giving examples, specifying audience, requiring citations, or asking the model to say when it does not know.
- What to understand
- Identify the difference first: format, length, tone, grounding, reasoning or refusal behavior. Then change only what addresses it. Few-shot examples shift style; explicit constraints control length; instructions to use only provided context reduce hallucination. Version prompts and compare outputs on the same inputs to confirm improvement.
- How it shows up
- A baseline and target output are shown and you pick the prompt revision that produces the target.
- Common confusion
- Switching to a larger model when a prompt instruction would close the gap more cheaply.
3.6 Implement LLM guardrails to prevent negative outcomesInput and output filtering, safety models and policy checks.
- What it means
- Guardrails are controls around the model that block or correct harmful inputs and outputs. This objective covers safety classifiers on prompts and responses, topic restrictions, PII detection, output validation and system prompt rules, implemented in code or through Mosaic AI Gateway settings.
- What to understand
- Input guardrails catch jailbreaks, off-topic requests and unsafe content before the model runs. Output guardrails catch toxic text, leaked personal data or format violations before the user sees them. AI Gateway on serving endpoints offers safety filtering and PII detection. Guardrails complement, not replace, good prompts and access controls.
- How it shows up
- An application produces or receives harmful content and you choose the guardrail that prevents it.
- Common confusion
- Treating a system prompt instruction as a sufficient guardrail; determined users can override prompt-only rules.
3.7 Select the best LLM based on the attributes of the application to be developedWeighing quality, context window, latency, cost, license and modality.
- What it means
- Applications differ in what they need from a model. This objective asks you to match attributes such as task complexity, context length, latency target, throughput, cost, language, license terms and tool-calling support to an available model.
- What to understand
- Smaller models suit high-volume, simple tasks with tight latency; larger models suit complex reasoning. Long documents need a long context window. Commercial use requires a compatible license. Agents need reliable function calling. Databricks Foundation Model APIs and external models through serving endpoints make it easy to compare candidates on the same evaluation set.
- How it shows up
- Application requirements are listed and you choose the model that meets all of them.
- Common confusion
- Choosing the largest, most capable model when the requirement emphasizes latency or cost.
3.8 Select an embedding model context length based on source documents, expected queries, and optimization strategyMatching embedding input limits to chunk sizes and queries.
- What it means
- Embedding models have maximum input lengths. This objective asks you to choose one whose context length suits the chunk sizes you plan, the length of expected queries and your optimization goal, such as cost, speed or retrieval precision.
- What to understand
- If chunks are long, a model with a larger context avoids truncation. If documents are short FAQs, a smaller, faster model may be enough. Queries and documents must use the same embedding model. Longer contexts cost more compute and can blur meaning, so chunk size and model limit should be chosen together.
- How it shows up
- Document lengths and query patterns are described and you select the embedding model or chunk size that fits.
- Common confusion
- Embedding queries with a different model from the one used to embed documents.
3.9 Select a model from a model hub or marketplace for a task based on model metadata/model cardsReading model cards for task fit, license, limits and evaluation.
- What it means
- Model hubs such as Hugging Face and Databricks Marketplace publish model cards. This objective asks you to read that metadata (intended use, training data, languages, context length, license, size and benchmark results) and pick a model suitable for a task.
- What to understand
- Check intended use first; a code model is a poor fit for summarizing contracts. Confirm license terms allow your use, especially commercial deployment. Check language coverage, context length and size against serving limits. Benchmarks help only when they resemble your task. Missing or vague cards are a risk signal.
- How it shows up
- Several model cards are summarized and you choose the model that fits the task and constraints.
- Common confusion
- Choosing by benchmark rank alone without checking license or intended use.
3.10 Select the best model for a given task based on common metrics generated in experimentsComparing MLflow runs on quality, latency, cost and safety metrics.
- What it means
- Experiments produce metrics: judge scores, exact match, ROUGE for summaries, toxicity, latency and token counts. This objective asks you to read a comparison and choose the model that best fits the stated priority.
- What to understand
- Identify the primary requirement first, then check other constraints are met. ROUGE suits summarization overlap; exact match suits extraction; LLM judges assess correctness and relevance; latency and tokens drive cost. Lower perplexity indicates better fit to text but not task success. Use MLflow experiments to compare runs on identical data.
- How it shows up
- A metrics table for candidate models is given and you select the one that best meets the requirement.
- Common confusion
- Choosing the top quality score when another model meets the quality bar at far lower latency or cost.
3.11 Utilize MLflow and Agent Framework for developing agentic systemsAuthoring, tracing and logging agents with MLflow and Mosaic AI Agent Framework.
- What it means
- Mosaic AI Agent Framework and MLflow together support building agents on Databricks. You author an agent with any library, wrap it in the MLflow ResponsesAgent interface, trace it, log it as a model from code, and evaluate it before deployment.
- What to understand
- MLflow Tracing records each step, tool call and retrieval with inputs and outputs, which is how you debug agents. Autologging covers common frameworks. Logging from code stores the agent definition file. Declaring resources such as Vector Search indexes and serving endpoints enables automatic authentication at deployment. The ResponsesAgent interface gives a standard schema for chat-style agents.
- How it shows up
- An agent workflow step is described and you choose the MLflow or Agent Framework feature that supports it.
- Common confusion
- Debugging an agent by printing outputs instead of using MLflow Tracing to see each intermediate step.
3.12 Compare the evaluation and monitoring phases of the Gen AI application life cycleOffline evaluation before release versus production monitoring afterwards.
- What it means
- Evaluation happens during development on curated datasets, often with expected answers, to decide whether a version is good enough. Monitoring happens in production on real traffic, usually without labels, to detect quality drops, safety issues, latency and cost changes over time.
- What to understand
- Evaluation uses fixed datasets, repeatable runs and judges that may need ground truth. Monitoring samples live requests, applies label-free judges, and watches operational metrics. Findings from monitoring feed new evaluation examples. The same MLflow scorers can serve both phases, which keeps measurements consistent.
- How it shows up
- An activity is described and you identify whether it belongs to evaluation or monitoring, or what each phase needs.
- Common confusion
- Expecting ground-truth answers to be available for live production traffic.
3.13 Enable multi-agent systems to leverage Genie Spaces or conversational API to retrieve dataUsing Genie for natural-language SQL over governed tables inside agents.
- What it means
- A Genie space lets users ask questions about curated Unity Catalog tables in plain language and returns SQL results. This objective covers adding a Genie space as an agent or tool in a multi-agent system, through Agent Bricks Multi-Agent Supervisor, a managed MCP server or the Genie Conversation API.
- What to understand
- Use Genie for structured, aggregate questions over tables and a retriever for unstructured documents. The Conversation API lets code start conversations, ask follow-ups and fetch query results. Genie respects Unity Catalog permissions of the caller. Good table descriptions and sample questions in the space improve SQL accuracy.
- How it shows up
- An agent must answer questions about table data and you choose Genie or its API as the component.
- Common confusion
- Embedding table rows into a vector index to answer aggregate numeric questions.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
LangGraph
A library for stateful, cyclic agent graphs.
- Remember
- Use it when agents loop or branch.
- Tested as
- Framework selection questions.
- Trap
- Using a linear chain for tool loops.
-
databricks-langchain
Package connecting LangChain to Databricks models and Vector Search.
- Remember
- Provides chat model and retriever wrappers.
- Tested as
- Integration questions.
- Trap
- Writing custom HTTP calls instead.
-
Hallucination vs missing context
Unsupported claims versus a retrieval gap.
- Remember
- Check the context before blaming the model.
- Tested as
- Response assessment questions.
- Trap
- Treating every error as hallucination.
-
Metadata filters
Restricting vector search by fields such as product or date.
- Remember
- Use entities extracted from the query.
- Tested as
- Prompt augmentation questions.
- Trap
- Relying on similarity alone.
-
Mosaic AI Gateway guardrails
Safety filtering and PII detection on serving endpoints.
- Remember
- They act on inputs and outputs.
- Tested as
- Guardrail questions.
- Trap
- Relying only on system prompts.
-
Model attributes
Context window, latency, cost, license, tool calling.
- Remember
- Requirements decide, not size.
- Tested as
- LLM selection questions.
- Trap
- Defaulting to the largest model.
-
Embedding context length
Maximum tokens an embedding model can encode.
- Remember
- Chunks must fit within it.
- Tested as
- Embedding selection questions.
- Trap
- Mixing embedding models for queries and documents.
-
Model card
Published metadata on use, data, license and limits.
- Remember
- Check license and intended use first.
- Tested as
- Model hub questions.
- Trap
- Ranking by benchmarks only.
-
ROUGE vs exact match
Overlap metric for summaries versus strict match for extraction.
- Remember
- Pick the metric that fits the task.
- Tested as
- Experiment metric questions.
- Trap
- Using ROUGE for classification.
-
MLflow Tracing
Step-level record of agent calls, tools and retrievals.
- Remember
- Primary debugging tool for agents.
- Tested as
- Agent Framework questions.
- Trap
- Relying on print statements.
-
ResponsesAgent
MLflow interface standardizing agent input and output.
- Remember
- Recommended for authoring agents for deployment.
- Tested as
- Agent authoring questions.
- Trap
- Assuming a framework-specific schema is required.
-
Evaluation vs monitoring
Offline labeled testing versus live label-free tracking.
- Remember
- Monitoring feeds new evaluation data.
- Tested as
- Lifecycle questions.
- Trap
- Expecting labels in production.
-
Genie space
Natural-language SQL over curated Unity Catalog tables.
- Remember
- Use it for structured data questions.
- Tested as
- Multi-agent data questions.
- Trap
- Vectorizing tables for aggregates.
-
Genie Conversation API
API to ask Genie questions and read results from code.
- Remember
- Supports follow-up questions in a conversation.
- Tested as
- Integration questions.
- Trap
- Thinking Genie works only in the UI.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Evaluation vs monitoring
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Evaluation | Offline tests on curated datasets | Can use ground truth | Running it once and stopping |
| Monitoring | Tracking live traffic over time | Label-free judges and ops metrics | Expecting expected answers |
| Shared scorers | Same MLflow scorers in both phases | Consistent measurement | Defining metrics twice differently |
Data access for agents
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Vector Search retriever | Similarity search over document chunks | Unstructured text | Using it for aggregates |
| Genie space | Natural language to SQL over tables | Structured data | Skipping table descriptions |
| Unity Catalog function tool | Governed deterministic logic | Exact operations | Duplicating logic in prompts |
Choosing a model
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Small LLM | Fast and cheap for simple tasks | Lower latency | Using it for complex reasoning |
| Large LLM | Stronger reasoning and instruction following | Higher cost and latency | Using it for simple routing |
| Embedding model | Turns text into vectors | Context length limits chunks | Mixing models across index and query |
| Marketplace or hub model | Third-party model with a model card | License and intended use vary | Ignoring license terms |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Biggest model wins
- What candidates pick
- Select the top-scoring large model for a high-volume classifier.
- Why it looks right
- Higher quality scores look safest.
- Why it's wrong
- If a smaller model meets the quality bar, it satisfies latency and cost requirements that the large model may violate.
Clue to watch for The scenario stresses latency, throughput or budget.
-
Prompt-only guardrail
- What candidates pick
- Add 'never discuss competitors' to the system prompt as the full control.
- Why it looks right
- It is quick and needs no infrastructure.
- Why it's wrong
- Prompt instructions can be bypassed by crafted input; input and output filters provide enforcement outside the model.
Clue to watch for The requirement is to prevent an outcome reliably.
-
Hallucination label
- What candidates pick
- Call a wrong answer a hallucination and change the prompt.
- Why it looks right
- Hallucination is the best-known failure.
- Why it's wrong
- If the retrieved context lacked the fact or was wrong, the fix is retrieval or data, not the prompt.
Clue to watch for The shown context does not contain the correct fact.
-
Vectors for tables
- What candidates pick
- Embed sales table rows so the agent can total revenue by region.
- Why it looks right
- Vector Search is already set up.
- Why it's wrong
- Similarity search cannot aggregate; Genie generates SQL that computes totals over governed tables.
Clue to watch for Questions ask for sums, counts or trends across rows.
-
Mixed embedding models
- What candidates pick
- Use a newer embedding model for queries while keeping the old index.
- Why it looks right
- Newer models perform better.
- Why it's wrong
- Vectors from different models are not comparable, so similarity scores become meaningless until the index is rebuilt.
Clue to watch for Only the query side is changed.
-
Benchmark-only model choice
- What candidates pick
- Pick the hub model with the best leaderboard score.
- Why it looks right
- Benchmarks are objective.
- Why it's wrong
- The model may have a non-commercial license, the wrong language coverage or a context too short for the task.
Clue to watch for Model cards list differing licenses or intended uses.
-
Labels in production
- What candidates pick
- Use correctness judges with expected answers on live traffic.
- Why it looks right
- They were used during evaluation.
- Why it's wrong
- Production requests have no expected answers, so monitoring relies on label-free judges and operational metrics.
Clue to watch for The activity runs on live user requests.
Answer framework
For an Application Development scenario, identify the deciding attribute and then pick the component that satisfies it.
- Find the stated priority in the scenario, such as latency, cost, context length, safety, data type or license, and use it to rank options.
- For model choices, confirm task fit, license, context window and tool support from model cards, then choose by experiment metrics that match the priority.
- For prompt problems, compare baseline and target output, then add the specific instruction, example, constraint or extracted field that closes the gap.
- For safety, place input and output guardrails around the model, using AI Gateway features or safety models rather than prompt rules alone.
- For agents, use MLflow Tracing and ResponsesAgent, route structured data questions to Genie and document questions to a retriever, then evaluate before monitoring.
Practice question
Want more like this? The Databricks GenAI Assoc practice test has questions for every domain.
Checklist
0 of 13
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Give this section the most time, since it carries 30% of the score. Build one small agent end to end: a LangGraph or LangChain agent that calls a Foundation Model API endpoint, retrieves from a Vector Search index and calls one Unity Catalog function. Enable MLflow autologging and inspect the traces. Wrap the agent with the ResponsesAgent interface and log it from code. This single exercise covers several objectives at once.
Next, practice prompt work in the AI Playground. Take a baseline answer and change one thing at a time: add an example, a length limit, an instruction to cite context. Note which change produced which effect.
For model selection, compare two or three models on the same small evaluation set and record latency, token usage and judge scores in an MLflow experiment. Read two model cards on Databricks Marketplace and note license and context limits. Then create a Genie space over a sample table and call it from code. Finish by writing a one-page comparison of evaluation and monitoring.
FAQ
How much of the Databricks GenAI Associate exam is Application Development?
Application Development is 30% of the exam, about 13 or 14 of the 45 scored questions, making it the largest section.
Do I need to know LangChain specifically?
You should know LangChain and LangGraph building blocks, but the objective says LangChain or similar tools. Agent Framework works with many libraries.
What is ResponsesAgent?
An MLflow interface that standardizes how an agent receives requests and returns responses, so it can be logged, evaluated and deployed consistently on Databricks.
Where do guardrails run on Databricks?
In application code, in safety models you call, and in AI Gateway settings on model serving endpoints, which can filter unsafe content and detect PII.
How do I choose an embedding model?
Match its context length to your chunk sizes, check language support and cost, and use the same model for both documents and queries.
When should an agent use Genie instead of Vector Search?
When the question needs SQL over structured tables, such as totals, counts or filters. Vector Search suits questions answered by document text.
What is the difference between evaluation and monitoring?
Evaluation tests a version offline on curated data before release. Monitoring tracks live traffic for quality, safety, latency and cost after release.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 3 knowledge
You have covered frameworks, prompt work, guardrails, model selection, MLflow agents and Genie integration. Test whether you can pick the right component when several look plausible.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the Databricks GenAI Assoc AI tutor Get any concept explained your way