Overview
Domain 5 assumes a generative AI system is already running and asks how to make it better. Microsoft lists two functional groups. The RAG group covers tuning similarity thresholds, chunk sizes and retrieval strategies, selecting and fine-tuning embedding models for a domain, combining semantic and keyword retrieval in hybrid search, and judging changes with relevance metrics and A/B tests.
The fine-tuning group covers designing advanced fine-tuning methods, creating and managing synthetic training data, monitoring and optimising fine-tuned models, and taking a fine-tuned model from development to production deployment. Azure AI Search, embedding models in Microsoft Foundry and Foundry fine-tuning are the main services, and the evaluation skills from Domain 4 provide the measurements that tell you whether an optimisation actually helped.
How it's tested
Questions present a generative AI system with a quality, cost or latency gap and ask what to change. You decide whether the cause lies in retrieval or in the model, then pick chunking, threshold, embedding, hybrid search or fine-tuning settings, and the metric that proves the change worked.
-
Retrieval or model
Symptoms are described and you decide whether to fix retrieval settings or customise the model.
-
Keyword meets vector
Exact codes or names are missed by vector search, and you choose hybrid search with semantic ranking.
-
Pick the tuning method
Training data and goals are given, and you choose supervised fine-tuning, preference optimisation or distillation from synthetic data.
An engineering assistant answers well about concepts but misses questions that quote exact part numbers such as PX-4471. Pure vector search treats the code as loosely similar text, so the right chunk ranks too low. Adding keyword retrieval through hybrid search lets BM25 match the exact token, reciprocal rank fusion merges both lists and the semantic ranker reorders the top results. Fine-tuning would not fix a retrieval miss.
Functional groups 2
Official functional groups from the Microsoft outline, in outline order.
5.1 Optimize retrieval-augmented generation (RAG) performance and accuracySimilarity thresholds, chunking, retrieval strategies, embedding models, hybrid search, relevance metrics and A/B tests.
- What it means
- This group covers improving the retrieval half of a RAG system so the model receives the right context. It spans how documents are split and embedded, how results are scored and filtered, how keyword and vector retrieval are combined, and how improvements are measured.
- What to understand
- Chunk size and overlap trade precision against context: small chunks match narrowly but lose surrounding meaning, large chunks dilute similarity. Top-k and minimum similarity thresholds decide how many results reach the prompt. Embedding model choice affects quality, dimensions and cost; models such as text-embedding-3-large can be shortened to fewer dimensions, and domain data can justify fine-tuning an open embedding model. In Azure AI Search, hybrid queries run BM25 keyword and vector search together and merge results with reciprocal rank fusion, and the semantic ranker reranks the top results. Retrieval metrics and document retrieval evaluation, combined with A/B tests on live traffic, show whether a change helped.
- How it shows up
- Expect a missed-context symptom and choose the retrieval change, or choose how to prove a change improved results.
- Common confusion
- Raising top-k adds more context but can lower groundedness by introducing noise and increases token cost.
5.2 Implement advanced fine-tuning and model customizationFine-tuning methods, synthetic data, monitoring fine-tuned models and production lifecycle.
- What it means
- This group covers customising foundation models when prompting and retrieval are not enough. It includes choosing a fine-tuning method, producing training data including synthetic data, checking training results and moving a fine-tuned model through deployment and ongoing operation.
- What to understand
- Supervised fine-tuning teaches format, tone or task behaviour from example conversations in JSONL chat format. Direct preference optimisation learns from pairs of preferred and rejected responses, and reinforcement fine-tuning uses graders to improve reasoning models on verifiable tasks. Distillation uses a larger teacher model's outputs, often stored or synthetic, to train a smaller, cheaper student. Synthetic data needs review for quality, bias and leakage of evaluation items. Watch training and validation loss for overfitting, then evaluate the fine-tuned model against the base model. Fine-tuned deployments carry hosting charges, so deploy deliberately, version models and retire unused deployments.
- How it shows up
- Look for whether the goal is new knowledge, which points to RAG, or changed behaviour, which points to fine-tuning, and which data is available.
- Common confusion
- Fine-tuning shapes how a model responds; it is a poor way to keep a model current with frequently changing facts.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Chunk size and overlap
How documents are split before embedding.
- Remember
- Overlap preserves meaning across chunk boundaries.
- Tested as
- Missing or partial context scenarios.
- Trap
- Assuming bigger chunks always improve answers.
-
Similarity threshold
Minimum score a result needs to enter the prompt.
- Remember
- Too high drops useful context; too low adds noise.
- Tested as
- Irrelevant context or empty results questions.
- Trap
- Changing top-k when the threshold is the problem.
-
Hybrid search
Combined keyword (BM25) and vector retrieval.
- Remember
- Results are merged with reciprocal rank fusion.
- Tested as
- Exact terms, codes or names being missed.
- Trap
- Relying on vector search alone for identifiers.
-
Semantic ranker
Azure AI Search reranking of top results with language understanding.
- Remember
- Applies after initial retrieval, improving ordering.
- Tested as
- Relevance ordering questions.
- Trap
- Expecting it to retrieve documents that were never returned.
-
Embedding model selection
Choosing the model that converts text to vectors.
- Remember
- Changing the model requires re-embedding the corpus.
- Tested as
- Domain accuracy and cost questions.
- Trap
- Mixing vectors from different embedding models in one index field.
-
A/B testing
Comparing two variants on split traffic with defined metrics.
- Remember
- Needs enough traffic and a clear success metric.
- Tested as
- Proving an optimisation helped.
- Trap
- Judging by a handful of manual tests.
-
Supervised fine-tuning
Training on example inputs and ideal outputs.
- Remember
- Best for format, tone and task behaviour.
- Tested as
- Consistent output style scenarios.
- Trap
- Using it to inject fast-changing facts.
-
Direct preference optimization
Training from preferred versus rejected response pairs.
- Remember
- Needs comparison data rather than single ideal answers.
- Tested as
- Aligning responses to human preferences.
- Trap
- Supplying single-answer data for DPO.
-
Distillation and synthetic data
Using a teacher model's outputs to train a smaller model.
- Remember
- Review synthetic data and keep evaluation sets separate.
- Tested as
- Cost reduction with quality retention.
- Trap
- Letting evaluation examples leak into training data.
-
Fine-tuned model hosting
Deployment of a customised model for inference.
- Remember
- Hosting is billed hourly while deployed, in addition to tokens.
- Tested as
- Cost and lifecycle questions.
- Trap
- Leaving idle fine-tuned deployments running.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
RAG versus fine-tuning
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| RAG | Retrieves current documents at query time. | Adds knowledge without retraining. | Fine-tuning when facts change weekly. |
| Supervised fine-tuning | Retrains on example conversations. | Changes behaviour, style and format. | Expecting it to cite source documents. |
| Prompt engineering | Instructions and examples in the prompt. | Cheapest first step; no training. | Skipping it before fine-tuning. |
Retrieval strategies in Azure AI Search
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Keyword (BM25) | Term matching on text. | Strong on exact identifiers. | Missing paraphrased meaning. |
| Vector | Similarity between embeddings. | Strong on meaning and paraphrase. | Missing exact codes and rare terms. |
| Hybrid with semantic ranker | Both retrievals merged, then reranked. | Best overall relevance for mixed queries. | Thinking the ranker replaces retrieval. |
Fine-tuning methods
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| SFT | Learns from ideal responses. | Needs input and target output pairs. | Using preference pairs. |
| DPO | Learns from preferred and rejected responses. | Needs comparison data. | Expecting it to teach new formats from scratch. |
| Reinforcement fine-tuning | Optimises with graders on reasoning tasks. | Applies to supported reasoning models. | Choosing it for simple tone changes. |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Fine-tune for fresh facts
- What candidates pick
- Fine-tune the model on new product data each month
- Why it looks right
- Fine-tuning sounds like teaching the model.
- Why it's wrong
- Facts that change often belong in a retrieval index; fine-tuning shapes behaviour and needs retraining for every update.
Clue to watch for Frequently updated documents or a need for citations.
-
More context is better
- What candidates pick
- Raise top-k to 50 to stop missed answers
- Why it looks right
- More chunks seem to guarantee the answer is present.
- Why it's wrong
- Extra chunks add noise and tokens and can lower groundedness; fix chunking, thresholds or hybrid retrieval instead.
Clue to watch for Rising cost or worse answers after adding context.
-
Vector search handles everything
- What candidates pick
- Switch to a larger embedding model to find part numbers
- Why it looks right
- Bigger models sound more accurate.
- Why it's wrong
- Exact identifiers are matched best by keyword retrieval, so hybrid search fixes the miss more directly.
Clue to watch for Queries containing codes, SKUs or names.
-
Ranker retrieves documents
- What candidates pick
- Enable the semantic ranker to find documents that never appear
- Why it looks right
- Ranking and retrieval are often blurred.
- Why it's wrong
- The semantic ranker only reorders results already retrieved; missing documents need retrieval or chunking changes.
Clue to watch for The right document is absent from the top results entirely.
-
Train on everything
- What candidates pick
- Use all synthetic data plus the evaluation set for fine-tuning
- Why it looks right
- More data feels better.
- Why it's wrong
- Training on evaluation items inflates scores and hides real performance; keep test data separate and review synthetic samples.
Clue to watch for Requirements about trustworthy evaluation.
-
Lower loss means ready
- What candidates pick
- Deploy as soon as training loss falls
- Why it looks right
- Loss curves look like progress.
- Why it's wrong
- Falling training loss with rising validation loss shows overfitting; evaluate against the base model before production.
Clue to watch for Validation metrics are given in the scenario.
Answer framework
Use these steps when a scenario asks how to improve a generative AI system's quality or cost.
- Diagnose whether the gap is missing or noisy context, which is retrieval, or wrong behaviour, tone or format, which is the model.
- For retrieval gaps, adjust chunk size and overlap, thresholds and top-k, and add hybrid search with semantic ranking for identifiers.
- Revisit the embedding model when domain vocabulary is poorly captured, and re-embed the corpus after any change.
- For behaviour gaps, try prompting first, then choose SFT, DPO, reinforcement fine-tuning or distillation based on the data you have.
- Prove each change with relevance metrics, evaluators or an A/B test, then deploy, version and retire fine-tuned models deliberately to control hosting cost.
Practice question
Want more like this? The AI-300 practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Split your time between retrieval and fine-tuning, but start with retrieval because it is cheaper to experiment with. Build a small Azure AI Search index over a handful of documents, try two chunk sizes, then run the same questions with keyword, vector and hybrid queries plus the semantic ranker. Record which settings retrieve the right chunk and how many tokens reach the prompt. Swap the embedding model once to feel the re-embedding cost.
For fine-tuning, read the Foundry documentation on supported methods and data formats, then prepare a small SFT dataset and, if quota allows, run a short job. Study the training and validation loss charts and compare the result with the base model using Domain 4 evaluators. Note hosting charges and deployment options for fine-tuned models. Finish by writing a decision table that maps symptoms to fixes: missing context, noisy context, exact-term misses, wrong format, wrong tone and stale facts. Most questions in this domain test that mapping, so practise reading scenarios for the root cause before looking at options.
FAQ
How much of AI-300 is Domain 5, Optimize generative AI systems and model performance?
Optimize generative AI systems and model performance is weighted 10–15% of AI-300. With about 50 scored items assumed, that is roughly 5 to 7 questions.
When should I fine-tune instead of using RAG?
Fine-tune when you need consistent behaviour, tone, format or task skill that prompting cannot achieve. Use RAG when answers depend on current or private documents.
What is hybrid search?
Hybrid search runs keyword and vector retrieval in one query and merges results with reciprocal rank fusion. In Azure AI Search, the semantic ranker can then rerank the top results.
Does changing chunk size require re-indexing?
Yes. Chunks are created and embedded at indexing time, so changing size or overlap means reprocessing the documents and rebuilding their vectors.
What is synthetic data used for?
It supplements scarce real examples for fine-tuning or evaluation, often generated by a stronger model. Review it for quality and keep it separate from test data.
What should I check before deploying a fine-tuned model?
Compare training and validation loss for overfitting, evaluate the model against the base model on held-out data, and plan hosting cost and versioning.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 5 knowledge
Test whether you can trace a quality gap to retrieval or the model and choose the optimisation that fixes it.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the AI-300 AI tutor Get any concept explained your way