Overview
Domain 5 covers how to prove a GenAI application works and how to fix it when it does not, in two task statements with fourteen skills between them. Evaluation covers quality metrics beyond classic ML accuracy, such as relevance, factual accuracy, consistency and fluency, Amazon Bedrock model evaluations, A/B and canary testing, user feedback and annotation workflows, regression testing and quality gates, RAG evaluation, LLM-as-a-judge, retrieval relevance and latency testing, agent task completion and tool-use evaluation, stakeholder reporting and deployment validation with synthetic workflows. Troubleshooting covers context window overflow and truncation, FM API integration errors, prompt engineering and prompt maintenance problems, and retrieval issues such as weak embeddings, poor chunking, drift and slow vector search.
At 11%, it is the lightest domain, but it is dense, and the evaluation ideas also support questions in the safety and optimization domains.
How it's tested
Domain 5 questions describe an evaluation need or a symptom in a running GenAI app. You either choose how to measure quality, comparing methods such as automatic metrics, LLM-as-a-judge and human review, or trace a symptom to its root cause in context, prompt, API or retrieval.
-
Choose the evaluator
A quality question is described, and you pick automatic metrics, LLM-as-a-judge, human annotation or user feedback.
-
Isolate the layer
Answers are wrong, and you decide whether retrieval, prompt, model or integration is at fault before changing anything.
-
Gate the release
A model or prompt update is planned, and you choose regression tests, canary releases or quality gates.
After a team re-chunked its document store, answers became vague although the model and prompt were unchanged. Evaluating the generator first would waste time, because only retrieval changed. Running retrieval tests with relevance scoring on a fixed query set shows that the new chunks split key passages. Since the change was isolated to retrieval, retrieval testing is the right first diagnostic.
Task statements 2
Official task statements from the Amazon Web Services outline, in outline order.
5.1 Implement evaluation systems for GenAIQuality metrics, model evaluations, user feedback, QA processes, RAG and agent evaluation, reporting and deployment validation.
- What it means
- This task covers evaluating FM outputs with metrics for relevance, factual accuracy, consistency and fluency, comparing configurations with Amazon Bedrock evaluations, A/B and canary tests and cost-performance analysis, collecting ratings and annotations, running regression tests and quality gates, evaluating RAG with LLM-as-a-judge and human feedback, testing retrieval relevance and latency, measuring agent task completion and tool use, reporting to stakeholders and validating deployments.
- What to understand
- Bedrock evaluations support automatic metrics, LLM-as-a-judge scoring and human evaluation jobs, plus knowledge base (RAG) evaluation of retrieval and generation. Judge models scale scoring but should be calibrated against human ratings. Canary releases expose a new model to a small share of traffic first. Agent evaluation looks at whether the goal was met, which tools were called and how many steps it took.
- How it shows up
- An evaluation goal is described, and you choose the method, metric or release gate that measures it reliably.
- Common confusion
- Using exact-match accuracy for open-ended answers; GenAI quality needs semantic, judge-based or human evaluation.
5.2 Troubleshoot GenAI applicationsContext handling, API integration, prompt engineering, retrieval and prompt maintenance issues.
- What it means
- This task covers diagnosing context window overflow and truncation with dynamic chunking and prompt redesign, fixing FM API problems through error logging, request validation and response analysis, troubleshooting prompts with testing frameworks and version comparison, resolving retrieval issues such as poor embeddings, drift, bad chunking and slow vector search, and maintaining prompts with template tests, CloudWatch Logs, X-Ray and schema validation.
- What to understand
- Truncated answers usually trace to a max token limit or an oversized prompt that leaves little room for output. Validation exceptions often come from a request body that does not match the model's schema. Prompt regressions are found by comparing versions against the same test set. Retrieval faults show up as low relevance scores even when the model behaves normally.
- How it shows up
- A symptom such as truncation, errors, inconsistency or irrelevant answers is described, and you choose the diagnostic or fix.
- Common confusion
- Changing several things at once; isolating one layer at a time is how the root cause is found.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Amazon Bedrock evaluations
Managed jobs that score models or knowledge bases with metrics, judge models or human reviewers.
- Remember
- Supports model comparison and RAG evaluation.
- Tested as
- Evaluation method questions.
- Trap
- Building a custom harness when the managed one fits.
-
LLM-as-a-judge
An FM grades outputs against a rubric.
- Remember
- Scales well; calibrate against human ratings.
- Tested as
- Open-ended quality questions.
- Trap
- Using the same model to judge its own output uncritically.
-
Human evaluation
Reviewers rate or annotate responses.
- Remember
- Best for nuance, tone and domain judgment.
- Tested as
- Subjective quality questions.
- Trap
- Using it for every regression run.
-
RAG evaluation
Separately scoring retrieval quality and answer faithfulness.
- Remember
- Shows whether the retriever or generator is at fault.
- Tested as
- RAG quality questions.
- Trap
- Scoring only the final answer.
-
Faithfulness and groundedness
Whether the answer is supported by retrieved context.
- Remember
- Core hallucination metric for RAG.
- Tested as
- Factual accuracy questions.
- Trap
- Confusing it with relevance.
-
Canary testing
Sending a small share of traffic to a new model or prompt first.
- Remember
- Limits blast radius during updates.
- Tested as
- Deployment validation questions.
- Trap
- Switching all traffic at once.
-
Regression testing
Rerunning a fixed test set after each change.
- Remember
- Catches quality drops from prompt or model updates.
- Tested as
- QA process questions.
- Trap
- Testing only new features.
-
Agent evaluation metrics
Task completion rate, tool selection accuracy and step efficiency.
- Remember
- Measures the whole multi-step workflow.
- Tested as
- Agent performance questions.
- Trap
- Judging only the final message.
-
Context window overflow
Prompt plus retrieved content exceeds or crowds the model limit.
- Remember
- Causes truncation or missing context.
- Tested as
- Content handling questions.
- Trap
- Switching models instead of trimming context.
-
Max tokens limit
The cap on generated output length.
- Remember
- Too low causes cut-off answers.
- Tested as
- Truncation questions.
- Trap
- Blaming the model for a configuration limit.
-
Embedding drift
Mismatch between how documents and queries are embedded over time.
- Remember
- Re-embed after changing the embedding model.
- Tested as
- Retrieval troubleshooting questions.
- Trap
- Mixing vectors from two embedding models.
-
Prompt version comparison
Running old and new prompt versions on the same inputs.
- Remember
- Shows whether a change helped.
- Tested as
- Prompt troubleshooting questions.
- Trap
- Judging prompts on one example.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Evaluation methods
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Automatic metrics | Computed scores such as similarity or robustness | Fast and cheap at scale | Missing nuance |
| LLM-as-a-judge | A model scores against a rubric | Scalable open-ended scoring | Skipping human calibration |
| Human evaluation | People rate or annotate | Highest nuance, slowest | Using it for every build |
| User feedback | Ratings from real users | Reflects production experience | Treating sparse ratings as complete |
Troubleshooting by symptom
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Answer cut off | Output stops mid-sentence | Check max tokens and context size | Changing models first |
| Validation error | API rejects request | Check request schema for the model | Retrying unchanged requests |
| Irrelevant answer | Model responds off-topic | Check retrieval relevance first | Rewriting the prompt only |
| Inconsistent format | Output structure varies | Schema validation and prompt specs | Raising temperature |
Release validation approaches
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Regression test set | Fixed prompts with expected results | Detects quality drops before release | Too few cases |
| Canary release | Small traffic share to the new version | Limits user impact | No rollback criteria |
| A/B test | Split traffic to compare versions | Measures real-user preference | Ending tests too early |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Accuracy for free text
- What candidates pick
- Score chatbot answers by exact match against a reference.
- Why it looks right
- Accuracy is the familiar ML metric.
- Why it's wrong
- Correct answers can be phrased many ways; semantic metrics, judge models or human ratings evaluate open-ended text properly.
Clue to watch for Responses are open-ended or conversational.
-
Blame the generator
- What candidates pick
- Tune the prompt or swap the model when RAG answers degrade.
- Why it looks right
- The answer is what users see.
- Why it's wrong
- If retrieval changed or returns poor passages, generator changes cannot help; test retrieval relevance first.
Clue to watch for Recent change to chunking, embeddings or index.
-
Bigger model for truncation
- What candidates pick
- Move to a larger model when responses cut off.
- Why it looks right
- Larger models handle more.
- Why it's wrong
- Truncation usually comes from a low max tokens setting or a prompt that fills the context window.
Clue to watch for Answers stop mid-sentence consistently.
-
Judge without calibration
- What candidates pick
- Rely only on LLM-as-a-judge scores for a release.
- Why it looks right
- It is fast and automatic.
- Why it's wrong
- Judge models have biases; checking them against a sample of human ratings keeps scores trustworthy.
Clue to watch for High-stakes quality decision.
-
Mixed embeddings
- What candidates pick
- Change the embedding model and keep the old index.
- Why it looks right
- Re-indexing seems expensive.
- Why it's wrong
- Vectors from different embedding models are not comparable, so the corpus must be re-embedded.
Clue to watch for Retrieval collapsed after an embedding model change.
-
Big-bang release
- What candidates pick
- Switch all traffic to a new model after offline tests.
- Why it looks right
- Offline tests passed.
- Why it's wrong
- Production traffic differs from test sets; a canary or A/B rollout with rollback limits harm.
Clue to watch for Model or prompt update to a live application.
Answer framework
Apply these steps to a Domain 5 evaluation or troubleshooting scenario.
- Decide whether the scenario asks you to measure quality, compare options, gate a release or diagnose a specific symptom.
- For measurement, match the evaluator to the quality aspect: automatic metrics for scale, LLM-as-a-judge for open-ended text and humans for nuance.
- For RAG and agents, evaluate components separately, retrieval relevance and faithfulness for RAG, task completion and tool use for agents.
- For symptoms, isolate one layer at a time, checking configuration limits, request schemas, retrieval and prompts before changing the model. Many failures trace back to token limits, throttling or an empty retrieval result rather than the model itself.
- Protect releases with regression test sets, canary or A/B rollouts, automated quality gates and clear rollback criteria.
Practice question
Want more like this? The AIP-C01 practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Although this is the lightest domain, it ties the exam together, so study it after Domains 1 and 2 when you already have a working RAG or agent example. Start by running one Amazon Bedrock model evaluation job that compares two models, then a second job using a judge model. Look at what each metric means and where the judge disagrees with your own rating.
Next, evaluate a knowledge base so you see retrieval and generation scored separately. Change the chunk size, rerun the evaluation and note which metric moves. This makes the 'isolate the layer' habit concrete.
For troubleshooting, build a short symptom table: truncation, validation errors, irrelevant answers, inconsistent formats and slow retrieval, each with its first diagnostic step. Practice by deliberately breaking your sample app, for example lowering max tokens or mixing embedding models, and fixing it.
Finish with release practice: a small regression set, a quality gate in a pipeline and a canary plan. Review practice questions by naming the layer at fault before reading the options.
FAQ
How much of AIP-C01 is Domain 5?
Testing, Validation, and Troubleshooting is 11% of the scored content, the smallest domain, with two task statements: evaluation systems and troubleshooting.
Why not use accuracy to evaluate an LLM?
Open-ended answers can be correct in many wordings, so exact-match accuracy undercounts good responses. Semantic metrics, judge models and human ratings fit better.
What can Amazon Bedrock evaluations do?
Run automatic metric, LLM-as-a-judge and human evaluation jobs to compare models, and evaluate knowledge bases for both retrieval and generated answer quality.
How do I evaluate a RAG application?
Score retrieval relevance and context coverage separately from answer faithfulness and correctness, so you know which component needs work.
How do I evaluate an agent?
Measure task completion rate, whether the right tools were chosen, how many steps were taken and the quality of reasoning across the workflow.
Why are my model responses cut off?
Usually the max tokens setting is too low or the prompt and retrieved context leave little room for output. Check both before changing models.
How should model updates be released?
Run regression tests and quality gates first, then roll out through a canary or A/B test with clear rollback criteria.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 5 knowledge
You have covered GenAI evaluation, Bedrock model evaluations, RAG and agent testing, release validation and troubleshooting. Test whether you can isolate the failing layer under exam conditions.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the AIP-C01 AI tutor Get any concept explained your way