Overview
Experimentation lists seven objectives on NVIDIA's exam page. Two concern training: assisting with model training and optimization, and preparing large datasets for pretraining, fine-tuning and RLHF through steps such as scraping and tokenization. One covers designing hardware or software tests for LLM applications, and another covers judging current or emerging technologies on cost, portability, compatibility and usability. A further objective concerns collecting data from human subjects, with RLHF as the example. The last two are about evaluation: refining existing models through benchmarking, and executing experiments that evaluate models and complete pipelines.
In NVIDIA terms this domain maps to the NeMo framework for customization and alignment, NeMo Curator for data preparation, and evaluation practices for both raw models and RAG systems. The study guide's readings add A/B testing, zero-shot testing, hallucination, GLUE and RAG evaluation.
How it's tested
Questions here are scenario-driven. A team has a training goal, a dataset, limited GPUs or an evaluation result, and you pick the method, preparation step or test that fits. Expect terminology on PEFT methods, RLHF stages and benchmarks, plus judgment about trade-offs such as memory, cost and how far a result generalizes.
-
Choose the customization method
A model must be adapted under stated limits on memory, data or time, and you choose full fine-tuning, LoRA, prompt tuning or prompting.
-
Order the pipeline
Steps for preparing data or aligning a model are listed, and you identify the missing or correctly ordered stage.
-
Judge the evaluation
A benchmark or A/B result is reported and you decide whether it is valid or which metric or test would be better.
A team fine-tunes a model and reports a large jump on a public benchmark. Later, users see little improvement. Checking the fine-tuning corpus reveals that it was scraped from sites that also host the benchmark questions. The gain likely came from contamination, not skill. The fix is deduplicating training data against evaluation sets and confirming with a private held-out test or an A/B test with real users.
Objectives 7
Official objectives from the NVIDIA outline, in outline order.
3.1 Assist in model training and training optimization under the supervision of a senior team member.Training settings, mixed precision and parameter-efficient fine-tuning.
- What it means
- This covers supporting training runs: setting hyperparameters, watching loss, and applying techniques that make training fit the available GPUs and finish faster.
- What to understand
- Learning rate, warmup, batch size and epochs drive results. Mixed precision with BF16 or FP16 cuts memory and speeds up Tensor Core math. Gradient accumulation simulates larger batches, and activation checkpointing trades compute for memory. Supervised fine-tuning updates all weights, while parameter-efficient methods such as LoRA, QLoRA, adapters and p-tuning train a small set of parameters, which NeMo supports.
- How it shows up
- Memory or cost limits are stated and you choose the training technique or PEFT method that fits.
- Common confusion
- Assuming LoRA retrains every weight; it freezes the base model and learns small low-rank updates.
3.2 Assist in preparing (e.g., scraping, tokenization) large datasets for pretraining, fine-tuning, and RLHF.Collecting, cleaning, deduplicating and tokenizing training data.
- What it means
- Each training stage needs its own data shape: raw text for pretraining, instruction and response pairs for supervised fine-tuning, and ranked or paired responses for RLHF or preference optimization.
- What to understand
- Preparation steps include scraping or collecting text, extracting clean content, language identification, quality filtering, exact and fuzzy deduplication, PII redaction and toxicity filtering. NeMo Curator runs these steps at scale on GPUs. Tokenization converts text into token IDs with the tokenizer that matches the model, and sequences are packed or truncated to the context length.
- How it shows up
- A data problem such as duplicates or leaked personal data is described and you pick the preparation step that addresses it.
- Common confusion
- Using a different tokenizer from the one the model was trained with, which produces meaningless token IDs.
3.3 Assist in the design and conduct of hardware or software tests for LLM applications.Load, regression and comparison tests for LLM systems.
- What it means
- This is about planning and running tests that show how an LLM application behaves on given hardware and software, from functional checks to performance under load.
- What to understand
- Software tests include regression suites of prompts with expected behaviors, checks on output format, and A/B tests that compare a new prompt or model against the current one with real traffic. Hardware tests measure tokens per second, time to first token and memory use across GPU types, batch sizes and precision settings. Keep all other variables fixed so differences can be attributed.
- How it shows up
- A test plan is described and you identify the flaw or the measurement it lacks.
- Common confusion
- Changing hardware and model version at once, which makes the cause of any difference unknowable.
3.4 Assist in the evaluation of current or emerging technologies to consider factors such as cost, portability, compatibility, or usability.Weighing models, frameworks and formats on practical criteria.
- What it means
- Teams regularly choose between models, serving stacks and formats. This objective asks you to compare options on total cost, how easily they move between environments, whether they work with existing hardware and software, and how easy they are to use.
- What to understand
- Cost includes GPU hours, memory footprint and per-token pricing for hosted APIs. Portability covers open formats such as ONNX and containerized delivery such as NIM microservices that run on cloud, data center or workstation GPUs. Compatibility checks include CUDA versions, supported architectures and licensing terms. Usability covers APIs, documentation and tooling.
- How it shows up
- Two technologies are compared and you identify which criterion favors one of them.
- Common confusion
- Choosing the highest-scoring model without counting serving cost or hardware fit.
3.5 Awareness of, and/or participation in, data collection from human subjects (e.g., RLHF).Human preference data and the RLHF pipeline.
- What it means
- Reinforcement learning from human feedback uses people's judgments of model outputs to align behavior with human preferences. Associates should know the stages and the care that human data collection requires.
- What to understand
- RLHF typically follows supervised fine-tuning. Annotators rank or compare responses, a reward model learns to predict those preferences, and the policy model is optimized against the reward model, classically with PPO. Direct preference optimization skips the separate reward model. Clear annotator guidelines, inter-annotator agreement checks, informed consent and fair treatment of annotators all affect data quality.
- How it shows up
- RLHF stages are listed and you identify the missing step or what human annotators contribute.
- Common confusion
- Thinking humans score every training step live; they produce preference data that trains a reward model.
3.6 Evaluate and refine existing models / benchmarking.Benchmarks, contamination and iterative improvement.
- What it means
- This covers measuring a model on standard and custom benchmarks, then using the results to decide what to refine.
- What to understand
- Common benchmarks include GLUE and SuperGLUE for language understanding, MMLU for broad knowledge, HellaSwag for commonsense and HumanEval for code. Zero-shot and few-shot evaluation show how a model handles tasks without task-specific training. Results can be inflated by contamination when benchmark items leaked into training data. Domain-specific test sets and LLM-as-a-judge scoring complement public benchmarks.
- How it shows up
- A benchmark result is described and you judge whether it supports the claimed improvement.
- Common confusion
- Treating a public leaderboard score as proof of performance on a company's own task.
3.7 Executes experimentation to evaluate models and pipelines.Controlled experiments on models and full RAG pipelines.
- What it means
- Beyond single models, teams evaluate whole pipelines, such as retriever, reranker and generator together, by running controlled experiments and tracking results.
- What to understand
- Change one variable at a time, such as chunk size, embedding model, reranker or prompt, while holding the rest fixed. Track runs with configuration and seeds. For RAG, measure retrieval quality with recall at k and generation quality with faithfulness to retrieved context, answer relevance and hallucination rate. Repeat runs or use enough samples to tell real gains from noise.
- How it shows up
- A pipeline experiment is described and you pick the metric or design change that makes it valid.
- Common confusion
- Judging a RAG pipeline only by answer fluency, ignoring whether answers are grounded in retrieved context.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Parameter-efficient fine-tuning
Training a small subset or add-on of parameters while the base model stays frozen.
- Remember
- It cuts memory and storage compared with full fine-tuning.
- Tested as
- Customization method questions.
- Trap
- Thinking it always matches full fine-tuning quality.
-
LoRA
Adds trainable low-rank matrices to selected weight layers.
- Remember
- Base weights are frozen; adapters are small and swappable.
- Tested as
- PEFT questions.
- Trap
- Saying LoRA updates all weights.
-
QLoRA
LoRA applied on top of a base model quantized to 4 bits.
- Remember
- It lets large models be tuned on a single GPU.
- Tested as
- Memory-limited training questions.
- Trap
- Confusing it with inference-only quantization.
-
Prompt tuning and p-tuning
Learn soft prompt embeddings rather than model weights.
- Remember
- Very few trainable parameters.
- Tested as
- NeMo customization questions.
- Trap
- Treating them as manually written prompts.
-
Mixed precision
Training with BF16 or FP16 while keeping key values in higher precision.
- Remember
- It reduces memory and speeds up Tensor Core math.
- Tested as
- Training optimization questions.
- Trap
- Assuming it always hurts accuracy.
-
RLHF stages
Supervised fine-tuning, human preference collection, reward model, policy optimization.
- Remember
- The reward model stands in for human judgment.
- Tested as
- Alignment questions.
- Trap
- Skipping the reward model stage.
-
Direct preference optimization
Optimizes the model directly on preference pairs.
- Remember
- No separate reward model is trained.
- Tested as
- Alignment comparison questions.
- Trap
- Calling it a type of supervised pretraining.
-
Deduplication
Removing exact and near-duplicate documents from training data.
- Remember
- It reduces memorization and benchmark contamination.
- Tested as
- Data preparation questions.
- Trap
- Only removing exact copies.
-
NeMo Curator
NVIDIA's GPU-accelerated toolkit for curating LLM training data.
- Remember
- Handles filtering, deduplication and PII redaction at scale.
- Tested as
- NVIDIA tooling questions.
- Trap
- Thinking it serves models.
-
Benchmark contamination
Test items appear in training data, inflating scores.
- Remember
- Check overlap before trusting results.
- Tested as
- Evaluation validity questions.
- Trap
- Accepting leaderboard gains at face value.
-
A/B testing
Comparing two variants on live or held-out traffic under the same conditions.
- Remember
- Randomize assignment and change one thing.
- Tested as
- Testing questions.
- Trap
- Comparing variants on different user groups or time periods.
-
RAG evaluation metrics
Retrieval recall, context relevance, answer faithfulness and answer relevance.
- Remember
- Evaluate retrieval and generation separately.
- Tested as
- Pipeline evaluation questions.
- Trap
- Scoring only the final answer's fluency.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
Model customization options
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Full fine-tuning | Update every weight on task data | Most flexible, highest memory and cost | Choosing it on a single small GPU |
| LoRA or QLoRA | Train low-rank adapters on a frozen base | Small, cheap, swappable adapters | Expecting it to add retrievable facts reliably |
| Prompt or p-tuning | Learn soft prompt vectors only | Fewest trainable parameters | Confusing soft prompts with written prompts |
| Prompt engineering | Change inputs only | No training at all | Expecting it to teach a new skill |
Alignment approaches
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Supervised fine-tuning | Learn from demonstration responses | Teaches the format of good answers | Assuming it captures preferences between answers |
| RLHF with PPO | Reward model trained on rankings guides optimization | Separate reward model, more complex | Leaving out the reward model |
| Direct preference optimization | Optimize directly on chosen versus rejected pairs | Simpler, no reward model | Thinking it needs no human preference data |
Evaluation methods
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Public benchmarks | Standard tasks such as MMLU or GLUE | Comparable across models | Ignoring contamination and task mismatch |
| Custom held-out set | Examples from the target use case | Most relevant to the real task | Letting them leak into training |
| A/B test | Two variants with real users | Measures actual impact | Changing several variables at once |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Full fine-tune on one GPU
- What candidates pick
- Fine-tune all weights of a large model on a single memory-limited GPU.
- Why it looks right
- It sounds like the most powerful option.
- Why it's wrong
- All weights plus optimizer states will not fit; LoRA or QLoRA trains a small set of parameters within the budget.
Clue to watch for The scenario stresses limited GPU memory or cost.
-
Humans grade every step
- What candidates pick
- Describe RLHF as people scoring outputs live during optimization.
- Why it looks right
- Human feedback is in the name.
- Why it's wrong
- Humans produce preference rankings once, a reward model learns from them, and optimization uses the reward model.
Clue to watch for The question asks what annotators actually produce.
-
Leaderboard equals readiness
- What candidates pick
- Ship a model because it tops a public benchmark.
- Why it looks right
- Benchmarks look objective.
- Why it's wrong
- Public scores may be contaminated or unrelated to the target task; a domain test set or A/B test is needed.
Clue to watch for The use case differs from the benchmark's tasks.
-
Any tokenizer works
- What candidates pick
- Tokenize fine-tuning data with a convenient generic tokenizer.
- Why it looks right
- Tokenization seems like simple text splitting.
- Why it's wrong
- Token IDs only mean something to the model's own vocabulary, so mismatched tokenizers corrupt the input.
Clue to watch for The data preparation step mentions a different tokenizer.
-
Exact dedup is enough
- What candidates pick
- Remove only identical documents from a scraped corpus.
- Why it looks right
- Exact matching is easy and cheap.
- Why it's wrong
- Near-duplicates with small edits still cause memorization and contamination; fuzzy deduplication catches them.
Clue to watch for The corpus comes from scraping many mirrored sites.
-
Many changes per run
- What candidates pick
- Swap the embedding model, chunk size and prompt in one experiment.
- Why it looks right
- It saves time.
- Why it's wrong
- With several changes at once, any improvement cannot be traced to its cause.
Clue to watch for The experiment reports a change but several settings differ.
Answer framework
Use these steps when a Domain 3 scenario asks how to train, prepare data for, or evaluate an LLM.
- Identify the stage involved, whether data preparation, training, alignment, testing or evaluation, before looking at the options.
- For training, compare the goal with the memory, data and time available, then pick the lightest method that achieves it.
- For data preparation, match the defect, such as duplicates, noise, personal data or wrong format, to the curation step that removes it.
- For alignment, check the order of supervised fine-tuning, preference collection, reward modeling and optimization, or note DPO's shortcut.
- For evaluation, confirm that only one variable changed, the test data is clean and the metric fits the task before accepting a result.
Practice question
Want more like this? The NCA-GENL practice test has questions for every domain.
Checklist
0 of 13
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Split this domain into three blocks: customization, data and evaluation. For customization, build a comparison table of full fine-tuning, LoRA, QLoRA, adapters and p-tuning with columns for trainable parameters, memory need and typical use. Then add mixed precision and gradient accumulation as memory tools. If you can, run a short LoRA job with NeMo or Hugging Face PEFT on a small open model to see the trainable parameter count drop.
For data, walk through a curation pipeline in order: collect, extract, filter, deduplicate, redact, tokenize, then format for pretraining, SFT or preference training. Learn the RLHF sequence until you can draw it from memory, and note where DPO differs.
For evaluation, learn what the main benchmarks measure, why contamination matters, and how to design a fair A/B test. Evaluate a small RAG pipeline twice with one setting changed, recording recall at k and faithfulness. Set aside about two weeks of part-time study, then use practice questions and review every miss by asking which stage of the workflow it belonged to.
FAQ
How much of NCA-GENL is Experimentation?
Experimentation is 22% of the exam, roughly eleven to thirteen questions, making it the third-largest domain after Core ML and AI Knowledge and Software Development.
Why does the study guide PDF show different objectives for this domain?
The January 2025 study guide repeats the Data Analysis objectives under Experimentation, which appears to be a copy error. NVIDIA's exam web page lists the seven Experimentation objectives used on this page.
Do I need to know specific PEFT methods?
Yes, at a conceptual level. Know that LoRA trains low-rank adapters on a frozen model, QLoRA adds 4-bit quantization of the base, and prompt or p-tuning learns soft prompt vectors.
What are the stages of RLHF?
Supervised fine-tuning, collecting human preference rankings, training a reward model on them, and optimizing the model against that reward, classically with PPO.
Which NVIDIA tools appear in this domain?
NeMo framework for training, customization and alignment, and NeMo Curator for large-scale data preparation, along with general GPU training practices such as mixed precision.
How should a RAG pipeline be evaluated?
Measure retrieval and generation separately: recall or relevance of retrieved chunks, then faithfulness of the answer to those chunks and its relevance to the question.
What makes an experiment trustworthy?
One variable changed at a time, clean held-out data with no contamination, tracked configurations and enough samples to separate a real gain from noise.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 3 knowledge
You have covered training optimization, PEFT, data curation, RLHF and benchmarking. Test whether you can pick the right method and spot a flawed experiment.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the NCA-GENL AI tutor Get any concept explained your way