NCP-AII · Domain 5 of 5

NCP-AII Domain 5: Troubleshooting and Optimization

Domain 5 covers diagnosing and replacing failed GPUs, network cards, fans and power supplies, then tuning AMD and Intel host settings and storage so GPU servers deliver expected performance.

Exam weight
12%
Official objectives
5
Estimated questions estimate
~9
Suggested study time estimate
10–15 hrs

About 12 min read Reviewed October 2026

Overview

Troubleshooting and Optimization has five objectives. Three deal with hardware faults: identifying and troubleshooting faults in components such as GPUs, fans and network cards, identifying which card, GPU or power supply is faulty, and replacing those parts correctly. The tools are the ones used throughout the exam, applied to failure cases: nvidia-smi and DCGM for GPU health, kernel logs for XID errors, the BMC event log and sensors for fans and power supplies, and NVIDIA System Management on DGX systems. The other two objectives deal with performance. Executing performance optimization for AMD and Intel servers covers BIOS and OS settings such as NUMA layout, PCIe and IOMMU options, power profiles and CPU and NIC affinity. Storage optimization covers client settings, network tuning and GPUDirect Storage.

At 12%, this domain is about nine questions.

How it's tested

Questions in this domain present symptoms and evidence, such as an XID message, a BMC event, a DCGM failure or a performance gap, and ask which component is at fault or which change will help. Some questions ask for the correct replacement procedure or the right BIOS or OS setting for a CPU platform.

  • Read the evidence

    A log line or alert is shown, and you identify the failing component from XID codes, sensor readings or event entries.

  • Replace safely

    A part has failed, and you choose the correct order of drain, power down, replace, update firmware and revalidate.

  • Tune the host

    A node underperforms, and you choose the BIOS or OS setting for AMD or Intel that fixes GPU, NIC or storage throughput.

Reasoning it through

Kernel logs on one node show a GPU falling off the bus, and jobs on that node fail. Reinstalling the driver will not fix a GPU that disappears from PCIe. The engineer should drain the node in Slurm, collect an nvidia-bug-report and BMC logs, reseat or replace the GPU or tray following the service manual, then rerun DCGM diagnostics before returning the node to service.

Objectives 5

Official objectives from the NVIDIA outline, in outline order.

5.1 Identify and troubleshoot hardware faults (e.g., GPU/fan/network card)Diagnosing faults from logs, sensors and diagnostics.
What it means
This covers diagnosing hardware problems in GPUs, fans, network cards and related components by reading kernel XID errors, DCGM diagnostic results, BMC sensor data and event logs, and link diagnostics.
What to understand
XID errors in the kernel log identify GPU events, such as memory errors or a GPU dropping off the bus. Fan faults show as BMC sensor alarms and often cause thermal throttling. Network card faults appear as link errors, flapping or missing devices. nvidia-bug-report.sh and nvsm dump health collect evidence for support.
How it shows up
A symptom and log excerpt are given, and you identify the fault and the next diagnostic step.
Common confusion
Treating every XID as a software problem that a driver reinstall will fix.
5.2 Identify faulty cards/GPUs/power suppliesPinpointing which specific component has failed.
What it means
This covers narrowing a fault to the exact GPU, network card or power supply, using serial numbers, PCI bus IDs, slot locations, BMC inventory and sensor data so the correct part is replaced.
What to understand
nvidia-smi reports bus IDs and serial numbers, which map to physical slots or tray positions through system documentation. Power supply faults appear in the BMC event log with the supply number. Correct identification prevents removing a healthy part and leaving the faulty one in place.
How it shows up
Inventory and log data are provided, and you choose the component to replace.
Common confusion
Mapping a GPU index from nvidia-smi directly to a physical slot without checking the bus ID.
5.3 Replace faulty cards/GPUs/power suppliesSafe, documented replacement and revalidation.
What it means
This covers replacing failed GPUs, cards and power supplies: draining workloads, following the service manual for power-down and handling, installing the part, updating its firmware to the validated level and revalidating the node.
What to understand
Many power supplies are hot-swappable in redundant configurations, while GPUs and most cards need the node powered down. Replacement parts may arrive with different firmware, so they must be brought to the cluster baseline. Revalidation with diagnostics, and NCCL where relevant, is required before returning the node.
How it shows up
A replacement scenario is described, and you choose the next step or the missing step.
Common confusion
Returning a node to service right after replacing a part, without firmware alignment or tests.
5.4 Execute performance optimization for AMD and Intel ServersHost BIOS and OS tuning for GPU workloads.
What it means
This covers tuning AMD EPYC and Intel Xeon based GPU servers through BIOS and OS settings, including NUMA configuration, PCIe and IOMMU options, power and performance profiles, CPU frequency governors and binding processes to the CPU and NIC nearest each GPU.
What to understand
On AMD, NUMA nodes per socket (NPS) settings change memory and PCIe locality; on Intel, sub-NUMA clustering plays a similar role. IOMMU and PCIe ACS settings can block or slow peer-to-peer traffic needed by GPUDirect RDMA. Performance power profiles and correct CPU and NIC affinity avoid latency spikes and cross-socket traffic.
How it shows up
A node underperforms on one CPU platform, and you choose the BIOS or OS change.
Common confusion
Applying one vendor's BIOS option names and defaults to the other platform.
5.5 Storage OptimizationImproving storage throughput to GPU nodes.
What it means
This covers improving storage performance for training and checkpointing through client mount and tuning options, network settings such as MTU and RDMA, local NVMe caching, and GPUDirect Storage where supported.
What to understand
Storage bottlenecks show as low GPU utilization during data loading or long checkpoint times. Fixes include using RDMA-capable protocols, matching MTU, adding parallel connections, using local NVMe as a cache and enabling GPUDirect Storage to move data directly into GPU memory. Changes should be measured with the same tests used in Domain 4.
How it shows up
GPUs sit idle waiting for data, and you choose the storage change that helps most.
Common confusion
Adding GPUs to fix a job that is actually limited by storage throughput.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • XID errors

    GPU driver event codes written to the kernel log.

    Remember
    They point to specific GPU fault types.
    Tested as
    GPU fault questions.
    Trap
    Ignoring them as noise.

  • GPU fell off the bus

    An XID event where the GPU disappears from PCIe.

    Remember
    Usually hardware, power or seating related.
    Tested as
    Fault identification questions.
    Trap
    Reinstalling the driver.

  • ECC and row remapping

    Memory error correction and spare-row replacement on data center GPUs.

    Remember
    Pending remaps need a GPU reset.
    Tested as
    Memory fault questions.
    Trap
    Replacing the GPU after one correctable error.

  • nvidia-bug-report.sh

    Script that gathers GPU and system diagnostics for support.

    Remember
    Run it before changing hardware.
    Tested as
    Evidence collection questions.
    Trap
    Collecting it after the part is swapped.

  • BMC event log (SEL)

    Record of power, fan, thermal and hardware events.

    Remember
    First stop for PSU and fan faults.
    Tested as
    PSU and fan questions.
    Trap
    Searching OS logs for PSU faults first.

  • Bus ID mapping

    Use PCI bus IDs and serials to locate the physical GPU or card.

    Remember
    GPU index order may not match slot order.
    Tested as
    Identification questions.
    Trap
    Assuming index equals slot.

  • Drain before repair

    Remove the node from scheduling before servicing.

    Remember
    Protects running jobs.
    Tested as
    Replacement questions.
    Trap
    Powering off a node with active jobs.

  • Firmware after replacement

    Bring new parts to the validated firmware set.

    Remember
    Spares may ship with other versions.
    Tested as
    Replacement questions.
    Trap
    Skipping the firmware check.

  • NPS and SNC

    AMD NUMA-per-socket and Intel sub-NUMA clustering settings.

    Remember
    They change memory and PCIe locality.
    Tested as
    Host tuning questions.
    Trap
    Mixing vendor terminology.

  • IOMMU and ACS

    PCIe isolation features that can block peer-to-peer traffic.

    Remember
    Check them for GPUDirect RDMA performance.
    Tested as
    Host tuning questions.
    Trap
    Leaving ACS redirecting P2P traffic.

  • CPU and NIC affinity

    Bind each GPU's process to its nearest CPU and NIC.

    Remember
    Use nvidia-smi topo -m to choose.
    Tested as
    Performance questions.
    Trap
    Letting traffic cross sockets.

  • GPUDirect Storage

    Direct data path between storage and GPU memory.

    Remember
    Bypasses the CPU bounce buffer.
    Tested as
    Storage questions.
    Trap
    Assuming any storage supports it.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

Where to look for each fault

ConceptMeaningKey differenceCommon trap
GPU fault XID in kernel log, DCGM results Software-visible events Checking only the BMC
Fan fault BMC sensors and event log Often causes thermal throttling Blaming the GPU first
PSU fault BMC event log with supply number May be hot-swappable Searching OS logs
Network card fault Link errors, missing device Use link diagnostics Assuming a switch fault

AMD vs. Intel host tuning

ConceptMeaningKey differenceCommon trap
AMD EPYC NPS setting controls NUMA per socket Affects PCIe and memory locality Using Intel option names
Intel Xeon Sub-NUMA clustering controls NUMA split Similar goal, different setting Using AMD option names
Both platforms Performance profile, IOMMU, ACS, affinity Shared tuning principles Leaving power-saving defaults

Storage optimization options

ConceptMeaningKey differenceCommon trap
RDMA transport Remote direct memory access for storage traffic Lower CPU use and latency Running over TCP by mistake
Local NVMe cache Caches datasets on local drives Cuts repeated remote reads Using it for durable checkpoints
GPUDirect Storage Storage to GPU memory directly Skips CPU bounce buffer Expecting it without support

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. Driver fixes hardware

    What candidates pick
    Reinstall the driver after an XID showing the GPU fell off the bus.
    Why it looks right
    Driver issues are common.
    Why it's wrong
    A GPU disappearing from PCIe points to hardware, seating or power, which software changes cannot repair.

    Clue to watch for The log shows the device lost from the bus.

  2. Index equals slot

    What candidates pick
    Pull the GPU in the slot numbered after the nvidia-smi index.
    Why it looks right
    The numbers look aligned.
    Why it's wrong
    GPU enumeration order can differ from physical layout; bus IDs and serial numbers identify the device.

    Clue to watch for The question gives bus IDs or serial numbers.

  3. Swap and go

    What candidates pick
    Return the node to Slurm right after the new part is installed.
    Why it looks right
    The hardware is new.
    Why it's wrong
    Spare parts may carry different firmware and must be tested, so firmware alignment and diagnostics come first.

    Clue to watch for The scenario ends right after physical replacement.

  4. Replace on first ECC

    What candidates pick
    Replace a GPU after a single correctable memory error.
    Why it looks right
    Any memory error sounds serious.
    Why it's wrong
    Correctable errors and row remapping are handled by the GPU; replacement is for persistent or uncorrectable faults.

    Clue to watch for Errors are correctable or a remap is pending.

  5. More GPUs for slow I/O

    What candidates pick
    Add GPUs to speed up a job with idle GPUs during data loading.
    Why it looks right
    More compute usually helps.
    Why it's wrong
    If GPUs wait on storage, extra GPUs wait too; the storage path needs tuning.

    Clue to watch for GPU utilization drops during data loading.

  6. Power-saving defaults

    What candidates pick
    Leave BIOS power profiles at energy-saving defaults.
    Why it looks right
    Defaults seem safe.
    Why it's wrong
    Energy-saving states add latency and reduce throughput for GPU and network-heavy workloads.

    Clue to watch for Performance varies or lags on otherwise healthy nodes.

Answer framework

Use these steps to move from symptom to fix in a troubleshooting or tuning scenario.

  1. Classify the symptom as a GPU, fan, power, network card, host configuration or storage problem using the evidence given.
  2. Collect evidence from the right source: XID and DCGM for GPUs, BMC sensors and event log for fans and power, link diagnostics for network cards.
  3. Identify the exact component using bus IDs, serial numbers or supply numbers, and drain the node before any service work.
  4. Replace the part following the service manual, align its firmware with the validated set and rerun diagnostics before returning the node.
  5. For performance gaps on healthy hardware, check NUMA, IOMMU, ACS, power profile and affinity on the CPU platform, then the storage client and network settings.

Practice question

A DGX node's BMC event log reports that power supply 3 has failed, while the remaining supplies are healthy and jobs are still running. What is the best course of action?

Want more like this? The NCP-AII practice test has questions for every domain.

Checklist

0 of 12

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

Approach this domain as practice in reasoning from evidence. Start by building a symptom table: for GPUs, fans, power supplies and network cards, list the log or tool that shows the fault and the usual fix. Learn the common XID codes NVIDIA documents, such as those for memory errors and a GPU falling off the bus, and what each implies.

Next, read the DGX service manual and best practices guide linked on the certification page, focusing on replacement procedures, hot-swap rules and the steps after a replacement. Practice mapping nvidia-smi bus IDs to physical positions using system documentation.

Then study host tuning for both CPU vendors. Write down the AMD and Intel names for NUMA, power and PCIe settings and the recommended values for GPU servers, and practice reading nvidia-smi topo -m to set CPU and NIC affinity. Finally, review storage tuning, including RDMA transports, MTU, local caching and GPUDirect Storage, and connect each change to the tests in Domain 4 that would measure it. Use practice questions to check that you choose evidence-based steps rather than quick fixes.

FAQ

How much of the NCP-AII is Troubleshooting and Optimization?

Troubleshooting and Optimization is 12% of the NCP-AII, about nine of the 70 to 75 questions.

What are XID errors?

XID errors are event codes the NVIDIA driver writes to the kernel log when a GPU problem occurs. Different codes indicate issues such as memory errors, application faults or a GPU dropping off the bus.

Where do power supply faults show up?

In the BMC sensors and system event log, which name the failed supply. The host OS may show nothing if redundancy keeps the node running.

What should happen after replacing a GPU?

Bring the new part to the validated firmware level, run diagnostics such as DCGM and NCCL as appropriate, confirm no errors in logs, then return the node to the scheduler.

Why does the exam mention AMD and Intel servers?

GPU servers use both CPU platforms, and their BIOS settings for NUMA, PCIe and power differ in name and defaults. You need to tune each correctly for GPU and network throughput.

What is GPUDirect Storage?

A technology that moves data directly between storage and GPU memory, avoiding a copy through CPU memory, which can improve throughput and reduce CPU load where storage supports it.

Other Artificial Intelligence & Machine Learning certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 5 knowledge

You have covered fault diagnosis, part identification and replacement, AMD and Intel host tuning and storage optimization. Check whether you can choose the evidence-based fix.

Take the NCP-AII practice test Check your NCP-AII readiness

Keep going