NCP-AII · Domain 4 of 5

NCP-AII Domain 4: Cluster Test and Verification

Domain 4, the largest NCP-AII domain, covers proving a cluster works: single-node stress, HPL, NCCL and NVLink tests, cable and firmware checks, ClusterKit, burn-in runs and storage testing.

Exam weight
33%
Official objectives
14
Estimated questions estimate
~24
Suggested study time estimate
25–35 hrs

About 16 min read Reviewed October 2026

Overview

Cluster Test and Verification is the heaviest NCP-AII domain and has the most objectives, fourteen in total. They fall into four groups. Node tests cover single-node stress testing, running High-Performance Linpack (HPL) and running single-node NCCL tests that also confirm the NVLink Switch fabric inside the node. Fabric and component checks cover validating cable signal quality, confirming cabling matches the design, and confirming firmware and software on switches, BlueField-3 devices and transceivers. Cluster-level assessment covers running ClusterKit for a multifaceted node assessment and running NCCL across nodes to verify east-west fabric bandwidth. Burn-in covers extended NCCL, HPL and NeMo runs, plus storage testing.

At 33%, this domain is about 24 questions. NVIDIA's own wording for objective 4.1 reads 'single-nod', which this guide treats as a typo for single-node.

How it's tested

Expect output-reading questions here. A scenario shows an nccl-tests table, an HPL summary, a link diagnostic or a firmware listing and asks what it means or what to do next. Other questions ask which test proves a specific property, such as NVLink health, rail bandwidth or long-run stability.

  • Pick the test

    A property to prove is named, such as intra-node bandwidth or fabric stability, and you choose the test that measures it.

  • Read the result

    Test output is shown, and you judge whether bus bandwidth, TFLOPS or link counters are acceptable or point to a fault.

  • Isolate the outlier

    A multi-node test underperforms, and you narrow it to a node, cable or switch with targeted tests.

Reasoning it through

A 32-node all_reduce_perf run reaches far less bus bandwidth than expected, but every node passed single-node NCCL. Because NVLink inside each node is healthy, the problem is in the east-west fabric. The engineer should check link diagnostics for errors or downgraded speed, verify cabling against the plan, then run NCCL on smaller node groups to find the rail or switch that drags the result down.

Objectives 14

Official objectives from the NVIDIA outline, in outline order.

4.1 Perform single-node stress testStressing GPUs, memory and power in one node. Official text reads 'single-nod'.
What it means
This covers running stress tests on one node at a time to prove GPUs, memory, power delivery and cooling hold up under sustained load, using tools such as DCGM diagnostics at higher run levels.
What to understand
Single-node stress tests find weak GPUs, thermal problems and power issues before the node is combined with others. Watch for throttling, ECC errors, XID messages and clock drops during the run. A node that fails here must be fixed or excluded before any multi-node test.
How it shows up
A stress run shows throttling or errors, and you decide whether the node passes.
Common confusion
Passing a node because the test finished, without checking for throttling or logged errors.
4.2 Execute HPL (High-Performance Linpack)Running HPL and reading its performance and residual check.
What it means
This covers running HPL, usually from the NVIDIA HPC Benchmarks container, on one or more nodes and interpreting the reported performance in TFLOPS and the residual check that confirms the answer was correct.
What to understand
HPL solves a dense linear system, stressing GPU compute and memory. Problem size N, block size NB and the P by Q process grid affect results. A run is valid only if the residual check passes, and results are compared with expected values for the GPU type and node count.
How it shows up
HPL output is shown, and you judge the result or choose a parameter change.
Common confusion
Reporting a high TFLOPS figure from a run whose residual check failed.
4.3 Perform single-node NCCL (including verify NVIDIA NVLink™ Switch)Intra-node NCCL tests that confirm NVLink and NVSwitch.
What it means
This covers running NCCL tests such as all_reduce_perf across all GPUs in one node to measure intra-node collective bandwidth and confirm that the NVLink and NVSwitch fabric is working.
What to understand
Single-node NCCL traffic stays on NVLink, so low bus bandwidth points to an NVLink, NVSwitch or Fabric Manager problem rather than the network. nvidia-smi nvlink status shows link state per GPU. Compare bus bandwidth, not algorithm bandwidth, against the expected figure for the platform.
How it shows up
Single-node NCCL results are low, and you identify NVLink or Fabric Manager as the likely cause.
Common confusion
Blaming the InfiniBand network for poor single-node NCCL results.
4.4 Validate cables by verifying signal qualityReading link diagnostics for errors and margins.
What it means
This covers checking the physical quality of each link using tools such as mlxlink to read bit error rates, symbol errors, effective errors, link speed and width, and transceiver data.
What to understand
A link can be up and still unhealthy. Rising raw or effective bit errors, symbol errors or a link negotiated below rated speed or width all indicate a bad cable, dirty connector or failing optic. These checks are done before and after burn-in to catch marginal links.
How it shows up
Link diagnostic output is shown, and you decide whether the cable needs attention.
Common confusion
Treating a link as good simply because its state is up.
4.5 Confirm cabling is correctChecking connections against the cabling plan.
What it means
This covers verifying that every cable connects the intended ports, using fabric discovery tools such as ibdiagnet or UFM topology checks on InfiniBand and LLDP on Ethernet, compared against the designed topology.
What to understand
Miswired rails still pass traffic but break the rail-optimized design and reduce collective performance. Discovery tools list each neighbor, so the output can be compared with the cabling plan to find swapped or misplaced cables before performance testing.
How it shows up
A topology report differs from the plan, and you identify the miswired link.
Common confusion
Assuming working links mean correct cabling.
4.6 Confirm FW/SW on switchesChecking switch firmware and operating system versions.
What it means
This covers confirming that InfiniBand and Ethernet switches run the firmware and network operating system versions specified for the deployment, and that all switches in the fabric are consistent.
What to understand
Mixed switch versions can cause unexpected behavior and are hard to troubleshoot under load. Versions are read from the switch CLI or a fabric manager such as UFM and compared with the validated release. Updates are planned and rolled out before acceptance testing, not during it.
How it shows up
A version listing shows inconsistent switches, and you choose the action.
Common confusion
Testing performance before aligning switch versions.
4.7 Confirm FW/SW on BlueField 3Checking BlueField-3 firmware, BFB and DOCA versions.
What it means
This covers confirming that each BlueField-3 device runs the intended NIC firmware, BFB image and DOCA software version, and that versions are consistent across the cluster.
What to understand
BlueField firmware is read with host tools such as flint or mlxfwmanager, and the Arm OS and DOCA version from the card itself. Firmware, BFB and DOCA versions should come from the same release. Inconsistent BlueField versions can cause uneven network performance between nodes.
How it shows up
BlueField version output is shown, and you identify the device that is out of compliance.
Common confusion
Checking only the host driver version and ignoring the card firmware and BFB.
4.8 Confirm FW on transceiversChecking optical module firmware.
What it means
This covers reading and confirming the firmware version of optical transceivers in switches and adapters, and updating modules that are below the supported level.
What to understand
Transceivers have their own firmware that affects link stability. Module details, including firmware version, part number and vendor, can be read with mlxlink and switch commands. Mixed or outdated module firmware is a known cause of flapping links.
How it shows up
Links flap on some modules, and you link the symptom to transceiver firmware.
Common confusion
Assuming transceivers have no firmware to check.
4.9 Run ClusterKit to perform a multifaceted node assessmentUsing ClusterKit to compare nodes across many metrics.
What it means
This covers running ClusterKit, an NVIDIA tool that assesses nodes across several dimensions, such as GPU and memory bandwidth, network latency and bandwidth between nodes, and comparing results to find outliers.
What to understand
ClusterKit runs a set of tests across many nodes and reports results per node and per pair, which makes slow nodes or links stand out. It is used after single-node checks and before full burn-in to catch nodes that pass alone but lag behind their peers.
How it shows up
ClusterKit results show one node behind the others, and you decide what to do next.
Common confusion
Using ClusterKit as a replacement for long burn-in runs.
4.10 Run NCCL to verify E/W fabric bandwidthMulti-node NCCL to test the compute fabric.
What it means
This covers running NCCL tests across multiple nodes to measure east-west fabric bandwidth, using MPI or Slurm to launch them, and comparing the bus bandwidth with expected values for the network design.
What to understand
Multi-node NCCL tests exercise the compute network rails. Results that fall short point to the fabric, cabling, NIC settings or one slow node. Testing progressively larger groups of nodes helps isolate where performance drops.
How it shows up
Multi-node NCCL bandwidth is low, and you choose how to isolate the cause.
Common confusion
Comparing algorithm bandwidth with link speed instead of using bus bandwidth.
4.11 Perform NCCL burn-inLong-running NCCL to prove fabric stability.
What it means
This covers running NCCL collective tests for an extended period across the cluster to prove the fabric and NICs remain stable and performant under sustained load.
What to understand
Burn-in looks for problems that short tests miss, such as links that degrade when warm or optics that start failing. Monitor bandwidth over time along with link error counters and logs. Any node or link that degrades is fixed and the burn-in repeated.
How it shows up
A burn-in shows bandwidth falling after hours, and you pick the likely cause or next step.
Common confusion
Treating a short successful NCCL run as proof of fabric stability.
4.12 Perform HPL burn-inLong-running HPL to prove compute stability.
What it means
This covers running HPL repeatedly or for an extended time to prove sustained compute, memory, power and cooling stability across nodes.
What to understand
HPL burn-in reveals GPUs that fail under prolonged heat and power load, and racks whose cooling cannot keep up. Results should stay consistent across iterations, residual checks must pass every time, and logs should show no XID errors.
How it shows up
HPL iterations vary or fail over time, and you identify the stability issue.
Common confusion
Accepting a single good HPL run as a burn-in result.
4.13 Perform Nemo burn-inRunning a NeMo training workload as a full-stack test.
What it means
This covers running an NVIDIA NeMo training job, typically a large language model training example, across the cluster as a realistic burn-in that exercises GPUs, NVLink, the network, storage and the software stack together.
What to understand
Synthetic tests stress one area at a time, while a NeMo training run combines compute, communication, checkpointing and data loading. Throughput and step time should stay stable over the run. Failures here often reveal storage or software issues that HPL and NCCL miss.
How it shows up
A NeMo burn-in fails while HPL and NCCL passed, and you identify what it exercised that they did not.
Common confusion
Assuming NeMo burn-in adds nothing once HPL and NCCL pass.
4.14 Test storageMeasuring storage throughput and latency from the nodes.
What it means
This covers testing storage performance from compute nodes using tools such as fio and, where used, GPUDirect Storage tests, measuring read and write bandwidth, IOPS and latency against requirements.
What to understand
Storage tests should run from many nodes at once to reflect real job load. Results depend on block size, file count and parallelism. GPUDirect Storage tests check the direct path between storage and GPU memory. Poor results lead to the client and network checks in Domain 5.
How it shows up
Storage test results are below target, and you choose the next check or test change.
Common confusion
Testing storage from one node and assuming the result scales.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • Bus bandwidth (busbw)

    nccl-tests figure normalized for the collective so it can be compared with hardware link speed.

    Remember
    Compare busbw, not algbw, to expectations.
    Tested as
    NCCL output questions.
    Trap
    Reading algbw as hardware bandwidth.

  • all_reduce_perf

    The most common nccl-tests binary for collective bandwidth.

    Remember
    Run single-node first, then multi-node.
    Tested as
    Test selection questions.
    Trap
    Jumping straight to multi-node.

  • HPL residual check

    Confirms the HPL answer is numerically correct.

    Remember
    A failed residual invalidates the score.
    Tested as
    HPL output questions.
    Trap
    Trusting TFLOPS alone.

  • HPL N, NB, P x Q

    Problem size, block size and process grid.

    Remember
    They tune performance and memory use.
    Tested as
    HPL parameter questions.
    Trap
    Making N too large for memory.

  • NVLink status

    nvidia-smi nvlink output showing link state and speed per GPU.

    Remember
    Low single-node NCCL points here.
    Tested as
    Intra-node questions.
    Trap
    Blaming the network instead.

  • mlxlink

    Tool to read link state, speed, errors and module details.

    Remember
    Errors matter even when the link is up.
    Tested as
    Signal quality questions.
    Trap
    Checking only link state.

  • ibdiagnet

    InfiniBand fabric diagnostic and discovery tool.

    Remember
    Compare discovered topology with the plan.
    Tested as
    Cabling questions.
    Trap
    Assuming traffic means correct wiring.

  • ClusterKit

    NVIDIA multifaceted node assessment tool.

    Remember
    Finds outliers across many nodes.
    Tested as
    Assessment questions.
    Trap
    Using it in place of burn-in.

  • Burn-in

    Extended runs that expose failures short tests miss.

    Remember
    Watch for degradation over time.
    Tested as
    Stability questions.
    Trap
    Stopping after one clean run.

  • NeMo burn-in

    Realistic training job that tests the whole stack.

    Remember
    Includes data loading and checkpoints.
    Tested as
    Full-stack questions.
    Trap
    Treating it as redundant.

  • Firmware consistency

    Switches, BlueField-3 and transceivers on validated, uniform versions.

    Remember
    Align before acceptance tests.
    Tested as
    Version check questions.
    Trap
    Testing with mixed versions.

  • Storage tests

    fio and GPUDirect Storage tests from many nodes.

    Remember
    Parallelism changes results.
    Tested as
    Storage questions.
    Trap
    Testing from one node only.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

What each test proves

ConceptMeaningKey differenceCommon trap
Single-node stress Node hardware under sustained load One node at a time Ignoring throttling during the run
HPL Compute and memory performance Needs a passing residual Trusting the score alone
Single-node NCCL NVLink and NVSwitch bandwidth No network involved Blaming InfiniBand
Multi-node NCCL East-west fabric bandwidth Uses the network rails Skipping single-node first
NeMo burn-in Whole-stack stability Realistic workload Calling it redundant

Fabric checks

ConceptMeaningKey differenceCommon trap
Signal quality Error rates and link margins Physical health of each link Up equals healthy
Cabling correctness Ports connect as designed Topology versus plan Working equals correct
Firmware checks Switch, DPU and optic versions Consistency with validated set Ignoring transceivers

NCCL bandwidth figures

ConceptMeaningKey differenceCommon trap
algbw Data size divided by time Depends on the collective Comparing it to link speed
busbw Normalized hardware bandwidth Comparable across collectives Ignoring it in favor of algbw

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. Network blamed for NVLink

    What candidates pick
    Investigate InfiniBand when single-node NCCL is slow.
    Why it looks right
    NCCL is associated with networking.
    Why it's wrong
    Single-node NCCL runs over NVLink and NVSwitch, so the network is not involved.

    Clue to watch for The test ran on one node only.

  2. Score without residual

    What candidates pick
    Report HPL TFLOPS from a run that failed the residual check.
    Why it looks right
    The number looks good.
    Why it's wrong
    A failed residual means the computation was wrong, so the run is invalid.

    Clue to watch for The output shows FAILED next to the residual.

  3. Link up means good

    What candidates pick
    Pass a link because its state is active.
    Why it looks right
    Traffic flows.
    Why it's wrong
    Bit errors, symbol errors or reduced speed and width can exist on active links and cause poor performance.

    Clue to watch for Error counters or speed differ from expected.

  4. One clean run is burn-in

    What candidates pick
    Sign off after one successful NCCL or HPL run.
    Why it looks right
    It shows the system works.
    Why it's wrong
    Burn-in exists to expose failures that appear only after hours of heat and load.

    Clue to watch for The question mentions sustained or extended testing.

  5. algbw versus link speed

    What candidates pick
    Compare algbw directly with NIC or NVLink speed.
    Why it looks right
    It is the first bandwidth column.
    Why it's wrong
    Bus bandwidth is the figure normalized for comparison with hardware speed.

    Clue to watch for Both algbw and busbw are shown.

  6. Skip transceiver firmware

    What candidates pick
    Ignore module firmware when links flap.
    Why it looks right
    Optics seem like passive parts.
    Why it's wrong
    Transceivers run firmware that affects link stability and must match the supported level.

    Clue to watch for Only some links using certain modules flap.

  7. One-node storage test

    What candidates pick
    Accept storage performance from a single client test.
    Why it looks right
    It is quick and simple.
    Why it's wrong
    Shared storage behaves differently under many parallel clients, which is how training jobs use it.

    Clue to watch for The requirement is cluster-wide throughput.

Answer framework

Use these steps to choose the right test and interpret its output in a verification scenario.

  1. Confirm the prerequisite tests have passed, working from single-node stress, HPL and NCCL toward multi-node and burn-in tests.
  2. Match the property in question to the test that measures it: NVLink with single-node NCCL, rails with multi-node NCCL, compute with HPL, the full stack with NeMo.
  3. Read the output against the correct metric, such as bus bandwidth, TFLOPS with a passing residual, or link error counters and negotiated speed.
  4. If a result falls short, isolate it with smaller node groups, link diagnostics, cabling checks and firmware version comparisons before replacing parts.
  5. For acceptance, require consistent results across extended burn-in runs and storage tests from many clients, with no new errors in logs.

Practice question

A single DGX node shows single-node all_reduce_perf bus bandwidth far below expected, while the other nodes in the cluster are normal. nvidia-smi shows all GPUs present. Which check is most relevant next?

Want more like this? The NCP-AII practice test has questions for every domain.

Checklist

0 of 13

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

This is the largest domain, so plan the most time here alongside Domain 1. Start by drawing the test ladder: single-node stress, HPL, single-node NCCL, then multi-node NCCL, ClusterKit, burn-ins and storage. For each step, write what it proves and what a failure points to.

Next, get hands-on with nccl-tests and HPL if you can reach any multi-GPU machine. Build nccl-tests, run all_reduce_perf across message sizes, and learn to read the algbw and busbw columns. Run the HPL container from the NVIDIA HPC Benchmarks collection and change N and the process grid to see the effect. Read the NCCL documentation linked on the certification page.

Then study fabric checks. Learn the main fields in mlxlink output, how ibdiagnet reports topology and errors, and how to read firmware versions from switches, BlueField-3 and optics. Finally, review burn-in strategy: duration, what to monitor and why a NeMo training run catches storage and software problems. Practice questions in this domain usually show output, so spend time reading real or sample outputs rather than only memorizing tool names.

FAQ

How much of the NCP-AII is Cluster Test and Verification?

Cluster Test and Verification is 33% of the NCP-AII, the largest domain, about 24 of the 70 to 75 questions. It also has the most objectives, fourteen.

What is the difference between algbw and busbw in NCCL tests?

Algorithm bandwidth is data size divided by time. Bus bandwidth applies a correction for the collective so it reflects the hardware bandwidth used and can be compared across operations and with link speeds.

Why run single-node tests before multi-node ones?

Single-node tests isolate GPU, NVLink and node problems. If you skip them, a bad node can look like a fabric problem in multi-node results and waste troubleshooting time.

What is ClusterKit?

ClusterKit is an NVIDIA tool that runs several tests across many nodes, such as bandwidth and latency checks, and reports results so slow or faulty nodes stand out.

How long should burn-in last?

NVIDIA's objectives do not set a single duration. Burn-in runs for hours rather than minutes, long enough to expose heat and stability problems, and is repeated after fixes.

Why include NeMo burn-in?

A NeMo training job combines compute, NVLink, network, storage reads and checkpoint writes, so it can reveal problems that focused synthetic tests do not exercise.

Is objective 4.1 really 'single-nod'?

NVIDIA's published wording reads 'Perform single-nod stress test', which appears to be a typo for single-node. This guide uses single-node.

Other Artificial Intelligence & Machine Learning certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 4 knowledge

You have covered node stress tests, HPL, NCCL, cabling and firmware checks, ClusterKit, burn-in and storage tests. Check whether you can read test output and pick the next step.

Take the NCP-AII practice test Check your NCP-AII readiness

Keep going