Overview
Domain 5 has five official objectives, all about using tools to find faults. On Spectrum-X switches, cl-resource-query reports how much of the ASIC's forwarding resources, such as routes, MAC entries and ACL rules, are in use. WJH provides real-time drop events with reasons. Verifying low-latency interconnects means checking the paths between GPUs, CPUs, NICs and storage, using tools such as nvidia-smi topo, perftest and NCCL tests.
On InfiniBand, UFM system health reports fabric and server problems, and the command-line utilities cover each layer: ibstat for local ports, ibnodes and iblinkinfo for topology and link width and speed, ibping for reachability, ibdiagnet for fabric-wide checks, and ib_write_bw and ib_write_lat for RDMA performance. At 20%, about one question in five involves choosing a tool or interpreting its output.
How it's tested
Expect symptom-first questions. A fabric, host or switch behaves badly, and you choose the tool that isolates the cause or interpret a short output excerpt. Distractors are usually real tools that work at the wrong layer or scope, so knowing exactly what each command reports is the key skill.
-
Pick the right command
A symptom such as slow bandwidth or an unreachable node is described, and you choose the diagnostic command that targets it.
-
Read the output
An excerpt from ibstat, iblinkinfo or cl-resource-query is shown, and you identify the problem it reveals.
-
Check the GPU path
Inter-GPU or GPU-to-storage latency is poor, and you choose how to verify topology, affinity or RDMA performance.
One node in a training job shows half the expected all-reduce bandwidth. ibstat on that node shows the port Active but at a lower rate than its peers. That points to a link negotiating at reduced width or speed. Running iblinkinfo confirms which switch port is degraded, and replacing the cable or transceiver is the likely fix.
Objectives 5
Official objectives from the NVIDIA outline, in outline order.
5.1 Use tools like cl-resource-query to check resource allocation in Spectrum-X environmentsChecking ASIC forwarding table usage on Cumulus Linux.
- What it means
- cl-resource-query reports how much of the Spectrum ASIC's hardware resources are used, including IPv4 and IPv6 routes, host entries, MAC addresses, ECMP next hops and ACL entries, against the available maximum.
- What to understand
- When tables fill, new routes or entries cannot be installed in hardware, causing traffic to fail or fall back to software. Forwarding table profiles change how space is allocated between resource types. In NVUE, nv show system forwarding shows similar data. Check resources after adding many tenants, VRFs or ACL rules.
- How it shows up
- A switch drops traffic after a large route or ACL change, and you choose cl-resource-query to check hardware table usage.
- Common confusion
- Reading interface counters when the problem is exhausted ASIC table space.
5.2 Use WJH services for real-time event analysisReal-time drop analysis on Spectrum switches.
- What it means
- WJH reports drop events from the switch ASIC as they happen, with the reason, ingress port, VLAN and five-tuple. It turns an anonymous discard counter into a specific explanation.
- What to understand
- Drop groups include layer 1 errors, layer 2 drops such as VLAN mismatch, router drops such as missing routes, tunnel issues, ACL denies and buffer drops caused by congestion. Results are viewable on the switch and fabric-wide through NetQ. Severity and recommended actions accompany many events. WJH works per packet rather than by polling.
- How it shows up
- A drop event is described, and you interpret the WJH reason or choose WJH as the fastest tool.
- Common confusion
- Assuming WJH only reports congestion; it also reports ACL, routing, layer 2 and physical layer drops.
5.3 Verify low-latency interconnects between GPUs, CPUs, and storage systemsValidating topology, affinity and RDMA paths on the host.
- What it means
- This objective checks that GPUs, CPUs, NICs and storage connect along the fastest paths. nvidia-smi topo -m shows whether GPU and NIC pairs share a PCIe switch, NUMA node or NVLink, and performance tests confirm actual latency and bandwidth.
- What to understand
- Best paths are NVLink between GPUs and a shared PCIe switch, shown as PIX, between GPU and NIC; SYS means traffic crosses the CPU interconnect and adds latency. Confirm GPUDirect RDMA is active, run ib_write_lat and ib_write_bw with GPU memory options, and use NCCL tests such as all_reduce_perf for collective performance. GPUDirect Storage tools check storage paths.
- How it shows up
- A slow GPU path is described, and you choose the tool or interpret topology output to locate the bottleneck.
- Common confusion
- Assuming any PCIe path is equal; paths crossing the CPU socket link add significant latency.
5.4 Use UFM system health to diagnose InfiniBand issuesUFM health reports for fabric and server problems.
- What it means
- UFM system health includes fabric health reports, the UFM server health monitor and system dump collection. Fabric health checks run diagnostics across the InfiniBand fabric and summarize problems such as duplicate GUIDs, link errors and SM issues.
- What to understand
- Use the UFM health tab to run a fabric health report, review the results by severity, and drill down to specific ports or devices. Server health monitoring tracks the UFM host's own processes, disk and memory. A system dump collects logs for support cases. Pair health reports with events and port counters to separate one-off faults from trends.
- How it shows up
- A fabric-wide problem is described, and you choose the UFM health feature that surfaces or documents it.
- Common confusion
- Confusing UFM server health, which monitors the UFM host, with fabric health, which checks the network.
5.5 Use commands like ib_write_lat, ib_write_bw, ibping, ibstat, ibdiagnet, ibnodes, and iblinkinfo to diagnose connectivity issuesInfiniBand command-line diagnostics from port to fabric.
- What it means
- These utilities from the InfiniBand diagnostic and perftest packages examine connectivity at different scopes, from a single host adapter to the whole fabric, and measure RDMA performance between two endpoints.
- What to understand
- ibstat shows local port state, rate, LID and SM LID. ibnodes lists fabric nodes, and iblinkinfo shows every link with width and speed. ibping tests reachability by LID or GUID but needs ibping -S on the target. ibdiagnet scans the whole fabric for errors and writes reports. ib_write_bw and ib_write_lat run as a server and client pair to measure RDMA write bandwidth and latency.
- How it shows up
- A connectivity or performance symptom is described, and you choose the command with the right scope or interpret its output.
- Common confusion
- Running ibping without starting the responder side, or using ibdiagnet for a local port check.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
cl-resource-query
Shows Spectrum ASIC table usage on Cumulus Linux.
- Remember
- Covers routes, hosts, MACs, ECMP and ACLs.
- Tested as
- Resource exhaustion questions.
- Trap
- Checking counters instead.
-
Forwarding profiles
Allocate ASIC table space between resource types.
- Remember
- Change when one table type fills.
- Tested as
- Capacity planning.
- Trap
- Assuming tables are fixed.
-
WJH drop groups
Layer 1, layer 2, router, tunnel, ACL and buffer.
- Remember
- Buffer drops indicate congestion.
- Tested as
- Drop interpretation.
- Trap
- Reading every drop as congestion.
-
nvidia-smi topo -m
Shows GPU, NIC and CPU affinity and link types.
- Remember
- PIX is best, SYS crosses sockets.
- Tested as
- Interconnect verification.
- Trap
- Ignoring NUMA placement.
-
NCCL tests
Benchmarks such as all_reduce_perf.
- Remember
- Measure collective bus bandwidth.
- Tested as
- End-to-end GPU checks.
- Trap
- Testing only point-to-point.
-
UFM fabric health report
Fabric-wide diagnostic summary in UFM.
- Remember
- Groups problems by severity.
- Tested as
- Fabric diagnosis.
- Trap
- Confusing it with server health.
-
UFM system dump
Bundle of logs and state for support.
- Remember
- Collect it before escalating.
- Tested as
- Support workflow.
- Trap
- Collecting logs manually.
-
ibstat
Local HCA port state, rate, LID and SM LID.
- Remember
- Fast first check on a host.
- Tested as
- Port questions.
- Trap
- Using it to see remote links.
-
iblinkinfo
All fabric links with width and speed.
- Remember
- Finds degraded links.
- Tested as
- Topology questions.
- Trap
- Expecting performance numbers.
-
ibnodes
Lists switches and HCAs discovered on the fabric.
- Remember
- Confirms a node is visible.
- Tested as
- Discovery questions.
- Trap
- Using it to test reachability.
-
ibping
InfiniBand-level ping by LID or GUID.
- Remember
- Target must run ibping -S.
- Tested as
- Reachability questions.
- Trap
- Forgetting the server side.
-
ibdiagnet
Fabric-wide scan for errors and configuration problems.
- Remember
- Writes detailed reports for analysis.
- Tested as
- Fabric diagnosis.
- Trap
- Using it for a single port.
-
ib_write_bw / ib_write_lat
Perftest RDMA write bandwidth and latency tests.
- Remember
- Run a server, then a client pointing at it.
- Tested as
- Performance questions.
- Trap
- Running only one side.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
InfiniBand diagnostic scope
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| ibstat | Local adapter port status | Single host | Expecting remote link data |
| iblinkinfo | Link width and speed fabric-wide | Topology and link health | Using it for bandwidth tests |
| ibdiagnet | Full fabric error scan | Broadest scope | Using it for quick local checks |
| ibping | Reachability by LID or GUID | Needs a responder | Starting only the client |
Performance tests
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| ib_write_bw | RDMA write bandwidth between two endpoints | Throughput | Judging latency from it |
| ib_write_lat | RDMA write latency between two endpoints | Latency | Judging bandwidth from it |
| NCCL all_reduce_perf | Collective performance across GPUs | Workload realistic | Treating it as a link test |
Spectrum-X troubleshooting tools
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| cl-resource-query | ASIC table usage | Capacity problems | Confusing it with counters |
| WJH | Drop events with reasons | Per-packet explanation | Assuming it covers only congestion |
| NetQ | Fabric history and validation | Change tracking | Expecting per-drop reasons |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
ibping just works
- What candidates pick
- Run ibping against a target and conclude it is unreachable when it fails.
- Why it looks right
- Ping usually needs nothing on the target.
- Why it's wrong
- ibping needs ibping -S running on the target; without it, failure says nothing about connectivity.
Clue to watch for The answer choices mention a server mode.
-
ibdiagnet for everything
- What candidates pick
- Run ibdiagnet to check one host's port state.
- Why it looks right
- It is the most thorough tool.
- Why it's wrong
- ibstat answers local port questions instantly; ibdiagnet scans the whole fabric and is better for broad problems.
Clue to watch for The symptom is on one host.
-
Bandwidth test measures latency
- What candidates pick
- Use ib_write_bw to judge latency.
- Why it looks right
- Both are perftest tools.
- Why it's wrong
- ib_write_bw reports throughput; ib_write_lat measures latency.
Clue to watch for The question asks about round-trip or one-way delay.
-
Counters show table exhaustion
- What candidates pick
- Check interface counters when new routes fail to install.
- Why it looks right
- Counters are the usual first look.
- Why it's wrong
- Hardware table exhaustion shows in cl-resource-query, not interface statistics.
Clue to watch for Problems began after adding many routes, VRFs or ACLs.
-
Any PCIe path is fine
- What candidates pick
- Ignore SYS entries in nvidia-smi topo output.
- Why it looks right
- PCIe bandwidth looks sufficient.
- Why it's wrong
- SYS means traffic crosses the CPU socket interconnect, adding latency and reducing GPUDirect efficiency.
Clue to watch for Topology output shows SYS between a GPU and its NIC.
-
WJH means congestion
- What candidates pick
- Treat every WJH event as a buffer problem.
- Why it looks right
- Drops are often congestion.
- Why it's wrong
- WJH also reports ACL denies, routing failures, VLAN issues and physical errors.
Clue to watch for The drop reason names ACL or router.
Answer framework
Use these steps to choose and read troubleshooting tools in Domain 5 scenarios.
- Identify the fabric type and scope of the symptom: single host port, single link, whole fabric, switch resources, or GPU path.
- For a single InfiniBand host, start with ibstat; for link width or speed, use iblinkinfo; for broad problems, run ibdiagnet or a UFM fabric health report.
- For reachability, use ibping with its responder; for throughput or delay, run ib_write_bw or ib_write_lat as a server and client pair.
- On Spectrum-X, use WJH to explain specific drops and cl-resource-query when hardware tables may be full after scale changes.
- For GPU paths, check nvidia-smi topo -m for affinity, confirm GPUDirect RDMA, then validate with NCCL tests across the job's nodes.
Practice question
Want more like this? The NCP-AIN practice test has questions for every domain.
Checklist
0 of 13
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Build a one-page tool matrix before anything else: each tool, the layer it works at, its scope, what it reports and one common misuse. Use the InfiniBand fabric utilities documentation for ibstat, ibnodes, iblinkinfo, ibping and ibdiagnet, and the perftest repository for ib_write_bw and ib_write_lat. If you have access to InfiniBand hardware, run each command and save the output so you recognize healthy and unhealthy results.
Next, cover Spectrum-X tools. In NVIDIA Air you can run cl-resource-query on a virtual switch to see its output format, and read the WJH documentation for drop groups and example events. Then study UFM system health: the health tab, fabric health reports and system dumps.
Finally, practice host-level checks. Run nvidia-smi topo -m on any multi-GPU system you can reach and explain each entry. Read the NCCL troubleshooting guide and the perftest options for GPU memory. Close with timed practice questions that show command output, since reading output quickly is what this domain rewards.
FAQ
How much of NCP-AIN is Troubleshooting Tools?
Troubleshooting Tools is 20% of the exam, the third-largest domain, with five official objectives.
What does cl-resource-query show?
Usage of Spectrum ASIC hardware tables such as routes, host entries, MAC addresses, ECMP next hops and ACL rules, against their limits.
Which command checks a single InfiniBand port?
ibstat shows local port state, physical state, rate, LID and the subnet manager LID for each adapter port.
Why does ibping fail even though the node is up?
ibping needs a responder started with ibping -S on the target. Without it, the client gets no reply.
When should I run ibdiagnet?
When the problem may be anywhere in the fabric. It scans all nodes and links and writes detailed reports.
How do I measure RDMA bandwidth between two hosts?
Start ib_write_bw on one host as the server, then run it on the other host pointing at the first.
What does SYS mean in nvidia-smi topo output?
The path crosses the CPU socket interconnect, the slowest option, which can hurt GPU-to-NIC performance.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 5 knowledge
You have covered cl-resource-query, WJH, interconnect checks, UFM health and the InfiniBand commands. Test your tool selection against timed symptom questions.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the NCP-AIN AI tutor Get any concept explained your way