Overview
Domain 3 has four official objectives covering the NVIDIA Quantum InfiniBand fabric that many AI clusters use for GPU traffic. Initial configuration and provisioning include setting up switches with MLNX-OS, enabling a subnet manager on a switch or on a UFM server, and making that subnet manager highly available with master and standby instances. Partition keys provide secure multi-tenancy by controlling which ports can communicate. QoS uses service levels, virtual lanes and arbitration tables, while adaptive routing lets switches choose less congested output ports. UFM monitors link status, port counters and bandwidth.
This domain carries 30% of the exam with only four objectives, so each one is tested in depth. Expect questions about subnet manager priority and failover, full versus limited PKey membership, SL-to-VL mapping, and what the UFM dashboards reveal about link health.
How it's tested
InfiniBand questions often describe fabric behavior and ask for the cause or the configuration step. You may see subnet manager states, partition definitions, SL-to-VL tables or UFM event descriptions. The best answers reflect how the subnet manager controls the fabric, rather than Ethernet habits such as VLANs or pause frames.
-
Subnet manager state
A fabric where ports stay in Initialize or a failover occurs is described, and you identify the subnet manager cause or HA setting.
-
Partition membership
Hosts with specific PKeys are listed, and you decide which pairs can communicate based on full or limited membership.
-
UFM evidence
A UFM dashboard, alarm or counter is described, and you interpret what it says about link health or utilization.
After cabling a new InfiniBand switch, ibstat on the hosts shows the port physical state as LinkUp but the logical state stuck at Initialize. A physical link exists, but no subnet manager has assigned LIDs and moved the port to Active. The correct step is to enable or reach a subnet manager, on a managed switch or UFM, rather than changing cables.
Objectives 4
Official objectives from the NVIDIA outline, in outline order.
3.1 Perform initial configuration and provisioning, including high availability (HA)Bringing up switches, the subnet manager and HA.
- What it means
- Initial setup includes configuring a managed Quantum switch with MLNX-OS, assigning management addresses, updating software and enabling the subnet manager. The subnet manager discovers the fabric, assigns LIDs, computes routes and moves ports to Active. HA means a standby subnet manager takes over if the master fails.
- What to understand
- Only one subnet manager is master per subnet; others are standby, chosen by priority and then GUID. The SM can run on a switch for small fabrics or on UFM for large ones, where UFM HA pairs two servers in active and standby roles. Ports that stay in Initialize indicate no working subnet manager. Keep firmware and SM versions consistent.
- How it shows up
- A new fabric or failover scenario is described, and you pick the step that enables or protects the subnet manager.
- Common confusion
- Running several equal-priority masters intentionally; only one master is active and others stand by.
3.2 Configure partition keys (PKeys) to ensure secure multi-tenancy in InfiniBand networksPartitioning the fabric so tenants cannot communicate.
- What it means
- A PKey is a 16-bit value that defines a partition, similar in purpose to a VLAN. The subnet manager programs PKey tables into ports, and switches and HCAs enforce them, so only ports sharing a partition can exchange traffic. The default partition is 0x7fff.
- What to understand
- The top bit of the PKey sets membership: full members, shown with the high bit set such as 0xffff or 0x8001, can talk to all members, while limited members can talk only to full members. Partitions are defined in the subnet manager configuration or through UFM. Hosts use IPoIB child interfaces or verbs with the PKey index to join a partition.
- How it shows up
- A partition list is given, and you decide which hosts can communicate or how to isolate a tenant.
- Common confusion
- Assuming two limited members of the same partition can communicate; they cannot.
3.3 Configure QoS and adaptive routing to dynamically adjust paths based on congestionService levels, virtual lanes and adaptive routing in InfiniBand.
- What it means
- InfiniBand QoS assigns traffic a service level, which the subnet manager maps to a virtual lane on each link, while VL arbitration tables weight how lanes share bandwidth. Adaptive routing lets switches choose among several output ports per destination based on current load.
- What to understand
- There are 16 service levels and up to 15 data VLs plus VL15 for management. Configure SL2VL and arbitration in the subnet manager QoS policy. Adaptive routing is enabled in the SM routing configuration and works alongside self-healing networking, which reroutes around failed links within the switches far faster than a full SM sweep.
- How it shows up
- A congestion or traffic priority need is described, and you choose the SL, VL or adaptive routing setting.
- Common confusion
- Looking for PFC or DSCP on InfiniBand; QoS uses SLs, VLs and credit-based flow control.
3.4 Use UFM to monitor InfiniBand link status and bandwidth utilizationMonitoring fabric health and traffic with UFM.
- What it means
- NVIDIA Unified Fabric Manager provides a network map, per-port counters, link status, events and alarms, and bandwidth and congestion views for InfiniBand fabrics. UFM Telemetry and Cyber-AI extend it with high-frequency data and anomaly detection.
- What to understand
- Use the network map to see topology and down links, port counters for symbol errors, link-downed counts and bandwidth, and events for threshold alarms. Monitoring degraded links, such as a port running at lower width or speed, is a common task. UFM can also manage partitions and run fabric health reports.
- How it shows up
- A monitoring question asks where in UFM to find link state, utilization or an error trend.
- Common confusion
- Using UFM only for configuration and ignoring its continuous telemetry and event history.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Subnet manager
Software that discovers the fabric, assigns LIDs and computes routes.
- Remember
- No SM means ports stay in Initialize.
- Tested as
- Bring-up questions.
- Trap
- Thinking switches route without an SM.
-
SM priority
Value from 0 to 15 that decides which SM becomes master.
- Remember
- Highest priority wins, then lowest GUID.
- Tested as
- HA questions.
- Trap
- Assuming the first SM started always wins.
-
UFM HA
Active and standby UFM servers with replicated data.
- Remember
- Failover keeps fabric management running.
- Tested as
- HA design.
- Trap
- Running two active UFMs.
-
LID
Local identifier assigned by the SM for routing.
- Remember
- GUIDs are fixed; LIDs are assigned.
- Tested as
- Addressing questions.
- Trap
- Confusing LIDs with GUIDs.
-
PKey
16-bit partition key that controls communication.
- Remember
- Default partition is 0x7fff.
- Tested as
- Multi-tenancy.
- Trap
- Treating it like an IP ACL.
-
Full vs. limited membership
High bit set means full member.
- Remember
- Limited members only reach full members.
- Tested as
- Partition reachability.
- Trap
- Letting limited members talk to each other.
-
Service level
Traffic class from 0 to 15 carried in packets.
- Remember
- Mapped to VLs per link by SL2VL tables.
- Tested as
- QoS questions.
- Trap
- Confusing SLs with VLs.
-
Virtual lane
Separate buffer and credit channel on a link.
- Remember
- VL15 is reserved for subnet management.
- Tested as
- QoS mechanics.
- Trap
- Assigning data to VL15.
-
Credit-based flow control
Senders transmit only with receiver buffer credits.
- Remember
- InfiniBand is lossless by design.
- Tested as
- Comparison with Ethernet.
- Trap
- Looking for PFC on InfiniBand.
-
Adaptive routing
Switches pick among several output ports by load.
- Remember
- Enabled through the SM routing configuration.
- Tested as
- Congestion questions.
- Trap
- Assuming static routes only.
-
Self-healing networking
Switch-level fast rerouting around failed links.
- Remember
- Recovers faster than an SM resweep.
- Tested as
- Resilience questions.
- Trap
- Thinking only the SM can reroute.
-
UFM network map and counters
Topology view with link state and per-port statistics.
- Remember
- Watch symbol errors and downshifted links.
- Tested as
- Monitoring questions.
- Trap
- Ignoring degraded but up links.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
PKey membership outcomes
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Full to full | Both ports full members | Communication allowed | Forgetting to set the high bit |
| Full to limited | One full, one limited | Communication allowed | Blocking it unnecessarily |
| Limited to limited | Both limited members | Communication blocked | Expecting it to work |
InfiniBand vs. Ethernet concepts
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Subnet manager | Central fabric control and routing | Ethernet uses distributed protocols | Expecting spanning tree |
| Credit-based flow control | Lossless by design | Ethernet needs PFC for lossless | Configuring PFC on InfiniBand |
| PKey | Partition membership | Similar purpose to a VLAN | Treating it as a routed boundary |
| SL and VL | Class and lane mapping | Similar purpose to priority queues | Confusing SL with VL |
Subnet manager hosting options
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Switch-based SM | SM on a managed Quantum switch | Suited to smaller fabrics | Using it for very large fabrics |
| UFM-based SM | SM on a UFM server | Scales and adds monitoring | Running it without HA |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
Cables are the problem
- What candidates pick
- Replace cables when ports show LinkUp but Initialize.
- Why it looks right
- Link problems usually mean cabling.
- Why it's wrong
- LinkUp physical state proves the cable works; Initialize means the subnet manager has not configured the port.
Clue to watch for Physical state is LinkUp.
-
Limited members talk
- What candidates pick
- Assume all members of a partition can communicate.
- Why it looks right
- They share a PKey.
- Why it's wrong
- Limited members can only exchange traffic with full members, never with other limited members.
Clue to watch for PKey values without the high bit set.
-
PFC on InfiniBand
- What candidates pick
- Enable priority flow control to make InfiniBand lossless.
- Why it looks right
- That is how Ethernet becomes lossless.
- Why it's wrong
- InfiniBand uses credit-based link-level flow control and is lossless natively.
Clue to watch for The fabric is InfiniBand, not RoCE.
-
Two masters for HA
- What candidates pick
- Run two master subnet managers for redundancy.
- Why it looks right
- More masters sound safer.
- Why it's wrong
- Only one SM is master per subnet; HA uses standby SMs chosen by priority.
Clue to watch for Question mentions failover or SM priority.
-
SL equals VL
- What candidates pick
- Treat service level and virtual lane as the same thing.
- Why it looks right
- Both describe traffic classes.
- Why it's wrong
- The SL is carried in the packet; SL2VL tables map it to a VL that can differ per link.
Clue to watch for Wording about mapping tables.
-
Up means healthy
- What candidates pick
- Consider every Active link healthy in UFM.
- Why it looks right
- Active state looks fine.
- Why it's wrong
- Links can be Active at reduced width or speed or with rising symbol errors, which UFM counters reveal.
Clue to watch for Throughput is below expected on one path.
Answer framework
Use these steps to reason through InfiniBand questions from fabric control down to monitoring.
- Check whether a working master subnet manager exists, since port state, LIDs and routes all depend on it; ports stuck in Initializing usually mean no SM has swept the fabric.
- For redundancy questions, identify the standby subnet manager or UFM HA pair and the priority values that decide which becomes master.
- For isolation questions, list each port's PKey and membership type, then apply the full and limited membership rules: limited members can talk to full members but not to each other.
- For performance questions, consider SL-to-VL mapping, arbitration weights and adaptive routing in the subnet manager configuration, plus congestion control when many senders target one destination.
- Confirm the outcome in UFM by checking link state, width, speed, error counters and bandwidth utilization on the affected ports.
Practice question
Want more like this? The NCP-AIN practice test has questions for every domain.
Checklist
0 of 12
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Start by building a clear mental model of how the subnet manager controls an InfiniBand fabric, since every objective depends on it. Read the InfiniBand Essentials course material and the subnet manager chapters of the MLNX-OS manual, and make sure you can explain discovery, LID assignment, routing and port state transitions. Then study HA: master election by priority, standby behavior and the UFM active and standby pair.
Next, spend focused time on partitions. Write out several PKey values, mark each as full or limited, and predict who can talk. Practice until the membership rule is automatic, because it is easy to get wrong under time pressure. Then cover QoS using the InfiniBand QoS documentation, drawing an SL2VL table and explaining what VL15 is reserved for.
Finish with adaptive routing, self-healing networking and UFM. If you have access to a lab or UFM demo, explore the network map, port counters and events. Otherwise, study the UFM Enterprise user manual screens closely so you recognize where link state and utilization appear.
FAQ
How much of NCP-AIN is NVIDIA InfiniBand Networking?
NVIDIA InfiniBand Networking is 30% of the exam, tied with Spectrum Networking, and has four official objectives.
What happens if there is no subnet manager?
Ports can reach LinkUp physically but stay in the Initialize logical state, with no LIDs or routes, so no data traffic flows.
How does subnet manager HA work?
Several SMs can run, but only the highest priority one becomes master. Others stay in standby and take over if the master fails.
What is the default PKey?
The default partition is 0x7fff, with 0xffff representing full membership. Custom partitions isolate tenants.
Can limited PKey members communicate?
Limited members can communicate with full members of the same partition, but not with other limited members.
Does InfiniBand use PFC?
No. InfiniBand uses credit-based flow control and is lossless natively. QoS relies on service levels and virtual lanes.
What does UFM monitor?
Topology, link state, width and speed, port error counters, bandwidth utilization, congestion and events across the InfiniBand fabric.
Other Artificial Intelligence & Machine Learning certifications
Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.
Test your Domain 3 knowledge
You have covered subnet manager setup and HA, PKey multi-tenancy, QoS, adaptive routing and UFM. Check whether you can apply the InfiniBand rules quickly in exam scenarios.
Keep going
- Training Training Built on your Weak areas
- Practice test Questions by domain, with explanations
- Mock exam Full length, timed like the real exam
- Readiness test Find your weakest domain in minutes
- Cheat sheet Key terms and formulas on one page
- Ask the NCP-AIN AI tutor Get any concept explained your way