NCP-AIN · Domain 2 of 6

NCP-AIN Domain 2: NVIDIA Spectrum Networking

The joint-largest NCP-AIN domain covers the Spectrum-X Ethernet platform: lossless RoCE, congestion control and adaptive routing, multi-tenant EVPN, simulation in NVIDIA Air, telemetry with WJH and NetQ, DOCA installation and SuperNIC setup.

Exam weight
30%
Official objectives
8
Estimated questions estimate
~22
Suggested study time estimate
25–35 hrs

About 14 min read Reviewed October 2026

Overview

Domain 2 has eight official objectives built around NVIDIA Spectrum-X, the Ethernet platform that pairs Spectrum switches running Cumulus Linux with BlueField-3 or ConnectX-8 SuperNICs. You configure the switches for RDMA over Converged Ethernet, then enable and verify QoS, ECN, PFC, adaptive routing and telemetry. You build multi-tenant BGP-EVPN with VRFs and VXLAN to isolate tenant workloads.

The operations side asks you to model a network in NVIDIA Air before deployment, diagnose congestion and drops with in-band telemetry and What Just Happened, and monitor latency and congestion in NetQ. On the host side, you install NVIDIA DOCA and configure SuperNIC functions for packet processing and congestion control. At 30%, tied with InfiniBand, this domain deserves the most lab time. Expect NVUE commands such as nv set qos roce and nv show outputs to appear in scenarios.

How it's tested

Spectrum questions mix configuration and diagnosis. Some show an NVUE snippet or nv show output and ask what is missing or what the result means. Others describe symptoms such as pause storms, tail latency or tenant leakage and ask which feature, tool or setting addresses them, including host-side DOCA and SuperNIC steps.

  • Complete the RoCE config

    Partial NVUE configuration is shown, and you choose the command or setting that makes RoCE lossless or switches it to lossy mode.

  • Pick the visibility tool

    A drop, congestion or latency symptom is described, and you choose between WJH, NetQ, in-band telemetry or NVIDIA Air.

  • Isolate the tenant

    Two tenants share a fabric, and you choose the EVPN, VRF or VNI design that keeps their traffic separate.

Reasoning it through

A team reports that RoCE throughput collapses when several jobs share uplinks, and switch counters show many PFC pause frames but no ECN marks. Pause frames without marking suggest ECN thresholds are missing or misconfigured, so congestion is only stopped by PFC. The fix is to verify the RoCE QoS profile with nv show qos roce and confirm ECN marking is active.

Objectives 8

Official objectives from the NVIDIA outline, in outline order.

2.1 Configure NVIDIA Spectrum-X switches for RDMA over Converged Ethernet (RoCE) to enable high-speed, low-latency communicationEnabling RoCE mode on Spectrum switches with NVUE.
What it means
This objective covers turning a Spectrum switch running Cumulus Linux into a RoCE-ready device. In NVUE, nv set qos roce applies a profile that trusts DSCP, maps RoCE traffic to a lossless priority, enables PFC and ECN, and reserves buffers. The mode can be lossless or lossy.
What to understand
Know that the default RoCE profile is lossless, with RoCE data on switch priority 3 and congestion notification packets given strict priority. Apply with nv config apply and verify with nv show qos roce. Lossy mode keeps ECN but drops PFC, relying on end-to-end congestion control. Spectrum-X deployments also apply the platform's recommended RoCE profile across leaves and spines consistently.
How it shows up
Given a partial switch configuration, choose the NVUE command that enables or verifies RoCE, or identify the mode in use.
Common confusion
Thinking RoCE requires InfiniBand hardware; RoCE runs RDMA over standard Ethernet with proper QoS.
2.2 Enable and verify QoS, explicit congestion notification (ECN), and priority flow control (PFC), as well as advanced features like adaptive routing and telemetryCongestion handling, adaptive routing and telemetry on Spectrum-X.
What it means
ECN marks packets when queues pass a threshold so senders slow down, PFC pauses a specific priority when buffers fill, and QoS maps traffic classes to queues. Spectrum-X adds RoCE adaptive routing, which sprays packets across all equal paths, plus telemetry that feeds congestion data to SuperNICs.
What to understand
ECN is the first line of defense and PFC the safety net; if PFC fires constantly, ECN tuning is wrong. Adaptive routing is enabled globally and per interface with nv set router adaptive-routing and works with SuperNICs that reorder packets through direct data placement. Verification uses nv show qos counters, interface PFC statistics and adaptive routing state.
How it shows up
A symptom such as pause storms, hot uplinks or ECMP collisions is described, and you choose the feature or verification command.
Common confusion
Expecting ECMP hashing to balance a few large AI flows; elephant flows collide, which adaptive routing addresses.
2.3 Configure multi-tenancy Border Gateway Protocol Ethernet VPN (BGP-EVPN) to isolate tenant workloadsVXLAN, VRFs and EVPN routes for tenant isolation.
What it means
BGP-EVPN is the control plane that distributes MAC and IP reachability for VXLAN overlays. Each tenant gets its own VRF and layer-3 VNI, with layer-2 VNIs for its VLANs, so tenants share the physical fabric but cannot see each other's routes.
What to understand
Know EVPN route types 2 for MAC and IP hosts and 5 for IP prefixes, the difference between symmetric and asymmetric integrated routing, and why symmetric routing with an L3 VNI per VRF scales better. In NVUE you enable EVPN, define the VTEP source, map VLANs to VNIs and bind VRFs. Route leaking between VRFs is a deliberate exception.
How it shows up
A multi-tenant requirement is described, and you choose the VRF, VNI or EVPN setting that isolates or connects tenants correctly.
Common confusion
Confusing L2 VNIs, which stretch a VLAN, with L3 VNIs, which carry routed traffic per VRF.
2.4 Use NVIDIA Air to simulate network environments and identify potential issuesBuilding a digital twin of the fabric to test changes safely.
What it means
NVIDIA Air is a cloud-hosted simulation platform that runs virtual Cumulus Linux switches, servers and NetQ in a digital twin. You can build a topology from a template, a drawing or a custom topology file and apply production configuration before touching real hardware.
What to understand
Air is used to validate configuration changes, automation playbooks and upgrade steps, and to train staff. It reproduces control-plane behavior such as BGP, EVPN and NVUE, but it does not model ASIC-level performance, buffer behavior or real RoCE throughput. Teams commonly run Ansible against an Air twin before production.
How it shows up
A change-management scenario is described, and you decide whether NVIDIA Air is the right place to test it.
Common confusion
Expecting Air to measure real RoCE bandwidth or buffer congestion; it simulates control and configuration.
2.5 Diagnose congestion or packet loss using in-band telemetry and NVIDIA What Just Happened (WJH) servicesFinding why and where packets were dropped or delayed.
What it means
WJH streams packet drop events from the Spectrum ASIC with a reason, such as layer 1 errors, layer 2 or router issues, ACL denies, tunnel faults or buffer congestion. In-band telemetry adds queue depth and latency data carried in packets or exported from switch hardware.
What to understand
WJH answers why a specific packet was dropped and on which port, including the five-tuple, so it is faster than reading counters. Buffer drops point to congestion, while ACL drops point to policy. On Cumulus Linux you view events locally with the WJH commands, and NetQ collects them fabric-wide. In-band telemetry helps locate congested hops in a path.
How it shows up
A drop or slowdown is described, and you choose WJH or telemetry, or interpret a WJH drop reason.
Common confusion
Treating interface error counters as equivalent to WJH; counters show totals, WJH gives per-event reasons.
2.6 Use NetQ for real-time network monitoring, including congestion detection and latency measurementsFabric-wide monitoring, validation and flow analysis with NetQ.
What it means
NetQ is NVIDIA's network operations tool. Agents on switches and hosts stream state and telemetry to a NetQ server, on premises or in the cloud, which stores history and provides a UI, CLI and validation checks.
What to understand
Know netq show for current and historical state, netq check validations for BGP, EVPN, MTU and other protocols, threshold-crossing events for queue length and RoCE counters, and flow analysis for latency and path visibility. NetQ answers what changed and when across the fabric, which single-switch commands cannot.
How it shows up
A monitoring need is described, and you pick the NetQ feature such as validation, threshold events or flow analysis.
Common confusion
Assuming NetQ configures the switches; it monitors and validates, while NVUE or automation changes config.
2.7 Install NVIDIA DOCAInstalling the DOCA software framework on hosts and BlueField devices.
What it means
DOCA is NVIDIA's software framework and driver stack for BlueField and ConnectX adapters. On the host you install DOCA-Host packages using a profile, and on BlueField you deploy a BFB bundle containing the Arm operating system and DOCA runtime.
What to understand
DOCA-Host profiles include doca-ofed for the classic driver stack, doca-networking and doca-all for the full set. DOCA-OFED replaces the older MLNX_OFED packages. BlueField images are pushed over the RShim interface with bfb-install. After installation, confirm drivers with ibv_devinfo or devlink, and check firmware with mlxfwmanager.
How it shows up
An installation goal is described, and you choose the right DOCA profile, package or BFB step.
Common confusion
Installing MLNX_OFED on new systems when DOCA-OFED is the current supported stack.
2.8 Configure NVIDIA SuperNIC functionality for advanced packet processing and congestion controlSetting up BlueField-3 or ConnectX-8 SuperNICs for Spectrum-X.
What it means
SuperNICs are the host side of Spectrum-X. They implement Spectrum-X congestion control using telemetry from the switches, handle out-of-order packets caused by adaptive routing through direct data placement into GPU memory, and accelerate RoCE.
What to understand
BlueField-3 can run in DPU mode or NIC mode; Spectrum-X SuperNIC deployments typically set the device for NIC operation and apply the Spectrum-X profile with mlxconfig and DOCA tools. Congestion control and adaptive routing must be enabled on both the switches and the SuperNICs to work end to end. Verify with mlxconfig query and RoCE counters.
How it shows up
A host-side requirement such as handling reordered packets or enabling congestion control is described, and you pick the SuperNIC setting.
Common confusion
Enabling adaptive routing only on switches; without SuperNIC reordering support, spraying hurts RDMA performance.

High-yield concepts

The terms, rules and formulas this domain turns into questions most often.

  • RoCE lossless mode

    Default NVUE RoCE profile with PFC and ECN enabled.

    Remember
    nv set qos roce, then nv config apply.
    Tested as
    RoCE configuration questions.
    Trap
    Forgetting to apply the pending config.

  • RoCE lossy mode

    ECN-based RoCE without PFC.

    Remember
    Relies on end-to-end congestion control.
    Tested as
    Mode identification.
    Trap
    Thinking lossy means no congestion control.

  • PFC

    Per-priority pause frames that prevent buffer overflow.

    Remember
    A safety net, not the primary control.
    Tested as
    Pause storm scenarios.
    Trap
    Using it as the main congestion tool.

  • ECN

    Marks packets past a queue threshold so senders slow down.

    Remember
    Triggers congestion notification packets back to the sender.
    Tested as
    Congestion tuning.
    Trap
    Assuming ECN drops packets.

  • RoCE adaptive routing

    Per-packet spraying across all equal-cost paths.

    Remember
    Needs SuperNIC reordering support.
    Tested as
    Load-balancing questions.
    Trap
    Expecting ECMP to handle elephant flows.

  • BGP-EVPN route type 5

    Advertises IP prefixes into a tenant VRF.

    Remember
    Type 2 carries MAC and host IP entries.
    Tested as
    EVPN design.
    Trap
    Mixing up route types 2 and 5.

  • L3 VNI

    Per-VRF VNI used for symmetric routed traffic.

    Remember
    One per tenant VRF.
    Tested as
    Multi-tenancy.
    Trap
    Reusing a VLAN VNI for routing.

  • NVIDIA Air

    Cloud digital twin of the network.

    Remember
    Simulates control plane and config, not ASIC performance.
    Tested as
    Change validation.
    Trap
    Using it for throughput testing.

  • What Just Happened

    ASIC-reported drop events with reasons.

    Remember
    Reasons include buffer, ACL, L1, L2, router and tunnel.
    Tested as
    Drop diagnosis.
    Trap
    Treating it as a counter readout.

  • NetQ validation

    netq check runs fabric-wide protocol checks.

    Remember
    Covers BGP, EVPN, MTU and more.
    Tested as
    Monitoring questions.
    Trap
    Expecting NetQ to push configs.

  • DOCA-Host profiles

    Package sets such as doca-ofed, doca-networking and doca-all.

    Remember
    DOCA-OFED replaces MLNX_OFED.
    Tested as
    Install questions.
    Trap
    Installing legacy MLNX_OFED.

  • BFB bundle

    Boot image for BlueField's Arm OS and DOCA runtime.

    Remember
    Installed over RShim with bfb-install.
    Tested as
    BlueField install steps.
    Trap
    Installing it like a host package.

  • SuperNIC congestion control

    Host-side rate control using switch telemetry.

    Remember
    Works with Spectrum switches end to end.
    Tested as
    Spectrum-X tuning.
    Trap
    Enabling it on one side only.

Side by side

Concepts candidates mix up, compared on the point that decides the answer.

ECN vs. PFC vs. adaptive routing

ConceptMeaningKey differenceCommon trap
ECN Marks packets to slow senders Proactive, end to end Assuming marked packets are dropped
PFC Pauses one priority hop by hop Reactive backstop Letting it run constantly
Adaptive routing Spreads packets over free paths Avoids hot links rather than slowing senders Skipping SuperNIC support

Spectrum visibility tools

ConceptMeaningKey differenceCommon trap
WJH Per-packet drop events with reasons Explains why a drop happened Reading only interface counters
NetQ Fabric-wide state, history and validation Answers what changed and when Expecting it to configure devices
In-band telemetry Queue and latency data along a path Locates congested hops Confusing it with SNMP polling
NVIDIA Air Simulated digital twin Pre-deployment testing Using it to measure performance

EVPN building blocks

ConceptMeaningKey differenceCommon trap
L2 VNI Stretches a VLAN over VXLAN Bridged traffic Using it for inter-subnet routing
L3 VNI Routed traffic per VRF Symmetric routing Sharing one across tenants
VRF Separate routing table per tenant Isolation boundary Leaking routes accidentally

Exam traps

Each trap is an answer that looks right. Learn the clue that should change your mind.

  1. PFC solves congestion

    What candidates pick
    Increase PFC aggressiveness to stop drops.
    Why it looks right
    PFC makes traffic lossless.
    Why it's wrong
    PFC only pauses; heavy pausing spreads congestion and causes head-of-line blocking, so ECN and congestion control must handle sustained load.

    Clue to watch for Counters show many pause frames and few ECN marks.

  2. ECMP is enough

    What candidates pick
    Rely on ECMP hashing for AI flows.
    Why it looks right
    ECMP works well in enterprise networks.
    Why it's wrong
    A few large synchronized flows hash onto the same links, so Spectrum-X uses per-packet adaptive routing instead.

    Clue to watch for Some uplinks are saturated while others are idle.

  3. Air measures throughput

    What candidates pick
    Benchmark RoCE bandwidth in NVIDIA Air.
    Why it looks right
    Air runs real Cumulus Linux images.
    Why it's wrong
    Air virtualizes the control plane and configuration, not the Spectrum ASIC data path or buffers.

    Clue to watch for The goal is a performance number.

  4. NetQ pushes config

    What candidates pick
    Use NetQ to change QoS on all switches.
    Why it looks right
    NetQ sees every switch.
    Why it's wrong
    NetQ monitors and validates; configuration changes go through NVUE, its API or automation such as Ansible.

    Clue to watch for The task is to modify settings.

  5. Counters explain drops

    What candidates pick
    Read interface discard counters to find the cause.
    Why it looks right
    Counters are familiar.
    Why it's wrong
    Counters show totals only; WJH reports each drop with its reason, port and flow.

    Clue to watch for The question asks why a packet was dropped.

  6. One VRF for all tenants

    What candidates pick
    Put tenants in one VRF with separate VLANs.
    Why it looks right
    VLANs already separate broadcast domains.
    Why it's wrong
    Shared VRFs share a routing table, so tenants can route to each other; isolation needs a VRF and L3 VNI per tenant.

    Clue to watch for Requirement says tenants must not reach each other.

Answer framework

Work through Spectrum-X questions by locating the problem, then matching the configuration or tool.

  1. Decide whether the scenario is about switch configuration, tenant isolation, host or SuperNIC setup, pre-deployment testing, or live monitoring.
  2. For RoCE behavior, confirm the QoS profile mode, check that ECN marks before PFC pauses, and verify the RoCE priority is consistent end to end.
  3. For uneven link use with large flows, choose adaptive routing and confirm both switches and SuperNICs support packet spraying and reordering.
  4. For tenants, map each one to its own VRF and L3 VNI with EVPN, and treat any route leaking as an explicit design decision.
  5. For diagnosis, pick WJH to explain individual drops, NetQ for fabric-wide history and validation, and Air for testing changes before production.

Practice question

A Spectrum-X fabric shows several leaf uplinks at full utilization while others sit nearly idle during AI training. ECMP is enabled and all links are healthy. Which change best addresses this?

Want more like this? The NCP-AIN practice test has questions for every domain.

Checklist

0 of 13

Tick each one you can do without notes. Your ticks stay in this browser.

Study strategy

Plan the largest block of study time for this domain and do most of it hands-on. Create a free NVIDIA Air simulation with two spines, four leaves and a few servers, then use NVUE to enable RoCE, check nv show qos roce, and switch between lossless and lossy modes so you recognize both outputs.

In the same lab, build BGP-EVPN with two tenant VRFs, map VLANs to L2 VNIs, assign L3 VNIs and confirm the tenants cannot reach each other. Add NetQ to the simulation and run netq check for BGP and EVPN, then break something deliberately and watch the validation fail.

Because Air cannot model buffers or RoCE throughput, cover congestion behavior through reading. Study the Spectrum-X whitepaper for adaptive routing, congestion control and SuperNIC reordering, and the Cumulus Linux WJH pages for drop reasons. Finally, walk through the DOCA installation guide, noting profiles, the BFB install over RShim and mlxconfig queries. Practice questions that show NVUE output are the best check of readiness here.

FAQ

How much of NCP-AIN is NVIDIA Spectrum Networking?

NVIDIA Spectrum Networking is 30% of the exam, tied with InfiniBand as the largest domain, with eight official objectives.

Which NVUE command enables RoCE on a Spectrum switch?

nv set qos roce enables the RoCE profile, which defaults to lossless mode. Run nv config apply, then verify with nv show qos roce.

What is the difference between lossless and lossy RoCE?

Lossless mode uses PFC plus ECN to avoid drops. Lossy mode keeps ECN marking but disables PFC, relying on end-to-end congestion control.

Why does Spectrum-X need SuperNICs?

Adaptive routing sprays packets over many paths, which can reorder them. SuperNICs place out-of-order data correctly and run congestion control using switch telemetry.

When should I use WJH instead of NetQ?

Use WJH to learn why specific packets were dropped. Use NetQ for fabric-wide monitoring, history, validation and threshold events.

Can NVIDIA Air test RoCE performance?

No. Air simulates switch software, control plane and configuration. It does not model ASIC buffers or real throughput.

Does DOCA replace MLNX_OFED?

Yes. DOCA-Host, including the doca-ofed profile, is the current host driver stack for ConnectX and BlueField adapters.

Other Artificial Intelligence & Machine Learning certifications

Where the same subject shows up in other exams. Cards marked "Same subject" open the matching domain.

Test your Domain 2 knowledge

You have covered RoCE configuration, congestion features, EVPN tenancy, Air, WJH, NetQ, DOCA and SuperNICs. Test your Spectrum-X judgment on timed scenario questions.

Take the NCP-AIN practice test Check your NCP-AIN readiness

Keep going