Overview

This notebook emulates a BlueField-3 DPU + GPU installed in the same server and connected through PCIe — and exercises the DPU data paths between two such servers across FABRIC sites. Each node is a VM pinned to a physical host with a dedicated SmartNIC and a GPU passed through as PCI devices, so both share one PCIe fabric exactly as a DPU+GPU appliance would. Port 1 of each SmartNIC is joined by a wide-area L2STS circuit for real two-endpoint dataplane experiments.

What it does

• Site discovery: scans fablib resource advertisements for hosts offering a free dedicated SmartNIC and a free GPU on the same server, and selects two hosts on distinct sites. SmartNIC preference: BlueField-2 → ConnectX-7 → ConnectX-6 → ConnectX-5 (a BlueField pick yields a real DPU); GPU preference is configurable (Tesla T4 first by default). PREFER_SITES/AVOID_SITES knobs constrain the choice.
• Provisioning: the full two-node topology is created in a single slice request (no NIC hot-plug — see pitfalls below), with the build plan and slice UUID persisted per-orchestrator for safe, idempotent re-runs.
• Bring-up: PCIe placement verification, NVIDIA driver install, RDMA stack bring-up (including the linux-modules-extra/mlx5_ib fix for FABRIC Ubuntu images), GPUDirect-style PCIe path checks, and cross-site addressing.
• Example workloads: cross-site rping and ib_write_bw over RoCE; a PyTorch fp16 matmul GPU benchmark; and a cross-site network→GPU pipeline in which one node streams 256 MiB over the WAN into the other node's GPU with an on-GPU checksum — the data-movement pattern behind distributed/federated training and GPUDirect-style ingest.