# Evaluation of TCP Congestion Control for Public High-Performance Wide-Area Networks

## Overview

This artifact builds a large multi-site topology on the FABRIC testbed using L2PTP tunnels with reserved bandwidth and deploys a custom DPDK-based traffic shaping application on the router nodes. The purpose of the experiments is the assessment of the fitness of the tested congestion control schemes (TCP CUBIC, BBRv1, and BBRv3) for support of massive data transfers between remote data generation and data processing locations—a use case common in scientific computing, AI workloads, and GPU-as-a-service scenarios where compute and storage resources are tightly scheduled and expensive.

**Evaluation criteria.** Results are presented using two metrics: (1) **FCT efficiency** (or FCT ratio), defined as the ratio of ideal to measured flow completion time—the ideal FCT is the file size divided by the bottleneck capacity, adjusted for protocol headers; higher values (closer to 100\%) indicate more predictable and efficient transfers; (2) **Bandwidth overhead**, the percentage of extra bytes sent due to packet retransmissions relative to the data delivered; lower overhead indicates fewer losses and more efficient use of the reserved bandwidth.

**Paper.** This artifact is for the experiments in the paper *“Evaluation of TCP Congestion Control for Public High-Performance Wide-Area Networks,”* accepted to the **International Conference on High Performance Switching and Routing (IEEE HPSR 2026)**, Montreal, Canada, June 2026. A preprint of the paper is available on arXiv: https://arxiv.org/abs/2603.12660

## Authors

- Fatih Berkay Sarpkaya, New York University
- Andrea Francini, Nokia Bell Labs

## Topology

The experiment uses 4 FABRIC sites connected by 8 L2PTP tunnels (20 Gbps each by default):

- **Sender Site 1**: 4 senders (S1-S4), 2 inter-routers (ER1-ER2), Router1 (SmartNIC)
- **Sender Site 2**: 4 senders (S5-S8), 2 inter-routers (ER3-ER4), Router2 (SmartNIC)
- **Receiver Site 1**: 4 receivers (D1-D2, D5-D6), Router3 (SmartNIC)
- **Receiver Site 2**: 4 receivers (D3-D4, D7-D8), Router4 (SmartNIC)

See `large-topology.png` for the full topology diagram.

## Related Resources

Details of the DPDK application (tm10 / Routing-Aware Shaper) can be found in:

- [Nokia-Bell-Labs/DPDK-RoutingAwareShaper](https://github.com/Nokia-Bell-Labs/DPDK-RoutingAwareShaper)
- [Lightweight Determinism in Large-Scale Networks](https://ieeexplore.ieee.org/document/10529953) (IEEE)
- [Routing-Aware Shaping for Feasible Multi-Domain Determinism](https://dl.acm.org/doi/10.1145/3636534.3696727) (ACM)

## Concepts Used in This Notebook

This notebook demonstrates the following FABRIC and DPDK concepts:

- **L2PTP tunnels with reserved bandwidth**: Guidance on reserving L2PTP tunnels with specified bandwidth and ERO (Explicit Route Object) hops for inter-site connectivity.
- **VLAN sub-interfaces**: Using `add_sub_interface` on router NICs to multiplex multiple WAN tunnels (one VLAN per tunnel) over a single physical port.
- **Automatic resource discovery**: Querying `list_hosts`, `list_sites`, and `list_links` to select sites and hosts with sufficient cores, RAM, disk, and link capacity.
- **Programmatic interface resolution**: Assigning IPs using FABRIC L2 network membership and runtime device resolution (MAC/VLAN matching) so configuration is robust to Linux device name drift after reboot.

## Prerequisites

- Active FABRIC account with project membership
- Sufficient FABRIC resources (see notebook for automatic resource discovery)

## Artifact Contents

| File | Description |
|------|-------------|
| `ARTIFACT_inter_site_experiments_large_topology.ipynb` | Main experiment notebook |
| `large-topology.png` | Topology diagram |
| `node_toolsv2/` | Shell scripts for DPDK/node setup (install dependencies, VFIO). GRUB and hugepages are configured by the notebook. |
| `analysis_final.py` | Data analysis script uploaded to experiment nodes |

- The Experiment-output folder includes the output files from the example experiment in the notebook and is used for plotting within the notebook.
- The generated_tm10_cfg and generated_tm10_scripts folders include files for MAC address configuration, VLAN mapping, and run scripts. These files are generated automatically based on the node interfaces and topology, but are included here for reference.


## Reproduction Instructions

1. Upload this artifact to the FABRIC JupyterHub environment.
2. Open `ARTIFACT_inter_site_experiments_large_topology.ipynb`.
3. Run the cells sequentially. The notebook will:
   - **Automatically discover** available FABRIC sites with sufficient resources
   - Reserve a slice with 24 VMs across 4 sites
   - Install a BBRv3-capable kernel on sender/receiver nodes
   - Configure IOMMU, DPDK, and hugepages on router nodes
   - Set up IP addressing, routing, and ARP entries
   - Run experiments with configurable congestion control algorithms
   - Collect and visualize results

The automatic site selection queries live resource availability and selects sites with enough cores, RAM, disk, and SmartNICs. If resources are insufficient, it prints diagnostics showing why each site was rejected.

## Configurable Parameters

Key parameters in the configuration cell:
- `L2PTP_BW`: L2PTP tunnel bandwidth in Gbps (default: 20)
- `SENDER_CORES`, `SENDER_RAM`, etc.: per-VM resource specifications
- `exp_factors` (in the Experiments section): congestion control algorithms, flow counts, trial count

## Notes

- Slice creation may take 30-60 minutes.
- Delete the slice when finished to free resources for other experimenters.
