A New Architecture for Petabyte-scale File Transfer Evaluated in the FABRIC testbed ( public )
The Multi-Level Error Detection (MLED) framework is a recursive architecture designed to reduce the probability of undetected errors (UEP) during large-scale file transfers. It organizes communication in layers, each implementing its own error detection policy, and can be configured to replicate error detection mechanisms of existing file transfer frameworks. Through a mathematical formulation and experimental validation on the FABRIC testbed, MLED demonstrates reduced undetected errors by intercepting them at intermediate layers, minimizing the need for full file retransmission under non-zero error rates. This enhanced error detection capability makes MLED highly reliable for scientific applications requiring data integrity.
This artifact contains a comprehensive collection of Jupyter notebooks designed to facilitate the entire workflow for conducting and analyzing MLED experiments on the FABRIC testbed. This artifact includes the following components:
Setting Up a FABRIC Slice: Notebooks to assist with reserving and configuring slices on the FABRIC testbed, both with and without the use of facility ports. These notebooks guide the setup of the network topology, including the configuration of virtual machines and optional integration of a remote data source via a facility port for greater experimental flexibility. They ensure proper communication between nodes and provide a fully functional environment for MLED deployment.
Running Experiments: The MLED_run_experiments.ipynb notebook is included to automate and control the execution of experiments across multiple configurations. It allows users to deploy MLED processes on designated nodes and conduct file transfer experiments under varying conditions and error rates.
Analyzing Experimental Results: The MLED_result_analysis.ipynb notebook is tailored for processing and visualizing the results of the conducted experiments. It provides tools to calculate performance metrics such as goodput, memory usage, and undetected error probability (UEP), generating graphical representations for easier analysis and comparison between configurations.
These notebooks collectively ensure a reproducible, structured approach for configuring, executing, and analyzing MLED experiments on the FABRIC testbed. Users with an active FABRIC account can replicate the results presented in the research paper by following the provided instructions. If you have the experimental output of MLED then FABRIC account is not required for analyzing the resutls using the MLED_result_analysis.ipynb notebook.
4
2
(0)
1
Jan. 15, 2025, 6:35 a.m.
Jan. 15, 2025, 6:35 a.m.
Versions
| 2025-01-15 | Jan. 15, 2025, 6:39 a.m. | urn:fabric:contents:renci:c2152ef8-a82a-457c-a35c-e3043204f6d2 | 2 | download |
Authors
- Abraham I Matta , Boston University (matta@bu.edu)
- Arash SARABI , Arizona State University (sarabi.arash@asu.edu)
- Prateek Jain , Boston University (jainp@bu.edu)
- Violet Syrotiuk , Arizona State University (syrotiuk@asu.edu)