Pegasus/HTCondor Distributed Workflow Deployment on FABRIC ( public ) ( PegasusAI )

This Jupyter notebook automates the deployment of a distributed Pegasus/HTCondor infrastructure on the FABRIC testbed. It provisions a central submit node and multiple worker nodes across geographically distributed FABRIC sites, interconnects them using FABNetv4, and configures all components for scalable scientific workflow execution.

  • Submit Node: Hosts the HTCondor Central Manager and the Pegasus Workflow Management System.
  • Worker Nodes: Deployed across multiple FABRIC sites and configured as HTCondor execute nodes.
  • Automated Setup Includes:
    • Slice creation and multi-node provisioning
    • FABNetv4 network configuration between all nodes
    • Automated installation of HTCondor and Pegasus
    • Passwordless SSH key exchange across nodes
    • Custom /etc/hosts configuration for consistent hostname resolution
    • Role-based configuration for submit and worker nodes

This notebook is intended for researchers, educators, and workflow developers who want to deploy, run, and validate distributed scientific workflows on FABRIC using Pegasus and HTCondor in a reproducible and automated manner.


80    117 (0)    2
Jan. 28, 2026, 6:07 p.m.


Versions

2026-01-28 Jan. 28, 2026, 6:07 p.m. urn:fabric:contents:renci:54a4ccb0-43e0-4458-8a07-bf7771770ae1 55 download
2026-01-18 Jan. 18, 2026, 9:43 p.m. urn:fabric:contents:renci:5a5978b5-c92b-41b1-aae2-a47e39b81a6f 62 download

NOTE: Authors are not being displayed (double-blind, or other reason)