Pegasus/HTCondor Distributed Workflow Deployment on FABRIC ( public )

PegasusAI

This Jupyter notebook automates the deployment of a distributed Pegasus/HTCondor infrastructure on the FABRIC testbed. It provisions a central submit node and multiple worker nodes across geographically distributed FABRIC sites, interconnects them using FABNetv4, and configures all components for scalable scientific workflow execution.

  • Submit Node: Hosts the HTCondor Central Manager and the Pegasus Workflow Management System.
  • Worker Nodes: Deployed across multiple FABRIC sites and configured as HTCondor execute nodes.
  • Automated Setup Includes:
    • Slice creation and multi-node provisioning
    • FABNetv4 network configuration between all nodes
    • Automated installation of HTCondor and Pegasus
    • Passwordless SSH key exchange across nodes
    • Custom /etc/hosts configuration for consistent hostname resolution
    • Role-based configuration for submit and worker nodes

This notebook is intended for researchers, educators, and workflow developers who want to deploy, run, and validate distributed scientific workflows on FABRIC using Pegasus and HTCondor in a reproducible and automated manner.

Views
158
Downloads
15 active (142 retired)
Versions
1
Last Updated
March 31, 2026, 2:25 a.m.
Version Created URN Downloads Actions
2026-03-31.1 March 31, 2026, 10:44 p.m. urn:fabric:contents:renci:1a2f6d72-42b4-4140-aae0-aa39b02cd918 15 download
University of North Carolina at Chapel Hill — kthare10@email.unc.edu