# Federated Grid Fault Detection — FABRIC weave (base example)

A runnable **starter** for *network-aware, resilient federated learning (FL)* for
**grid incipient-fault detection** across geographically distributed "utilities"
on the [FABRIC](https://fabric-testbed.net) testbed.

> **This is v0 — the base example.** It does
> **star** and **mesh** FL over a real wide-area network, with a compact 1D-CNN, a
> turnkey synthetic dataset plus the real Kaggle **VSB** partial-discharge data,
> GPU by default, `tc netem` perturbation, and optional churn. The full north-star
> design (more datasets, P4 in-network aggregation, ring/hybrid topologies, OE-417
> priors) is roadmap. See **[`weave.md`](weave.md)**
> for the authoritative spec.

---

## Two phases: the weave brings up the slice, the notebook runs the FL

This is the key design point. Provisioning and experimentation are separated so
you provision **once** and iterate on the science **many times**.

```
  Phase A — the weave (bring-up only)          Phase B — the notebook (run FL)
  ┌───────────────────────────────┐           ┌────────────────────────────────┐
  │ ./weave.sh                     │           │ federated_learning.ipynb       │
  │  • create slice (agg + N       │  slice_   │  • connect to the slice        │
  │    clients on FABNetv4, GPU)   │  info.    │  • run_star(...) / run_mesh(...)│
  │  • install torch + flwr        │  json     │  • perturbation / churn        │
  │  • upload fl/                  │ ────────▶ │  • collect metrics + figures   │
  │  • write results/slice_info.json│           │  • repeat with new settings    │
  │  • print READY and EXIT        │           │    (no re-provisioning)        │
  └───────────────────────────────┘           └────────────────────────────────┘
```

The weave **does not run FL and does not poll in a loop** — it stands the slice up
and exits. All training happens interactively in the notebook.

---

## Quickstart

### 1. Bring up the slice (Phase A)

From LoomAI (Run button) or a shell on the LoomAI host:

```bash
# GPU profile (default), 4 client sites
SLICE_NAME=fl-grid NUM_CLIENTS=4 ./weave.sh

# CPU-edge profile instead
SLICE_NAME=fl-grid USE_GPU=false ./weave.sh
```

Wait for `### PROGRESS: READY`. This creates `agg` + `client-<iso>` nodes on a
`FABNetv4` network, installs PyTorch (GPU wheels by default) + Flower, uploads the
`fl/` package, and writes **`results/slice_info.json`**.

**Prefer a notebook?** Open **`provision_slice.ipynb`** instead — it does the same
bring-up interactively and lets you **preview the discovered GPU/host placement**
before committing resources.

### 2. Run the federated learning (Phase B)

Open **`federated_learning.ipynb`** and run the cells top to bottom — it's plain
fablib, no extra library to import. The flow:

1. **§1 Connect** to the slice (reads `results/slice_info.json`).
2. **§1b Choose the dataset** — set `DATASET` (`pq_synth` or `vsb`) and
   `CLASS_WEIGHTS`; for `vsb`, provide a Kaggle token (see *Kaggle setup for `vsb`*
   under [Data](#data-pq_synth-turnkey-or-real-vsb)) and confirm it prints
   `kaggle auth + rules: OK`.
3. **§3 Install** the FL code + stack on every node (run once per slice; this is
   also where the CUDA/torch install profile is set).
4. **§5 Star** (Flower FedAvg) and **§7 Mesh** (decentralized peer-averaging over
   FABNetv4) — each collects `metrics.json` and draws the convergence figures
   (§6, §8).

Change `DATASET`, `ROUNDS`, `AGG`, or `CLASS_WEIGHTS` and re-run — the same slice
is reused, no re-provisioning.

### 3. Tear down when finished

```bash
./weave_cleanup.sh fl-grid        # delete the FABRIC slice + clear results/
# keep the slice, just clear local run artifacts:
DELETE_SLICE=false ./weave_cleanup.sh fl-grid
```

---

## Troubleshooting (new to FABRIC? start here)

The notebook is self-contained — debug from its own cells and helpers
(`tail(node, logfile, n)`, `running(node, pattern)`).

- **First move for any FL failure: re-run §4 (quick check)** — it pings each client
  → aggregator and tests the FL port over FABNetv4, printing per-node reachability.
- **A node missing the venv / `fl/` code / imports** — re-run **§3** (re-uploads
  `fl/` and reinstalls the stack).
- **A run hangs at "waiting"** — a client died or never connected; tail its log
  (see table).

| Symptom | Cause | Fix |
|---------|-------|-----|
| §3 import check ≠ `OK` | torch/flwr didn't install (apt lock at boot) | re-run §3; setup waits out the apt lock and falls back to an apt-free venv |
| §3 `venv MISSING` | setup ran with `sudo` (venv went to `/root`) | re-run §3 **without** `sudo` |
| §4 `CANNOT CONNECT` | FABNetv4 path / firewall on the port | re-check bring-up; every node needs a `FABNET_IPv4_*` interface |
| §5 *server did not start* | usually `~/results` missing (auto-created) or a bad arg | §5 prints the server-log tail |
| run hangs at "waiting" | a client died / never connected | `tail(nodes["client-ercot"], "star_client-ercot.log", 40)` |
| `vsb` client crashes on download | 401 = bad/expired token; 403 = rules not accepted | re-run §1b (fresh `KGAT_` token); accept the competition rules |

**Golden rules**
- Run the setup scripts **as the login user (no `sudo`)** so the venv lands in
  `/home/ubuntu/fl-venv` where the FL runs. `sudo` puts it in `/root` → nothing works.
- After editing `fl/` code on the host, re-run **§3** to push it to the nodes.

---

## Compute profiles (GPU by default)

| Profile | How | Client node | PyTorch |
|---------|-----|-------------|---------|
| **GPU (default)** | `USE_GPU=true` | 8c / 32 GB / 100 GB + `GPU_MODEL` | CUDA wheels (`CUDA_TAG`, default `cu121`) |
| **CPU-edge** | `USE_GPU=false` | 4c / 16 GB / 50 GB | CPU wheels |

`tools/setup-common.sh` auto-detects the GPU (`lspci | grep nvidia`): it installs
the NVIDIA driver + CUDA PyTorch when a GPU is present and CPU wheels otherwise, so
a mis-scheduled node still runs. `fl/model.py` picks `cuda` vs `cpu` at run time.

### Placement is discovered at the host level, not hardcoded

Site-aggregate availability lies (a site can report a free GPU whose actual host
can't fit the VM, or lacks that GPU) — so the weave discovers at the **host
level**, exactly like `pegasus-fabric.ipynb`. On a **GPU run** it queries FABRIC
(`get_resources()` → `list_hosts()` + `get_hosts_by_site()`) for each host's
per-model GPU availability (`rtx6000_available`, `a40_available`, `a30_available`,
`tesla_t4_available`), keeps only **`Active` hosts that have a free GPU *and* room
for the 8c/32 GB/100 GB VM on that same host**, **pins each client to a specific
host** (`add_node(host=...)`), and requests the GPU model that host actually has.
Clients spread across distinct sites first. You'll see this during `start`:

```
### PROGRESS: GPU-capable hosts found: 4 across 3 sites
###   TACC   host=tacc-w1     gpu=GPU_A40      fits=4
###   STAR   host=star-w2     gpu=GPU_A30      fits=3
### PROGRESS: Client placement:
###   client-ercot     site=TACC   host=tacc-w1     gpu=GPU_A40
###   client-caiso     site=STAR   host=star-w2     gpu=GPU_A30
```

Controls:
- `GPU_MODEL=auto` (default) searches all models; pin one with `GPU_MODEL=GPU_A40`.
- **CPU runs** use the same host-level discovery for the 4c/16 GB/50 GB VM (no GPU);
  the **aggregator** auto-places unless `AGG_GPU=true`.
- `SITE_MAP="ercot=TACC,caiso=LOSA,agg=RENC"` forces sites (overrides discovery).
  Suggested realistic footprints: ERCOT→TACC, CAISO→LOSA, PJM→WASH, MISO→STAR,
  SPP→KANS, NYISO→NEWY, agg→RENC.
- `EXCLUDE_SITES="EDUKY,EDC"` skips problem sites during discovery.

The final per-node site, **host**, GPU model, and coordinates are recorded in
`results/slice_info.json` and drawn on `topology.png`.

---

## What the two topologies do

- **star** — a Flower FedAvg/FedProx server on `agg`; each client trains on its
  non-IID partition and reports weights. Classic centralized FL — single
  aggregator bottleneck + trust dependency (the baseline for communication-efficient
  aggregation).
- **mesh** — **no central aggregator.** Each peer trains locally, serves its
  weights on a tiny HTTP endpoint, pulls its neighbors' weights **directly over
  FABNetv4**, and peer-averages (sync barrier or async gossip). This is the
  decentralized setting for network-aware convergence and resilience; the network
  carries the real per-round weight exchange, so convergence reflects true testbed
  behavior.

Both share `fl/model.py` (a ~10⁴-parameter 1D-CNN) and `fl/data.py`, so star vs
mesh results are directly comparable.

## Data (`pq_synth` turnkey, or real `vsb`)

`pq_synth` *(default)* generates IEEE-1159-style power-quality disturbance
waveforms **on the node** — no download, no account — with a deliberately
**non-IID** partition per client (a client-specific class mix + noise level).
Classes include an **incipient-fault** transient signature (the headline
detection target).

`vsb` uses the **real** Kaggle *VSB Power Line Fault Detection* partial-discharge
data (measured on medium-voltage covered conductors). Each client downloads it
once (cached at `~/.cache/kagglehub`) and trains on a **disjoint set of measurement
lines** — physical, not-injected non-IID. Binary PD-vs-normal; each 800k-sample
line is peak-preserving-downsampled so the same 1D-CNN applies. Select via
`DATASET = "vsb"` in the notebook's **§1b** cell.

### Kaggle setup for `vsb` (one-time)

Download uses [`kagglehub`](https://pypi.org/project/kagglehub/) with a modern
**`KGAT_` access token** (the legacy `kaggle.json` username+key pair still works as a
fallback). The competition (rules + data) lives at
**<https://www.kaggle.com/competitions/vsb-power-line-fault-detection>** — *not* any
`/code/...` notebook link.

1. **Accept the rules** on that page → **Rules** tab → *I Understand and Accept*.
   (Required even with a valid token; skipping this gives HTTP **403**.)
2. **Create an access token**: kaggle.com → **Settings** → **API** → **Create New
   Token** → a `KGAT_…` token. This **invalidates any previous token**.
3. **Give the token to §1b** any one way: paste it into `KAGGLE_API_TOKEN` (most
   reliable), export the `KAGGLE_API_TOKEN` env var, or place it in
   `~/.kaggle/access_token`. §1b writes it to `~/.kaggle/access_token` on every node
   and verifies it — do not proceed until it prints `kaggle auth + rules: OK`.

**Reading the errors** on a `vsb` run: **401 Unauthorized** = bad/expired/missing
token (regenerate it, step 2); **403 Forbidden** = token OK but rules not accepted
(step 1). The first `vsb` run is slower — each client pulls the ~GB parquet once,
then it's cached under `~/.cache/kagglehub`.

Both datasets share one loader interface (`load_partition`), so swapping the
remaining roadmap datasets (PSML, feeder_sim, …) is a `fl/data.py` change only.

---

## Outputs

Written to `results/` by the notebooks:

| Output | What it shows |
|--------|---------------|
| `convergence_star.png` | star accuracy / fault-F1 vs round (§6) |
| `convergence.png` | star vs mesh accuracy vs round (§8) |
| `topology.png` | deployed slice: sites + star/mesh communication graph (from `provision_slice.ipynb`) |
| `metrics.json` / `metrics-<peer>.json` | all raw per-round metrics + measured model footprint (params, KB) |

Suggested experiment sweeps are in `weave.md` §9. *(Communication-efficiency and
churn-resilience figures are on the roadmap — see `weave.md`.)*

---

## Files

| Path | Purpose |
|------|---------|
| `weave.md` | authoritative spec |
| `weave.json` / `weave.sh` | Phase-A provisioning args + bring-up orchestrator |
| `experiment.py` | Phase-A: FABlib provisioning → `slice_info.json` (no FL loop) |
| `provision_slice.ipynb` | Phase-A notebook: interactive bring-up + placement preview (alt to `weave.sh`) |
| `federated_learning.ipynb` | Phase-B notebook: launch star/mesh FL over SSH, collect metrics, figures |
| `requirements.txt` | node/runtime Python deps (torch installed separately) |
| `fl/` | on-node FL code (model, data, star, mesh, metrics, perturb) |
| `tools/` | per-node setup scripts (`setup-torch.sh` = GPU/CUDA config) |

---

## Requirements

- A configured FABRIC environment (FABlib / `fabrictestbed-extensions`) on the
  host that runs the weave and the notebook — LoomAI provides this.
- The nodes install their own PyTorch/Flower during Phase A; nothing to install by
  hand on the VMs.
