# Federated Learning — Grid Fault Detection (FABRIC weave)

**Spec file.** This is the authoritative specification for the
`Federated_Grid_Fault_Detection` weave. Edit this file to change the design; the
weave code (`weave.json`, `experiment.py`, `fl/`, `tools/`) is built to match it.

> **Status — base example (v0).** The code shipped alongside this spec is a
> **runnable starter**, *not* the full
> design below. What is built now: **star + mesh** topologies over `FABNetv4`, a
> compact 1D-CNN, the turnkey **`pq_synth`** dataset (generated on-node, no
> downloads/accounts) **plus the real `vsb`** partial-discharge dataset (downloaded
> + cached on-node from Kaggle), **GPU by default** with a CPU-only toggle,
> `tc netem` perturbation, optional mid-training churn, and the convergence/topology
> figures + `metrics.json`. Everything else in this spec (the other five datasets,
> OE-417 priors, P4 in-network aggregation, `ring`/`hybrid`, async staleness
> weighting) is **roadmap** — the interfaces are written so those drop in without a
> redesign.
> §10 tracks what is v0 vs roadmap. The rest of this document is the north-star
> design; treat unbuilt items as intent, not current behavior.

---

## 1. Purpose

A self-contained FABRIC weave that deploys **network-aware, resilient federated
learning (FL)** for **grid incipient-fault detection** across geographically
distributed "utilities," and measures how **topology**, **communication pattern**,
and **network perturbation** govern **convergence, communication efficiency, and
resilience** — grounded in *real testbed network behavior*, not simulation.

### Capabilities

| Capability | Description | What the weave exercises |
|------------|-------------|--------------------------|
| Network-aware convergence | Decentralized FL across star/mesh/ring, sync vs async/gossip, under `tc netem` perturbation | `TOPOLOGY` + `SYNC` + `LINK_*` args; convergence-rate & final-quality metrics per condition |
| Communication-efficient aggregation | Quantify the single-aggregator bottleneck; in-network (P4 data-plane) aggregation | v1 measures per-round communication bytes; P4 in-network aggregation is a documented **roadmap** hook (§10) |
| Resilience | Graceful degradation under churn, partial connectivity, and stragglers | `CHURN_CLIENTS` + `CHURN_ROUND` + `LINK_LOSS_PCT`; bounded-degradation plot |
| Minimal on-device detection | Compact fault-detection models that train on edge / on-device hardware | compact 1D-CNN; reported params / RAM / train footprint; **GPU by default**, CPU-only option for edge realism |

---

## 2. Scenario & topology

Each FABRIC site plays a **utility / ISO-RTO footprint** holding a **non-IID**
slice of grid power-quality data. The weave supports two communication topologies,
**selectable at run time** via the `TOPOLOGY` arg, over the same physical slice:

- **`star`** — one central **aggregator** runs a Flower (`flwr`) FedAvg/FedProx
  server; each client site trains locally and reports to the aggregator. Classic
  centralized FL; single-aggregator bottleneck + trust dependency (the baseline
  for evaluating communication-efficient aggregation).
- **`mesh`** — **no central aggregator**. Each site is a peer that trains locally
  then exchanges weights with its neighbors and **peer-averages** (decentralized
  SGD / gossip). Supports **sync** (per-round barrier) and **async/gossip**
  (`SYNC=async`). This is the decentralized setting for evaluating network-aware
  convergence and resilience.

`ring` and `hybrid` are natural extensions of the mesh peer graph and are
documented as roadmap (§10); the mesh peer layer is written to accept an arbitrary
neighbor adjacency so they drop in without a redesign.

### Site placement — discovered at submit time (never hardcoded)

Geographic placement is part of the experiment — the physical geography sets the
WAN conditions between operators — but sites are **discovered dynamically at
submit time**, not hardcoded:

- **GPU runs (default):** the weave queries FABRIC at the **host level**
  (`list_hosts` + `get_hosts_by_site`) for the per-host available GPU inventory
  across all models (`rtx6000_available`, `a40_available`, `a30_available`,
  `tesla_t4_available`), keeps only **`Active` hosts that have a GPU free *and*
  room for the 8c/32 GB/100 GB client VM on that same host**, then **pins each
  client to a specific host** (`add_node(host=...)`) and requests the GPU model
  that host actually has. This avoids the two failure modes of site-aggregate
  selection — a site reporting a free GPU whose chosen host lacks it, and "no host
  fits the VM." Clients spread one-per-site first (geographic diversity), then
  fill. A run can mix RTX6000 / A40 / A30 / TeslaT4 across sites. `GPU_MODEL=auto`
  searches all models; pin it (e.g. `GPU_MODEL=GPU_A40`) to restrict discovery.
- **CPU runs:** the same host-level discovery pins each client to an `Active` host
  with room for the 4c/16 GB/50 GB VM, spread across sites (no GPU requested).
- **The aggregator:** auto-placed (`site=None`) unless `AGG_GPU=true` or overridden.
- **`SITE_MAP` override:** force a specific site per ISO when you need to, e.g.
  `SITE_MAP="ercot=TACC,caiso=LOSA,agg=RENC"`. Overrides win over discovery.
- **`EXCLUDE_SITES`:** comma-separated sites to skip (e.g. `EXCLUDE_SITES="EDUKY,EDC"`).

`NUM_CLIENTS=4` maps roles ERCOT, CAISO, PJM, MISO; `NUM_CLIENTS=6` adds SPP and
NYISO. The role→ISO name is fixed (it labels the client and shapes its non-IID
partition); the **site** each role lands on is whatever discovery/allocation
chooses and is recorded in `slice_info.json`. If fewer GPU-capable sites are found
than clients, the weave packs extra clients onto sites with spare GPUs, then
falls back to auto-placement for any remainder — logging every decision.

*(Suggested geographic targets if you want to pin realistic footprints via
`SITE_MAP`: ERCOT→TACC, CAISO→LOSA, PJM→WASH, MISO→STAR, SPP→KANS, NYISO→NEWY,
agg→RENC. These are hints, not defaults.)*

### Nodes

| Node | Count | Role | Site | Size — CPU / GPU profile |
|------|-------|------|------|--------------------------|
| `agg` | 1 | Flower server (star) / metrics coordinator (mesh) | `RENC` | 4c / 16 GB / 50 GB (GPU optional, `AGG_GPU`) |
| `client-<iso>` | `NUM_CLIENTS` | local training + (star) Flower client / (mesh) gossip peer | per map above | 8c / 32 GB / 100 GB + GPU (default) → 4c / 16 GB / 50 GB CPU-only |

Compute is a **profile** chosen at run time (see §3.1). **GPU is the default**
(`USE_GPU=true`) — a `GPU_MODEL` is attached to each client and CUDA + GPU
PyTorch is installed — for fast experiment turnaround and headroom on the
accuracy-vs-footprint upper end. Set `USE_GPU=false` for the CPU-only
edge-realism profile.

### Networks

- **`fabnet`** — a single **`FABNetv4`** routed-IPv4 network; every node gets one
  `NIC_Basic` on it (`<node>-nic1-p1`). This gives real, instrumented **wide-area
  L3 paths** between sites. All FL traffic (star reports / mesh peer exchange)
  rides this network.
- **Network perturbation** is applied with `tc netem` on each node's `fabnet`
  egress: `LINK_DELAY_MS`, `LINK_LOSS_PCT`, `LINK_RATE_MBIT`. Because the sites are
  real and geographically apart, base latency is already realistic; perturbation
  adds controlled, repeatable degradation on top.
- **Roadmap:** `L2STS` point-to-point L2 links and a P4/BMv2 aggregation node for
  in-network aggregation (§10).

```
                 ┌────────────── FABNetv4 (fabnet) ──────────────┐
                 │                                               │
   client-ercot(TACC)   client-caiso(LOSA)   client-pjm(WASH)   client-miso(STAR)
                 │                                               │
                 └──────────────── agg (RENC) ───────────────────┘
   star:  clients ─▶ agg (FedAvg)          mesh: clients ⇄ clients (peer-avg), agg = collector
```

---

## 3. Model — minimal on-device detector

- **Task:** classify short power-quality waveform windows into disturbance classes
  including an **incipient-fault** signature vs. normal operation (multi-class;
  fault-vs-normal precision/recall is the headline metric).
- **Architecture:** compact **1D-CNN** over single-cycle/multi-cycle voltage &
  current windows (2–3 conv blocks + global pooling + small dense head).
- **Budget target:** on the order of **10⁴–10⁵ parameters**, trainable on a CPU
  edge node in seconds/round. The weave **reports** parameter count, model size
  (KB), peak training RAM, and per-round train time so the "how compact can it be"
  question is answered with measured numbers, not claims.
- Defined once in `fl/model.py` and shared by both topologies so results are
  comparable across star vs mesh.

---

## 3.1 Compute profiles — CPU-edge & GPU variants

The same weave runs in two **compute profiles**, selected by `USE_GPU` (+
`GPU_MODEL`). This is one weave with a toggle, not two artifacts — the FL logic,
model, data, and topology code are identical; only the per-node hardware and the
PyTorch build differ. (If you later want two separately-publishable Marketplace
artifacts, it's a trivial fork: same code, different default args.)

### GPU-accelerated profile (default — `USE_GPU=true`)
- **Why:** (a) **faster experiment turnaround** for the large topology,
  perturbation, and churn sweeps
  (many topology × perturbation × churn runs); (b) headroom to study the
  **accuracy-vs-footprint upper end** — e.g., a heavier 2D CNN over
  spectrogram/scalogram transforms of the waveforms, or deeper temporal models —
  so "how compact can it be" is bracketed from both ends; (c) shrinks local
  compute time so **network effects dominate wall-clock**, giving cleaner
  network-vs-compute attribution.
- Clients: `8c / 32 GB / 100 GB` + a GPU component whose model is **discovered
  per site** (`GPU_RTX6000` / `GPU_A40` / `GPU_A30` / `GPU_TeslaT4`). `GPU_MODEL=auto`
  (default) searches all models; pin a value to restrict to one.
- Aggregator stays CPU unless `AGG_GPU=true` (aggregation is light; GPU on `agg`
  is only useful for future server-side secure-aggregation crypto or large
  server-side eval).
- **Install:** `tools/setup-common.sh` auto-detects the GPU (`lspci | grep -qi
  nvidia`), installs the matching NVIDIA driver + CUDA runtime and the CUDA
  PyTorch wheels; falls back to CPU wheels if no GPU is present, so a mis-scheduled
  node still runs. `experiment.py` selects `cuda` vs `cpu` device automatically.
- **FABRIC caveat (handled by host-level discovery):** GPUs are scarce and
  host-specific. On a GPU run the weave calls FABlib `get_resources()` then
  `list_hosts()` and, over **all** GPU models, finds the individual **`Active`
  hosts** that have a free GPU with room for the client VM on that same host,
  **pins** each client to such a host (`add_node(host=...)`), and requests that
  host's model. If fewer qualifying hosts exist than `NUM_CLIENTS`, it packs onto
  hosts with spare capacity and logs the shortfall; reduce `NUM_CLIENTS`, pin a
  `GPU_MODEL`, or set `SITE_MAP`/`EXCLUDE_SITES` if needed.

### CPU-edge profile (option — `USE_GPU=false`)
- **Why:** models the on-device / edge-controller reality of the target setting
  ("trains directly on smart meters and edge controllers with limited disk
  and RAM"). The compact 1D-CNN trains on CPU in seconds/round.
- Clients: `4c / 16 GB / 50 GB`, CPU PyTorch wheels. No GPU components → schedules
  on any FABRIC site, so the ISO/RTO site mapping is unconstrained.
- This is the scientifically-motivated configuration for the edge setting; use
  it when studying the compact-model-on-constrained-hardware end of the question.

### Preset arg-sets (the two "variants")
| Preset | Key args |
|--------|----------|
| **Accelerated (GPU)** | `USE_GPU=true GPU_MODEL=auto` (client 8c/32 GB + discovered GPU) — default |
| **Edge (CPU)** | `USE_GPU=false` (client 4c/16 GB) |

---

## 4. Data — open datasets (default), with the EPRI/EIA path

EPRI's waveform library is access-gated, so the weave's **practical default is a
tier of open, verified datasets**; EPRI/EIA stays available as an access-gated
option. All are exposed through one loader interface in
`fl/data.py` — `load_partition(site_iso, dataset) -> (train, test)` — so swapping
datasets never touches the FL or topology code. Each of the `NUM_CLIENTS` clients
gets a **non-IID** partition.

Selected by the `DATASET` arg:

| `DATASET` | Source (license / access) | Type | Role | Non-IID split |
|-----------|---------------------------|------|------|---------------|
| `pq_synth` *(default, **built v0**)* | Machlev/Chachkes IEEE-1159 generator, GitHub (open) | synthetic PQ waveforms, 8 IEEE-1159 classes, 2 channels | **turnkey** — generates on-node, no download/account; full run in minutes | per-site class-mix + SNR priors |
| `psml` | PSML — Zenodo DOI `10.5281/zenodo.5663995` (CC BY 4.0) | simulated PMU-scale waveforms w/ **event type + location** labels | **headline open** — multi-node, purpose-built, ships baselines | by node / region / bus (built in) |
| `vsb` *(**built v0**)* | VSB Power Line Fault Detection — Kaggle (competition rules; account) | **real** partial-discharge on MV covered conductors; binary PD-vs-normal, single channel, each 800k-sample line peak-downsampled | **flagship incipient-fault** result — PD is the canonical insulation-degradation precursor | disjoint measurement lines per client (`id_measurement % num_clients`) |
| `feeder_sim` | IEEE 13/34/123-bus + OpenDSS/GridLAB-D (open / BSD) | synthesized fault waveforms (SLG/LL/LLG/3φ, incipient/high-Z) | **scalable non-IID workhorse** — each feeder = a client with distinct topology | by feeder (intrinsic) |
| `kaggle_faults` | Kaggle "Electrical Fault Detection & Classification" (account) | tiny simulated tabular LG/LL/LLG/LLL | **smoke test** — trains in seconds | by fault type / generator |
| `epri_eia` | EPRI PQ library + EIA OE-417 (access-gated) | real PQ waveforms + regional event reports | **access-gated option** — stage under `data/epri/`, `data/eia_oe417/` or via `DATA_URL` | OE-417 regional priors |
| `cifar10` | torchvision (open) | image benchmark | **network-only baseline** — isolates network-aware convergence and resilience from the grid domain | Dirichlet-α |

**OE-417 as a non-IID shaper (applies to any dataset):** EIA/DOE OE-417 disturbance
summaries are **public US-gov data** (event type, region, cause, customers
affected) — no waveforms. With `OE417_PRIORS=true`, per-ISO label distributions are
weighted by each region's OE-417 event profile, giving *deliberately* non-IID,
regionally realistic partitions on top of whatever `DATASET` supplies.

**Turnkey testing:** the default `pq_synth` runs immediately (no download, no
account). `vsb` is also built: set `DATASET=vsb` and provide a modern Kaggle
`KGAT_` access token (via `$KAGGLE_API_TOKEN` or `~/.kaggle/access_token`; the
notebook's §1b cell pushes it to the nodes). Download uses `kagglehub`, which caches
under `~/.cache/kagglehub` on first use (a pre-staged `$FL_DATA_DIR/vsb` copy
short-circuits it for offline runs). For the
still-roadmap `psml`/`kaggle_faults`/`epri_eia`, stage data under `data/<name>/` or
set `DATA_URL`; unknown dataset names fall back to `pq_synth` with a warning so a
run never silently fails.

**Recommended:** `psml` (open headline) + `vsb` (real
incipient-fault flagship), with `feeder_sim` for scalable per-feeder non-IID and
`OE417_PRIORS=true` for regional realism. `epri_eia` remains the faithful target
once EPRI access is in hand. See §4.1 for sources/links and `README.md` for
per-dataset acquisition + licensing pointers.

### 4.1 Dataset sources (verified July 2026)
- **PSML** — https://github.com/tamu-engineering-research/Open-source-power-dataset · Zenodo `10.5281/zenodo.5663995` · paper https://arxiv.org/abs/2110.06324 (CC BY 4.0; confirm NC note on mirror)
- **VSB Power Line Fault Detection** — https://www.kaggle.com/c/vsb-power-line-fault-detection/data (real partial discharge; Kaggle account + competition rules)
- **IEEE PES Test Feeders** — https://cmte.ieee.org/pes-testfeeders/resources/ · OpenDSS https://sourceforge.net/projects/electricdss/ · GridLAB-D https://github.com/gridlab-d/gridlab-d (BSD)
- **IEEE-1159 PQ generator (Machlev/Chachkes)** — https://github.com/chachkes247/Power-Quality-Disturbances (open; replaces the access-gated IEEE DataPort PQ sets)
- **Kaggle Electrical Fault Detection & Classification** — https://www.kaggle.com/datasets/esathyaprakash/electrical-fault-detection-and-classification
- **DOE/EIA OE-417 disturbance summaries** — https://www.oe.netl.doe.gov/oe417_annual_summary.aspx (public US-gov; metadata only, no waveforms)
- **SoCal 28-Bus real synchrowaveforms** (emerging, 2026) — https://arxiv.org/abs/2504.06588 (real MV waveforms; verify dataset access)
- *Avoid as "open":* IEEE DataPort synthetic-PQ sets are password/subscription-gated — use the GitHub generator instead. UCI "Electrical Grid Stability" is grid *stability*, not fault/PQ — off-topic.

---

## 5. What a run does (lifecycle)

The weave is split into **two phases** so the slice is provisioned once and the
science is iterated many times interactively:

You can drive **Phase A** either from the CLI (`weave.sh`, the LoomAI "Run"
button) or interactively from **`provision_slice.ipynb`** — the notebook calls the
same `experiment.py` functions (`make_config` → `preview_placement` → `start`) and
adds a placement **preview** so you can see the discovered host/GPU plan before
committing resources.

### Phase A — bring up the slice (the weave; `experiment.py` via `weave.sh`)
`experiment.py` runs on the LoomAI host over FABlib + SSH, with `start` / `stop` /
`status`. **`start` brings the slice up and then exits — it does not run FL and
does not poll in a loop:**

1. **Provision** — resolve ISO→site placement, create the slice (nodes + `fabnet`,
   GPU components by default), submit, `wait_ssh`.
2. **Configure** — run `tools/setup-common.sh` + role script on every node:
   install Python, PyTorch (GPU or CPU wheels, auto-detected), `flwr`; upload the
   `fl/` package.
3. **Publish topology** — discover each node's `fabnet` IP and write
   `results/slice_info.json` (node names, roles, fabnet IPs, ISO map, device
   profile). This is the hand-off contract the notebook reads.
4. **Report `READY`** and exit. The slice stays up for the notebook to drive.
   `status` is a **one-shot** health check (state + SSH reachability); there is no
   long-running monitor. `stop` tears the slice down (unless `KEEP_SLICE=true`).

### Phase B — run the federated learning (the notebook)
`federated_learning.ipynb` is where the experiment actually happens. The user
opens it **after** the slice is `READY` and drives one or many FL runs against the
same slice — changing `TOPOLOGY`, perturbation, churn, rounds, etc. between runs
without re-provisioning. The notebook is **self-contained plain fablib** (no
external run library) and, per run:

1. **Connect** — `fablib.get_slice(SLICE_NAME)`, load `results/slice_info.json`.
2. **Configure the run** — pick `TOPOLOGY`, `ROUNDS`, `LOCAL_EPOCHS`, `LR`,
   `AGGREGATION`, `SYNC`, perturbation (`LINK_*`), churn (`CHURN_*`).
3. **Run FL** for `ROUNDS` rounds (launched over SSH):
   - *star:* start the Flower server on `agg`, a Flower client on each `client-*`;
     server runs `AGGREGATION` (fedavg|fedprox).
   - *mesh:* each peer runs `LOCAL_EPOCHS`, exchanges weights with neighbors
     **directly over `fabnet`** (small per-peer HTTP weight server), peer-averages
     (`SYNC=sync|async`); the coordinator only starts/stops peers and collects.
   - if `CHURN_CLIENTS>0`, drop that many clients at round `CHURN_ROUND`.
4. **Collect** per-round metrics from every node into `results/metrics.json`.
5. **Analyze inline** — regenerate the figures (§7) and print a summary (final
   fault precision/recall, rounds-to-target-accuracy, total comm bytes).

Orchestration patterns (per the weave-architect guidance): aggregator/peer
addresses come from `results/slice_info.json` (discovered from the `fabnet`
subnet); clients use background retry loops until the server/peers are reachable;
perturbation and churn are applied over SSH so a run is reproducible from the
notebook config alone.

---

## 6. Run arguments

Because the weave only **brings up the slice** and the notebook **runs the FL**,
the arguments are split by phase.

### 6.1 Provisioning args — Phase A (`weave.json` `args`)
These shape the physical slice. Set them when you launch the weave.

| Arg | Type | Default | Purpose |
|-----|------|---------|---------|
| `SLICE_NAME` | string | `fl-grid` | FABRIC slice name |
| `NUM_CLIENTS` | int | `4` | client sites (4 → +SPP,NYISO at 6) |
| `SITE_MAP` | string | *(ISO→site table above)* | override ISO→FABRIC-site placement |
| `USE_GPU` | bool | `true` | **default** GPU profile: add GPU + CUDA PyTorch to clients (§3.1). `false` = CPU-edge |
| `GPU_MODEL` | enum | `auto` | `auto` = discover & use **all** models per site; or pin `GPU_RTX6000` \| `GPU_A40` \| `GPU_A30` \| `GPU_TeslaT4` |
| `AGG_GPU` | bool | `false` | also put a GPU on the aggregator (rarely needed) |
| `SITE_MAP` | string | `""` | optional per-ISO site override, e.g. `ercot=TACC,agg=RENC` (else hosts are discovered) |
| `EXCLUDE_SITES` | string | `""` | comma-separated sites to skip during discovery, e.g. `EDUKY,EDC` |
| `KEEP_SLICE` | bool | `true` | keep the slice up after `start` (needed so the notebook can drive it; the weave never auto-deletes) |

### 6.2 FL run args — Phase B (notebook variables)
These are set in `federated_learning.ipynb` and can change between runs on the
**same** slice — no re-provisioning.

| Arg | Type | Default | Purpose |
|-----|------|---------|---------|
| `TOPOLOGY` | enum | `star` | `star` \| `mesh` (roadmap: `ring`, `hybrid`) |
| `DATASET` | enum | `pq_synth` | `pq_synth` \| `vsb` (both v0). Roadmap: `psml` \| `feeder_sim` \| `kaggle_faults` \| `epri_eia` \| `cifar10` (§4) |
| `ROUNDS` | int | `20` | FL rounds |
| `LOCAL_EPOCHS` | int | `1` | local epochs per round |
| `BATCH_SIZE` | int | `64` | local batch size |
| `LR` | number | `0.01` | learning rate |
| `AGGREGATION` | enum | `fedavg` | star: `fedavg` \| `fedprox`; mesh: `peer_avg` \| `gossip` |
| `SYNC` | enum | `sync` | mesh comms: `sync` \| `async` |
| `LINK_DELAY_MS` | int | `0` | injected one-way latency (tc netem) |
| `LINK_LOSS_PCT` | number | `0` | injected packet loss % |
| `LINK_RATE_MBIT` | int | `0` | egress rate cap (0 = unlimited) |
| `CHURN_CLIENTS` | int | `0` | clients dropped mid-training |
| `CHURN_ROUND` | int | `0` | round at which churn occurs |

*(`vsb` reads a Kaggle `KGAT_` access token from `$KAGGLE_API_TOKEN` or
`~/.kaggle/access_token` (legacy `kaggle.json` still works), caches via `kagglehub`
under `~/.cache/kagglehub`, and honors a pre-staged `$FL_DATA_DIR/vsb`. Other roadmap
run args — `DATA_URL`, `OE417_PRIORS`, `ALLOW_SYNTHETIC_FALLBACK` — land with the
remaining §4 datasets; `pq_synth` needs none of them.)*

---

## 7. Deliverables (shareable results)

The **`federated_learning.ipynb`** notebook is the primary interactive deliverable
(launch FL → collect → plot). Each run writes to `results/`:

- **`slice_info.json`** — the Phase-A → Phase-B hand-off (node names, roles,
  fabnet IPs, ISO map, device profile). Written by the weave, read by the notebook.
- **`convergence.png`** — accuracy / fault-detection F1 vs. round, one curve per
  topology & network condition (the network-aware convergence figure).
- **`topology.png`** — the deployed slice: sites on a US map with ISO labels + the
  active communication graph (star vs mesh).
- **`churn_resilience.png`** *(when churn is set)* — accuracy vs. round with the
  churn event marked; quantifies graceful degradation.
- **`comm_efficiency.png`** — cumulative communication bytes vs. accuracy per
  topology (motivates in-network aggregation).
- **`metrics.json`** — all raw per-round metrics (accuracy, precision/recall, loss,
  comm bytes, wall-clock, per-client staleness) + the model footprint numbers.
- **`README.md`** — detailed description of what it deploys, how to run each
  capability scenario, and how figures map to the system capabilities.

*(v0 emits `convergence.png`, `topology.png`, `comm_efficiency.png`, and
`metrics.json`; `churn_resilience.png` appears when a churn run is executed. No
`.docx` is generated — the README + figures are ready for reuse.)*

---

## 8. File layout

```
federated-learning/                 # weave root (this spec lives here)
├── weave.md                 # this spec (authoritative)
├── weave.json               # metadata + PROVISIONING args (Phase A)
├── weave.sh                 # thin start/stop/status orchestrator (bring-up only)
├── weave_cleanup.sh         # teardown: delete slice + clear results/ (LoomAI cleanup script)
├── experiment.py            # Phase A: FABlib provisioning → slice_info.json (no FL loop)
├── provision_slice.ipynb    # Phase A notebook: interactive bring-up (alt to weave.sh)
├── federated_learning.ipynb # Phase B notebook: launch star/mesh over SSH, collect, plot (plain fablib)
├── README.md                # bring up slice → open notebook → run each capability scenario
├── requirements.txt         # node/runtime Python deps (uploaded; torch installed separately)
├── fl/                      # FL logic (uploaded to nodes, runs ON nodes)
│   ├── __init__.py
│   ├── model.py             # compact 1D-CNN (shared)
│   ├── data.py              # pq_synth generator + vsb (Kaggle) loader; one non-IID loader interface
│   ├── star_server.py       # Flower server (FedAvg/FedProx)
│   ├── star_client.py       # Flower client
│   ├── mesh_peer.py         # decentralized gossip / peer-averaging (HTTP over fabnet)
│   ├── perturb.py           # tc netem apply/clear
│   └── metrics.py           # per-round metric capture → metrics.json
├── tools/                   # per-node setup (uploaded, run during Phase A)
│   ├── setup-lib.sh         # shared apt-lock helpers (sourced)
│   ├── setup-torch.sh       # *** edit here to change GPU/CUDA/driver/torch ***
│   ├── setup-common.sh      # venv + setup-torch.sh + pip install -r requirements.txt
│   ├── setup-agg.sh         # aggregator/coordinator (forwards args to common)
│   └── setup-client.sh      # client/peer (forwards args to common)
├── results/                 # (git-ignored) slice_info.json, metrics.json, *.png
└── .weaveignore             # exclude data/, results/, __pycache__
```

*(`vsb` caches its `kagglehub` download under `~/.cache/kagglehub` on each node at
run time — not checked in. A staged `data/` tree for the remaining
`psml`/`epri_eia`/`feeders` loaders lands with §4; `pq_synth` generates on-node so it
needs no `data/`.)*

---

## 9. Suggested experiment sweeps

1. **Topology baseline:** `TOPOLOGY=star` vs `mesh`, no perturbation →
   convergence + final F1. "Does decentralized match/exceed centralized?"
2. **Network-aware convergence:** fix `mesh`, sweep
   `LINK_DELAY_MS ∈ {0,25,100}`, `LINK_LOSS_PCT ∈ {0,1,5}`, `SYNC=sync` vs `async`.
3. **Resilience:** `CHURN_CLIENTS=1..2` at `CHURN_ROUND=ROUNDS/2`, star vs
   mesh → bounded degradation.
4. **Communication efficiency:** compare cumulative comm bytes to
   reach target F1 across topologies.
5. **Model minimality:** shrink the 1D-CNN, record F1 vs params/RAM.

---

## 10. Scope & roadmap

**v0 (this weave — the base example):**
- Two-phase design: the **weave brings up the slice** (`experiment.py start`,
  no FL loop / no polling), a **notebook runs the FL** (`federated_learning.ipynb`).
- **star + mesh** over `FABNetv4`; Flower star + custom HTTP peer-averaging mesh.
- **`pq_synth`** turnkey dataset (generated on-node, non-IID per site) **and
  `vsb`** — the real Kaggle partial-discharge dataset, downloaded + cached on-node,
  partitioned into disjoint measurement lines per client.
- **GPU by default**, CPU-only via `USE_GPU=false`.
- `tc netem` perturbation; optional mid-training churn.
- `convergence.png` / `topology.png` / `comm_efficiency.png` / `metrics.json`
  (`churn_resilience.png` on churn runs).

**Roadmap (documented hooks, not built in v0):**
- **More datasets** — `psml`, `feeder_sim`, `kaggle_faults`, `epri_eia`,
  `cifar10` behind the same `fl/data.py` `load_partition()` interface; `DATA_URL`,
  `OE417_PRIORS`, `ALLOW_SYNTHETIC_FALLBACK`. (`pq_synth` + `vsb` are built — see v0.)
- **In-network aggregation** — add a P4/BMv2 node on `fabnet` and move gradient
  summation into the data plane; `mesh_peer.py` already isolates the aggregation
  step behind an interface so an in-network aggregator can replace peer-averaging.
- **`ring` / `hybrid` topologies** — supply a neighbor adjacency to the mesh layer.
- **`L2STS` point-to-point links** for controlled per-link L2 perturbation.
- Async staleness-weighted aggregation and straggler-aware scheduling.
```
