# Testing Plan: [TITLE]

> **How to use this template:** Run `/design-experiment` and Claude will fill
> this in with test procedures specific to your experiment.

---

## Smoke Test (5-10 minutes)

**Goal:** Verify the basic deployment pipeline works.

1. Run with minimal configuration:
   - Slice name: `test-smoke`
   - Duration: shortest useful value (e.g., 10 seconds)
   - Smallest parameter set

2. Verify:
   - [ ] Slice reaches StableOK state
   - [ ] SSH access works to all nodes
   - [ ] Software installation completes without errors
   - [ ] One data point is collected in correct format
   - [ ] `### PROGRESS:` markers appear in Build Log
   - [ ] Stop button triggers clean teardown (slice deleted)
   - [ ] No orphaned slices remain after teardown

### Walkthrough Questions
> 1. "What does a successful single data point look like?"
> 2. "Are there any setup steps that might fail on first try?"

---

## Integration Test (30-60 minutes)

**Goal:** Verify the complete flow with a reduced parameter set.

1. Run with reduced configuration:
   - 2-3 parameter values instead of the full set
   - 1 repetition per configuration
   - Shorter duration if applicable

2. Verify:
   - [ ] All parameter combinations produce valid output
   - [ ] Data files appear in `data/` directory with correct naming
   - [ ] JSON/CSV parses correctly (no malformed records)
   - [ ] Progress reporting updates throughout the run
   - [ ] No error accumulation in `weave.log`
   - [ ] Metadata file records correct run information
   - [ ] Analysis produces summary with expected metrics
   - [ ] Notebooks render correctly with the collected data

3. Check resource cleanup:
   - [ ] All slices deleted after Stop
   - [ ] No leftover processes on the VMs

### Walkthrough Questions
> 1. "Which parameter combinations are most likely to reveal problems?"
> 2. "Are there edge cases in your parameter space (zero, max, boundary values)?"

---

## Full Run

**Goal:** Execute the complete experiment with all parameters.

1. Deploy with full configuration:
   - All parameter values
   - All planned repetitions
   - Full measurement duration

2. Monitoring:
   - [ ] Check progress via Build Log every [N] minutes
   - [ ] Use `notebooks/progress.ipynb` for detailed status
   - [ ] Watch for error patterns in output

3. Post-completion verification:
   - [ ] All expected data files present (count matches configurations x repetitions)
   - [ ] No missing or truncated data files
   - [ ] Summary statistics computed and saved
   - [ ] Visualizations generated in `notebooks/results.ipynb`
   - [ ] No orphaned slices (check with `fabric_list_slices`)

### Walkthrough Questions
> 1. "How long do you expect the full run to take?"
> 2. "Should it run overnight, or do you need to monitor it?"

---

## Resume Test

**Goal:** Verify the experiment can be stopped and restarted.

1. Start a full run
2. Stop after approximately 25% completion
3. Restart with the same slice name and parameters
4. Verify:
   - [ ] Previously completed work is preserved (not re-run)
   - [ ] Only remaining configurations execute
   - [ ] Final data set is complete and consistent
   - [ ] No duplicate entries in results

*Note: If your experiment does not support resume, skip this test.*

### Walkthrough Questions
> 1. "Does your experiment need checkpoint/resume?"
> 2. "What state needs to be saved for resume to work?"

---

## Failure Recovery Test

**Goal:** Verify graceful handling of common failures.

### Site Unavailable
- [ ] Run with a site that is full or down
- [ ] Verify: falls back to another site or reports clear error
- [ ] No resources left allocated

### Node Failure (SSH timeout)
- [ ] Deploy successfully, then simulate SSH failure
- [ ] Verify: detects the problem in monitor, reports clear error
- [ ] Cleanup runs even on failure

### Measurement Tool Failure
- [ ] Remove or break a measurement tool on the VM
- [ ] Verify: reports which metric failed, continues with others or fails cleanly

### Token Expiry
- [ ] Run a long experiment that spans a token refresh
- [ ] Verify: reports auth error clearly, advises user to refresh token

### Walkthrough Questions
> 1. "What are the most likely failure modes for your experiment?"
> 2. "Should failures retry, skip, or halt the entire experiment?"

---

## Validation Checklist

After all tests pass, confirm:

- [ ] All `### PROGRESS:` markers display correctly in WebUI Build Log
- [ ] Stop button triggers clean teardown within 10 seconds
- [ ] Data directory structure matches specification
- [ ] All result files parse correctly (valid JSON/CSV)
- [ ] No hardcoded site names in topology or scripts
- [ ] Scripts are idempotent (safe to re-run)
- [ ] Notebooks produce correct charts from collected data
- [ ] `weave.json` has meaningful defaults for all args
- [ ] README accurately describes the experiment
- [ ] `weave.log` contains useful debugging information
