# Benchmarking Pipelines A / B / C

`scripts/benchmark_pipelines.py` compares the three segmentation pipelines on
**one GPU** with a unified protocol:

| pipeline | backbone | data protocol |
|---|---|---|
| A | in-house `Unet3D(base=16, depth=4)` | processed 1 mm head-cropped percentile-normalized volumes (`pimg`/`plabel`) |
| B | nnU-Net v2 `3d_fullres` | native source volumes (`img`/`label`), per Pipeline B's design |
| C | MONAI `UNet(16→128)` | processed 1 mm volumes (`pimg`/`plabel`) |

Each pipeline is benchmarked with **its own standard settings** (batch, patch,
augmentation, TTA, optimizer), so the results reflect the pipelines as used in
the study.

## Metrics

- **DICE** — tumor probability thresholded at 0.5 vs ground truth on the
  held-out patient-level test split (`split_test.jsonl`), the primary-metric
  convention of all three pipelines. B additionally reports the **hard-seg**
  (argmax) Dice it exports natively.
- **Training** — wall-clock total and s/epoch for `--train-epochs` on the
  first `--max-rows` train rows (single GPU). For B, the one-time raw-dataset
  build and `plan_and_preprocess` wall time are reported separately
  (`raw_build_s`, `plan_preprocess_s`).
- **Inference** — mean **s/volume** on `--max-eval-rows` held-out test volumes
  (first `--eval-warmup` cases untimed), each pipeline's standard
  sliding-window + TTA protocol (`--no-tta` disables TTA everywhere).
- **Model size** — parameter count, checkpoint weight file size (MB), and peak
  **VRAM** (MB) for the training run and for the inference run.
  (A/C training VRAM via `torch.cuda.max_memory_allocated`; B via nvidia-smi
  polling around the `nnUNetv2_train` subprocess.)

## Prerequisites

- The `longitudinal` conda env and the GPU(s) it sees:
  ```bash
  source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
  ```
- Data for Pipelines A and C (and split membership for all three):
  `data/proc/*` + `data/manifests/split_{train,val,test}.jsonl`, i.e.
  `scripts/preprocess.py` and `scripts/05_build_splits.py` have been run.
- Data for Pipeline B (train/eval mode): the labeled source manifests
  `data/manifests/{ntuh,m6_labeled}.jsonl` with reachable native NIfTIs
  (the benchmark joins native `img`/`label` paths by case key).
- `--skip-train` mode additionally needs the study checkpoints:
  `runs/round0/best.pt` (A), `runs_monai/round0/best.pt` (C), and the study
  nnU-Net results tree `nnu/results/Dataset210_NTUH_T1C_PL/...` (B).

## Quick start

```bash
cd /mnt/b4/xfr/git26/longitudinal
conda activate longitudinal

# small/fast check (what the validation run used): ~minutes per pipeline
python scripts/benchmark_pipelines.py \
    --train-epochs 1 --max-rows 24 --nnu-cases 32 --max-eval-rows 4

# default full benchmark: train 2 epochs on 120 rows, eval on 12 test volumes
python scripts/benchmark_pipelines.py
```

Outputs (repo root):

```
results/benchmark_pipelines.json        # settings + all metrics, per pipeline
results/benchmark_pipelines_per_row.jsonl
results/benchmark_pipelines_bench.png   # 6-panel bar chart
```

plus the artifacts it created (safe to delete): `runs_bench/{A,C}/best.pt`,
`nnu/{raw,preprocessed,results}/Dataset221_T1C_BENCH`,
`logs/nnu/bench_{plan,train}.log`.

## All options

| flag | default | meaning |
|---|---|---|
| `--pipelines` | `A B C` | space-separated subset, e.g. `"A C"` |
| `--gpu` | `0` | physical GPU index; all pipelines run serially on this one GPU |
| `--train-epochs` | `2` | training budget for A and C (and B via `NNU_PL_EPOCHS`) |
| `--max-rows` | `120` | train rows for A/C (first N of `split_train.jsonl` that have labels) |
| `--nnu-cases` | `120` | train cases for B (first N labeled native cases of the same split) |
| `--max-eval-rows` | `12` | held-out test volumes for inference + DICE |
| `--eval-warmup` | `1` | untimed warmup cases before inference timing |
| `--workers` | `4` | data-loader workers (A/C) |
| `--batch` | `3` | per-GPU batch (A/C) |
| `--patch` | `96` | patch/sliding-window size (A/C) |
| `--no-tta` | off | disable each pipeline's standard test-time augmentation |
| `--skip-train` | off | evaluate existing study checkpoints instead of training |
| `--nnu-dsid` | `221` train / `210` skip-train | nnU-Net dataset id: in training mode an isolated `Dataset{id}_T1C_BENCH` (the study's 210 is never touched); in `--skip-train` mode the study's 210 by default, or a previously benchmarked id to re-evaluate it |
| `--nnu-skip-prep` | off | B (train mode): skip `plan_and_preprocess` and reuse the existing raw + preprocessed tree for `--nnu-dsid` (e.g. retime one more training epoch on an already prepared dataset) |
| `--out` | `results/benchmark_pipelines.json` | output JSON path (the `_per_row.jsonl` / `_bench.png` are derived from it) |

## Examples

```bash
# longer training budget, bigger eval set (quality-focused run)
python scripts/benchmark_pipelines.py --train-epochs 5 --max-rows 240 --max-eval-rows 20

# timing only, no TTA
python scripts/benchmark_pipelines.py --no-tta --max-eval-rows 20

# re-evaluate the study's trained checkpoints (no training)
python scripts/benchmark_pipelines.py --skip-train --max-eval-rows 24

# benchmark B only; keep the trained Dataset221 model for a second inference-only pass
python scripts/benchmark_pipelines.py --pipelines B
python scripts/benchmark_pipelines.py --pipelines B --skip-train --nnu-dsid 221

# use GPU 1 and write to a custom location
python scripts/benchmark_pipelines.py --gpu 1 --out results/bench_gpu1.json
```

## Interpreting the results

- **"1 epoch" is pipeline-native.** A/C: one pass over the N train volumes
  (batch 3, 96³ random patches, AdamW 3e-4 + warmup/cosine, bf16).
  B: one pass over the N train cases at nnU-Net's planned batch and patch size
  (typically 2 and ~256³, AdamW 1e-2 PolyLR, fp16). Epoch times are therefore
  comparable as *wall-clock cost per pass over the training set in each
  pipeline's standard configuration*, not per identical number of patches.
- **DICE reflects the training budget.** Short benchmarks measure the
  machinery, not the final quality of the pipelines. Compare DICE across
  pipelines only at the same `--train-epochs`/`--max-rows` budget, or use
  `--skip-train` on fully trained study checkpoints.
- **Inference times are single-GPU, standard-protocol numbers**
  (A: 96³ step 48 + 4-flip TTA; B: nnU-Net gaussian sliding window +
  mirroring, native volumes; C: MONAI gaussian 50% + 4-flip TTA).
  B is the slowest per volume on this data set because it runs on uncropped
  native volumes with full mirroring TTA.
- **B's one-time cost.** `raw_build_s` + `plan_preprocess_s` are paid once per
  (dataset, round) — in the iterative study every round repays the
  plan/preprocess on the growing dataset; the benchmark reports it for
  context only.
- **VRAM notes.** Training VRAM includes optimizer states (AdamW) and the
  largest batch; B's number is the nvidia-smi peak of the whole
  `nnUNetv2_train` process tree on that GPU. Inference VRAM is per
  pipeline as measured in-process (B: torch peak over the timed cases).

## Troubleshooting

| symptom | fix |
|---|---|
| `split_train.jsonl has no labeled processed rows` | run `scripts/preprocess.py` over the manifests and `scripts/05_build_splits.py` first |
| `B: key ... not in ntuh/m6_labeled manifests` | build the source manifests (`scripts/01_build_ntuh_manifest.py`, `scripts/02_build_m6_dataset.py`) and confirm the native NIfTIs are reachable |
| `--skip-train: .../best.pt not found` | the study checkpoint for that pipeline doesn't exist yet — train a round first, or drop `--skip-train` |
| B `--skip-train` fails on the model folder | the study's `nnu/results/Dataset210_NTUH_T1C_PL/NTUHLPLTrainer__nnUNetPlans__3d_fullres/fold_0/` must exist (produced by `scripts_nnu/03_nnu_train.py`) |
| OOM on B | B trains at the planned 256³-class patches and needs ~10–20 GB; free other GPU users or run with a smaller `--nnu-cases` set (patch size derives from the median case size) |
| inflated/inconsistent timings | do not run two benchmarks on the same `--gpu` concurrently; note that data-loader (NFS) latency is included in training and inference wall times |