Furen Xiao
8fe1818c7d
- Updated regex patterns for improved matching of series names and tags. - Added functionality to reject non-head series based on study and series descriptions. - Implemented max voxel spacing check to filter out series with excessive spacing. - Enhanced the reconstruction script to handle dynamic-frame series exclusions and artifact pruning. - Modified output paths for reconstructed NIfTI files and added QA screenshot generation. - Improved argument parsing in benchmark and pseudo-labeling scripts for better flexibility. - Introduced a new script for generating QA screenshots from reconstructed volumes.
152 lines
No EOL
8.2 KiB
Markdown
152 lines
No EOL
8.2 KiB
Markdown
# Benchmarking Pipelines A / B / C
|
||
|
||
`scripts/benchmark_pipelines.py` compares the three segmentation pipelines on
|
||
**one GPU** with a unified protocol:
|
||
|
||
| pipeline | backbone | data protocol |
|
||
|---|---|---|
|
||
| A | in-house `Unet3D(base=16, depth=4)` | processed 1 mm head-cropped percentile-normalized volumes (`pimg`/`plabel`) |
|
||
| B | nnU-Net v2 `3d_fullres` | native source volumes (`img`/`label`), per Pipeline B's design |
|
||
| C | MONAI `UNet(16→128)` | processed 1 mm volumes (`pimg`/`plabel`) |
|
||
|
||
Each pipeline is benchmarked with **its own standard settings** (batch, patch,
|
||
augmentation, TTA, optimizer), so the results reflect the pipelines as used in
|
||
the study.
|
||
|
||
## Metrics
|
||
|
||
- **DICE** — tumor probability thresholded at 0.5 vs ground truth on the
|
||
held-out patient-level test split (`split_test.jsonl`), the primary-metric
|
||
convention of all three pipelines. B additionally reports the **hard-seg**
|
||
(argmax) Dice it exports natively.
|
||
- **Training** — wall-clock total and s/epoch for `--train-epochs` on the
|
||
first `--max-rows` train rows (single GPU). For B, the one-time raw-dataset
|
||
build and `plan_and_preprocess` wall time are reported separately
|
||
(`raw_build_s`, `plan_preprocess_s`).
|
||
- **Inference** — mean **s/volume** on `--max-eval-rows` held-out test volumes
|
||
(first `--eval-warmup` cases untimed), each pipeline's standard
|
||
sliding-window + TTA protocol (`--no-tta` disables TTA everywhere).
|
||
- **Model size** — parameter count, checkpoint weight file size (MB), and peak
|
||
**VRAM** (MB) for the training run and for the inference run.
|
||
(A/C training VRAM via `torch.cuda.max_memory_allocated`; B via nvidia-smi
|
||
polling around the `nnUNetv2_train` subprocess.)
|
||
|
||
## Prerequisites
|
||
|
||
- The `longitudinal` conda env and the GPU(s) it sees:
|
||
```bash
|
||
source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
|
||
```
|
||
- Data for Pipelines A and C (and split membership for all three):
|
||
`data/proc/*` + `data/manifests/split_{train,val,test}.jsonl`, i.e.
|
||
`scripts/preprocess.py` and `scripts/05_build_splits.py` have been run.
|
||
- Data for Pipeline B (train/eval mode): the labeled source manifests
|
||
`data/manifests/{ntuh,m6_labeled}.jsonl` with reachable native NIfTIs
|
||
(the benchmark joins native `img`/`label` paths by case key).
|
||
- `--skip-train` mode additionally needs the study checkpoints:
|
||
`runs/round0/best.pt` (A), `runs_monai/round0/best.pt` (C), and the study
|
||
nnU-Net results tree `nnu/results/Dataset210_NTUH_T1C_PL/...` (B).
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
cd /mnt/b4/xfr/git26/longitudinal
|
||
conda activate longitudinal
|
||
|
||
# small/fast check (what the validation run used): ~minutes per pipeline
|
||
python scripts/benchmark_pipelines.py \
|
||
--train-epochs 1 --max-rows 24 --nnu-cases 32 --max-eval-rows 4
|
||
|
||
# default full benchmark: train 100 epochs on 80% of train cases (~1986 rows),
|
||
# eval on 10% of test volumes (~30)
|
||
python scripts/benchmark_pipelines.py
|
||
```
|
||
|
||
Outputs (repo root):
|
||
|
||
```
|
||
results/benchmark_pipelines.json # settings + all metrics, per pipeline
|
||
results/benchmark_pipelines_per_row.jsonl
|
||
results/benchmark_pipelines_bench.png # 6-panel bar chart
|
||
```
|
||
|
||
plus the artifacts it created (safe to delete): `runs_bench/{A,C}/best.pt`,
|
||
`nnu/{raw,preprocessed,results}/Dataset221_T1C_BENCH`,
|
||
`logs/nnu/bench_{plan,train}.log`.
|
||
|
||
## All options
|
||
|
||
| flag | default | meaning |
|
||
|---|---|---|
|
||
| `--pipelines` | `A B C` | space-separated subset, e.g. `"A C"` |
|
||
| `--gpu` | `0` | physical GPU index; all pipelines run serially on this one GPU |
|
||
| `--train-epochs` | `100` | training budget for A and C (and B via `NNU_PL_EPOCHS`) |
|
||
| `--max-rows` | `80%` of `split_train` | train rows for A/C (first N of `split_train.jsonl` that have labels; default 80% of the split, ~1986 rows) |
|
||
| `--nnu-cases` | `80%` of `split_train` | train cases for B (first N labeled native cases of the same split; default same as `--max-rows`) |
|
||
| `--max-eval-rows` | `10%` of `split_test` | held-out test volumes for inference + DICE (default 10% of the split, ~30 rows) |
|
||
| `--eval-warmup` | `1` | untimed warmup cases before inference timing |
|
||
| `--workers` | `4` | data-loader workers (A/C) |
|
||
| `--batch` | `3` | per-GPU batch (A/C) |
|
||
| `--patch` | `96` | patch/sliding-window size (A/C) |
|
||
| `--no-tta` | off | disable each pipeline's standard test-time augmentation |
|
||
| `--skip-train` | off | evaluate existing study checkpoints instead of training |
|
||
| `--nnu-dsid` | `221` train / `210` skip-train | nnU-Net dataset id: in training mode an isolated `Dataset{id}_T1C_BENCH` (the study's 210 is never touched); in `--skip-train` mode the study's 210 by default, or a previously benchmarked id to re-evaluate it |
|
||
| `--nnu-skip-prep` | off | B (train mode): skip `plan_and_preprocess` and reuse the existing raw + preprocessed tree for `--nnu-dsid` (e.g. retime one more training epoch on an already prepared dataset) |
|
||
| `--out` | `results/benchmark_pipelines.json` | output JSON path (the `_per_row.jsonl` / `_bench.png` are derived from it) |
|
||
|
||
## Examples
|
||
|
||
```bash
|
||
# longer training budget, bigger eval set (quality-focused run)
|
||
python scripts/benchmark_pipelines.py --train-epochs 5 --max-rows 240 --max-eval-rows 20
|
||
|
||
# timing only, no TTA
|
||
python scripts/benchmark_pipelines.py --no-tta --max-eval-rows 20
|
||
|
||
# re-evaluate the study's trained checkpoints (no training)
|
||
python scripts/benchmark_pipelines.py --skip-train --max-eval-rows 24
|
||
|
||
# benchmark B only; keep the trained Dataset221 model for a second inference-only pass
|
||
python scripts/benchmark_pipelines.py --pipelines B
|
||
python scripts/benchmark_pipelines.py --pipelines B --skip-train --nnu-dsid 221
|
||
|
||
# use GPU 1 and write to a custom location
|
||
python scripts/benchmark_pipelines.py --gpu 1 --out results/bench_gpu1.json
|
||
```
|
||
|
||
## Interpreting the results
|
||
|
||
- **"1 epoch" is pipeline-native.** A/C: one pass over the N train volumes
|
||
(batch 3, 96³ random patches, AdamW 3e-4 + warmup/cosine, bf16).
|
||
B: one pass over the N train cases at nnU-Net's planned batch and patch size
|
||
(typically 2 and ~256³, AdamW 1e-2 PolyLR, fp16). Epoch times are therefore
|
||
comparable as *wall-clock cost per pass over the training set in each
|
||
pipeline's standard configuration*, not per identical number of patches.
|
||
- **DICE reflects the training budget.** Short benchmarks measure the
|
||
machinery, not the final quality of the pipelines. Compare DICE across
|
||
pipelines only at the same `--train-epochs`/`--max-rows` budget, or use
|
||
`--skip-train` on fully trained study checkpoints.
|
||
- **Inference times are single-GPU, standard-protocol numbers**
|
||
(A: 96³ step 48 + 4-flip TTA; B: nnU-Net gaussian sliding window +
|
||
mirroring, native volumes; C: MONAI gaussian 50% + 4-flip TTA).
|
||
B is the slowest per volume on this data set because it runs on uncropped
|
||
native volumes with full mirroring TTA.
|
||
- **B's one-time cost.** `raw_build_s` + `plan_preprocess_s` are paid once per
|
||
(dataset, round) — in the iterative study every round repays the
|
||
plan/preprocess on the growing dataset; the benchmark reports it for
|
||
context only.
|
||
- **VRAM notes.** Training VRAM includes optimizer states (AdamW) and the
|
||
largest batch; B's number is the nvidia-smi peak of the whole
|
||
`nnUNetv2_train` process tree on that GPU. Inference VRAM is per
|
||
pipeline as measured in-process (B: torch peak over the timed cases).
|
||
|
||
## Troubleshooting
|
||
|
||
| symptom | fix |
|
||
|---|---|
|
||
| `split_train.jsonl has no labeled processed rows` | run `scripts/preprocess.py` over the manifests and `scripts/05_build_splits.py` first |
|
||
| `B: key ... not in ntuh/m6_labeled manifests` | build the source manifests (`scripts/01_build_ntuh_manifest.py`, `scripts/02_build_m6_dataset.py`) and confirm the native NIfTIs are reachable |
|
||
| `--skip-train: .../best.pt not found` | the study checkpoint for that pipeline doesn't exist yet — train a round first, or drop `--skip-train` |
|
||
| B `--skip-train` fails on the model folder | the study's `nnu/results/Dataset210_NTUH_T1C_PL/NTUHLPLTrainer__nnUNetPlans__3d_fullres/fold_0/` must exist (produced by `scripts_nnu/03_nnu_train.py`) |
|
||
| OOM on B | B trains at the planned 256³-class patches and needs ~10–20 GB; free other GPU users or run with a smaller `--nnu-cases` set (patch size derives from the median case size) |
|
||
| inflated/inconsistent timings | do not run two benchmarks on the same `--gpu` concurrently; note that data-loader (NFS) latency is included in training and inference wall times | |