longitudinal/BENCHMARKING.md

Benchmarking Pipelines A / B / C

scripts/benchmark_pipelines.py compares the three segmentation pipelines on one GPU with a unified protocol:

pipeline	backbone	data protocol
A	in-house Unet3D(base=16, depth=4)	processed 1 mm head-cropped percentile-normalized volumes (pimg/plabel)
B	nnU-Net v2 3d_fullres	native source volumes (img/label), per Pipeline B's design
C	MONAI UNet(16→128)	processed 1 mm volumes (pimg/plabel)

Each pipeline is benchmarked with its own standard settings (batch, patch, augmentation, TTA, optimizer), so the results reflect the pipelines as used in the study.

Metrics

• DICE — tumor probability thresholded at 0.5 vs ground truth on the held-out patient-level test split (split_test.jsonl), the primary-metric convention of all three pipelines. B additionally reports the hard-seg (argmax) Dice it exports natively.
• Training — wall-clock total and s/epoch for --train-epochs on the first --max-rows train rows (single GPU). For B, the one-time raw-dataset build and plan_and_preprocess wall time are reported separately (raw_build_s, plan_preprocess_s).
• Inference — mean s/volume on --max-eval-rows held-out test volumes (first --eval-warmup cases untimed), each pipeline's standard sliding-window + TTA protocol (--no-tta disables TTA everywhere).
• Model size — parameter count, checkpoint weight file size (MB), and peak VRAM (MB) for the training run and for the inference run. (A/C training VRAM via torch.cuda.max_memory_allocated; B via nvidia-smi polling around the nnUNetv2_train subprocess.)

Prerequisites

• The longitudinal conda env and the GPU(s) it sees:

  source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
  

• Data for Pipelines A and C (and split membership for all three): data/proc/* + data/manifests/split_{train,val,test}.jsonl, i.e. scripts/preprocess.py and scripts/05_build_splits.py have been run.
• Data for Pipeline B (train/eval mode): the labeled source manifests data/manifests/{ntuh,m6_labeled}.jsonl with reachable native NIfTIs (the benchmark joins native img/label paths by case key).
• --skip-train mode additionally needs the study checkpoints: runs/round0/best.pt (A), runs_monai/round0/best.pt (C), and the study nnU-Net results tree nnu/results/Dataset210_NTUH_T1C_PL/... (B).

Quick start

cd /mnt/b4/xfr/git26/longitudinal
conda activate longitudinal

# small/fast check (what the validation run used): ~minutes per pipeline
python scripts/benchmark_pipelines.py \
    --train-epochs 1 --max-rows 24 --nnu-cases 32 --max-eval-rows 4

# default full benchmark: train 100 epochs on 80% of train cases (~1986 rows),
# eval on 10% of test volumes (~30)
python scripts/benchmark_pipelines.py

Outputs (repo root):

results/benchmark_pipelines.json        # settings + all metrics, per pipeline
results/benchmark_pipelines_per_row.jsonl
results/benchmark_pipelines_bench.png   # 6-panel bar chart

plus the artifacts it created (safe to delete): runs_bench/{A,C}/best.pt, nnu/{raw,preprocessed,results}/Dataset221_T1C_BENCH, logs/nnu/bench_{plan,train}.log.

All options

flag	default	meaning
--pipelines	A B C	space-separated subset, e.g. "A C"
--gpu	0	physical GPU index; all pipelines run serially on this one GPU
--train-epochs	100	training budget for A and C (and B via NNU_PL_EPOCHS)
--max-rows	80% of split_train	train rows for A/C (first N of split_train.jsonl that have labels; default 80% of the split, ~1986 rows)
--nnu-cases	80% of split_train	train cases for B (first N labeled native cases of the same split; default same as --max-rows)
--max-eval-rows	10% of split_test	held-out test volumes for inference + DICE (default 10% of the split, ~30 rows)
--eval-warmup	1	untimed warmup cases before inference timing
--workers	4	data-loader workers (A/C)
--batch	3	per-GPU batch (A/C)
--patch	96	patch/sliding-window size (A/C)
--no-tta	off	disable each pipeline's standard test-time augmentation
--skip-train	off	evaluate existing study checkpoints instead of training
--nnu-dsid	221 train / 210 skip-train	nnU-Net dataset id: in training mode an isolated Dataset{id}_T1C_BENCH (the study's 210 is never touched); in --skip-train mode the study's 210 by default, or a previously benchmarked id to re-evaluate it
--nnu-skip-prep	off	B (train mode): skip plan_and_preprocess and reuse the existing raw + preprocessed tree for --nnu-dsid (e.g. retime one more training epoch on an already prepared dataset)
--out	results/benchmark_pipelines.json	output JSON path (the _per_row.jsonl / _bench.png are derived from it)

Examples

# longer training budget, bigger eval set (quality-focused run)
python scripts/benchmark_pipelines.py --train-epochs 5 --max-rows 240 --max-eval-rows 20

# timing only, no TTA
python scripts/benchmark_pipelines.py --no-tta --max-eval-rows 20

# re-evaluate the study's trained checkpoints (no training)
python scripts/benchmark_pipelines.py --skip-train --max-eval-rows 24

# benchmark B only; keep the trained Dataset221 model for a second inference-only pass
python scripts/benchmark_pipelines.py --pipelines B
python scripts/benchmark_pipelines.py --pipelines B --skip-train --nnu-dsid 221

# use GPU 1 and write to a custom location
python scripts/benchmark_pipelines.py --gpu 1 --out results/bench_gpu1.json

Interpreting the results

• "1 epoch" is pipeline-native. A/C: one pass over the N train volumes (batch 3, 96³ random patches, AdamW 3e-4 + warmup/cosine, bf16). B: one pass over the N train cases at nnU-Net's planned batch and patch size (typically 2 and ~256³, AdamW 1e-2 PolyLR, fp16). Epoch times are therefore comparable as wall-clock cost per pass over the training set in each pipeline's standard configuration, not per identical number of patches.
• DICE reflects the training budget. Short benchmarks measure the machinery, not the final quality of the pipelines. Compare DICE across pipelines only at the same --train-epochs/--max-rows budget, or use --skip-train on fully trained study checkpoints.
• Inference times are single-GPU, standard-protocol numbers (A: 96³ step 48 + 4-flip TTA; B: nnU-Net gaussian sliding window + mirroring, native volumes; C: MONAI gaussian 50% + 4-flip TTA). B is the slowest per volume on this data set because it runs on uncropped native volumes with full mirroring TTA.
• B's one-time cost. raw_build_s + plan_preprocess_s are paid once per (dataset, round) — in the iterative study every round repays the plan/preprocess on the growing dataset; the benchmark reports it for context only.
• VRAM notes. Training VRAM includes optimizer states (AdamW) and the largest batch; B's number is the nvidia-smi peak of the whole nnUNetv2_train process tree on that GPU. Inference VRAM is per pipeline as measured in-process (B: torch peak over the timed cases).

Troubleshooting

symptom	fix
split_train.jsonl has no labeled processed rows	run scripts/preprocess.py over the manifests and scripts/05_build_splits.py first
B: key ... not in ntuh/m6_labeled manifests	build the source manifests (scripts/01_build_ntuh_manifest.py, scripts/02_build_m6_dataset.py) and confirm the native NIfTIs are reachable
--skip-train: .../best.pt not found	the study checkpoint for that pipeline doesn't exist yet — train a round first, or drop --skip-train
B --skip-train fails on the model folder	the study's nnu/results/Dataset210_NTUH_T1C_PL/NTUHLPLTrainer__nnUNetPlans__3d_fullres/fold_0/ must exist (produced by scripts_nnu/03_nnu_train.py)
OOM on B	B trains at the planned 256³-class patches and needs ~10–20 GB; free other GPU users or run with a smaller --nnu-cases set (patch size derives from the median case size)
inflated/inconsistent timings	do not run two benchmarks on the same --gpu concurrently; note that data-loader (NFS) latency is included in training and inference wall times