Commit graph

1 commit

Author SHA1 Message Date
a491ae0523 feat(train): implement checkpointing and robust data loading
Enhance the training pipeline with stateful checkpointing and improve
the resilience of the data loading process against filesystem latency
and transient I/O errors.

- Implement auto-resuming in `train_ddp` by loading model, optimizer,
  and scheduler states from `state.pt`.
- Add atomic state saving using temporary files to prevent corruption.
- Introduce `_read_nii` with exponential backoff retries to handle
  transient NFS/filesystem failures during NIfTI reading.
- Add explicit error handling for missing or unreadable label files in
  `PatchDataset`.
- Update `sliding_window_probs` to conditionally apply Test-Time
  Augmentation (TTA) based on the `tta` parameter.
- Add `scripts/test_dataloader.py` for verifying dataset integrity.
2026-09-26 03:44:12 +08:00