Enhance the training pipeline with stateful checkpointing and improve the resilience of the data loading process against filesystem latency and transient I/O errors. - Implement auto-resuming in `train_ddp` by loading model, optimizer, and scheduler states from `state.pt`. - Add atomic state saving using temporary files to prevent corruption. - Introduce `_read_nii` with exponential backoff retries to handle transient NFS/filesystem failures during NIfTI reading. - Add explicit error handling for missing or unreadable label files in `PatchDataset`. - Update `sliding_window_probs` to conditionally apply Test-Time Augmentation (TTA) based on the `tta` parameter. - Add `scripts/test_dataloader.py` for verifying dataset integrity. |
||
|---|---|---|
| .. | ||
| common.py | ||
| dataset.py | ||
| losses.py | ||
| training.py | ||
| unet3d.py | ||