Trainer

This module provides the Trainer class, which is responsible for managing the training, validation, and testing loops of machine learning models. It supports distributed training, early stopping, and various tracking mechanisms.

Training Process

The train() method orchestrates the training process, including validation and testing phases. Here’s a detailed explanation of its internal workings:

  1. Initialization: - The method begins by initializing the trackers if the current process is the main GPU. - It sets up a progress bar for tracking epochs.

  2. Training Loop: - For each epoch, the method runs the training loop (_run_loop_train) to compute the training loss. - It then runs the validation loop (_run_loop_validation) if a validation dataset is provided.

  3. Logging and Saving: - Metrics, including training and validation loss, are logged using the trackers. - The model is saved if the current training loss is the best observed so far. - The model is also saved at regular intervals defined by save_interval. - If save_last is enabled, the model is saved at the end of each epoch.

  4. Early Stopping: - The method checks if early stopping criteria are met using the early_stopper. If so, training is halted early.

  5. Finalization: - After training, the trackers are finalized, and a summary of the training process is logged.

Subtleties:

  • Distributed Training: The trainer supports distributed training by wrapping the model with DistributedDataParallel if multiple GPUs are available.

  • Metrics Tracking: Metrics are updated and logged during both training and validation loops.

  • Snapshot Saving: Snapshots of the model are saved with detailed information, including model state, optimizer state, and training information.

snapshot = {
        "MODEL": {
            "MODEL_CONFIG": self.config["model"],
            "MODEL_STATE": self.model.state_dict(),
        },
        "TRAIN_INFO": {
            "EPOCHS_RUN": epoch,
            "BEST_LOSS": loss,
            "OPTIMIZER_STATE": self.optimizer.state_dict(),
            "SCHEDULER_STATE": (
                self.scheduler.state_dict() if self.scheduler else None
            ),
        },
        "GLOBAL_CONFIG": self.global_config.to_dict(),
    }

Trainer Class

The Trainer class is responsible for managing the complete lifecycle of model training. It handles configuration setup, data loading, training loops, validation, and testing.

class lis2.trainer.Trainer(*args, **kwargs)[source]

Bases: ITrainer

Trainer class responsible for managing the training, validation, and testing loops.

The Trainer class is designed to handle the complete lifecycle of model training, including configuration setup, data loading, training loops, validation, and testing. It supports distributed training, early stopping, and various tracking mechanisms.

Configuration:
  • name (str): The name of the training run.

  • output_dir (Union[Path, str]): Directory to save outputs.

  • model (Config): Configuration for the model (BaseModel).

  • train_dataset (Config): Configuration for the training dataset (BaseDataset).

  • val_dataset (Config): Configuration for the validation dataset (BaseDataset).

  • test_dataset (Optional[Config]): Configuration for the test dataset (BaseDataset). Default is None.

  • optimizer (Config): Configuration for the optimizer (BaseOptimizer).

  • scheduler (Optional[Config]): Configuration for the learning rate scheduler (BaseScheduler). Default is None.

  • early_stopper (Optional[Union[Config, bool]]): Configuration for early stopping (BaseEarlyStopping), work with loss of validation. If True, using LossEarlyStopping by default. If False, early stopping is disabled.

  • batch_size (int): Batch size for training. Default is 256.

  • num_workers (int): Number of workers for data loading. Default is the number of available CPUs.

  • epochs (int): Number of training epochs. Default is 100.

  • save_interval (int): Interval for saving model checkpoints. Default is 10.

  • trackers (Config): Configuration for trackers (BaseTracker). Default is an empty dictionary.

  • save_last (bool): Whether to save the last model checkpoint. Default is False.

  • metrics (list[Config]): List of metrics to track (BaseMetric). Default is a list with ‘map’, ‘dice’, and ‘roc_auc’ metrics.

Example Configuration (YAML):
name: "example_run"
output_dir: "/path/to/output"
model:
    type: "ExampleModel"
    params: {}
train_dataset:
    type: "ExampleDataset"
    params: {}
val_dataset:
    type: "ExampleDataset"
    params: {}
test_dataset:
    type: "ExampleDataset"
    params: {}
optimizer:
    type: "Adam"
    params:
        lr: 0.001
scheduler:
    type: "StepLR"
    params:
        step_size: 10
early_stopper:
    type: "EarlyStopping"
    params:
        patience: 5
batch_size: 256
num_workers: 8
epochs: 100
save_interval: 10
trackers: {}
save_last: False
metrics:
    - type: "map"
    - type: "dice"
    - type: "roc_auc"
Aliases:

epoch

config_schema: Dict = {'batch_size': Schema(type=<class 'int'>, aliases=[], optional=True, default=256), 'early_stopper': Schema(type=typing.Union[dict, str, bool], aliases=[], optional=True, default=None), 'epochs': Schema(type=<class 'int'>, aliases=['epoch'], optional=True, default=100), 'metrics': Schema(type=list[typing.Union[dict, str]], aliases=[], optional=True, default=[{'type': 'map'}, {'type': 'dice'}, {'type': 'roc_auc'}]), 'model': Schema(type=typing.Union[dict, str], aliases=[], optional=False, default=None), 'num_workers': Schema(type=<class 'int'>, aliases=[], optional=True, default=4), 'optimizer': Schema(type=typing.Union[dict, str], aliases=[], optional=False, default=None), 'output_dir': Schema(type=<class 'pathlib.Path'>, aliases=[], optional=False, default=None), 'run_name': Schema(type=<class 'str'>, aliases=[], optional=False, default=None), 'save_interval': Schema(type=<class 'int'>, aliases=[], optional=True, default=10), 'save_last': Schema(type=<class 'bool'>, aliases=[], optional=True, default=False), 'scheduler': Schema(type=typing.Union[dict, str], aliases=[], optional=True, default=None), 'test_dataset': Schema(type=typing.Union[dict, str], aliases=[], optional=True, default=None), 'trackers': Schema(type=typing.Union[dict, str], aliases=[], optional=True, default={}), 'train_dataset': Schema(type=typing.Union[dict, str], aliases=[], optional=False, default=None), 'val_dataset': Schema(type=typing.Union[dict, str], aliases=[], optional=False, default=None)}
classmethod from_snapshot(snapshot_path: str) → Trainer[source]

Load a snapshot of the training progress.

Parameters:

snapshot_path (str) – Path to the snapshot file.

Returns:

Trainer instance loaded from the snapshot.

Return type:

Trainer

get_final_info() → dict[str, Any][source]

Return the final training information including metrics, best model path, and other relevant details.

preconditions() → None[source]

Check preconditions before starting the training process.

run_final_evaluation(csv_path: str = 'test_results.csv') → dict[str, Any][source]

Execute the test phase on a specified test dataset and saveDict results to a CSV file.

Parameters:
  • test_dataset (Optional[BaseDataset]) – Dataset object for testing. If None, uses self.test_dataloader.

  • csv_path (str) – Path to save the test results as a CSV file.

Returns:

Dictionary containing the average loss and computed metrics.

Return type:

dict[str, Any]

setup_device(force_device=None, multi_gpu=False)[source]

Sets up the computing device, handling GPU assignments in a distributed setting.

Parameters:

force_device (str or None) – Manually specified device (e.g., “cuda:0”, “cpu”). If None, it assigns the appropriate device based on availability.

train(*args, **kwargs) → dict[str, Any][source]

Start the training process, including validation and test phases.

Returns:

Final training information including metrics and model paths.

Return type:

dict[str, Any]