Optimizers

This module provides a framework for integrating PyTorch optimizers with a configurable schema, allowing dynamic configuration and extension of optimizers.

Adding a Custom Optimizer

To add a custom optimizer, you need to define a new class that extends the BaseOptimizer class. You can configure the optimizer using a schema and implement the optimization logic in the step method.

Registering Optimizers

The module automatically registers all valid optimizer classes from torch.optim as subclasses of BaseOptimizer. This allows for seamless integration and configuration of PyTorch’s built-in optimizers.

For more information on the available optimizers and their parameters, refer to the official PyTorch documentation: torch.optim.

BaseOptimizer

class lis2.optimizer.BaseOptimizer(*args, **kwargs)[source]

Bases: TypedConfigurable, Optimizer

Base class for PyTorch optimizers integrated with TypedConfigurable.

Enables dynamic subclass generation for each optimizer. Check torch.optim documentation for more information on how to implement custom optimizers.

Configuration:
  • lr (float): The learning rate for the optimizer.

Example

Here’s how you can create and use a custom optimizer by inheriting from BaseOptimizer:

import torch
from torch.optim.optimizer import Optimizer

class MyCustomOptimizer(BaseOptimizer):

    schema = {
        "lr": Schema(float, optional=True, default=0.01),
    }

    def __init__(self, params):
        defaults = {"lr": self.lr}
        super().__init__(params, defaults)

    def step(self, closure=None):
        # Implementation of the optimization step
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                d_p = p.grad.data
                p.data.add_(-group['lr'], d_p)

# Usage example
model = MyModel()  # Your PyTorch model
optimizer = MyCustomOptimizer(model.parameters(), lr=0.01)
config_schema: Dict = {'lr': Schema(type=<class 'float'>, aliases=[], optional=True, default=0.01)}

The BaseOptimizer class serves as the base class for integrating PyTorch optimizers with the TypedConfigurable framework. It enables dynamic subclass generation for each optimizer, allowing for easy configuration and extension.