drytorch.lib.schedulers

Module containing schedulers for the learning rates.

Functions

rescale(factor)	Create a scaling transformation.
restart(restart_interval[, ...])	Create a restart transformation.
warmup([warmup_steps])	Create a warmup transformation.

Classes

AbstractScheduler(*args, **kwargs)	Abstract class for the scheduler.
BaseScheduler()	Base class for schedulers that use dataclasses.
ComposedScheduler(base_scheduler, ...)	A scheduler produced by composing transformations.
ConstantScheduler()	Constant learning rate.
CosineScheduler([decay_steps, min_decay])	Schedule cosine decay: f(x) = C0 + C1(1 + cos(πx/C2)).
ExponentialScheduler([exp_decay, min_decay])	Schedule exponential decay: f(x) = C0 + C1(C2^x).
PolynomialScheduler([max_epochs, power, ...])	Polynomial learning rate scheduler: f(x) = C0 + C1(1 - x/C2)^C3.
RescaleScheduler(logic, factor)	Scheduler adding scaling to existing logic.
RestartScheduler(logic, restart_interval[, ...])	Scheduler adding periodic restarts to existing logic.
StepScheduler(milestones, gamma)	Step-wise learning rate scheduler.
TransformScheduler(logic)	Base class for scheduler transformations.
WarmupScheduler(logic, warmup_steps)	Scheduler adding warmup to existing logic.

class AbstractScheduler(*args, **kwargs)

Bases: SchedulerProtocol, ABC

Abstract class for the scheduler.

base_scheduler_name

name of the base scheduler for representation.

Type:

str

parameters

metadata associated with the scheduler.

Type:

dict[str, Any]

__call__(base_lr: float, epoch: int) → float

Modify the learning rate according to a schedule.

Parameters:

• base_lr (float) – initial learning rate.

• epoch (int) – the current epoch.

Returns:

scheduled value for the learning rate.

Raises:

ValueError – if base_lr or epoch are non-positive.

Return type:

float

bind(f: Callable[[Callable[[float, int], float]], AbstractScheduler], /) → ComposedScheduler

Allow transformation of the scheduler.

Parameters:

f (Callable[[Callable[[float, int], float]], AbstractScheduler]) – a function specifying the transformation.

Returns:

the transformed scheduler.

Return type:

ComposedScheduler

class ConstantScheduler

Bases: BaseScheduler

Constant learning rate.

class CosineScheduler(decay_steps: int = 250, min_decay: float = 0.01)

Bases: BaseScheduler

Schedule cosine decay: f(x) = C0 + C1(1 + cos(πx/C2)).

C0, C1, and C2 are defined so that: - f(x) = base_value when epoch = 0 and, - f(x) = min value when epoch is C2 = number of decay steps.

After the number of decay steps, returns min value.

Parameters:

• decay_steps (int)

• min_decay (float)

decay_steps

number of steps (epochs) to reach maximum decay.

Type:

int

min_decay

fraction of base_value for the minimum value.

Type:

float

__post_init__()

Input Validation.

class ExponentialScheduler(exp_decay: float = 0.975, min_decay: float = 0.0)

Bases: BaseScheduler

Schedule exponential decay: f(x) = C0 + C1(C2^x).

C0, C1, and C2 are defined so that: - f(x) = base_value when epoch = 0, - f(x) = min value when the epoch goes to infinite - f(x) is an exponential function with decay factor C2.

After the number of decay steps, returns min value.

Parameters:

• exp_decay (float)

• min_decay (float)

exp_decay

exponential decay parameter d for the curve: f(x) = Cd^x.

Type:

float

min_decay

proportion of base learning rate for the minimum CO.

Type:

float

__post_init__()

Input Validation.

class PolynomialScheduler(max_epochs: int = 1000, power: float = 1.0, min_decay: float = 0.0)

Bases: BaseScheduler

Polynomial learning rate scheduler: f(x) = C0 + C1(1 - x/C2)^C3.

C0, C1, C2, C3 are defined so that: - f(x) = base_value when epoch = 0, - f(x) = min value when epoch is C2 = number of decay steps - f(x) is a polynomial of degree C3.

After the number of decay steps, returns min value.

Parameters:

• max_epochs (int)

• power (float)

• min_decay (float)

max_epochs

maximum number of epochs.

Type:

int

power

polynomial power.

Type:

float

min_decay

minimum fraction of the initial learning rate.

Type:

float

__post_init__()

Input Validation.

class RescaleScheduler(logic: Callable[[float, int], float], factor: float)

Bases: TransformScheduler

Scheduler adding scaling to existing logic.

logic

callable to calculate the scheduling.

factor

factor to multiply the output by.

parameters

metadata associated with the scheduler.

Type:

dict[str, Any]

base_scheduler_name

name of the base scheduler for representation.

Type:

str

Initialize.

Parameters:

• logic (_SchedulingLogic) – callable to calculate the scheduling.

• factor (float) – factor to multiply the output by.

class RestartScheduler(logic: Callable[[float, int], float], restart_interval: int, restart_fraction: float = 1.0, max_restart: int | None = None)

Bases: TransformScheduler

Scheduler adding periodic restarts to existing logic.

logic

callable to calculate the scheduling.

restart_interval

number of epochs between restarts.

restart_fraction

fraction to use when restarting.

max_restart

maximum number of restarts before deactivating.

parameters

metadata associated with the scheduler.

Type:

dict[str, Any]

base_scheduler_name

name of the base scheduler for representation.

Type:

str

Initialize.

Parameters:

• logic (_SchedulingLogic) – callable to calculate the scheduling.

• restart_interval (int) – number of epochs between restarts.

• restart_fraction (float) – fraction to use when restarting.

• max_restart (int | None) – maximum number of restarts before deactivating.

class StepScheduler(milestones: Iterable[int] = <factory>, gamma: float = 0.1)

Bases: BaseScheduler

Step-wise learning rate scheduler.

Reduces learning rate by a factor at specified milestones.

Parameters:

• milestones (Iterable[int])

• gamma (float)

milestones

iterable of epochs at which to reduce the learning rate.

Type:

collections.abc.Iterable[int]

gamma

factor by which to reduce learning rate.

Type:

float

__post_init__()

Input Validation.

class WarmupScheduler(logic: Callable[[float, int], float], warmup_steps: int)

Bases: TransformScheduler

Scheduler adding warmup to existing logic.

logic

callable to calculate the scheduling.

warmup_steps

number of warmup steps.

parameters

metadata associated with the scheduler.

Type:

dict[str, Any]

base_scheduler_name

name of the base scheduler for representation.

Type:

str

Initialize.

Parameters:

• logic (_SchedulingLogic) – callable to calculate the scheduling.

• warmup_steps (int) – number of warmup steps.

rescale(factor: float) → Callable[[Callable[[float, int], float]], RescaleScheduler]

Create a scaling transformation.

Parameters:

factor (float) – factor that rescales the value.

Returns:

A decorator that adds scaling to the scheduling logic.

Return type:

Callable[[Callable[[float, int], float]], RescaleScheduler]

restart(restart_interval: int, restart_fraction: float = 1.0, max_restart: int | None = None) → Callable[[Callable[[float, int], float]], RestartScheduler]

Create a restart transformation.

Parameters:

• restart_interval (int) – number of epochs between restarts.

• restart_fraction (float) – fraction to use when restarting.

• max_restart (int | None) – Maximum number of restarts before deactivating.

Returns:

A decorator that adds restarting to the scheduling logic.

Return type:

Callable[[Callable[[float, int], float]], RestartScheduler]

warmup(warmup_steps: int = 10) → Callable[[Callable[[float, int], float]], WarmupScheduler]

Create a warmup transformation.

Parameters:

warmup_steps (int) – number of warmup steps.

Returns:

A decorator that adds warmup to the scheduling logic.

Return type:

Callable[[Callable[[float, int], float]], WarmupScheduler]