drytorch.lib.schedulers

Module containing schedulers for the learning rates.

Functions

rescale(factor)	Create a scaling transformation.
restart(restart_interval[, restart_fraction])	Create a restart transformation.
warmup([warmup_steps])	Create a warmup transformation.

Classes

AbstractScheduler(*args, **kwargs)	Abstract class for the scheduler.
ConstantScheduler()	Constant learning rate.
CosineScheduler([decay_steps, min_decay])	Schedule cosine decay: f(x) = C0 + C1(1 + cos(πx/C2)).
ExponentialScheduler([exp_decay, min_decay])	Schedule exponential decay: f(x) = C0 + C1(C2^x).
PolynomialScheduler([max_epochs, power, ...])	Polynomial learning rate scheduler: f(x) = C0 + C1(1 - x/C2)^C3.
RescaleScheduler(base_scheduler, factor)	Scale the output of an existing scheduler.
RestartScheduler(base_scheduler, ...[, ...])	Wraps another scheduler to provide periodic restarts.
StepScheduler(milestones, gamma)	Step-wise learning rate scheduler.
WarmupScheduler(base_scheduler[, warmup_steps])	Adds a warmup phase to any scheduler.

class AbstractScheduler(*args, **kwargs)

Bases: SchedulerProtocol, ABC

Abstract class for the scheduler.

__call__(base_lr: float, epoch: int) → float

Modify the learning rate according to a schedule.

Parameters:

• base_lr (float) – initial learning rate.

• epoch (int) – the current epoch.

Returns:

scheduled value for the learning rate.

Raises:

ValueError – if base_lr or epoch are non-positive.

Return type:

float

bind(f: Callable[[AbstractScheduler], AbstractScheduler], /) → AbstractScheduler

Allow transformation of the scheduler.

Parameters:

f (Callable[[AbstractScheduler], AbstractScheduler]) – a function specifying the transformation.

Returns:

the transformed scheduler.

Return type:

AbstractScheduler

class ConstantScheduler

Bases: AbstractScheduler

Constant learning rate.

class CosineScheduler(decay_steps: int = 250, min_decay: float = 0.01)

Bases: AbstractScheduler

Schedule cosine decay: f(x) = C0 + C1(1 + cos(πx/C2)).

C0, C1 and C2 are defined so that: - f(x) = base_value when epoch = 0 and, - f(x) = min value when epoch is C2 = number of decay steps.

After the number of decay steps, returns min value.

Parameters:

• decay_steps (int)

• min_decay (float)

decay_steps

number of steps (epochs) to reach maximum decay.

Type:

int

min_decay

fraction of base_value for the minimum value.

Type:

float

__post_init__()

Input Validation.

class ExponentialScheduler(exp_decay: float = 0.975, min_decay: float = 0.0)

Bases: AbstractScheduler

Schedule exponential decay: f(x) = C0 + C1(C2^x).

C0, C1 and C2 are defined so that: - f(x) = base_value when epoch = 0, - f(x) = min value when the epoch goes to infinite and, - f(x) is an exponential function with decay factor C2.

After the number of decay steps, returns min value.

Parameters:

• exp_decay (float)

• min_decay (float)

exp_decay

exponential decay parameter d for the curve: f(x) = Cd^x.

Type:

float

min_decay

proportion of base learning rate for the minimum CO.

Type:

float

__post_init__()

Input Validation.

class PolynomialScheduler(max_epochs: int = 1000, power: float = 1.0, min_decay: float = 0.0)

Bases: AbstractScheduler

Polynomial learning rate scheduler: f(x) = C0 + C1(1 - x/C2)^C3.

C0, C1, C2, C3 are defined so that: - f(x) = base_value when epoch = 0, - f(x) = min value when epoch is C2 = number of decay steps and, - f(x) is a polynomial of degree C3.

After the number of decay steps, returns min value.

Parameters:

• max_epochs (int)

• power (float)

• min_decay (float)

max_epochs

maximum number of epochs.

Type:

int

power

polynomial power.

Type:

float

min_decay

minimum fraction of the initial learning rate.

Type:

float

__post_init__()

Input Validation.

class StepScheduler(milestones: Iterable[int] = <factory>, gamma: float = 0.1)

Bases: AbstractScheduler

Step-wise learning rate scheduler.

Reduces learning rate by a factor at specified milestones.

Parameters:

• milestones (Iterable[int])

• gamma (float)

milestones

iterable of epochs at which to reduce the learning rate.

Type:

collections.abc.Iterable[int]

gamma

factor by which to reduce learning rate.

Type:

float

__post_init__()

Input Validation.

rescale(factor: float) → Callable[[AbstractScheduler], AbstractScheduler]

Create a scaling transformation.

Parameters:

factor (float) – factor that rescales the value.

Returns:

A decorator that wraps a scheduler with scaling.

Return type:

Callable[[AbstractScheduler], AbstractScheduler]

restart(restart_interval: int, restart_fraction: float = 1.0) → Callable[[AbstractScheduler], AbstractScheduler]

Create a restart transformation.

Parameters:

• restart_interval (int) – number of epochs between restarts.

• restart_fraction (float) – fraction to use when restarting.

Returns:

A decorator that wraps a scheduler with restarts.

Return type:

Callable[[AbstractScheduler], AbstractScheduler]

warmup(warmup_steps: int = 10) → Callable[[AbstractScheduler], AbstractScheduler]

Create a warmup transformation.

Parameters:

warmup_steps (int) – number of warmup steps.

Returns:

A decorator that wraps a scheduler with warmup.

Return type:

Callable[[AbstractScheduler], AbstractScheduler]