Table of Contents
Shortcuts

# CosineAnnealingWarmRestarts

class torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0, T_mult=1, eta_min=0, last_epoch=- 1, verbose=False)

  Set the learning rate of each parameter group using a cosine annealing schedule, where η m a x\eta_{max}η ma x​ is set to the initial lr, T c u r T_{cur}T c u r​ is the number of epochs since the last restart and T i T_{i}T i​ is the number of epochs between two warm restarts in SGDR:

  η t=η m i n+1 2(η m a x−η m i n)(1+cos⁡(T c u r T i π))\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{i}}\pi\right)\right) η t​=η min​+2 1​(η ma x​−η min​)(1+cos(T i​T c u r​​π))

  When T c u r=T i T_{cur}=T_{i}T c u r​=T i​, set η t=η m i n\eta_t = \eta_{min}η t​=η min​. When T c u r=0 T_{cur}=0 T c u r​=0 after restart, set η t=η m a x\eta_t=\eta_{max}η t​=η ma x​.

  It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts.

  Parameters:

    - optimizer (Optimizer) – Wrapped optimizer.

    - T_0 (int) – Number of iterations for the first restart.

    - T_mult (int, optional) – A factor increases T i T_{i}T i​ after a restart. Default: 1.

    - eta_min (float, optional) – Minimum learning rate. Default: 0.

    - last_epoch (int, optional) – The index of last epoch. Default: -1.

    - verbose (bool) – If `True`, prints a message to stdout for each update. Default: `False`.

  get_last_lr()

    Return last computed learning rate by current scheduler.

  load_state_dict(state_dict)

    Loads the schedulers state.

    Parameters:

      state_dict (dict) – scheduler state. Should be an object returned from a call to `state_dict()`.

  print_lr(is_verbose, group, lr, epoch=None)

    Display the current learning rate.

  state_dict()

    Returns the state of the scheduler as a `dict`.

    It contains an entry for every variable in self.__dict__ which is not the optimizer.

  step(epoch=None)

    Step could be called after every batch update

    Example

        >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
        >>> iters = len(dataloader)
        >>> for epoch in range(20):
        >>>     for i, sample in enumerate(dataloader):
        >>>         inputs, labels = sample['inputs'], sample['labels']
        >>>         optimizer.zero_grad()
        >>>         outputs = net(inputs)
        >>>         loss = criterion(outputs, labels)
        >>>         loss.backward()
        >>>         optimizer.step()
        >>>         scheduler.step(epoch + i / iters)

    This function can be called in an interleaved way.

    Example

        >>> scheduler = CosineAnnealingWarmRestarts(optimizer, T_0, T_mult)
        >>> for epoch in range(20):
        >>>     scheduler.step()
        >>> scheduler.step(26)
        >>> scheduler.step() # scheduler.step(27), instead of scheduler(20)

## Docs

Access comprehensive developer documentation for PyTorch

View Docs

## Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

## Resources

Find development resources and get your questions answered

View Resources

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: Cookies Policy.