Table of Contents
Shortcuts

# Adam

class torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False, *, foreach=None, maximize=False, capturable=False, differentiable=False, fused=None)

  Implements Adam algorithm.

  input:γ (lr),β 1,β 2 (betas),θ 0 (params),f(θ) (objective)λ (weight decay), amsgrad, maximize initialize:m 0←0 ( first moment),v 0←0 (second moment), v 0^m a x←0 for t=1 to … do if maximize:g t←−∇θ f t(θ t−1)else g t←∇θ f t(θ t−1)if λ≠0 g t←g t+λ θ t−1 m t←β 1 m t−1+(1−β 1)g t v t←β 2 v t−1+(1−β 2)g t 2 m t^←m t/(1−β 1 t)v t^←v t/(1−β 2 t)if a m s g r a d v t^m a x←m a x(v t^m a x,v t^)θ t←θ t−1−γ m t^/(v t^m a x+ϵ)else θ t←θ t−1−γ m t^/(v t^+ϵ)r e t u r n θ t\begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: \textit{amsgrad}, \:\textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned}​input:γ (lr),β 1​,β 2​ (betas),θ 0​ (params),f(θ) (objective)λ (weight decay),amsgrad,maximize initialize:m 0​←0 ( first moment),v 0​←0 (second moment),v 0​
  ​ma x←0 for t=1 to…do if maximize:g t​←−∇θ​f t​(θ t−1​)else g t​←∇θ​f t​(θ t−1​)if λ=0 g t​←g t​+λ θ t−1​m t​←β 1​m t−1​+(1−β 1​)g t​v t​←β 2​v t−1​+(1−β 2​)g t 2​m t​
  ​←m t​/(1−β 1 t​)v t​
  ​←v t​/(1−β 2 t​)if am s g r a d v t​
  ​ma x←max(v t​
  ​ma x,v t​
  ​)θ t​←θ t−1​−γ m t​
  ​/(v t​
  ​ma x
  ​+ϵ)else θ t​←θ t−1​−γ m t​
  ​/(v t​
  ​
  ​+ϵ)return θ t​​

  For further details regarding the algorithm we refer to Adam: A Method for Stochastic Optimization.

  Parameters:

    - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

    - lr (float, Tensor, optional) – learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True.

    - betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))

    - eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

    - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

    - amsgrad (bool, optional) – whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond (default: False)

    - foreach (bool, optional) – whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. If memory is prohibitive, batch fewer parameters through the optimizer at a time or switch this flag to False (default: None)

    - maximize (bool, optional) – maximize the params based on the objective, instead of minimizing (default: False)

    - capturable (bool, optional) – whether this instance is safe to capture in a CUDA graph. Passing True can impair ungraphed performance, so if you don’t intend to graph capture this instance, leave it False (default: False)

    - differentiable (bool, optional) – whether autograd should occur through the optimizer step in training. Otherwise, the step() function runs in a torch.no_grad() context. Setting to True can impair performance, so leave it False if you don’t intend to run autograd through this instance (default: False)

    - fused (bool, optional) – whether the fused implementation (CUDA only) is used. Currently, torch.float64, torch.float32, torch.float16, and torch.bfloat16 are supported. (default: None)

  Note

  The foreach and fused implementations are typically faster than the for-loop, single-tensor implementation. Thus, if the user has not specified BOTH flags (i.e., when foreach = fused = None), we will attempt defaulting to the foreach implementation when the tensors are all on CUDA. For example, if the user specifies True for fused but nothing for foreach, we will run the fused implementation. If the user specifies False for foreach but nothing for fused (or False for fused but nothing for foreach), we will run the for-loop implementation. If the user specifies True for both foreach and fused, we will prioritize fused over foreach, as it is typically faster. We attempt to use the fastest, so the hierarchy goes fused -> foreach -> for-loop. HOWEVER, since the fused implementation is relatively new, we want to give it sufficient bake-in time, so we default to foreach and NOT fused when the user has not specified either flag.

  add_param_group(param_group)

    Add a param group to the `Optimizer` s param_groups.

    This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the `Optimizer` as training progresses.

    Parameters:

      param_group (dict) – Specifies what Tensors should be optimized along with group specific optimization options.

  load_state_dict(state_dict)

    Loads the optimizer state.

    Parameters:

      state_dict (dict) – optimizer state. Should be an object returned from a call to `state_dict()`.

  register_load_state_dict_post_hook(hook, prepend=False)

    Register a load_state_dict post-hook which will be called after `load_state_dict()` is called. It should have the following signature:

        hook(optimizer) -> None

    The `optimizer` argument is the optimizer instance being used.

    The hook will be called with argument `self` after calling `load_state_dict` on `self`. The registered hook can be used to perform post-processing after `load_state_dict` has loaded the `state_dict`.

    Parameters:

      - hook (Callable) – The user defined hook to be registered.

      - prepend (bool) – If True, the provided post `hook` will be fired before all the already registered post-hooks on `load_state_dict`. Otherwise, the provided `hook` will be fired after all the already registered post-hooks. (default: False)

    Returns:

      a handle that can be used to remove the added hook by calling `handle.remove()`

    Return type:

      `torch.utils.hooks.RemoveableHandle`

  register_load_state_dict_pre_hook(hook, prepend=False)

    Register a load_state_dict pre-hook which will be called before `load_state_dict()` is called. It should have the following signature:

        hook(optimizer, state_dict) -> state_dict or None

    The `optimizer` argument is the optimizer instance being used and the `state_dict` argument is a shallow copy of the `state_dict` the user passed in to `load_state_dict`. The hook may modify the state_dict inplace or optionally return a new one. If a state_dict is returned, it will be used to be loaded into the optimizer.

    The hook will be called with argument `self` and `state_dict` before calling `load_state_dict` on `self`. The registered hook can be used to perform pre-processing before the `load_state_dict` call is made.

    Parameters:

      - hook (Callable) – The user defined hook to be registered.

      - prepend (bool) – If True, the provided pre `hook` will be fired before all the already registered pre-hooks on `load_state_dict`. Otherwise, the provided `hook` will be fired after all the already registered pre-hooks. (default: False)

    Returns:

      a handle that can be used to remove the added hook by calling `handle.remove()`

    Return type:

      `torch.utils.hooks.RemoveableHandle`

  register_state_dict_post_hook(hook, prepend=False)

    Register a state dict post-hook which will be called after `state_dict()` is called. It should have the following signature:

        hook(optimizer, state_dict) -> state_dict or None

    The hook will be called with arguments `self` and `state_dict` after generating a `state_dict` on `self`. The hook may modify the state_dict inplace or optionally return a new one. The registered hook can be used to perform post-processing on the `state_dict` before it is returned.

    Parameters:

      - hook (Callable) – The user defined hook to be registered.

      - prepend (bool) – If True, the provided post `hook` will be fired before all the already registered post-hooks on `state_dict`. Otherwise, the provided `hook` will be fired after all the already registered post-hooks. (default: False)

    Returns:

      a handle that can be used to remove the added hook by calling `handle.remove()`

    Return type:

      `torch.utils.hooks.RemoveableHandle`

  register_state_dict_pre_hook(hook, prepend=False)

    Register a state dict pre-hook which will be called before `state_dict()` is called. It should have the following signature:

        hook(optimizer) -> None

    The `optimizer` argument is the optimizer instance being used. The hook will be called with argument `self` before calling `state_dict` on `self`. The registered hook can be used to perform pre-processing before the `state_dict` call is made.

    Parameters:

      - hook (Callable) – The user defined hook to be registered.

      - prepend (bool) – If True, the provided pre `hook` will be fired before all the already registered pre-hooks on `state_dict`. Otherwise, the provided `hook` will be fired after all the already registered pre-hooks. (default: False)

    Returns:

      a handle that can be used to remove the added hook by calling `handle.remove()`

    Return type:

      `torch.utils.hooks.RemoveableHandle`

  register_step_post_hook(hook)

    Register an optimizer step post hook which will be called after optimizer step. It should have the following signature:

        hook(optimizer, args, kwargs) -> None

    The `optimizer` argument is the optimizer instance being used.

    Parameters:

      hook (Callable) – The user defined hook to be registered.

    Returns:

      a handle that can be used to remove the added hook by calling `handle.remove()`

    Return type:

      `torch.utils.hooks.RemovableHandle`

  register_step_pre_hook(hook)

    Register an optimizer step pre hook which will be called before optimizer step. It should have the following signature:

        hook(optimizer, args, kwargs) -> None or modified args and kwargs

    The `optimizer` argument is the optimizer instance being used. If args and kwargs are modified by the pre-hook, then the transformed values are returned as a tuple containing the new_args and new_kwargs.

    Parameters:

      hook (Callable) – The user defined hook to be registered.

    Returns:

      a handle that can be used to remove the added hook by calling `handle.remove()`

    Return type:

      `torch.utils.hooks.RemovableHandle`

  state_dict()

    Returns the state of the optimizer as a `dict`.

    It contains two entries:

    - `state`: a Dict holding current optimization state. Its content

        differs between optimizer classes, but some common characteristics hold. For example, state is saved per parameter, and the parameter itself is NOT saved. `state` is a Dictionary mapping parameter ids to a Dict with state corresponding to each parameter.

    - `param_groups`: a List containing all parameter groups where each

        parameter group is a Dict. Each parameter group contains metadata specific to the optimizer, such as learning rate and weight decay, as well as a List of parameter IDs of the parameters in the group.

    NOTE: The parameter IDs may look like indices but they are just IDs associating state with param_group. When loading from a state_dict, the optimizer will zip the param_group `params` (int IDs) and the optimizer `param_groups` (actual `nn.Parameter` s) in order to match state WITHOUT additional verification.

    A returned state dict might look something like:

        {
            'state': {
                0: {'momentum_buffer': tensor(...), ...},
                1: {'momentum_buffer': tensor(...), ...},
                2: {'momentum_buffer': tensor(...), ...},
                3: {'momentum_buffer': tensor(...), ...}
            },
            'param_groups': [
                {
                    'lr': 0.01,
                    'weight_decay': 0,
                    ...
                    'params': [0]
                },
                {
                    'lr': 0.001,
                    'weight_decay': 0.5,
                    ...
                    'params': [1, 2, 3]
                }
            ]
        }

    Return type:

      Dict[str, Any]

  step(closure=None)

    Performs a single optimization step.

    Parameters:

      closure (Callable, optional) – A closure that reevaluates the model and returns the loss.

  zero_grad(set_to_none=True)

    Resets the gradients of all optimized `torch.Tensor` s.

    Parameters:

      set_to_none (bool) – instead of setting to zero, set the grads to None. This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests `zero_grad(set_to_none=True)` followed by a backward pass, `.grad`s are guaranteed to be None for params that did not receive a gradient. 3. `torch.optim` optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).

## Docs

Access comprehensive developer documentation for PyTorch

View Docs

## Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

## Resources

Find development resources and get your questions answered

View Resources

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: Cookies Policy.