Optimizers

The General Form

$${\theta }_{\text{new}}={\theta }_{\text{old}}+\Delta \theta$$

where $\Delta \theta$ is computed from the gradient $\frac{\partial L}{\partial \theta }$.

Stochastic Gradient Descent (SGD)

$${\theta }_{\text{new}}={\theta }_{\text{old}}-\eta \frac{\partial L}{\partial \theta }$$

where:

• $\eta$ = learning rate (hyperparameter, e.g., 0.001)
• $\frac{\partial L}{\partial \theta }$ = gradient you computed
• negative because gradient is calculated to be in the direction of greater loss

SGD with Momentum

Adds “velocity” to smooth updates:

$$v_t=\beta v_{t-1}+\frac{\partial L}{\partial W}$$ $$W_{\text{new}}=W_{\text{old}}-\eta v_t$$

where:

• $v_t$ = velocity (exponential moving average of gradients)
• $\beta$ = momentum coefficient (e.g., 0.9)

Adam

Combines momentum + adaptive learning rates:

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)\frac{\partial L}{\partial W}$$

(first moment, like momentum)

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2){\left(\frac{\partial L}{\partial W}\right)}^2$$

(second moment, squared gradients)

Bias correction:

$${\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t},{\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}$$

Update:

$$W_{\text{new}}=W_{\text{old}}-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$$

where:

• ${\beta }_1=0.9$ (first moment decay)
• ${\beta }_2=0.999$ (second moment decay)
• $ϵ=10^{-8}$ (numerical stability)
• Adam adapts learning rate per parameter based on gradient history.