ml
Optimizer Comparator
SGD, Momentum, RMSProp, Adam, AdaGrad side by side — trajectory overlay on f(x), simultaneous loss curves, play/pause step controls.
Function
Parameters
Optimizers
1 / 60
Optimizer Trajectories on f(x)
Loss Curves
—— SGD—— Momentum—— RMSProp—— Adam—— AdaGrad
Step 1 — Current f(x)
SGD16.5
Momentum16.5
RMSProp16.5
Adam16.5
AdaGrad16.5
Final State (all steps)
Optimizerxf(x)Steps
SGD10.560
Momentum0.90370.509360
RMSProp1.0140.500260
Adam1.0150.500260
AdaGrad3.7057.81560
SGD: x ← x − η∇f · Momentum: v ← βv + η∇f, x ← x − v
RMSProp: s ← ρs + (1-ρ)g², x ← x − η/√(s+ε) · g
Adam: bias-corrected first & second moments, β₁=0.9, β₂=0.999
AdaGrad: G ← G + g², x ← x − η/√(G+ε) · g
RMSProp: s ← ρs + (1-ρ)g², x ← x − η/√(s+ε) · g
Adam: bias-corrected first & second moments, β₁=0.9, β₂=0.999
AdaGrad: G ← G + g², x ← x − η/√(G+ε) · g