Different kinds of deep networks typically need different optimizers, which have to be chosen after ?multiple trials, making the training process inefficient. To relieve ?this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm ?(Adan). Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, then adopts ?NME to estimate the first- and second-order moments of the gradient ?for convergence acceleration. Besides, we prove that Adan finds an ?-approximate first-order stationary point within $O(?^{?3.5})$ stochastic gradient complexity on the non-convex stochastic problems, matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for ?many popular networks, e.g. ResNet, ConvNext, ViT, Swin, MAE, DETR, ?GPT-2, Transformer-XL. More surprisingly, Adan can use half of the ?epochs of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc, and also shows great tolerance to a large range of minibatch size, e.g. from 1k to 32k. Code is released at https://github.com/sail-sg/Adan.