OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

OmniOpt presents a unified survey and benchmarking framework for a fragmented ecosystem of more than one hundred optimization methods. It models optimizer updates as a five-stage meta-pipeline and uses norm-constrained linear minimization oracles (LMOs) to provide a shared geometric view. The paper introduces a two-dimensional taxonomy: one axis describes the optimizer’s mechanism family, while the other records the measurable training objectives it targets. Its benchmark cookbook spans representative optimizers, model scales, and training regimes, including language-model pretraining and image classification, to analyze trade-offs across objective dimensions.