Differentially Private Natural Gradient Descent

This paper proposes DP-NGD, a framework for applying natural-gradient optimization to differentially private training. It addresses three obstacles: privacy costs from estimating curvature, the mismatch between isotropic DP mechanisms and anisotropic preconditioning, and instability caused by inverse curvature in flat directions. The method estimates curvature independently of private data, performs the private mechanism in whitened space, and dynamically clamps curvature. The authors report state-of-the-art accuracy on standard benchmarks and up to a 10x convergence-speed improvement over first-order baselines under the same privacy budget.