Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows - Université PSL (Paris Sciences & Lettres) Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows

Résumé

Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles for non-Euclidean geometries and momentum-based dynamics remain largely unknown. In this paper, we characterize "all" conservation laws in this general setting. In stark contrast to the case of gradient flows, we prove that the conservation laws for momentum-based dynamics exhibit temporal dependence. Additionally, we often observe a "conservation loss" when transitioning from gradient flow to momentum dynamics. Specifically, for linear networks, our framework allows us to identify all momentum conservation laws, which are less numerous than in the gradient flow case except in sufficiently over-parameterized regimes. With ReLU networks, no conservation law remains. This phenomenon also manifests in non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF): all conservation laws can be determined in the gradient flow context, yet none persists in the momentum case.

Dates et versions

hal-04590932 , version 1 (28-05-2024)

Licence

Identifiants

Citer

Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré. Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows. 41st International Conference on Machine Learning, Jul 2024, Vienna, Austria. ⟨hal-04590932⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More