TY - GEN

T1 - On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data

AU - Behnia, Tina

AU - Kini, Ganesh Ramachandra

AU - Vakilian, Vala

AU - Thrampoulidis, Christos

N1 - KAUST Repository Item: Exported on 2023-07-28
Acknowledged KAUST grant number(s): CRG8
Acknowledgements: This work is supported by an NSERC Discovery Grant, NSF Grant CCF-2009030, and by a CRG8-KAUST award. The authors also acknowledge use of the Sockeye cluster by UBC Advanced Research Computing.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.

PY - 2023/1/1

Y1 - 2023/1/1

N2 - Various logit-adjusted parameterizations of the cross-entropy (CE) loss have been proposed as alternatives to weighted CE for training large models on label-imbalanced data far beyond the zero train error regime. The driving force behind those designs has been the theory of implicit bias, which for linear(ized) models, explains why they successfully induce bias on the optimization path towards solutions that favor minorities. Aiming to extend this theory to non-linear models, we investigate the implicit geometry of classifiers and embeddings that are learned by different CE parameterizations. Our main result characterizes the global minimizers of a non-convex cost-sensitive SVM classifier for the unconstrained features model, which serves as an abstraction of deep-nets. We derive closed-form formulas for the angles and norms of classifiers and embeddings as a function of the number of classes, the imbalance and the minority ratios, and the loss hyperparameters. Using these, we show that logit-adjusted parameterizations can be appropriately tuned to learn symmetric geometries irrespective of the imbalance ratio. We complement our analysis with experiments and an empirical study of convergence accuracy in deep-nets.

AB - Various logit-adjusted parameterizations of the cross-entropy (CE) loss have been proposed as alternatives to weighted CE for training large models on label-imbalanced data far beyond the zero train error regime. The driving force behind those designs has been the theory of implicit bias, which for linear(ized) models, explains why they successfully induce bias on the optimization path towards solutions that favor minorities. Aiming to extend this theory to non-linear models, we investigate the implicit geometry of classifiers and embeddings that are learned by different CE parameterizations. Our main result characterizes the global minimizers of a non-convex cost-sensitive SVM classifier for the unconstrained features model, which serves as an abstraction of deep-nets. We derive closed-form formulas for the angles and norms of classifiers and embeddings as a function of the number of classes, the imbalance and the minority ratios, and the loss hyperparameters. Using these, we show that logit-adjusted parameterizations can be appropriately tuned to learn symmetric geometries irrespective of the imbalance ratio. We complement our analysis with experiments and an empirical study of convergence accuracy in deep-nets.

UR - http://hdl.handle.net/10754/693261

UR - https://proceedings.mlr.press/v206/behnia23a.html

UR - http://www.scopus.com/inward/record.url?scp=85164889479&partnerID=8YFLogxK

M3 - Conference contribution

SP - 10815

EP - 10838

BT - 26th International Conference on Artificial Intelligence and Statistics, AISTATS 2023

PB - ML Research Press

ER -