Modeling the Machine Learning Multiverse

Samuel J. Bell; Onno P. Kampman; Jesse Dodge; Neil D. Lawrence

Back to publications

Modeling the Machine Learning Multiverse

Samuel J. Bell, Onno P. Kampman, Jesse Dodge, Neil D. Lawrence

Advances in Neural Information Processing Systems (NeurIPS), 2022.

Abstract

Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis introduced in response to psychology’s own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers. Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, a multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.

Links

Cite this Paper

BibTeX


@InProceedings{publications/modeling-the-machine-learning-multiverse,
  title = 	 {Modeling the Machine Learning Multiverse},
  author = 	 {Bell, Samuel J. and Kampman, Onno P. and Dodge, Jesse and Lawrence, Neil D.},
  booktitle = 	 {Advances in Neural Information Processing Systems (NeurIPS)},
  year = 	 {2022},
  url = 	 {/publications/modeling-the-machine-learning-multiverse.html},
  abstract = 	 {Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis.
Our framework builds upon the multiverse analysis introduced in response to psychology's own reproducibility crisis.
To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design.
Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization.
In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers.
Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap.
For the machine learning community, a multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.}
}

Endnote

%0 Conference Paper
%T Modeling the Machine Learning Multiverse
%A Samuel J. Bell
%A Onno P. Kampman
%A Jesse Dodge
%A Neil D. Lawrence
%B Advances in Neural Information Processing Systems (NeurIPS)
%D 2022	
%F publications/modeling-the-machine-learning-multiverse
%U /publications/modeling-the-machine-learning-multiverse.html
%X Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis.
Our framework builds upon the multiverse analysis introduced in response to psychology's own reproducibility crisis.
To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design.
Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization.
In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers.
Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap.
For the machine learning community, a multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.

RIS


TY  - CPAPER
TI  - Modeling the Machine Learning Multiverse
AU  - Samuel J. Bell
AU  - Onno P. Kampman
AU  - Jesse Dodge
AU  - Neil D. Lawrence
BT  - Advances in Neural Information Processing Systems (NeurIPS)
DA  - 2022/10/22	
ID  - publications/modeling-the-machine-learning-multiverse
UR  - /publications/modeling-the-machine-learning-multiverse.html
AB  - Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis.
Our framework builds upon the multiverse analysis introduced in response to psychology's own reproducibility crisis.
To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design.
Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization.
In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers.
Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap.
For the machine learning community, a multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.
ER  -

APA


Bell, S.J., Kampman, O.P., Dodge, J. & Lawrence, N.D.. (2022). Modeling the Machine Learning Multiverse. Advances in Neural Information Processing Systems (NeurIPS) Available from /publications/modeling-the-machine-learning-multiverse.html.