Deep Neural Networks as Point Estimates for Deep Gaussian Processes

Vincent Dutordoir; James Hensman; Mark van der Wilk; Carl Henrik Ek; Zoubin Ghahramani; Nicolas Durrande

Back to publications

Deep Neural Networks as Point Estimates for Deep Gaussian Processes

Vincent Dutordoir, James Hensman, Mark van der Wilk, Carl Henrik Ek, Zoubin Ghahramani, Nicolas Durrande

Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021.

Abstract

Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.

Links

Cite this Paper

BibTeX


@InProceedings{publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes,
  title = 	 {Deep Neural Networks as Point Estimates for Deep Gaussian Processes},
  author = 	 {Dutordoir, Vincent and Hensman, James and van der Wilk, Mark and Ek, Carl Henrik and Ghahramani, Zoubin and Durrande, Nicolas},
  booktitle = 	 {Advances in Neural Information Processing Systems 34 (NeurIPS 2021)},
  year = 	 {2021},
  pdf = 	 {https://proceedings.neurips.cc/paper/2021/file/4e6cd95227cb0c280e99a195be5f6615-Paper.pdf},
  url = 	 {/publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes.html},
  abstract = 	 {Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.}
}

Endnote

%0 Conference Paper
%T Deep Neural Networks as Point Estimates for Deep Gaussian Processes
%A Vincent Dutordoir
%A James Hensman
%A Mark van der Wilk
%A Carl Henrik Ek
%A Zoubin Ghahramani
%A Nicolas Durrande
%B Advances in Neural Information Processing Systems 34 (NeurIPS 2021)
%D 2021	
%F publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes
%U /publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes.html
%X Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.

RIS


TY  - CPAPER
TI  - Deep Neural Networks as Point Estimates for Deep Gaussian Processes
AU  - Vincent Dutordoir
AU  - James Hensman
AU  - Mark van der Wilk
AU  - Carl Henrik Ek
AU  - Zoubin Ghahramani
AU  - Nicolas Durrande
BT  - Advances in Neural Information Processing Systems 34 (NeurIPS 2021)
DA  - 2021/11/22	
ID  - publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes
L1  - https://proceedings.neurips.cc/paper/2021/file/4e6cd95227cb0c280e99a195be5f6615-Paper.pdf
UR  - /publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes.html
AB  - Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.
ER  -

APA


Dutordoir, V., Hensman, J., van der Wilk, M., Ek, C.H., Ghahramani, Z. & Durrande, N.. (2021). Deep Neural Networks as Point Estimates for Deep Gaussian Processes. Advances in Neural Information Processing Systems 34 (NeurIPS 2021) Available from /publications/deep-neural-networks-as-point-estimates-for-deep-gaussian-processes.html.