Empirical Bayes Transductive Meta-Learning with Synthetic Gradients

Shell Xu Hu; Pablo Garcia Moreno; Yang Xiao; Xi Shen; Guillaume Obozinski; Neil D. Lawrence; Andreas Damianou

Back to publications

Empirical Bayes Transductive Meta-Learning with Synthetic Gradients

Shell Xu Hu, Pablo Garcia Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D. Lawrence, Andreas Damianou

International Conference on Learning Representations, 2020.

Abstract

We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging the unlabeled query set in addition to the support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior on the query set of each task. We derive a novel amortized variational inference that couples all the variational posteriors via a meta-model, which consists of a synthetic gradient network and an initialization network. Each variational posterior is derived from synthetic gradient descent to approximate the true posterior on the query set, although where we do not have access to the true gradient. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods. Besides, we conduct two zero-shot learning experiments to further explore the potential of the synthetic gradient.

Links

Cite this Paper

BibTeX


@InProceedings{Hu-empirical20,
  title = 	 {Empirical {B}ayes Transductive Meta-Learning with Synthetic Gradients},
  author = 	 {Hu, Shell Xu and Garcia Moreno, Pablo and Xiao, Yang and Shen, Xi and Obozinski, Guillaume and Lawrence, Neil D. and Damianou, Andreas},
  booktitle = 	 {International Conference on Learning Representations},
  year = 	 {2020},
  pdf = 	 {https://openreview.net/attachment?id=Hkg-xgrYvH&name=original_pdf},
  url = 	 {/publications/empirical-bayes-transductive-meta-learning-with-synthetic-gradients.html},
  abstract = 	 {We propose a meta-learning approach that learns from multiple tasks
in a transductive setting, by leveraging the unlabeled query set in
addition to the support set to generate a more powerful model for
each task. To develop our framework, we revisit the empirical Bayes
formulation for multi-task learning.  The evidence lower bound of
the marginal log-likelihood of empirical Bayes decomposes as a sum
of local KL divergences between the variational posterior and the
true posterior on the query set of each task.  We derive a novel
amortized variational inference that couples all the variational
posteriors via a meta-model, which consists of a synthetic gradient
network and an initialization network. Each variational posterior is
derived from synthetic gradient descent to approximate the true
posterior on the query set, although where we do not have access to
the true gradient.  Our results on the Mini-ImageNet and CIFAR-FS
benchmarks for episodic few-shot classification outperform previous
state-of-the-art methods. Besides, we conduct two zero-shot learning
experiments to further explore the potential of the synthetic
gradient.
}
}

Endnote

%0 Conference Paper
%T Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
%A Shell Xu Hu
%A Pablo Garcia Moreno
%A Yang Xiao
%A Xi Shen
%A Guillaume Obozinski
%A Neil D. Lawrence
%A Andreas Damianou
%B International Conference on Learning Representations
%D 2020	
%F Hu-empirical20
%U /publications/empirical-bayes-transductive-meta-learning-with-synthetic-gradients.html
%X We propose a meta-learning approach that learns from multiple tasks
in a transductive setting, by leveraging the unlabeled query set in
addition to the support set to generate a more powerful model for
each task. To develop our framework, we revisit the empirical Bayes
formulation for multi-task learning.  The evidence lower bound of
the marginal log-likelihood of empirical Bayes decomposes as a sum
of local KL divergences between the variational posterior and the
true posterior on the query set of each task.  We derive a novel
amortized variational inference that couples all the variational
posteriors via a meta-model, which consists of a synthetic gradient
network and an initialization network. Each variational posterior is
derived from synthetic gradient descent to approximate the true
posterior on the query set, although where we do not have access to
the true gradient.  Our results on the Mini-ImageNet and CIFAR-FS
benchmarks for episodic few-shot classification outperform previous
state-of-the-art methods. Besides, we conduct two zero-shot learning
experiments to further explore the potential of the synthetic
gradient.

RIS


TY  - CPAPER
TI  - Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
AU  - Shell Xu Hu
AU  - Pablo Garcia Moreno
AU  - Yang Xiao
AU  - Xi Shen
AU  - Guillaume Obozinski
AU  - Neil D. Lawrence
AU  - Andreas Damianou
BT  - International Conference on Learning Representations
DA  - 2020/04/26	
ID  - Hu-empirical20
L1  - https://openreview.net/attachment?id=Hkg-xgrYvH&name=original_pdf
UR  - /publications/empirical-bayes-transductive-meta-learning-with-synthetic-gradients.html
AB  - We propose a meta-learning approach that learns from multiple tasks
in a transductive setting, by leveraging the unlabeled query set in
addition to the support set to generate a more powerful model for
each task. To develop our framework, we revisit the empirical Bayes
formulation for multi-task learning.  The evidence lower bound of
the marginal log-likelihood of empirical Bayes decomposes as a sum
of local KL divergences between the variational posterior and the
true posterior on the query set of each task.  We derive a novel
amortized variational inference that couples all the variational
posteriors via a meta-model, which consists of a synthetic gradient
network and an initialization network. Each variational posterior is
derived from synthetic gradient descent to approximate the true
posterior on the query set, although where we do not have access to
the true gradient.  Our results on the Mini-ImageNet and CIFAR-FS
benchmarks for episodic few-shot classification outperform previous
state-of-the-art methods. Besides, we conduct two zero-shot learning
experiments to further explore the potential of the synthetic
gradient.

ER  -

APA


Hu, S.X., Garcia Moreno, P., Xiao, Y., Shen, X., Obozinski, G., Lawrence, N.D. & Damianou, A.. (2020). Empirical Bayes Transductive Meta-Learning with Synthetic Gradients. International Conference on Learning Representations Available from /publications/empirical-bayes-transductive-meta-learning-with-synthetic-gradients.html.