Environment is a nexus: generalization process for domain generalization

Abstract

Domain Generalization (DG) aims to learn a well-generalized model for unseen target domains from multiple observed source domains. Most current approaches, e.g., domain-invariant representation, typically focus on learning a universal sample-to-label mapping (function) across domains and have achieved impressive performance. Nonetheless, they usually overlook semantically intrinsic domain-specific information, resulting in limited generalizability. For compensation, in this paper, we adopt a novel standpoint, that is, a domain can be regarded as a meta-sample sampling from a certain meta-distribution, namely an environment distribution. With this perspective, in DG, functions learned from individual domains can be seen as a collective set of functional samples. Consequently, we can establish a meta-function, mapping from the environment to functions, to induce specific functions for unseen domains from the above function set effectively. To achieve this meta-function, we propose a learning paradigm based on Gaussian process with theoretical guarantee, namely Generalization Process for Domain Generalization (GPDG). Specifically, analogous to traditional Gaussian process, we describe the inference process in DG as Gaussian process fed with samples and their corresponding domain distributions. Furthermore, we employ a domain augmentation strategy to refine its smoothness. Extensive experiments are constructed to demonstrate the effectiveness of GPDG.

Graphical abstract

Keywords

domain generalization / generalization process for domain generalization / generalization bound / Gaussian process / meta-function

References

[1] Krueger D, Caballero E, Jacobsen J H, Zhang A, Binas J, Zhang D, Le Priol R, Courville A. Out-of-distribution generalization via risk extrapolation (REx). In: Proceedings of the 38th International Conference on Machine Learning. 2021, 5815–5826

[2] Gulrajani I, Lopez-Paz D. In search of lost domain generalization. In: Proceedings of the 9th International Conference on Learning Representations. 2021, 1–29

[3] Wu K, Jia F, Han Y . Domain-specific feature elimination: multi-source domain adaptation for image classification. Frontiers of Computer Science, 2023, 17( 4): 174705

[4] Fan J, Wang Y, Guan H, Song C, Zhang Z . Toward few-shot domain adaptation with perturbation-invariant representation and transferable prototypes. Frontiers of Computer Science, 2022, 16( 3): 163347

[5] Li H, Pan S J, Wang S, Kot A C. Domain generalization with adversarial feature learning. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 5400–5409

[6] Hu H, Yang Y, Yin Y, Wu J . Metric learning for domain adversarial network. Frontiers of Computer Science, 2022, 16( 5): 165341

[7] Rame A, Dancette C, Cord M. Fishr: invariant gradient variances for out-of-distribution generalization. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 18347–18377

[8] Wang P, Zhang Z, Lei Z, Zhang L. Sharpness-aware gradient matching for domain generalization. In: Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 3769–3778

[9] Arjovsky M, Bottou L, Gulrajani I, Lopez-Paz D. Invariant risk minimization. 2019, arXiv preprint arXiv: 1907.02893

[10] Salaudeen O, Koyejo S. Causally inspired regularization enables domain general representations. In: Proceedings of the 27th International Conference on Artificial Intelligence and Statistics. 2024, 3124–3132

[11] Cao M, Chen S. Mixup-induced domain extrapolation for domain generalization. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 11168–11176

[12] Xie B, Chen Y, Wang J, Zhou K, Han B, Meng W, Cheng J. Enhancing evolving domain generalization through dynamic latent representations. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 16040–16048

[13] Lu W, Wang J, Li H, Chen Y, Xie X . Domain-invariant feature exploration for domain generalization. Transactions on Machine Learning Research, 2022, 2022: 1–20

[14] Zhou K, Yang Y, Qiao Y, Xiang T . MixStyle neural networks for domain generalization and adaptation. International Journal of Computer Vision, 2024, 132( 3): 822–836

[15] Cha J, Chun S, Lee K, Cho H C, Park S, Lee Y, Park S. SWAD: domain generalization by seeking flat minima. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 1716

[16] Li D, Yang Y, Song Y Z, Hospedales T. Learning to generalize: meta-learning for domain generalization. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018, 3490–3497

[17] Seeger M . PAC-Bayesian generalisation error bounds for Gaussian process classification. Journal of Machine Learning Research, 2002, 3: 233–269

[18] Sun B, Saenko K. Deep CORAL: correlation alignment for deep domain adaptation. In: Hua G, Jégou H. Computer Vision–ECCV 2016 Workshops. Cham: Springer, 2016, 443–450

[19] Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, March M, Lempitsky V . Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016, 17( 1): 2096–2030

[20] Nguyen T, Do K, Duong B, Nguyen T. Domain generalisation via risk distribution matching. 2023, arXiv preprint arXiv: 2310.18598

[21] Li B, Shen Y, Wang Y, Zhu W, Reed C, Li D, Keutzer K, Zhao H. Invariant information bottleneck for domain generalization. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. 2022, 7399–7407

[22] Yan S, Song H, Li N, Zou L, Ren L. Improve unsupervised domain adaptation with Mixup training. 2020, arXiv preprint arXiv:2001.00677

[23] Shankar S, Piratla V, Chakrabarti S, Chaudhuri S, Jyothi P, Sarawagi S. Generalizing across domains via cross-gradient training. In: Proceedings of the 6th International Conference on Learning Representations. 2018, 1–12

[24] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th International Conference on Machine Learning. 2017, 1126–1135

[25] Balaji Y, Sankaranarayanan S, Chellappa R. MetaReg: towards domain generalization using meta-regularization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 1006–1016

[26] Du Y, Xu J, Xiong H, Qiu Q, Zhen X, Snoek C G M, Shao L. Learning to learn with variational information bottleneck for domain generalization. In: Proceedings of the 16th Computer Vision–ECCV 2020. 2020, 200–216

[27] Du Y, Zhen X, Shao L, Snoek C G M. MetaNorm: learning to normalize few-shot batches across domains. In: Proceedings of the 9th International Conference on Learning Representations. 2020, 1–23

[28] Liu Y, Kothari P, van Delft B, Bellot-Gurlet B, Mordan T, Alahi A. TTT++: when does self-supervised test-time training fail or thrive? In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 1669

[29] Khoee A G, Yu Y, Feldt R . Domain generalization through meta-learning: A survey. Artificial Intelligence Review, 2024, 57( 10): 285

[30] Huang J, Qiu Q, Church K. Exploiting a zoo of checkpoints for unseen tasks. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 1486

[31] Holzleitner M, Pereverzyev S V, Zellinger W . Domain generalization by functional regression. Numerical Functional Analysis and Optimization, 2024, 45( 3): 259–281

[32] Rudner T G J, Sejdinovic D, Gal Y. Inter-domain deep Gaussian processes. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 767

[33] Achituve I, Navon A, Yemini Y, Chechik G, Fetaya E. GP-tree: a Gaussian process classifier for few-shot incremental learning. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 54–65

[34] Wilson A G, Hu Z, Salakhutdinov R, Xing E P. Deep kernel learning. In: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics. 2016, 370–378

[35] Snelson E, Ghahramani Z. Sparse Gaussian processes using pseudo-inputs. In: Proceedings of the 19th International Conference on Neural Information Processing Systems. 2005, 1257–1264

[36] Wilson A G, Hu Z, Salakhutdinov R, Xing E P. Stochastic variational deep kernel learning. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 2594–2602

[37] Polson N G, Scott J G, Windle J . Bayesian inference for logistic models using Pólya–Gamma latent variables. Journal of the American statistical Association, 2013, 108( 504): 1339–1349

[38] Lim H, Kim B, Choo J, Choi S. TTN: a domain-shift aware batch normalization in test-time adaptation. In: Proceedings of the 11th International Conference on Learning Representations. 2023, 1–19

[39] Zhu Y, Wu X, Qiang J, Yuan Y, Li Y . Representation learning via an integrated autoencoder for unsupervised domain adaptation. Frontiers of Computer Science, 2023, 17( 5): 175334

[40] Choi S, Yang S, Choi S, Yun S. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In: Proceedings of the 17th European Conference on Computer Vision. 2022, 440–458

[41] Sun Y, Wang X, Liu Z, Miller J, Efros A A, Hardt M. Test-time training with self-supervision for generalization under distribution shifts. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 856

[42] McAllester D A. Some PAC-Bayesian theorems. In: Proceedings of the 11th Annual Conference on Computational Learning Theory. 1998, 230–234

[43] Alquier P . User-friendly introduction to PAC-Bayes bounds. Foundations and Trends® in Machine Learning, 2024, 17( 2): 174–303

[44] Wang J, Lan C, Liu C, Ouyang Y, Qin T, Lu W, Chen Y, Zeng W, Philip S Y . Generalizing to unseen domains: a survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 8): 8052–8072

[45] Mohajerin Esfahani P, Kuhn D . Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations. Mathematical Programming, 2018, 171( 1): 115–166

[46] Yu K, Chu W. Gaussian process models for link analysis and transfer learning. In: Proceedings of the 21st International Conference on Neural Information Processing Systems. 2007, 1657–1664

[47] Dong X, Wen J . A pixel-based outlier-free motion estimation algorithm for scalable video quality enhancement. Frontiers of Computer Science, 2015, 9( 5): 729–740

[48] Shu Y, Cao Z, Wang C, Wang J, Long M. Open domain generalization with domain-augmented meta-learning. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 9619–9628

[49] Sun S, Zhao B, Mateen M, Chen X, Wen J . Mask guided diverse face image synthesis. Frontiers of Computer Science, 2022, 16( 3): 163311

[50] Shi Y, Seely J, Torr P, Siddharth N, Hannun A, Usunier N, Synnaeve G. Gradient matching for domain generalization. In: Proceedings of the 10th International Conference on Learning Representations. 2020, 1−19

[51] Eastwood C, Robey A, Singh S, von Kügelgen J, Hassani H, Pappas G J, Schölkopf B. Probable domain generalization via quantile risk minimization. In: Proceedings of the 36th International Conference on Neural Information Processing Systems.2022, 1261

[52] Shi Y, Seely J, Torr P, Siddharth N, Hannun A, Usunier N, Synnaeve G. Gradient matching for domain generalization. In: Proceedings of the 10th International Conference on Learning Representations.2022, 1−28

[53] Shi Y, Seely J, Torr P, Siddharth N, Hannun A, Usunier N, Synnaeve G. Gradient matching for domain generalization. In: Proceedings of the 10th International Conference on Learning Representations. 2023, 1–21

[54] Nam H, Lee H, Park J, Yoon W, Yoo D. Reducing domain gap by reducing style bias. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 8686–8695

[55] Vapnik V N. Statistical Learning Theory. New York: Wiley, 1998

[56] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16x16 words: transformers for image recognition at scale. In: Proceedings of the 9th International Conference on Learning Representations. 2021, 1–21

[57] Amit R, Meir R. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In: Proceedings of the 35th International Conference on Machine Learning. 2018, 205–214

[58] Albuquerque I, Monteiro J, Darvishi M, Falk T H, Mitliagkas I. Generalizing to unseen domains via distribution matching. 2019, arXiv preprint arXiv: 1911.00804

[59] Lu W, Wang W, Yidong J, Xie X. Towards optimization and model selection for domain generalization: a Mixup-guided solution. In: Proceedings of the KDD’23 Workshop on Causal Discovery, Prediction and Decision. 2023, 75–97

[60] Dai R, Zhang Y, Fang Z, Han B, Tian X. Moderately distributional exploration for domain generalization. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 272

RIGHTS & PERMISSIONS

Higher Education Press