Inverse distillation for source-free unsupervised domain adaptation

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

Unsupervised domain adaptation (UDA) aims to leverage labeled source domain knowledge to improve the target domain’s performance. Source-free domain adaptation (SFDA), a recent research focus, addresses challenges such as data privacy by relying solely on a pretrained source model and unlabeled target data, thus eliminating the need for direct access to source domain data. Although many studies have proposed methods such as generating a source domain, using a proxy source domain, or using pseudo-label training, these approaches directly fine-tune the source model, overlooking the excessive bias towards the source domain data in SFDA. The source domain model contains numerous source domain-specific features, and directly updating it to shift towards the target domain is hindered by these domain-specific features. To address this issue, we propose an inverse distillation-based SFDA method. By constructing an initial target domain model, the method extracts pure target domain features and distills them back into the source model, facilitating a smoother transition towards the target domain. Additionally, it identifies stable and active target samples from both structural and scoring perspectives, applying distinct matching strategies for pseudo-label selection. Extensive experiments and ablation studies on public datasets (Digits, Office-31, Office-Home and VisDA-2017) demonstrate the superior performance of our approach in SFDA tasks.

Keywords

source-free domain adaptation / knowledge distillation / image classification

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] V.K.Kurmi,V.K.Subramanian,V.P.Namboodiri, Domain impression:A source data free domain adaptation method,in 2021 IEEE Winter Conference on Applications of Computer Vision(WACV),(2021), 615-625. https://doi.org/10.1109/WACV48630.2021.00066

[2] Y.Hou,L.Zheng, Visualizing adapted knowledge in domain transfer,in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),(2021), 13819-13828. https://doi.org/10.1109/CVPR46437.2021.01361

[3] Y.Ding,L.Sheng,J.Liang,A.Zheng,R.He, Proxymix:Proxy-based mixup training with label refinery for source-free domain adaptation, Neural Netw., 167(2023), 92-103. https://doi.org/10.1016/j.neunet.2023.08.005

[4] Y.Ganin,V.Lempitsky, Unsupervised domain adaptation by backpropagation,in Proceedings of the 32nd International Conference on Machine Learning, 37(2015), 1180-1189.

[5] H.Liu,J.Wang,M.Long, Cycle self-training for domain adaptation,in Advances in Neural Information Processing Systems, 34(2021), 22968-22981.

[6] W.Zhou,G.Guan,Y.Yi,W.Cui,Y.Chen, Progressive pseudo-labels enhancement for source-free domain adaptation medical image segmentation, Biomed.Signal Process.Control, 109(2025), 108053. https://doi.org/10.1016/j.bspc.2025.108053

[7] M.Long,Y.Cao,J.Wang,M.Jordan, Learning transferable features with deep adaptation networks,in Proceedings of the 32nd International Conference on Machine Learning, 37(2015), 97-105.

[8] S.Li,S.Song,G.Huang,Z.Ding,C.Wu, Domain invariant and class discriminative feature learning for visual domain adaptation, IEEE Trans.Image Process., 27(2018), 4260-4273. https://doi.org/10.1109/TIP.2018.2839528

[9] B.Xu,Z.Zeng,C.Lian,Z.Ding, Few-shot domain adaptation via mixup optimal transport, IEEE Trans.Image Process., 31(2022), 2518-2528. https://doi.org/10.1109/TIP.2022.3157139

[10] Y.H.Tsai,W.C.Hung,S.Schulter,K.Sohn,M.H.Yang,M.Chandraker, Learning to adapt structured output space for semantic segmentation,in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),(2018), 7472-7481. https://doi.org/10.1109/CVPR.2018.00780

[11] Y.Zou,Z.Yu,B.V.K.Kumar,J.Wang, Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,in Proceedings of the European Conference on Computer Vision(ECCV),(2018), 289-305.

[12] J.Peng,Y.Huang,W.Sun,N.Chen,Y.Ning,Q.Du, Domain adaptation in remote sensing image classification:A survey, IEEE J.Sel.Top.Appl.Earth Obs.Remote Sens., 15(2022), 9842-9859. https://doi.org/10.1109/JSTARS.2022.3220875

[13] Y.Huang,J.Peng,W.Sun,N.Chen,Q.Du,Y.Ning, et al., Two-branch attention adversarial domain adaptation network for hyperspectral image classification, IEEE Trans.Geosci.Remote Sens., 60(2022), 1-13. https://doi.org/10.1109/TGRS.2022.3215677

[14] Y.Ning,J.Peng,L.Sun,Y.Huang,W.Sun,Q.Du, Adaptive local discriminant analysis and distribution matching for domain adaptation in hyperspectral image classification, IEEE J.Sel.Top.Appl.Earth Obs.Remote Sens., 15(2022), 4797-4808. https://doi.org/10.1109/JSTARS.2022.3181577

[15] L.Zha,Y.Chen,P.Zhou,Y.Zhang, Intensifying the consistency of pseudo label refinement for unsupervised domain adaptation person re-identification,in 2023 IEEE International Conference on Multimedia and Expo(ICME),(2023), 1547-1552. https://doi.org/10.1109/ICME55011.2023.00267

[16] M.Long,H.Zhu,J.Wang,M.I.Jordan, Deep transfer learning with joint adaptation networks,in Proceedings of the 34th International Conference on Machine Learning , 70(2017), 2208-2217.

[17] K.Mei,C.Zhu,J.Zou,S.Zhang, Instance adaptive self-training for unsupervised domain adaptation,in Computer Vision-ECCV 2020:16th European Conference ,(2020), 415-430. https://doi.org/10.1007/978-3-030-58574-725

[18] W.Zhang,W.Ouyang,W.Li,D.Xu, Collaborative and adversarial network for unsupervised domain adaptation,in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,(2018), 3801-3809. https://doi.org/10.1109/CVPR.2018.00400

[19] Z.Zheng,Y.Yang, Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation, Int.J.Comput.Vis., 129(2021), 1106-1120. https://doi.org/10.1007/s11263-020-01395-y

[20] M.Shao,S.Chen,F.Wang,L.Zhang, Adaptive pseudo-label threshold for source-free domain adaptation, Neural Comput.Appl., 37(2025), 1875-1887. https://doi.org/10.1007/s00521-024-10697-y

[21] Z.Qiu,Y.Zhang,H.Lin,S.Niu,Y.Liu,Q.Du, et al., Source-free domain adaptation via avatar prototype generation and adaptation,in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence(IJCAI-21) ,(2021), 2921-2927. https://doi.org/10.24963/ijcai.2021/402

[22] R.Li,Q.Jiao,W.Cao,H.S.Wong,S.Wu, Model adaptation:Unsupervised domain adaptation without source data,in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) ,(2020), 9638-9647. https://doi.org/10.1109/CVPR42600.2020.00966

[23] H.Yan,Y.Guo,C.Yang, Source-free unsupervised domain adaptation with surrogate data generation,in BMVC ,(2021), 198. https://doi.org/10.5244/c.35.324

[24] Y.Du,H.Yang,M.Chen,H.Luo,J.Jiang,Y.Xin, et al., Generation,augmentation,and alignment:A pseudo-source domain based method for source-free domain adaptation, Mach.Learn., 113(2024), 3611-3631. https://doi.org/10.1007/s10994-023-06432-8

[25] J.Liang,D.Hu,J.Feng, Do we really need to access the source data?source hypothesis transfer for unsupervised domain adaptation,in Proceedings of the 37th International Conference on Machine Learning, 119(2020), 6028-6039.

[26] S.Yang,J.Van de Weijer,L.Herranz,S.Jui, Exploiting the intrinsic neighborhood structure for source-free domain adaptation,in Advances in Neural Information Processing Systems, 34(2021), 29393-29405.

[27] H.Xia,H.Zhao,Z.Ding, Adaptive adversarial network for source-free domain adaptation,in 2021 IEEE/CVF International Conference on Computer Vision(ICCV) ,(2021), 8990-8999. https://doi.org/10.1109/iccv48922.2021.00888

[28] J.Huang,D.Guan,A.Xiao,S.Lu, Model adaptation:Historical contrastive learning for unsupervised domain adaptation without source data,in Advances in Neural Information Processing Systems, 34(2021), 3635-3649.

[29] S.Qu,G.Chen,J.Zhang,Z.Li,W.He,D.Tao, General class-balanced multicentric dynamic prototype pseudo-labeling for source-free domain adaptation, Int.J.Comput.Vis., 133(2025), 3327-3348. https://doi.org/10.1007/s11263-024-02335-w

[30] J.Liang,D.Hu,Y.Wang,R.He,J.Feng, Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer, IEEE Trans.Pattern Anal.Mach.Intell., 44(2021), 8602-8617. https://doi.org/10.1109/tpami.2021.3103390

[31] K.Sohn,D.Berthelot,N.Carlini,Z.Zhang,H.Zhang,C.A.Raffel, et al., Fixmatch:Simplifying semi-supervised learning with consistency and confidence,in Advances in Neural Information Processing Systems, 33(2020), 596-608.

[32] B.Zhang,Y.Wang,W.Hou,H.Wu,J.Wang,M.Okumura, et al., Flexmatch:Boosting semi-supervised learning with curriculum pseudo labeling,in Advances in Neural Information Processing Systems, 34(2021), 18408-18419.

[33] C.Chen,W.Xie,W.Huang,Y.Rong,X.Ding,Y.Huang, et al., Progressive feature alignment for unsupervised domain adaptation,in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) ,(2019), 627-636. https://doi.org/10.1109/CVPR.2019.00072

[34] K.Tanwisuth,X.Fan,H.Zheng,S.Zhang,H.Zhang,B.Chen, et al., A prototype-oriented framework for unsupervised domain adaptation,in Advances in Neural Information Processing Systems, 34(2021), 17194-17208.

[35] Y.Yang,S.Chen,X.Li,L.Xie,Z.Lin,D.Tao, Inducing neural collapse in imbalanced learning:Do we really need a learnable classifier at the end of deep neural network?,in Advances in Neural Information Processing Systems, 35(2022), 37991-38002.

[36] J.Hoffman,E.Tzeng,T.Park,J.Y.Zhu,P.Isola,K.Saenko, et al., Cycada:Cycle-consistent adversarial domain adaptation,in International Conference on Machine Learning ,(2018), 1989-1998.

[37] K.Saenko,B.Kulis,M.Fritz,T.Darrell, Adapting visual category models to new domains,in Computer Vision-ECCV 2010:11th European Conference,(2010), 213-226.https://doi.org/10.1007/978-3-642-15561-1 16

[38] H.Venkateswara,J.Eusebio,S.Chakraborty,S.Panchanathan, Deep hashing network for unsupervised domain adaptation,in 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),(2017), 5385-5394. https://doi.org/10.1109/CVPR.2017.572

[39] X.Peng,B.Usman,N.Kaushik,J.Hoffman,D.Wang,K.Saenko, Visda:The visual domain adaptation challenge,preprint, arXiv:1710.06924.

[40] A.Paszke,S.Gross,F.Massa,A.Lerer,J.Bradbury,G.Chanan, et al., Pytorch:An imperative style,high-performance deep learning library,in Advances in Neural Information Processing Systems, 32(2019), 1-12.

[41] K.He,X.Zhang,S.Ren,J.Sun, Deep residual learning for image recognition,in 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),(2016), 770-778. https://doi.org/10.1109/CVPR.2016.90

[42] J.Deng,W.Dong,R.Socher,L.J.Li,K.Li,L.Fei-Fei, Imagenet:A large-scale hierarchical image database,in 2009 IEEE Conference on Computer Vision and Pattern Recognition,(2009), 248-255. https://doi.org/10.1109/CVPR.2009.5206848

[43] E.Tzeng,J.Hoffman,K.Saenko,T.Darrell, Adversarial discriminative domain adaptation,in 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),(2017), 2962-2971. https://doi.org/10.1109/CVPR.2017.316

[44] K.Saito,Y.Ushiku,T.Harada,K.Saenko, Adversarial dropout regularization,preprint, arXiv:1711.01575.

[45] Z.Deng,Y.Luo,J.Zhu, Cluster alignment with a teacher for unsupervised domain adaptation,in 2019 IEEE/CVF International Conference on Computer Vision(ICCV),(2019), 9943-9952. https://doi.org/10.1109/ICCV.2019.01004

[46] C.Y.Lee,T.Batra,M.H.Baig,D.Ulbricht, Sliced wasserstein discrepancy for unsupervised domain adaptation,in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),(2019), 10277-10287. https://doi.org/10.1109/CVPR.2019.01053

[47] Y.Kim,D.Cho,K.Han,P.Panda,S.Hong, Domain adaptation without source data, IEEE Trans.Artif.Intell., 2(2021), 508-518. https://doi.org/10.1109/TAI.2021.3110179

[48] S.Roy,M.Trapp,A.Pilzer,J.Kannala,N.Sebe,E.Ricci, et al., Uncertainty-guided source-free domain adaptation,in Computer Vision-ECCV 2022,(2022), 537-555. https://doi.org/10.1007/978-3-031-19806-931

[49] S.Yang,Y.Wang,L.Herranz,S.Jui,J.Van de Weijer, Casting a BAIT for offline and online source-free domain adaptation, Comput.Vis.Image Underst., 234(2023), 103747. https://doi.org/10.1016/j.cviu.2023.103747

[50] T.Chu,Y.Liu,J.Deng,W.Li,L.Duan, Denoised maximum classifier discrepancy for source-free unsupervised domain adaptation,in Proceedings of the AAAI Conference on Artificial Intelligence, 36(2022), 472-480. https://doi.org/10.1609/aaai.v36i1.19925

[51] N.Ding,Y.Xu,Y.Tang,C.Xu,Y.Wang,D.Tao, Source-free domain adaptation via distribution estimation,in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),(2022), 7202-7212. https://doi.org/10.1109/CVPR52688.2022.00707

[52] F.Wang,Z.Han,Y.Gong,Y.Yin, Exploring domain-invariant parameters for source free domain adaptation,in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) ,(2022), 7141-7150. https://doi.org/10.1109/CVPR52688.2022.00701

[53] J.He,L.Wu,C.Tao,F.Lv, Source-free domain adaptation with unrestricted source hypothesis, Pattern Recognit., 149(2024), 110246. https://doi.org/10.1016/j.patcog.2023.110246

[54] M.Jing,J.Li,K.Lu,L.Zhu,H.T.Shen, Visually source-free domain adaptation via adversarial style matching, IEEE Trans.Image Process., 33(2024), 1032-1044. https://doi.org/10.1109/TIP.2024.3353539

[55] L.Zhou,N.Li,M.Ye,X.Zhu,S.Tang, Source-free domain adaptation with class prototype discovery, Pattern Recognit., 145(2024), 109974. https://doi.org/10.1016/j.patcog.2023.109974

[56] S.Yang,Y.Wang,K.Wang,S.Jui,J.Van de Weijer, Attracting and dispersing:A simple approach for source-free domain adaptation,in Advances in Neural Information Processing Systems, 35(2022), 5802-5815. https://doi.org/10.52202/068431-0420

[57] J.Lee,D.Jung,J.Yim,S.Yoon, Confidence score for source-free unsupervised domain adaptation,in International Conference on Machine Learning, 162(2022), 12365-12377.

[58] L.Yi,G.Xu,P.Xu,J.Li,R.Pu,C.Ling, et al., When source-free domain adaptation meets learning with noisy labels,preprint, arXiv:2301.13381.

[59] M.Litrico,A.Del Bue,P.Morerio, Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation,in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),(2023), 7640-7650. https://doi.org/10.1109/cvpr52729.2023.00738

[60] S.Tang,A.Chang,F.Zhang,X.Zhu,M.Ye,C.Zhang, Source-free domain adaptation via target prediction distribution searching, Int.J.Comput.Vis., 132(2024), 654-672. https://doi.org/10.1007/s11263-023-01892-w

[61] R.R.Selvaraju,M.Cogswell,A.Das,R.Vedantam,D.Parikh,D.Batra, Grad-CAM:Visual explanations from deep networks via gradient-based localization, Int.J.Comput.Vis., 128(2020), 336-359. https://doi.org/10.1007/s11263-019-01228-7