IRFNet: implicit representation fusion for high-fidelity 3D facial reconstruction

Abstract

Existing three-dimensional (3D) face reconstruction methods struggle to capture high-frequency facial details, such as subtle expressions and fine skin textures, essential for accurate reconstruction and realistic user interaction. To address this limitation, we propose the implicit representation fusion network (IRFNet), a novel framework for precise facial geometry reconstruction. IRFNet integrates deformation-aware feature extraction and semantic facial segmentation, effectively combining local and global structural cues to optimize facial geometry accuracy. Additionally, a hybrid feature rendering mechanism enhances reconstruction consistency, particularly in complex environments. Compared to current approaches, IRFNet mitigates the geometric distortions inherent in explicit representations and better adapts to diverse facial morphologies and expression variations. Extensive experiments on real-world facial benchmarks demonstrate that IRFNet achieves state-of-the-art performance in 3D face reconstruction.

Keywords

A

References

[1] Zhang X, Yan C. E-MobileNeXt: face expression recognition model based on improved MobileNeXt. Optoelectronics letters, 2024, 20(2): 122-128 J]

[2] Li J, Chen Z, Li G, et al. . Automatic classification of ASD children using appearance-based features from videos. Neurocomputing, 2022, 470: 40-50 J]

[3] Xie E, Zhang Z, Zhang G, et al. . Light bottle transformer based large scale point cloud classification. Optoelectronics letters, 2023, 19(6): 377-384 J]

[4] Li M. Reconstruction meets recognition: exploring face identity embeddings. 2024, York, University of York[D]

[5] Tewari A, Zollhofer M, Kim H, et al. . MOFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. Proceedings of the IEEE International Conference on Computer Vision Workshops, October 22–29, 2017, Venice, Italy. 2017, New York, IEEE12741283[C]

[6] Jiang L, Zhang J, Deng B, et al. . 3D face reconstruction with geometry details from a single image. IEEE transactions on image processing, 2018, 27(10): 4756-4770 J]

[7] Qin D, Saito J, Aigerman N, et al. . Neural face rigging for animating and retargeting facial meshes in the wild. ACM SIGGRAPH 2023 Conference Proceedings, August 6–10, 2023, California, USA. 2023, New York, IEEE111[C]

[8] An S, Xu H, Shi Y, et al. . Panohead: geometry-aware 3D full-head synthesis in 360deg. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18–22, 2023, Vancouver, Canada. 2023, New York, IEEE2095020959[C]

[9] Gecer B, Ploumpis S, Kotsia I, et al. . GANFIT: generative adversarial network fitting for high fidelity 3D face reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 15–21, 2019, California, USA. 2019, New York, IEEE11551164[C]

[10] Wang Z, Zhu X, Zhang T, et al. . 3D face reconstruction with the geometric guidance of facial part segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 17–21, 2024, Washington, USA. 2024, New York, IEEE16721682[C]

[11] Wang T C, Liu M Y, Zhu J Y, et al. . High-resolution image synthesis and semantic manipulation with conditional GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 18–22, 2018, Utah, USA. 2018, New York, IEEE87988807[C]

[12] Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision, October 11–14, 2016, Amsterdam, The Netherland. 2016, Cham, Springer International Publishing694711[C]

[13] Guo J, Zhu X, Yang Y, et al. . Towards fast, accurate and stable 3D dense face alignment. European Conference on Computer Vision, August 23–28, 2020, Glasgow, UK. 2020, Cham, Springer International Publishing152168[C]

[14] Shang J, Shen T, Li S, et al. . Self-supervised monocular 3D face reconstruction by occlusion-aware multi-view geometry consistency. European Conference on Computer Vision, August 23–28, 2020, Glasgow, UK. 2020, Cham, Springer International Publishing5370[C]

[15] Dib A, Thebault C, Ahn J, et al. . Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. Proceedings of the IEEE/CVF International Conference on Computer Vision, October 11–17, 2021, Montreal, QC, Canada. 2021, New York, IEEE1281912829[C]

[16] Wu C Y, Xu Q, Neumann U. Synergy between 3DMM and 3D landmarks for accurate 3D facial geometry. 2021 International Conference on 3D Vision (3DV), December 1–3, 2021, London, UK. 2021, New York, IEEE453463[C]

[17] Zhang H, Tian Y, Zhang Y, et al. . PyMAF-X: towards well-aligned full-body model regression from monocular images. IEEE transactions on pattern analysis and machine intelligence, 2023, 45(10): 12287-12303 J]

[18] Yang W, Zhao Y, Yang B, et al. . Learning 3D face reconstruction from the cycle-consistency of dynamic faces. IEEE transactions on multimedia, 2023, 26: 3663-3675 J]

[19] Zainuddin Z, Tahir Z. Monocular 3D face reconstruction using 3D morphable model and elasticface. IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT), July 20–22, 2023, Bali, Indonesia. 2023, New York, IEEE320325[C]

[20] Zhou P, Liu X, Lu J, et al. . Shape attention module for regularizing 3D face reconstruction. International Conference on Wavelet Analysis and Pattern Recognition (ICWAPR), November 17–19, 2023, Guilin, China. 2023, New York, IEEE1318[C]

[21] Dib A, Ahn J, Thebault C, et al. . S2F2: self-supervised high fidelity face reconstruction from monocular image. IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), January 5–8, 2023, Washington, USA. 2023, New York, IEEE18[C]

[22] Deng Y, Yang J, Xu S, et al. . Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, June 16–20, 2019, Long Beach, USA. 2019, New York, IEEE[C]

[23] Hong F T, Zhang L, Shen L, et al. . Depth-aware generative adversarial network for talking head video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–24, 2022, New Orleans, USA. 2022, New York, IEEE33973406[C]

[24] Zeng B, Liu B, Li H, et al. . FNeVR: neural volume rendering for face animation. Advances in neural information processing systems, 2022, 35: 22451-22462 J]

[25] Zeng B, Liu X, Gao S, et al. . Face animation with an attribute-guided diffusion model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18–22, 2023, Vancouver, Canada. 2023, New York, IEEE628637[C]

RIGHTS & PERMISSIONS

Tianjin University of Technology