Perceptual point cloud quality assessment for immersive metaverse experience

Abstract

Perceptual quality assessment for point cloud is critical for immersive metaverse experience and is a challenging task. Firstly, because point cloud is formed by unstructured 3D points that makes the topology more complex. Secondly, the quality impairment generally involves both geometric attributes and color properties, where the measurement of the geometric distortion becomes more complex. We propose a perceptual point cloud quality assessment model that follows the perceptual features of Human Visual System (HVS) and the intrinsic characteristics of the point cloud. The point cloud is first pre-processed to extract the geometric skeleton keypoints with graph filtering-based re-sampling, and local neighboring regions around the geometric skeleton keypoints are constructed by K-Nearest Neighbors (KNN) clustering. For geometric distortion, the Point Feature Histogram (PFH) is extracted as the feature descriptor, and the Earth Mover's Distance (EMD) between the PFHs of the corresponding local neighboring regions in the reference and the distorted point clouds is calculated as the geometric quality measurement. For color distortion, the statistical moments between the corresponding local neighboring regions are computed as the color quality measurement. Finally, the global perceptual quality assessment model is obtained as the linear weighting aggregation of the geometric and color quality measurement. The experimental results on extensive datasets show that the proposed method achieves the leading performance as compared to the state-of-the-art methods with less computing time. Meanwhile, the experimental results also demonstrate the robustness of the proposed method across various distortion types. The source codes are available at https://github.com/llsurreal919/PointCloudQualityAssessment

Keywords

Metaverse / Point cloud / Quality assessment / Point feature histogram / Earth mover's distance

CRediT authorship contribution statement

Baoping Cheng: Writing - review & editing, Writing - original draft, Visualization, Validation, Supervision, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Lei Luo: Visualization, Validation, Supervision, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Ziyang He: Writing - review & editing, Writing - original draft, Visualization, Validation, Supervision, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Ce Zhu: Visualization, Validation, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Xiaoming Tao: Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant (62171257, U22B2001, U19A2052, 62020106011, 62061015), in part by the Natural Science Foundation of Chongqing under Grant (2023NSCQMSX2930), in part by the Youth Innovation Group Support Program of ICE Discipline of CQUPT under Grant (SCIE-QN-2022-05), and in part by the Graduate Scientifc Research and Innovation Project of Chongqing under Grant (CYS22469).

References

[1] Q. Liu, H. Su, Z. Duanmu, W. Liu, Z. Wang,Perceptual quality assessment of colored 3D point clouds, IEEE Trans. Vis. Comput. Graph. 29 (8) (2023) 3642-3655.

[2] H. Liu, H. Yuan, Q. Liu, J. Hou, J. Liu,A comprehensive study and comparison of core technologies for mpeg 3-d point cloud compression, IEEE Trans. Broadcast. 66 (3) (2020) 701-717.

[3] L. Zhao, K.-K. Ma, X. Lin, W. Wang, J. Chen, Real-time Lidar point cloud compression using bi-directional prediction and range-adaptive floating-point coding, IEEE Trans. Broadcast. 68 (3) (2022) 620-635.

[4] A.F. Guarda, N.M. Rodrigues, F. Pereira,Point cloud geometry scalable coding with a single end-to-end deep learning model, in: 2020 IEEE International Conference on Image Processing (ICIP), IEEE, 2020, pp. 3354-3358.

[5] M. Yang, Z. Luo, M. Hu, M. Chen, D. Wu, A comparative measurement study of point cloud-based volumetric video codecs, IEEE Trans. Broadcast. 69 (3) (2023) 715-726.

[6] X. Jia, S. Yang, Y. Wang, J. Zhang, Y. Peng, S. Chen,Dual-view 3D reconstruction via learning correspondence and dependency of point cloud regions, IEEE Trans. Image Process. 31 (2022) 6831-6846.

[7] X. Qiao, Y. Feng, S. Liu, T. Shan, R. Tao, Radar point clouds processing for human activity classification using convolutional multilinear subspace learning, IEEE Trans. Geosci. Remote Sens. 60 (2022) 1-17.

[8] F. Wang, W. Li, D. Xu, Cross-dataset point cloud recognition using deep-shallow domain adaptation network, IEEE Trans. Image Process. 30 (2021) 7364-7377.

[9] J. Zhang, G. Fang, B. Wang, X. Zhou, Q. Pei, C. Chen,Monocular 3D object detection with pseudo-lidar confidence sampling and hierarchical geometric feature extraction in 6G network, Digit. Commun. Netw. 9 (4) (2023) 827-835.

[10] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600-612.

[11] Z. Zhang, W. Sun, X. Min, W. Zhu, T. Wang, W. Lu, G. Zhai,A no-reference deep learning quality assessment method for super-resolution images based on frequency maps, in: 2022 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, 2022, pp. 3170-3174.

[12] S. Chen, D. Tian, C. Feng, A. Vetro, J. Kovačević, Fast resampling of three-dimensional point clouds via graphs, IEEE Trans. Signal Process. 66 (3) (2018) 666-681.

[13] R.B. Rusu, Z.C. Marton, N. Blodow, M. Beetz,Learning informative point classes for the acquisition of object model maps, in:2008 10th International Conference on Control, Automation, Robotics and Vision, 2008, pp. 643-650.

[14] O. Pele, M. Werman,A linear time histogram metric for improved sift matching, in:Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Mar-seille, France, October 12-18, 2008, in:Proceedings, Part III 10, Springer, Marseille, France, 2008, pp. 495-508.

[15] Q. Yang, H. Chen, Z. Ma, Y. Xu, R. Tang, J. Sun,Predicting the perceptual quality of point cloud: a 3D-to-2D projection-based exploration, IEEE Trans. Multimed. 23 (2020) 3877-3891.

[16] S. Perry, H.P. Cong, L.A. da Silva Cruz, J. Prazeres, M. Pereira, A. Pinheiro, E. Dumic, E. Alexiou, T. Ebrahimi, Quality evaluation of static point clouds encoded using MPEG codecs, in: 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 3428-3432.

[17] A. Javaheri, C. Brites, F.M.B. Pereira, J.M. Ascenso, Point cloud rendering after cod-ing: impacts on subjective and objective quality, IEEE Trans. Multimed. 23 (2021) 4049-4064.

[18] E. Alexiou, E. Upenik, T. Ebrahimi, Towards subjective quality assessment of point cloud imaging in augmented reality, in: 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), 2017, pp. 1-6.

[19] P. Cignoni, C. Rocchini, R. Scopigno, Metro: Measuring Error on Simplified Surfaces, Computer Graphics Forum, vol. 17, Wiley Online Library, 1998, pp. 167-174.

[20] D. Tian, H. Ochimizu, C. Feng, R. Cohen, A. Vetro, Geometric distortion metrics for point cloud compression, in: 2017 IEEE International Conference on Image Process-ing (ICIP), 2017, pp. 3460-3464.

[21] MPEG reference software, http://mpegx.int-evry.fr/software/MPEG/PCC/TM/mpeg-pcc-dmetric. (Accessed 13 September 2023).

[22] A. Javaheri, C. Brites, F. Pereira, J. Ascenso,A generalized Hausdorff distance based quality metric for point cloud geometry, in:2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1-6.

[23] E. Alexiou, T. Ebrahimi, Point cloud quality assessment metric based on angular similarity, in: 2018 IEEE International Conference on Multimedia and Expo (ICME), 2018, pp. 1-6.

[24] S. Perry, A. Pinheiro, E. Dumic, L.A. da Silva Cruz, Study of subjective and objective quality evaluation of 3D point cloud data by the JPEG committee, Electron. Imaging 2019 (10) (2019) 312.

[25] Y. Xu, Q. Yang, L. Yang, J.-N. Hwang, EPES: point cloud quality modeling using elastic potential energy similarity, IEEE Trans. Broadcast. 68 (1) (2021) 33-42.

[26] A. Laazoufi, M. El Hassouni,Saliency-based point cloud quality assessment method using aware features learning, in: 2022 9th International Conference on Wireless Networks and Mobile Communications (WINCOM), IEEE, 2022, pp. 1-5.

[27] G. Meynet, J. Digne, G. Lavoué,PC-MSDM: a quality metric for 3D point clouds,in:2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019, pp. 1-3.

[28] A. Javaheri, C. Brites, F. Pereira, J. Ascenso, Mahalanobis based point to distribution metric for point cloud geometry quality evaluation, IEEE Signal Process. Lett. 27 (2020) 1350-1354.

[29] G. Meynet, Y. Nehmé, J. Digne, G. Lavoué,PCQM: a full-reference quality metric for colored 3D point clouds,in:2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1-6.

[30] Q. Yang, Z. Ma, Y. Xu, Z. Li, J. Sun, Inferring point cloud quality via graph similarity, IEEE Trans. Pattern Anal. Mach. Intell. 44 (6) (2022) 3015-3029.

[31] E. Alexiou, T. Ebrahimi,Exploiting user interactivity in quality assessment of point cloud imaging, in:2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019, pp. 1-6.

[32] L. Zhang, L. Zhang, X. Mou, D. Zhang, FSIM: a feature similarity index for image quality assessment, IEEE Trans. Image Process. 20 (8) (2011) 2378-2386.

[33] R.B. Rusu, N. Blodow, M. Beetz,Fast point feature histograms (FPFH) for 3D regis-tration, in: 2009 IEEE International Conference on Robotics and Automation, 2009, pp. 3212-3217.

[34] O. Pele, M. Werman, Fast and robust Earth mover’s distances, in: 2009 IEEE 12th International Conference on Computer Vision, 2009, pp. 460-467.

[35] J.-M. Geusebroek, R. van den Boomgaard, A.W.M. Smeulders, H. Geerts, Color in-variance, IEEE Trans. Pattern Anal. Mach. Intell. 23 (12) (2001) 1338-1350.

[36] S. Schwarz, D. Flynn,Common test conditions for point cloud compression, in: Doc-ument N18175, ISO/IEC JTC1/SC29/WG11, MPEG, Marrakesh, Morocco, 2019.

[37] VQEG, Final report from the video quality experts group on the validation of ob-jective models of video quality assessment, http://www.its.bldrdoc.gov/vqeg/vqeg-home.aspx.assessment. (Accessed 13 September 2023).