Transparent Object Reconstruction Based on Compressive Sensing and Super-Resolution Convolutional Neural Network

Abstract

The detection and reconstruction of transparent objects have remained challenging due to the absence of their features and variations in the local features with variations in illumination. In this paper, both compressive sensing (CS) and super-resolution convolutional neural network (SRCNN) techniques are combined to capture transparent objects. With the proposed method, the transparent object’s details are extracted accurately using a single pixel detector during the surface reconstruction. The resultant images obtained from the experimental setup are low in quality due to speckles and deformations on the object. However, the implemented SRCNN algorithm has obviated the mentioned drawbacks and reconstructed images visually plausibly. The developed algorithm locates the deformities in the resultant images and improves the image quality. Additionally, the inclusion of compressive sensing minimizes the measurements required for reconstruction, thereby reducing image post-processing and hardware requirements during network training. The result obtained indicates that the visual quality of the reconstructed images has increased from a structural similarity index (SSIM) value of 0.2 to 0.53. In this work, we demonstrate the efficiency of the proposed method in imaging and reconstructing transparent objects with the application of a compressive single pixel imaging technique and improving the image quality to a satisfactory level using the SRCNN algorithm.

Keywords

Transparent object imaging / single-pixel imaging / compressive sensing / total-variation minimization / SRCNN algorithm

References

[1] Kutulakos K N, Steger E. A theory of refractive and specular 3D shape by light-path triangulation. International Journal of Computer Vision, 2008, 76(1): 13-29.

[2] V. Chari and P. Sturm, “A theory of refractive photo-light-path triangulation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Oregon, USA, 2013, pp. 1438–1445.

[3] Mathai A, Guo N, Liu D, Wang X. 3D transparent object detection and reconstruction based on passive mode single-pixel imaging. Sensors, 2020, 20(15): 4211.

[4] Tian X, Liu R, Wang Z, Ma J. High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision. Information Fusion, 2021, 77, 19-28.

[5] R. Rantoson, C. Stolz, D. Fofi, and F. Mériaudeau, “3D reconstruction of transparent objects exploiting surface fluorescence caused by UV irradiation,” in 2010 IEEE International Conference on Image Processing, Hong Kong, China, 2010, pp. 2965–2968.

[6] K. Han, K. Y. K. Wong, and M. Liu, “A fixed viewpoint approach for dense reconstruction of transparent objects,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015, pp. 4001–4008.

[7] C. J. Phillips, M. Lecce, and K. Daniilidis, “Seeing glassware: from edge detection to pose estimation and shape recovery,” in Robotics: Science and Systems, Michigan, USA, 2016: 3.

[8] G. Georgakis, M. A. Reza, A. Mousavian, P. H. Le, and J. KoŠecká, “Multiview RGB-D dataset for object instance detection,” in 2016 Fourth International Conference on 3D Vision (3DV), Stanford, USA, 2016, pp. 426–434.

[9] Wang Z, Zhou Q, Shuang Y. Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions. Measurement, 2020, 151, 107114.

[10] Wang Z. Review of real-time three-dimensional shape measurement techniques. Measurement, 2020, 156, 107624.

[11] Eren G, Aubreton O, Meriaudeau F, Secades L A S, Fofi D, Naskali A T, . Scanning from heating: 3D shape estimation of transparent objects from local surface heating. Optics Express, 2009, 17(14): 11457-11468.

[12] A. Brahm, C. Rößler, P. Dietrich, S. Heist, P. Kühmstedt, and G. Notni, “Non-destructive 3D shape measurement of transparent and black objects with thermal fringes,” in Dimensional Optical Metrology and Inspection for Practical Applications, vol. 9868: International Society for Optics and Photonics, Baltimore, Maryland, USA, 2016, pp. 98680C.

[13] U. Klank, D. Carton, and M. Beetz, “Transparent object detection and reconstruction on a mobile platform,” in 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 2011, pp. 5971–5978.

[14] Jiang H, Zhai H, Xu Y, Li X, Zhao H. 3D shape measurement of translucent objects based on Fourier single-pixel imaging in projector-camera system. Optics Express, 2019, 27(23): 33564-33574.

[15] Atcheson B, Ihrke I, Heidrich W, Tevs A, Bradley D, Magnor M, . Time-resolved 3d capture of non-stationary gas flows. ACM Transactions on Graphics, 2008, 27(5): 1-9.

[16] X. Fu, Y. Sun, M. LiWang, Y. Huang, X. P. Zhang, and X. Ding, “A novel retinex based approach for image enhancement with illumination adjustment,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, 2014, pp: 1190–1194.

[17] Glasbey C A. An analysis of histogram-based thresholding algorithms. CVGIP: Graphical Models and Image Processing, 1993, 55(6): 532-537.

[18] Guo X, Li Y, Ling H. LIME: low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 2016, 26(2): 982-993.

[19] Yang W, Zhang X, Tian Y, Wang W, Xue J H, Liao Q. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia, 2019, 21(12): 3106-3121.

[20] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, et al., “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015, pp. 1–9.

[21] Dong C, Loy C C, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307.

[22] Zhang L, Zhang Y, Zhang Z, Shen J, Wang H. Real-time water surface object detection based on improved faster R-CNN. Sensors, 2019, 19(16): 3523.

[23] P. J. Lai and C. S. Fuh, “Transparent object detection using regions with convolutional neural network,” in IPPR Conference on Computer Vision, Graphics, and Image Processing, Taiwan, China, 2015, pp. 2.

[24] E. Xie, W. Wang, W. Wang, M. Ding, C. Shen, and P. Luo, “Segmenting transparent objects in the wild,” in Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, 2020, pp. 696–711.

[25] M. P. Khaing and M. Masayuki, “Transparent object detection using convolutional neural network,” in International Conference on Big Data Analysis and Deep Learning Applications, Miyazaki, Japan, 2018, pp. 86–93.

[26] S. Song and H. Shim, “Depth reconstruction of translucent objects from a single time-of-flight camera using deep residual networks,” in Asian Conference on Computer Vision, Perth, Australia, 2018, pp. 641–657.

[27] Duarte M F, Davenport M A, Takhar D, Laska J N, Sun T, Kell K F, . Single-pixel imaging via compressive sampling. IEEE Signal Processing Magazine, 2008, 25(2): 83-91.

[28] Tropp J A. A mathematical introduction to compressive sensing [Book Review]. Bulletin of the American Mathematical Society, 2017, 54(1): 151-165.

[29] Romberg J. Imaging via compressive sampling. IEEE Signal Processing Magazine, 2008, 25(2): 14-20.

[30] Chua L O, Roska T. The CNN paradigm. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 1993, 40(3): 147-156.

[31] Elhoseny M, Shankar K. Optimal bilateral filter and convolutional neural network based denoising method of medical image measurements. Measurement, 2019, 143, 125-135.

[32] Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao, Z. Deng, et al., “Closed-loop matters: Dual regression networks for single image super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 5407–5416.

[33] Wang B, Zou Y, Zhang L, Hu Y, Yan H, Zuo C, . Low-light-level image super-resolution reconstruction based on a multi-scale features extraction network. Photonics, 2021, 8(8): 321.

[34] T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in Proceedings of the IEEE International Conference on Computer Vision, Honolulu, USA, 2017, pp. 4799–4807.

[35] Sara U, Akter M, Uddin M S. Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study. Journal of Computer and Communications, 2019, 7(3): 8-18.