Differentiable Deformation Graph-Based Neural Non-rigid Registration

Abstract

The traditional pipeline for non-rigid registration is to iteratively update the correspondence and alignment such that the transformed source surface aligns well with the target surface. Among the pipeline, the correspondence construction and iterative manner are key to the results, while existing strategies might result in local optima. In this paper, we adopt the widely used deformation graph-based representation, while replacing some key modules with neural learning-based strategies. Specifically, we design a neural network to predict the correspondence and its reliability confidence rather than the strategies like nearest neighbor search and pair rejection. Besides, we adopt the GRU-based recurrent network for iterative refinement, which is more robust than the traditional strategy. The model is trained in a self-supervised manner and thus can be used for arbitrary datasets without ground-truth. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art methods by a large margin.

Keywords

Differentiable deformation graph / Non-rigid registration

References

[1] Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2007)

[2] Aoki, Y., Goforth, H., Srivatsan, R.A., Lucey, S.: Pointnetlk: robust & efficient point cloud registration using pointnet. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7163–7172 (2019)

[3] Aubry, M., Schlickewei, U., Cremers, D.: The wave kernel signature: A quantum mechanical approach to shape analysis. In: IEEE International Conference on Computer Vision (ICCV), pp. 1626–1633 (2011)

[4] Aygün, M., Lähner, Z., Cremers, D.: Unsupervised dense shape correspondence using heat kernels. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 573–582 (2020)

[5] Besl PJ, McKay ND. A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell.. 1992, 14 2 239-256

[6] Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: Registering human bodies in motion. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

[7] Boykov, Y., Veksler, O., Zabih, R.: Markov random fields with efficient approximations. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 648–655 (1998)

[8] Bozic, A., Palafox, P.R., Zollhöfer, M., Dai, A., Thies, J., Nießner, M.: Neural non-rigid tracking. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2020)

[9] Bronstein, M.M., Kokkinos, I.: Scale-invariant heat kernel signatures for non-rigid shape recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1704–1711 (2010)

[10] Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. In: Proceedings of the National Academy of Sciences, 1168–1172 (2006)

[11] Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry of Non-Rigid Shapes. Monographs in Computer Science, (2009)

[12] Chen J, Ma J, Yang C, Ma L, Zheng S. Non-rigid point set registration via coherent spatial mapping. Signal Process. 2015, 106 62-72

[13] Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: encoder-decoder approaches. In: Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111 (2014)

[14] Chui H, Rangarajan A. A new point matching algorithm for non-rigid registration. Comput. Vis. Image Underst.. 2003, 89 2–3 114-141

[15] Coifman, R.R., Lafon, S., Lee, A.B., Maggioni, M., Nadler, B., Warner, F., Zucker, S.W.: Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. In: Proceedings of the National Academy of Sciences, pp. 7426–7431 (2005)

[16] Feng, W., Zhang, J., Cai, H., Xu, H., Hou, J., Bao, H.: Recurrent multi-view alignment network for unsupervised surface registration. CoRR abs/2011.12104 (2020)

[17] Golyanik, V., Taetz, B., Reis, G., Stricker, D.: Extended coherent point drift algorithm with correspondence priors and optimal subsampling. In: Winter Conference on Applications of Computer Vision (WACV), pp. 1–9 (2016)

[18] Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3d-coded: 3d correspondences by deep deformation. In: Computer Vision–ECCV 2018–15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, vol. 11206 (2018)

[19] Guo, K., Xu, F., Wang, Y., Liu, Y., Dai, Q.: Robust non-rigid motion tracking and surface reconstruction using L0 regularization. In: IEEE International Conference on Computer Vision (ICCV), pp. 3083–3091 (2015)

[20] Halimi, O., Litany, O., Rodolà, E., Bronstein, A.M., Kimmel, R.: Unsupervised learning of dense shape correspondence. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4370–4379 (2019)

[21] Hirose O. A Bayesian formulation of coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell.. 2020, 99 1-1

[22] Hurlburt NE, Jaffey S. A spectral optical flow method for determining velocities from digital imagery. Earth Sci. Inform.. 2015, 8 4 959-965

[23] Jian B, Vemuri BC. Robust point set registration using gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell.. 2011, 33 8 1633-1645

[24] Kim VG, Lipman Y, Funkhouser TA. Blended intrinsic maps. ACM Trans. Graph.. 2011, 30 79

[25] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)

[26] Kittenplon, Y., Eldar, Y.C., Raviv, D.: Flowstep3d: model unrolling for self-supervised scene flow estimation. CoRR abs/2011.10147 (2020)

[27] Kolesov I, Lee J, Sharp G, Vela PA, Tannenbaum AR. A stochastic approach to diffeomorphic point set registration with landmark constraints. IEEE Trans. Pattern Anal. Mach. Intell.. 2016, 38 2 238-251

[28] Li H, Sumner RW, Pauly M. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum. 2008, 27 5 1421-1430

[29] Li H, Adams B, Guibas LJ, Pauly M. Robust single-view geometry and motion reconstruction. ACM Trans. Graph.. 2009, 28 175

[30] Li, C., Simon, T., Saragih, J.M., Póczos, B., Sheikh, Y.: LBS autoencoder: self-supervised fitting of articulated meshes to point clouds. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11967–11976 (2019)

[31] Lipman Y, Funkhouser TA. Möbius voting for surface correspondence. ACM Trans. Graph.. 2009, 28 1-12

[32] Litany, O., Remez, T., Rodolà, E., Bronstein, A.M., Bronstein, M.M.: Deep functional maps: structured prediction for dense shape correspondence. In: IEEE International Conference on Computer Vision (ICCV) (2017)

[33] Liu, P., Lyu, M.R., King, I., Xu, J.: Selflow: self-supervised learning of optical flow. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4571–4580 (2019)

[34] Liu, X., Qi, C.R., Guibas, L.J.: Flownet3d: learning scene flow in 3d point clouds. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 529–537 (2019)

[35] Ma J, Zhao J, Yuille AL. Non-rigid point set registration by preserving global and local structures. IEEE Trans. Image Process. 2016, 25 1 53-64

[36] Mémoli F, Sapiro G. A theoretical and computational framework for isometry invariant recognition of point cloud data. Found. Comput. Math.. 2005, 45 313-347

[37] Mittal, H., Okorn, B., Held, D.: Just go with the flow: self-supervised scene flow estimation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11174–11182 (2020)

[38] Myronenko A, Song XB. Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell.. 2010, 32 12 2262-2275

[39] Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.W.: Kinectfusion: real-time dense surface mapping and tracking. In: IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136 (2011)

[40] Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: reconstruction and tracking of non-rigid scenes in real-time. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 343–352 (2015)

[41] Nguyen A, Ben-Chen M, Welnicka K, Ye Y, Guibas LJ. An optimization approach to improving collections of shape maps. Comput. Graph. Forum. 2011, 72 1481-1491

[42] Qian, C., Sun, X., Wei, Y., Tang, X., Sun, J.: Realtime and robust hand tracking from depth. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1106–1113 (2014)

[43] Ovsjanikov M, Ben-Chen M, Solomon J, Butscher A, Guibas LJ. Functional maps: a flexible representation of maps between shapes. ACM Trans. Graph.. 2012, 31 1-11

[44] Rodolà, E., Möller, M., Cremers, D.: Point-wise map recovery and refinement from functional correspondence. In: International Symposium on Vision Modeling and Visualization, pp. 25–32 (2015)

[45] Rustamov, R.M.: Laplace–Beltrami eigenfunctions for deformation invariant shape representation. In: Eurographics Symposium on Geometry Processing, pp. 225–233 (2007)

[46] Shimada, S., Golyanik, V., Tretschk, E., Stricker, D., Theobalt, C.: Dispvoxnets: non-rigid point set alignment with supervised learning proxies. In: International Conference on 3D Vision, pp. 27–36 (2019)

[47] Smith, L.N., Topin, N.: Super-convergence: very fast training of residual networks using large learning rates. CoRR abs/1708.07120 (2017)

[48] Sumner RW, Schmid J, Pauly M. Embedded deformation for shape manipulation. ACM Trans. Graph.. 2007, 26 3 80

[49] Sun J, Ovsjanikov M, Guibas LJ. A concise and provably informative multi-scale signature based on heat diffusion. Comput. Graph. Forum. 2009, 324 1383-1392

[50] Sun, D., Yang, X., Liu, M., Kautz, J.: PWC-net: CNNS for optical flow using pyramid, warping, and cost volume. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8934–8943 (2018)

[51] Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: European Conference on Computer Vision (ECCV), pp. 402–419 (2020)

[52] Teed, Z., Deng, J.: RAFT-3D: scene flow using rigid-motion embeddings. CoRR abs/2012.00726 (2020)

[53] Tombari, F., Salti, S., di Stefano, L.: Unique signatures of histograms for local surface description. In: European Conference on Computer Vision (ECCV), pp. 356–369 (2010)

[54] Tombari, F., Salti, S., di Stefano, L.: Unique signatures of histograms for local surface description. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) European Conference on Computer Vision (ECCV). Lecture Notes in Computer Science, vol. 6313, pp. 356–369 (2010)

[55] Vedula S, Baker S, Rander P, Collins RT, Kanade T. Three-dimensional scene flow. IEEE Trans. Pattern Anal. Mach. Intell.. 2005, 27 3 475-480

[56] Wang, L., Fang, Y.: Coherent point drift networks: unsupervised learning of non-rigid point set registration. CoRR abs/1906.03039 (2019)

[57] Wang, Y., Solomon, J.: Deep closest point: learning representations for point cloud registration. In: IEEE International Conference on Computer Vision (ICCV), pp. 3522–3531 (2019)

[58] Wang, L., Chen, J., Li, X., Fang, Y.: Non-rigid point set registration networks. CoRR abs/1904.01428 (2019)

[59] Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph.. 2019, 38 5 146-114612

[60] Yang Y, Ong SH, Foong KWC. A robust global and local mixture distance based non-rigid point set registration. Pattern Recognit.. 2015, 48 1 156-173

[61] Yao, Y., Deng, B., Xu, W., Zhang, J.: Quasi-newton solver for robust non-rigid registration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7597–7606 (2020)

[62] Yu, T., Zheng, Z., Guo, K., Zhao, J., Dai, Q., Li, H., Pons-Moll, G., Liu, Y.: Doublefusion: real-time capture of human performances with inner body shapes from a single depth sensor. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

[63] Zeng, Y., Qian, Y., Zhu, Z., Hou, J., Yuan, H., He, Y.: Corrnet3d: unsupervised end-to-end learning of dense correspondence for 3d point clouds. CoRR abs/2012.15638 (2020)

Funding

National Natural Science Foundation of China(62122071)

Youth Innovation Promotion Association of the Chinese Academy of Sciences(2018495)

Fundamental Research Funds for Central Universities of the Central South University(WK3470000021)