Local features and manifold ranking coupled method for sketch-based 3D model retrieval

Abstract

3D model retrieval can benefit many downstream virtual reality applications. In this paper, we propose a new sketch-based 3D model retrieval framework by coupling local features and manifold ranking. At technical fronts, we exploit spatial pyramids based local structures to facilitate the efficient construction of feature descriptors.Meanwhile, we propose an improved manifold ranking method, wherein all the categories between arbitrary model pairs will be taken into account. Since the smooth and detail-preserving line drawings of 3D model are important for sketch-based 3D model retrieval, the Difference of Gaussians (DoG) method is employed to extract the line drawings over the projected depth images of 3D model, and Bezier Curve is then adopted to further optimize the extracted line drawing. On that basis, we develop a 3D model retrieval engine to verify our method. We have conducted extensive experiments over various public benchmarks, and have made comprehensive comparisons with some state-of-the-art 3D retrieval methods. All the evaluation results based on the widely-used indicators prove the superiority of our method in accuracy, reliability, robustness, and versatility.

Keywords

sketch-based retrieval / 3D model / manifold ranking / line drawing / local features

References

[1] Zeleznik R C, Herndon K P, Hughes J F. SKETCH: an interface for sketching 3D scenes. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. 1996, 163–170

[2] Gao Y, Wang M, Tao D C, Ji R R, Dai Q H. 3-D object retrieval and recognition with hypergraph analysis. IEEE Transactions on Image Processing, 2012, 21(9): 4290–4303

[3] Walther D B, Chai B, Caddigan E, Beck D M, Li F F. Simple line drawings suffice for functional MRI decoding of natural scene categories. Proceedings of the National Academy of Sciences, 2011, 108(23): 9661–9666

[4] Eitz M, Hays J, Alexa M. How do humans sketch objects? ACMTransactions on Graphics, 2012, 31(4): 44

[5] Funkhouser T, Min P, Kazhdan M, Chen J, Halderman A, Dobkin D, Jacobs D. A search engine for 3D models. ACM Transactions on Graphics, 2003, 22(1): 83–105

[6] Li B, Lu Y J, Ghumman A, Strylowski B, Gutierrez M, Sadiq S, Forster S, Feola N, Bugerin T. 3D Sketch-Based 3D model retrieval. In: Proceedings of the 5th ACM International Conference on Multimedia Retrieval. 2015, 555–558

[7] Liu X L, Huang L, Deng C, Tao D C. Query-adaptive hash code ranking for large-scale multi-view visual search. IEEE Transactions on Image Processing, 2016, 25(10): 4514–4524

[8] Liu X L, Du B W, Deng C, Liu M, Lang B. Structure sensitive hashing with adaptive product quantization. IEEE Transactions on Cybernetics, 2016, 46(10): 2252–2264

[9] DeCarlo D, Finkelstein A. Rusinkiewicz S, Santella . ASuggestive contours for conveying shape. ACM Transactions on Graphics, 2003, 22(3): 848–855

[10] Saavedra J M, Bustos B, Schreck T, Yoon S M, Scherer M. Sketchbased 3D model retrieval using keyshapes for global and local representation. In: Proceedings of the 5th Eurographics Conference on 3D Object Retrieval. 2012, 47–50

[11] Yoon S, Scherer M, Schreck T, Kuijper A. A sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. In: Proceedings of the 18th ACM International Conference on Multimedia. 2010, 193–200

[12] Yoon S M, Kuijper A. Sketch-based 3D model retrieval using compressive sensing classification. Electronics Letters, 2011, 47(21): 1181–1183

[13] Judd T, Durand F, Adelson E. Apparent ridges for line drawing. ACM Transactions on Graphics, 2007, 26(3): 19

[14] Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M. Sketch-based shape retrieval. ACM Transactions on Graphics, 2012, 31(4): 1–10

[15] Li B, Schreck T, Godil A, Alexa M, Boubekeur T, Bustos B, Chen J, Eitz M, Furuya T, Hildebrand K, Huang S, Johan H, Kuijper A, Ohbuchi R, Richter R, Saavedra J M, Scherer M, Yanagimachi T, Yoon G J, Yoon S M. Track: sketch-based 3D shape retrieval. In: Proceedings of the 5th Eurographics Conference on 3D Object Retrieval. 2012, 109–118

[16] Shao T J, Xu W W, Yin K K, Wang J D, Zhou K, Guo B N. Discriminative sketch-based 3D model retrieval via robust shape matching. In: Proceedings of Comput Graph Forum. 2011, 2011–2020

[17] Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Fu H, Furuya T, Johan H, Liu J, Ohbuchi R, Tatsuma A, Zou C. SHREC’14 track: extended large scale sketch-based 3D shape retrieval. In: Proceedings of Eurographics Workshop on 3D Object Retrieval. 2014

[18] Li B, Lu Y, Godil A, Schreck T, Aono M, Johan H, Saavedra J M, Tashiro S. SHREC’13 track: scale sketch-based 3D shape retrieval. In: Proceedings of the 6th Eurographics Workshop on 3D Object Retrival. 2013, 89–96

[19] Yoon S M, Yoon G, Schreck T. User-drawn sketch-based 3D object retrieval using sparse coding. Multimedia Tools and Applications. 2015, 74(13): 4707–4722

[20] Hu R, Wang T H, Collomosse J. A bag-of-regions approach to sketchbased image retrieval. In: Proceedings of the 18th IEEE International Conference on Image Processing. 2011, 3661–3664

[21] Li B, Johan H. Sketch-based 3D model retrieval by incorporating 2D- 3D alignment. Multimedia Tools and Applications, 2013, 65(3): 363–385

[22] Li B, Johan H. A 3D model feature for retrieval. In: Proceedings of MMM. 2010, 185–195

[23] Belongie S, Malik J, Puzicha J. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(4): 509–522

[24] Ohbuchi R, Furuya T. Scale-weighted dense bag of visual features for 3D model retrieval from a partial view 3D model. In: Proceedings of the 12th IEEE International Conference on Computer Vision. 2009, 63–70

[25] Lowe D G. Distinctive Image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110

[26] Wang F, Lin L F, Tang M. A new sketch-based 3D model retrieval approach by using global and local features. Graphical Models, 2014, 76(3): 128–139

[27] Wang F, Le K, Li Y. Sketch-based 3D shape retrieval using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1875–1883

[28] Zhu F, Xie J, Fang Y. Learning cross-domain neural networks for sketch-based 3D shape retrieval. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016.

[29] He J R, Li M J, Zhang H J, Tong H H, Zhang C S. Manifold-ranking based image retrieval. In: Proceedings of the 12th Annual ACM International Conference on Multimedia. 2004, 9–16

[30] Tong H H, He J R, Li M J, Ma W Y, Zhan g H J, Zhang C S. Manifoldranking- based keyword propagation for image retrieval. EURASIP Journal on Applied Signal Processing, 2006, 2006: 190

[31] Guan Z Y, Bu J J, Mei Q Z, Chen C, Wang C. Personalized tag recommendation using graph-based ranking on multi-type interrelated objects. In: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2009, 540–547

[32] Agarwal S. Ranking on on graph data. In: Proceedings of the 23rd Internatioal Conference on Machine Learning. 2006, 25–32

[33] Bu J J, Tan S L, Chen C, Wang C, Wu H, Zhang L J, He X F. Music recommendation by unified hypergraph: combining social media information and music content. In: Proceedings of the 18th ACM International Conference on Multimedia. 2010, 391–400

[34] Furuya T, Ohbuchi R. Similarity metric learning for sketch-based 3D object retrieval. Multimedia Tools and Applications, 2015, 74(23): 10367–10392

[35] Liu X L, Lang B, Xu Y, Cheng B. Feature grouping and local soft match for mobile visual search. Pattern Recognition Letters, 2012, 33(3): 239–246

[36] Yang Y1, Nie F P, Xu D B, Luo J, Zhuang Y T, Pan Y H. Multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(4): 723–742

[37] Liu X L, Deng C, Lang B, Tao D C, Li X L. Query-adaptive reciprocal hash tables for nearest neighbor search. IEEE Transactions on Image Processing, 2015, 25(2): 907–919

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag Berlin Heidelberg, part of Springer Nature