Image categorization using a semantic hierarchy model with sparse set of salient regions

Abstract

Image categorization in massive image database is an important problem. This paper proposes an approach for image categorization, using sparse set of salient semantic information and hierarchy semantic label tree (HSLT) model. First, to provide more critical image semantics, the proposed sparse set of salient regions only at the focuses of visual attention instead of the entire scene was formed by our proposed saliency detection model with incorporating low and high level feature and Shotton’s semantic texton forests (STFs) method. Second, we also propose a new HSLT model in terms of the sparse regional semantic information to automatically build a semantic image hierarchy, which explicitly encodes a general to specific image relationship. And last, we archived image dataset using image hierarchical semantic, which is help to improve the performance of image organizing and browsing. Extension experimental results showed that the use of semantic hierarchies as a hierarchical organizing framework provides a better image annotation and organization, improves the accuracy and reduces human’s effort.

Keywords

salient region / sparse set / semantic hierarchy / image annotation / image categorization

References

[1] Grifflths T, Jordan M, Tenenbaum J, Blei D M. Hierarchical topic models and the nested chinese restaurant process. Advances in Neural Information Processing Systems, 2004, 16: 106-114

[2] Bannour H, Hudelot C. Towards ontologies for image interpretation and annotation. In: Proceedings of the 9th International Workshop on Content-Based Multimedia Indexing (CBMI). 2011, 211-216

[3] Tousch A M, Herbin S, Audibert J Y. Semantic hierarchies for image annotation: a survey. Pattern Recognition, 2012, 45(1): 333-345

[4] Marszalek M, Schmid C. Semantic hierarchies for visual object recognition. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2007, 1-7

[5] Wei X Y, Ngo C W. Ontology-enriched semantic space for video search. In: Proceedings of the 15th International Conference on Multimedia. 2007, 981-990

[6] Deng J, Dong W, Socher R, Li L J, Li K, Fei-Fei L. Imagenet: a largescale hierarchical image database. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2009, 248-255

[7] Snow R, Jurafsky D, Ng A Y. Semantic taxonomy induction from heterogenous evidence. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics. 2006, 801-808

[8] Miller G A. Wordnet: a lexical database for English. Communications of the ACM, 1995, 38(11): 39-41

[9] Jin Y, Khan L, Wang L, Awad M. Image annotations by combining multiple evidence & wordnet. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. 2005, 706-715

[10] Joshi D, Datta R, Zhuang Z, Weiss W, Friedenberg M, Li J, Wang J Z. Paragrab: a comprehensive architecture for web image management and multimodal querying. In: Proceedings of the 32nd International Conference on Very Large Data Bases. 2006, 1163-1166

[11] Datta R, Ge W, Li J, Wang J Z. Toward bridging the annotationretrieval gap in image search. IEEE MultiMedia, 2007, 14(3): 24-35

[12] Torralba A, Fergus R, Freeman W T. 80 million tiny images: a large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(11): 1958-1970

[13] Sivic J, Russell B C, Zisserman A, Freeman W T, Efros A A. Unsupervised discovery of visual object class hierarchies. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1-8

[14] Bart E, Porteous I, Perona P, Welling M. Unsupervised learning of visual taxonomies. In: Proceedings of the 2008 IEEE onference on Computer Vision and Pattern Recognition (CVPR). 2008, 1-8

[15] Yao B Z, Yang X, Lin L, Lee M W, Zhu S C. I2t: image parsing to text description. Proceedings of the IEEE, 2010, 98(8): 1485-1508

[16] Griffln G, Perona P. Learning and using taxonomies for fast visual categorization. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1-8

[17] Marszaek M, Schmid C. Constructing category hierarchies for visual recognition. In: Proceedings of the 10th European Conference on Computer Vision. 2008, 479-491

[18] Ahuja N, Todorovic S. Learning the taxonomy and models of categories present in arbitrary images. In: Proceedings of 11th IEEE International Conference on Computer Vision (ICCV). 2007, 1-8

[19] Li L J, Wang C, Lim Y, Blei D M, Fei-Fei L. Building and using a semantivisual image hierarchy. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2010, 3336-3343

[20] Fan J, Gao Y, Luo H. Hierarchical classification for automatic image annotation. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2007, 111-118

[21] Fan J, Gao Y, Luo H. Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. IEEE Transactions on Image Processing, 2008, 17(3): 407-426

[22] Fan J, Gao Y, Luo H, Jain R. Mining multilevel image semantics via hierarchical classification. IEEE Transactions on Multimedia, 2008, 10(2): 167-187

[23] Wu L, Hua X S, Yu N, Ma W Y, Li S. Flickr distance: a relationship measure for visual concepts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 863-875

[24] Bannour H, Hudelot C. Building semantic hierarchies faithful to image semantics. In: Proceedings of the 18th International Conference on Advances in Multimedia Modeling. 2012, 4-15

[25] Moosmann F, Nowak E, Jurie F. Randomized clustering forests for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(9): 1632-1646

[26] Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91-110

[27] Wu L, Hu Y, Li M, Yu N, Hua X S. Scale-invariant visual language modeling for object categorization. IEEE Transactions on Multimedia, 2009, 11(2): 286-294

[28] Bannour H, Hudelot C. Hierarchical image annotation using semantic hierarchies. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management. 2012, 2431-2434

[29] Deng J, Berg A C, Li K, Fei-Fei L. What does classifying more than 10 000 image categories tell us? In: Proceedings of the 11th European Conference on Computer Vision. 2010, 71-84

[30] Theeuwes J. Top–down and bottom–up control of visual selection. Acta Psychologica, 2010, 135(2): 77-99

[31] Hou X, Zhang L. Saliency detection: a spectral residual approach. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2007, 1-8

[32] Harel J, Koch C, Perona P. Graph-based visual saliency. Advances in Neural Information Processing Systems, 2006, 545-552

[33] Achanta R, Hemami S, Estrada F, Susstrunk S. Frequency-tuned salient region detection. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2009, 1597-1604

[34] Goferman S, Zelnik-Manor L, Tal A. Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10): 1915-1926

[35] Zheng Y, Liu C p, Liu G, Wang Z h. Saliency detection based on inhibition of blur regions. Microelectronics & Computer, 2012, 29(3): 84-88

[36] Yang Z, Chunping L, Zhaohui W, Yi J, Shengrong G. A saliency detection model based on multi-feature fusion. In: Proceedings of the 7th International Conference on Computational Intelligence and Security (CIS). 2011, 1062-1066

[37] Sivic J, Russell B C, Efros A A, Zisserman A, Freeman W T. Discovering objects and their location in images. In: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV). 2005, 370-377

[38] Li Z, Wang Y, Chen J, Xu J, Larid J. Image topic discovery with saliency detection. Journal of Machine Learning Research, 2003, 3: 993-1022

[39] Wu L, Hoi S C. Enhancing bag-of-words models with semanticspreserving metric learning. IEEE Multimedia Magazine, 2011, 18(1): 24-37

[40] Wu L, Hoi S C, Yu N. Semantics-preserving bag-of-words models and applications. IEEE Transactions on Image Processing, 2010, 19(7): 1908-1920

[41] Shotton J, Johnson M, Cipolla R. Semantic texton forests for image categorization and segmentation. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2008, 1-8

[42] Friston K, Kiebel S. Cortical circuits for perceptual inference. Neural Networks, 2009, 22(8): 1093-1104

[43] Shotton J, Winn J, Rother C, Criminisi A. The MSRC 21-class object recognition database, 2006

[44] Everingham M, Van Gool L, Williams C K, Winn J, Zisserman A. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 2010, 88(2): 303-338

[45] Shotton J, Winn J, Rother C, Criminisi A. Textonboost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International Journal of Computer Vision, 2009, 81(1): 2-23

RIGHTS & PERMISSIONS

2014 Higher Education Press and Springer-Verlag Berlin Heidelberg