A comprehensive review of significant researches on content based indexing and retrieval of visual information

Abstract

Developments in multimedia technologies have paved way for the storage of huge collections of video documents on computer systems. It is essential to design tools for content-based access to the documents, so as to allow an efficient exploitation of these collections. Content based analysis provides a flexible and powerfulway to access video data when compared with the other traditional video analysis techniques. The area of content based video indexing and retrieval (CBVIR), focusing on automating the indexing, retrieval and management of video, has attracted extensive research in the last decade. CBVIR is a lively area of research with enduring acknowledgments from several domains. Herein a vital assessment of contemporary researches associated with the content-based indexing and retrieval of visual information. In this paper, we present an extensive review of significant researches on CBVIR. Concise description of content based video analysis along with the techniques associated with the content based video indexing and retrieval is presented.

Keywords

nultimedia information / content based video retrieval (CBVR) / content based video indexing and retrieval (CBVIR) / shot segmentation / object segmentation / feature extraction / indexing / motion estimation / querying / key frame / retrieval / and indexing

References

[1] Hanis A, Sziranyi T. Measuring the motion similarity in video indexing. In: Proceedings of the 4th EURASIP Conference Focused on Video/Image Processing andMultimedia Communications. 2003, 507-512

[2] Calic J, Izuierdo E. Efficient key-frame extraction and video analysis. In: Proceedings of the 2002 International Conference on Information Technology: Coding and Computing. 2002, 28-33

[3] Carbonaro A. Ontology-based video retrieval in a semantic-based learning environment. Journal of e-Learning and Knowledge Society, 2009, 4(3): 203-212

[4] George A, Rajakumar B, Suresh B. Markov random field based image restoration with aid of local and global features. International Journal of Computer Applications, 2012, 48(8): 23-28

[5] Kundra E H, Verma E M, Aashima E. Filter for removal of impulse noise by using fuzzy logic. International Journal of Image Processing (IJIP), 2011, 3(5): 195-202

[6] Umamakeswari A, Rajaraman A. Object based video analysis, interpretation and tracking. Journal of Computer Science, 2007, 3(10): 818-822

[7] Amer A. Object-based video retrieval based on motion analysis and description. Technical Report, University du Québec, 1999

[8] Javed O, Shah M, Comaniciu D. A probabilistic framework for object recognition in video. In: Proceedings of the 2004 International Conference on Image Processing. 2004, 2713-2716

[9] Radhakrishnan R, Divakaran A, Xiong Z, Otsuka I. A content-adaptive analysis and representation framework for audio event discovery from unscripted multimedia. EURASIP Journal on Applied Signal Processing, 2006: 1-24

[10] Schnettler B, Raab J. Interpretative visual analysis developments: state of the art and pending problems. Historical Social Research/Historische Sozialforschung, 2009, 265-295

[11] Ramoser H, Schlogl T, Beleznai C, Winter M, Bischof H. Shape-based detection of humans for video surveillance applications. In: Proceedings of the 2003 IEEE International Conference on Image Processing. 2003, 3: 1013-1016

[12] Ahmad A M, Lee S Y. Fast and robust object-extraction framework for object-based streaming system. International Journal of Virtual Technology and Multimedia, 2008, 1(1): 39-60

[13] Ngo C W, Pong T C, Zhang H J. Motion-based video representation for scene change detection. International Journal of Computer Vision, 2002, 50(2): 127-142

[14] Zelnik-Manor L, Irani M. Event-based analysis of video. In: Proceedings of the 2001 IEEE Conference on Computer Vision and Pattern Recognition. 2001, II-123-II-130

[15] Ding Y, Fan G Camera view-based american football video analysis. In: Proceedings of the 8th IEEE International Symposium on Multimedia. 2006, 317-322

[16] Mohan C K, Dhananjaya N, Yegnanarayana B. Video shot segmentation using late fusion technique. In: Proceedings of the 7th International Conference on Machine Learning and Applications. 2008, 267-270

[17] Affendey L S, Mamat A, Ibrahim H, Ahmad F. Video data modeling to support hybrid query. International Journal of Computer Science and Network Security, 2007, 7(9): 53-61

[18] Aytar Y, Shah M, Luo J. Utilizing semantic word similarity measures for video retrieval. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1-8

[19] Dimitrova N, Zhang H J, Shahraray B, Sezan I, Huang T, Zakhor A. Applications of video-content analysis and retrieval. IEEE MultiMedia, 2002, 9(3): 42-55

[20] Bergman L D, Castelli V, Li C. Progressive content-based retrieval from satellite image archives. Technical Report, D-LibMagazine, 1997

[21] Chang S F, Smith J R, Meng H J, Wang H, Zhong D. Finding images/video in large archives. Technical Report, D-Lib Magazine, 1997

[22] Gupta A, Jain R. Visual information retrieval. Communications of the ACM, 1997, 40(5): 70-79

[23] Papadias D, Mantzourogiannis M, Kalnis P, Mamoulis N, Ahmad I. Content-based retrieval using heuristic search. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 1999, 168-175

[24] Koprinska I, Carrato S. Temporal video segmentation: a survey. Signal Processing: Image Communication, 2001, 16(5): 477-500

[25] Hanjalic A. Shot-boundary detection: unraveled and resolved? IEEE Transactions on Circuits and Systems for Video Technology, 2002, 12(2): 90-105

[26] Girgensohn A, Boreczky J. Time-constrained keyframe selection technique. In: Proceedings of the 1999 IEEE International Conference on Multimedia Computing and Systems. 1999, 756-761

[27] Liu T, Kender J ROptimization algorithms for the selection of key frame sequences of variable length. In: Proceedings of the 2002 European Conference on Computer Vision. 2002, 403-417

[28] Aslandogan Y A, Yu C T. Techniques and systems for image and video retrieval. IEEE Transactions on Knowledge and Data Engineering, 1999, 11(1): 56-63

[29] Lu G. Techniques and data structures for efficient multimedia retrieval based on similarity. IEEE Transactions on Multimedia, 2002, 4(3): 372-384

[30] Lelescu D, Schonfeld D. Video skimming and summarization based on principal component analysis. In: Proceedings of the 4th IFIP/IEEE International Conference on Management of Multimedia on the Internet. 2001, 128-141

[31] Gargi U, Kasturi R, Strayer S H. Performance characterization of video-shot-change detection methods. IEEE Transactions on Circuits and Systems for Video Technology, 2000, 10(1): 1-13

[32] Huang Y, Liu Q, Metaxas D. Video object segmentation by hypergraph cut. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1738-1745

[33] Aydm Alatan A, Tuncel E, Onural L. A rule-based method for object segmentation in video sequences. In: Proceedings of the 1997 International Conference on Image Processing. 1997, 522-525

[34] Otoom A F, Gunes H, Piccardi M. Feature extraction techniques for abandoned object classification in video surveillance. In: Proceedings of the 15th IEEE International Conference on Image Processing. 2008, 1368-1371

[35] Van Cauwelaert D. Generic models for adaptive robust feature extraction in video. In: Proceedings of the 9th FirW PhD Symposium. 2008, 148-149

[36] Zhong D, Chang S F. An integrated approach for content-based video object segmentation and retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 1999, 9(8): 1259-1268

[37] Avrithis Y S, Doulamis A D, Doulamis N D, Kollias S D. An adaptive approach to video indexing and retrieval using fuzzy classification. In: Proceedings of VLBV. 1998

[38] Idris F, Panchanathan S. Review of image and video indexing techniques. Journal of Visual Communication and Image Representation, 1997, 8(2): 146-166

[39] Christel M G, Smith M A, Taylor C R, Winkler D BEvolving video skims into useful multimedia abstractions. In: Proceedings of the 1998 SIGCHI Conference on Human factors in Computing Systems. 1998, 171-178

[40] Marchand-Maillet S. Content-based video retrieval: an overview. 2000

[41] Jawahar C, Chennupati B, Paluri B, Jammalamadaka N. Video Retrieval Based on Textual Queries. Technical Report, 2005

[42] Zhang X P, Chen Z. An automated video object extraction system based on spatiotemporal independent component analysis and multiscale segmentation. EURASIP Journal on Applied Signal Processing, 2006, 2006: 184

[43] Chang Y, Lee D J, Hong Y, Archibald J. Unsupervised video shot detection using clustering ensemble with a color global scale-invariant feature transform descriptor. Journal on Image and Video Processing, 2008, 9

[44] Kolekar M H, Palaniappan K, Sengupta S, Seetharaman G. Semantic concept mining based on hierarchical event detection for soccer video indexing. Journal of Multimedia, 2009, 4(5): 298-312

[45] Jiang R, Crookes D. Approach to automatic video motion segmentation. Electronics Letters, 2007, 43(18): 968-970

[46] Basharat A, Zhai Y, Shah M. Content based video matching using spatiotemporal volumes. Computer Vision and Image Understanding, 2008, 110(3): 360-377

[47] Kuo T C, Chen A L. A maskmatching approach for video segmentation on compressed data. Information Sciences, 2002, 141(1): 169-191

[48] Chen D Y, Hsiao MH, Lee S Y. Automatic closed caption detection and filtering in mpeg videos for video structuring. Journal of Information Science and Engineering, 2006, 22(5): 1145-1162

[49] Duan L Y, Xu M, Tian Q, Xu C S, Jin J S. A unified framework for semantic shot classification in sports video. IEEE Transactions on Multimedia, 2005, 7(6): 1066-1083

[50] Liu S, Xu M,Yi H, Chia L T, Rajan D. Multimodal semantic analysis and annotation for basketball video. EURASIP Journal on Applied Signal Processing, 2006: 182

[51] Han B, Gao X, Ji H. A shot boundary detection method for news video based on rough-fuzzy sets. International Journal of Information Technology, 2005, 11(7): 101-111

[52] Gao X, Tang X. Unsupervised video-shot segmentation and model-free anchorperson detection for news video story parsing. IEEE Transactions on Circuits and Systems for Video Technology, 2002, 12(9): 765-776

[53] Zhai Y, Shah M Video scene segmentation using markov chain monte carlo. IEEE Transactions on Multimedia, 2006, 8(4): 686-697

[54] Fan J, Aref W G, Elmagarmid A K, Hacid M S, Marzouk M S, Zhu X. Multiview: multilevel video content representation and retrieval. Journal of Electronic Imaging, 2001, 10(4): 895-908

[55] Dönderler M E, Ulusoy Ö, Güdükbay U. Rule-based spatiotemporal query processing for video databases. The VLDB Journal, 2004, 13(1): 86-103

[56] Erozel G, Cicekli N K, Cicekli I. Natural language querying for video databases. Information Sciences, 2008, 178(12): 2534-2552

[57] Smeaton A F, Wilkins P, Worring M, De Rooij O, Chua T S, Luan H. Content-based video retrieval: three example systems from trecvid. International Journal of Imaging Systems and Technology, 2008, 18(2-3): 195-201

[58] Schonfeld D, Lelescu D. Vortex: video retrieval and tracking from compressed multimedia databases—multiple object tracking from mpeg-2 bit stream. Journal of Visual Communication and Image Representation, 2000, 11(2): 154-182

[59] Vrochidis S, Doulaverakis C, Gounaris A, Nidelkou E, Makris L, Kompatsiaris I. A hybrid ontology and visual-based retrieval model for cultural heritage multimedia collections. International Journal of Metadata, Semantics and Ontologies (IJMSO), 2008, 3(3): 167-182

[60] Wen C Y, Chang L F, Li H H. Content based video retrieval with motion vectors and the rgb color model. Forensic Science Journal, 2007, 6(2): 1-36

[61] Lili N, Noah S, Khalid F. Extracting and integrating multimodality features via multidimensional approach for video retrieval. International Journal of Computer Science and Network Security, 2009, 9(2): 252

[62] Hoi S C, Lyu M R. A multimodal and multilevel ranking scheme for large-scale video retrieval. IEEE Transactions on Multimedia, 2008, 10(4): 607-619

[63] Tjondronegoro D, Chen Y P PContent-based indexing and retrieval using mpeg-7 and x-query in video data management systems. World Wide Web, 2002, 5(3): 207-227

[64] Ma Y F, Zhang H J. Motion pattern-based video classification and retrieval. EURASIP Journal on Applied Signal Processing, 2003: 199-208

[65] DeMenthon D, Doermann D. Video retrieval of near-duplicates using κ-nearest neighbor retrieval of spatio-temporal descriptors. Multimedia Tools and Applications, 2006, 30(3): 229-253

[66] Yang J, Li Q, Wenyin L, Zhuang Y. Searching for flash movies on the web: a content and context based framework. World Wide Web, 2005, 8(4): 495-517

[67] Dagtas S, Al-Khatib W, Ghafoor A, Kashyap R L. Models for motionbased video indexing and retrieval. IEEE Transactions on Image Processing, 2000, 9(1): 88-101

[68] Lu H, Ooi B C, Shen H T, Xue X. Hierarchical indexing structure for efficient similarity search in video retrieval. IEEE Transactions on Knowledge and Data Engineering, 2006, 18(11): 1544-1559

[69] Zhang D, Nunamaker J FA natural language approach to contentbased video indexing and retrieval for interactive e-learning. IEEE Transactions on Multimedia, 2004, 6(3): 450-458

[70] Amir A, Srinivasan S, Efrat A. Search the audio, browse the video—a generic paradigm for video collections. EURASIP Journal on Advances in Signal Processing, 1900, 2003(2): 209-222

[71] Chiu C Y, Chao S P, Wu M Y, Yang S N, Lin H C. Content-based retrieval for human motion data. Journal of Visual Communication and Image Representation, 2004, 15(3): 446-466

[72] Munesawang P, Guan L. Adaptive video indexing and automatic/semiautomatic relevance feedback. IEEE Transactions on Circuits and Systems for Video Technology, 2005, 15(8): 1032-1046

[73] Fablet R, Bouthemy P, Pérez P. Nonparametric motion characterization using causal probabilistic models for video indexing and retrieval. IEEE Transactions on Image Processing, 2002, 11(4): 393-407

[74] Yi H, Rajan D, Chia L T. A new motion histogram to index motion content in video segments. Pattern Recognition Letters, 2005, 26(9): 1221-1231

[75] Babu R V, Ramakrishnan K. Compressed domain video retrieval using object and global motion descriptors. Multimedia Tools and Applications, 2007, 32(1): 93-113

[76] Snoek C G, Worring M, Koelma D C, Smeulders A W. A learned lexicon-driven paradigm for interactive video retrieval. IEEE Transactions on Multimedia, 2007, 9(2): 280-292

[77] Abdelali A B, Mtibaa A, Bourennane E, Abid M. Design of hardware accelerators for content based video indexing. Asian Journal of Information Technology, 2006, 5(9): 976-984

[78] Doulamis A D, Doulamis N D, Kollias S D. A fuzzy video content representation for video summarization and content-based retrieval. Signal Processing, 2000, 80(6): 1049-1067

[79] Lee J, Dickinson B WHierarchical video indexing and retrieval for subband-coded video. IEEE Transactions on Circuits and Systems for Video Technology, 2000, 10(5): 824-829

[80] Fan J, Zhu X, Hacid M S, Elmagarmid A KModel-based video classification toward hierarchical representation, indexing and access. Multimedia Tools and Applications, 2002, 17(1): 97-120

[81] Albanese M, Chianese A, Moscato V, Sansone L. A formal model for video shot segmentation and its application via animate vision. Multimedia Tools and Applications, 2004, 24(3): 253-272

[82] Cheung R. Indexing an intelligent video database using evolutionary control. Journal of Digital Information Management, 2003, 1: 8-19

[83] Erol B, Kossentini F. Retrieval by local motion. EURASIP Journal on Advances in Signal Processing, 1900, 2003(1): 41-47

[84] Xu C, Cheng J, Zhang Y, Zhang Y, Lu H. Sports video analysis: semantics extraction, editorial content creation and adaptation. Journal of Multimedia, 2009, 4(2): 69-79

[85] Lee S, Yoo C D. Robust video fingerprinting for content-based video identification. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(7): 983-988

[86] Fan J, Elmagarmid A K, Zhu X, Aref W G, Wu L. Classview: hierarchical video shot classification, indexing, and accessing. IEEE Transactions on Multimedia, 2004, 6(1): 70-86

[87] Fan J, Luo H, Elmagarmid A K. Concept-oriented indexing of video databases: toward semantic sensitive retrieval and browsing. IEEE Transactions on Image Processing, 2004, 13(7): 974-992

[88] Khan A, Sun L, Ifeachor E. Content-based video quality prediction for mpeg4 video streaming over wireless networks. Journal ofMultimedia, 2009, 4(4): 228-239

[89] Dumont E, Merialdo B. Rushes video summarization and evaluation. Multimedia Tools and Applications, 2010, 48(1): 51-68

[90] Thyagharajan K, Ramachandran V. An effective transmission and browsing methodology for streaming video. Journal of Computer Science, 2006, 2(4): 326-332

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag Berlin Heidelberg