A survey on high coherence visual media retargeting: recent advances and applications

Abstract

The numerous works on media retargeting call for a thorough and comprehensive survey for reviewing and categorizing existing works and providing insights that can help future design of retargeting approaches and its applications. First, we present the basic problem of media retargeting and detail state-of-the-art retargeting methods devised to solve it. Second, we review recent works on objective quality assessment of media retargeting, where we find that although these works are designed to make the objective assessment result in accordance with the subjective evaluation, they are only suitable for certain situations. Considering the subjective nature of aesthetics, designing objective assessment metric for media retargeting could be a promising area for investigation. Third, we elaborate on other applications extended from retargeting techniques. We show how to apply the retargeting techniques in other fields to solve their challenging problems, and reveal that retargeting technique is not just a simple scaling algorithm, but a thought or concept, which has great flexibility and is quite useful.We believe this review can help researchers and practitioners to solve the existing problems of media retargeting and bring new ideas in their works.

Keywords

media retargeting / quality assessment / aesthetenhancement / image retrieval / video synopsis

References

[1] Grundmann M, Kwatra V, Han M, Essa I. Discontinuous seam-carving for video retargeting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 569–576

[2] Avidan S, Shamir A. Seam carving for content-aware image resizing. ACM Transactions on Graphics (TOG), 2007, 26(3): 10

[3] Panozzo D, Weber O, Sorkine O. Robust image retargeting via axisaligned deformation. Computer Graphics Forum, 2012, 31(2pt1): 229–236

[4] Wang Y S, Tai C L, Sorkine O, Lee T Y. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics (TOG), 2008, 27(5): 118

[5] Yan B, Li K, Yang X C, Hu T X. Seam searching based pixel fusion for image retargeting. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(1): 15–23

[6] Fang Y M, Chen Z Z, Lin W S, Lin C W. Saliency-based image retargeting in the compressed domain. In: Proceedings of the 19th ACM international conference on Multimedia. 2011, 1049–1052

[7] Mansfield A, Gehler P, Van Gool L, Rother C. Scene carving: scene consistent image retargeting. In: Daniilidis K, Maragos P, Paragios N, eds. Computer Vision–ECCV 2010. Springer Berlin Heidelberg, 2010, 143–156

[8] Qi S Y, Ho J. Seam segment carving: retargeting images to irregularlyshaped image domains. In: Fitzgibbon A, Lazebnik S, Perona P, et al, eds. Computer Vision–ECCV 2012, Springer Berlin Heidelberg, 2012, 314–326

[9] Shen J B, Wang D P, Li X L. Depth-aware image seam carving. IEEE Transactions on Cybernetics, 2013, 43(5): 1453–1461

[10] Noh H, Han B. Seam carving with forward gradient difference maps. In: Proceedings of the 20th ACM international conference on Multimedia. 2012, 709–712

[11] Battiato S, Farinella G M, Puglisi G, Ravi D. Saliency-based selection of gradient vector flow paths for content aware image resizing. IEEE Transactions on Image Processing, 2014, 23(5): 2081–2095

[12] Dong W M, Zhou N, Lee T Y, Wu F Z, Kong Y, Zhang X P. Summarization-based image resizing by intelligent object carving. IEEE Transactions on Visualization and Computer Graphics, 2014,20(1): 1

[13] Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M. Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2006, 771–780

[14] Zhang L M, Wang M, Nie L Q, Hong L, Rui Y, Tian Q. Retargeting semantically-rich photos. IEEE Transactions on Multimedia (TMM), 2015, 17(9): 1538–1549

[15] Chang C H, Chuang Y Y. A line-structure-preserving approach to image resizing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 1075–1082

[16] Lin S S, Yeh I C, Lin C H, Lee T Y. Patch-based image warping for content-aware retargeting. IEEE Transactions on Multimedia (TMM), 2013, 15(2): 359–368

[17] Felzenszwalb P F, Huttenlocher D P. Efficient graph-based image segmentation. International Journal of Computer Vision, 2004, 59(2): 167–181

[18] Wu Y C, Liu X T, Liu S X, Ma K L. ViSizer: a visualization resizing framework. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(2): 278–290

[19] Gallea R, Ardizzone E, Pirrone R. Physical metaphor for streaming media retargeting. IEEE Transactions on Multimedia, 2014, 16(4): 971–979

[20] Yan B, Yang X C, Li K. Efficient image retargeting via adaptive pixel fusion. In: Proceedings of the 22nd ACM International Conference on Multimedia. 2014, 929–932

[21] Rubinstein M, Shamir A, Avidan S. Multi-operator media retargeting. ACM Transactions on Graphics, 2009, 28(3): 23

[22] Dong W M, Zhou N, Paul J C, Zhang X P. Optimized image resizing using seam carving and scaling. ACM Transactions on Graphics, 2009, 28(5): 125

[23] Liu Z, Yan H B, L. Shen L Q, Ngan K N, Zhang Z Y. Adaptive image retargeting using saliency-based continuous seam carving. Optical Engineering, 2010, 49(1)

[24] Zhang G X, Cheng M M, Hu S M, Martin R R. A shape-preserving approach to image resizing. Computer Graphics Forum, 2009, 28(7): 1897–1906

[25] Liu Y, Sun L F, Yang S Q. A retargeting method for stereoscopic 3D video. Computational Visual Media, 2015, 1(2): 119–127

[26] Dong W M, Wu F Z, Kong Y, Mei X, Lee T Y, Zhang X P. Image retargeting by texture-aware synthesis. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2016, 22(2): 1088–1101

[27] Dong W M, Bao G B, Zhang X P, Paul J C. Fast multi-operator image resizing and evaluation. Journal of Computer Science and Technology, 2012, 27(1): 121–134

[28] Wu H, Wang Y S, Feng K C, Wong T T, Lee T Y, Heng P A. Resizing by symmetry-summarization. ACM Transactions on Graphics, 2010, 29(6): 159

[29] Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998 (11): 1254–1259

[30] Hu S M, Chen T, Xu K, Cheng M M, Martin R R. Internet visual media processing: a survey with graphics and vision applications. The Visual Computer, 2013, 29(5): 393–405

[31] Kraevoy V, Sheffer A, Shamir A, Cohen-Or D. Non-homogeneous resizing of complex models. ACM Transactions on Graphics, 2008, 27(5): 111

[32] Wang K P, Zhang C M. Content-aware model resizing based on surface deformation. Computers & Graphics, 2009, 33(3): 433–438

[33] Xiao C X, Jin L Q, Nie Y W, Wang R F, Sun H Q, Ma K L. Contentaware model resizing with symmetry-preservation. The Visual Computer, 2015, 31(2): 155–167

[34] Chen L, Meng X X. Anisotropic resizing of model with geometric textures. In: Proceedings of the 2009 SIAM/ACM Joint Conference on Geometric and Physical Modeling. 2009, 289–294

[35] Lin J J, Cohen-Or D, Zhang H, Liang C, Sharf A, Deussen O, Chen B Q. Structure-preserving retargeting of irregular 3D architecture. ACM Transactions on Graphics, 2011, 30(6): 183

[36] Shamir A, Sorkine O. Visual media retargeting. ACM SIGGRAPH ASIA 2009 Courses, 2009

[37] Rubinstein M, Shamir A, Avidan S. Improved seam carving for video retargeting. ACM Transactions on Graphics, 2008, 27(3): 1–9

[38] Chiang C K, Wang S F, Chen Y L, Lai S H. Fast JND-based video carving with GPU acceleration for real-time video retargeting. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(11): 1588–1597

[39] Chao W L, Su H H, Chien S Y, Hsu W, Ding J J. Coarse-to-fine temporal optimization for video retargeting based on seam carving. In: Proceedings of the 2011 IEEE International Conference on Multimedia and Expo. 2011, 1–6

[40] Deselaers T, Dreuw P, Ney H. Pan, zoom, scan – time-coherent, trained automatic video cropping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8

[41] Fan X, Xie X, Zhou H Q, Ma W Y. Looking into video frames on small displays. In: Proceedings of the 11th ACM international conference on Multimedia. 2003, 247–250

[42] Liu F, Gleicher M. Video retargeting: automating pan and scan. In: Proceedings of the 14th Annual ACM International Conference on Multimedia. 2006, 241–250

[43] Kopf S, Haenselmann T, Farin D, Effelsberg W. Automatic generation of summaries for the Web. In: Yeung M M, Lienhart R W, Li C S, eds. Storage and Retrieval for Image and Video Databases, 2004, 417–428

[44] Wolf L, Guttmann M, Cohen-Or D. Non-homogeneous content-driven video-retargeting. In: Proceedings of the 11th IEEE International Conference on Computer Vision. 2007, 1–6

[45] Zhang Y F, Hu S M, Martin R R. Shrinkability maps for content-aware video resizing. Computer Graphics Forum, 2008, 27(7): 1797–1804

[46] Wang Y S, Fu H, Sorkine O, Lee T Y, Seidel H P. Motion-aware temporal coherence for video resizing. ACMTransactions on Graphics, 2009, 28(5): 127

[47] Krähenbühl P, Lang M, Hornung A, Gross M. A system for retargeting of streaming video. ACM Transactions on Graphics, 2009, 28(5): 126

[48] Kim J S, Kim J H, Kim C S. Adaptive image and video retargeting technique based on Fourier analysis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1730–1737

[49] Wang S F, Lai S H. Compressibility-aware media retargeting with structure preserving. IEEE Transactions on Image Processing, 2011, 20(3): 855–865

[50] Shi L, Wang J Q, Duan L Y, Lu H Q. Consumer video retargeting: context assisted spatial-temporal grid optimization. In: Proceedings of the 17th ACM International Conference on Multimedia. 2009, 301–310

[51] Wang Y S, Lin H C, Sorkine O, Lee T Y. Motion-based video retargeting with optimized crop-and-warp. ACM Transactions on Graphics, 2010, 29(4): 90

[52] Wang Y S, Hsiao J H, Sorkine O, Lee T Y. Scalable and coherent video resizing with per-frame optimization. ACM Transactions on Graphics, 2011, 30(4): 88

[53] Yen T C, Tsai C M, Lin C W. Maintaining temporal coherence in video retargeting using mosaic-guided scaling. IEEE Transactions on Image Processing, 2011, 20(8): 2339–2351

[54] Khan S, Shah M. Object based segmentation of video using color, motion and spatial information. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2001

[55] Paris S. Edge-preserving smoothing and mean-shift segmentation of video streams. In: Forsyth D, Torr P, Zisserman A, eds. Computer Vision–ECCV 2008. Springer Berlin Heidelberg, 2008, 460–473

[56] Wang J, Thiesson B, Xu Y Q, Cohen M. Image and video segmentation by anisotropic kernel mean shift. In: Proceedings of the 10th European Conference on Computer Vision. 2004, 238–249

[57] Hu Y Q, Rajan D. Hybrid shift map for video retargeting. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 577–584

[58] Yan B, Sun K R, Liu L. Matching area based seam carving for video retargeting. IEEE Transactions on Circuits and Systems for Video Technology. 2013, 23(2): 302–310

[59] Lin S S, Lin C H, Yeh I C, Chang S H, Yeh C K, Lee T Y. Contentaware video retargeting using object-preserving warping. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(10): 1677–1686

[60] Qu Z, Wang J Q, Xu M, Lu H Q. Context-aware video retargeting via graph model. IEEE Transactions on Multimedia, 2013, 15(7): 1677–1687

[61] Yuan Z, Lu T R, Huang Y, Wu D P, Yu H. Addressing visual consistency in video retargeting: a refined homogeneous approach. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(6): 890–903

[62] Li B, Duan L Y, Wang J, Ji R, Lin C W, Gao W. Spatiotemporal grid flow for video retargeting. IEEE Transactions on Image Processing, 2014, 23(4): 1615–1628

[63] Nie Y W, Zhang Q, Wang R F, Xiao C X. Video retargeting combining warping and summarizing optimization. The Visual Computer, 2013, 29(6–8): 785–794

[64] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612

[65] Hsu C C, Lin C W, Fang Y, Lin W. Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss. IEEE Journal of Selected Topics in Signal Processing, 2014, 8(3): 377–389

[66] Bare B, Li K, Wang W Y, Yan B. Learning to assess image retargeting. In: Proceedings of the 22nd ACM International Conference on Multimedia. 2014, 925–928

[67] Rubinstein M, Gutierrez D, Sorkine O, Shamir A. A comparative study of image retargeting. ACM Transactions on Graphics, 2010, 29(6): 160

[68] Pele O, Werman M. Fast and robust earth mover’s distances. In: Proceedings of the 12th IEEE international conference on Computer vision. 2009, 460–467

[69] Liu C, Yuen J, Torralba A, Sivic J, Freeman W T. Sift flow: dense correspondence across different scenes. In: Proceedings of the 10th European Conference on Computer Vision. 2008, 28–42

[70] Liu Y J, Luo X, Xuan Y M, Chen W F, Fu X L. Image retargeting quality assessment. Computer Graphics Forum, 2011, 30(2): 583–592

[71] Zhang J, Kuo C C J. An objective quality of experience (QoE) assessment index for retargeted images. In: Proceedings of the ACM International Conference on Multimedia. 2014, 257–266

[72] Fang Y M, Zeng K, Wang Z, Lin W S, Fang Z J, Lin C W. Objective quality assessment for image retargeting based on structural similarity. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2014, 4(1): 95–105

[73] Barnes C, Shechtman E, Finkelstein A, Goldman D. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009, 28(3): 24

[74] Manjunath B S, Ohm J R, Vasudevan V V, Yamada A. Color and texture descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 2001, 11(6): 703–715

[75] Simakov D, Caspi Y, Shechtman E, Irani M. Summarizing visual data using bidirectional similarity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8

[76] Kasutani E, Yamada A. The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval. In: Proceedings of the 2001 International Conference on Image Processing. 2001, 674–677

[77] Yan B, Yuan B H, Yang B. Effective video retargeting with jittery assessment. IEEE Transactions on Multimedia, 2014, 16(1): 272–277

[78] Tsai S S, Chen D, Takacs G, Chandrasekhar V, Singh J P, Girod B. Location coding for mobile image retrieval. In: Proceedings of the 5th International ICST Mobile Multimedia Communications Conference. 2009

[79] V Chandrasekhar V, Takacs G, Chen D, Tsai S, Grzeszczuk R, Girod B. Chog: compressed histogram of gradients a low bit-rate feature descriptor. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009, 2504–2511

[80] Lowe D G. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2004, 60(2): 91–110

[81] Yang X Y, Liu L L, Qian X M, Mei T, Shen J L, Tian Q. Mobile visual search via hievarchical sparse coding. In: Proceedings of the 2014 IEEE International Conference on Multimedia and Expo. 2014, 1–6

[82] Tan W M, Yan B, Li K, Tian Q. Image retargeting for preserving robust local feature: application to mobile visual search. IEEE Transactions on Multimedia, 2016, 18(1): 128–137

[83] Ke Y, Sukthankar R. PCA-SIFT: a more distinctive representation for local image descriptors. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2004, 506–513

[84] Seber G A F. Multivariate observations. John Wiley & Sons, 2009

[85] Spath H. The cluster dissection and analysis theory FORTRAN programs examples. Prentice-Hall, Inc., 1985

[86] Philbin J, Chum O, Isard M, Sivic J, Zisserman A. Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8

[87] Nie L Q, Wang M, Gao Y, Zha Z J, Chua T S. Beyond text QA: multimedia answer generation by harvesting Web information. IEEE Transactions on Multimedia, 2013, 15(2): 426–441

[88] Nie L Q, Yan S C, Wang M, Hong R C, Chua T S. Harvesting visual concepts for image search with complex queries. In: Proceedings of the 20th ACM international conference on Multimedia. 2012, 59–68

[89] Nie L Q, Wang M, Zha Z J, Chua T S. Oracle in image search: a content-based approach to performance prediction. ACM Transactions on Information Systems, 2012, 30(2): 13

[90] Hong R C, Li G D, Nie L Q, Tang J H, Chua T S. Exploring large scale data for multimedia QA: an initial study. In: proceedings of the ACM International Conference on Image and Video Retrieval. 2010, 74–81

[91] Lu S P, Dauphin G, Lafruit G, Munteanu A. Color retargeting: interactive time-varying color image composition from time-lapse sequences. Computational Visual Media, 2015, 1(4): 321–330

[92] Guo Y W, Liu M, Gu T T, Wang W P. Improving photo composition elegantly: considering image similarity during composition optimization. Computer Graphics Forum, 2012, 31(7): 2193–2202

[93] Zhang F L, Wang M, Hu S M. Aesthetic image enhancement by dependence-aware object recomposition. IEEE Transactions on Multimedia, 2013, 15(7): 1480–1490

[94] Li K, Yan B, Li J, Majumder A. Seam carving based aesthetics enhancement for photos. Signal Processing: Image Communication, 2015, 39: 509–516

[95] Bertalmio M, Sapiro G, Caselles V, Ballester C. Image in-painting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. 2000, 417–424

[96] Yeung M M, Yeo B L. Video visualization for compact presentation and fast browsing of pictorial content. IEEE Transactions on Circuits and Systems for Video Technology, 1997, 7(5): 771–785

[97] Oh J, Wen Q, Lee J, Hwang S, Video abstraction. Hershey, PA: Idea Group Inc. and IRM Press, 2004

[98] Liu T M, Zhang H J, Qi F H. A novel video key-frame-extraction algorithm based on perceived motion energy model. IEEE Transactions on Circuits and Systems for Video Technology, 2003, 13(10): 1006–1013

[99] Hanjalic A, Zhang H J. An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis. IEEE Transactions on Circuits and Systems for Video Technology, 1999, 9(8): 1280–1289

[100] You J Y, Liu G Z, Sun L, Li H L. A multiple visual models based perceptive analysis framework for multilevel video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(3): 273–285

[101] Qu W, Zhang Y F, Wang D L, Feng S, Yu G. Semantic movie summarization based on string of IE-RoleNets. Computational Visual Media, 2015, 1(2): 129–141

[102] Pritch Y, Rav-Acha A, Peleg S. Nonchronological video synopsis and indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(11): 1971–1984

[103] Lu S P, Zhang S H, Wei J, Hu S M, Martin R R. Timeline editing of objects in video. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(7): 1218–1227

[104] Nie Y W, Sun H Q, Li P, Xiao C X, Ma K L. Object movements synopsis via part assembling and stitching. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(9): 1303–1315

[105] Nie Y W, Xiao C X, Sun H Q, Li P. Compact video synopsis via global spatiotemporal optimization. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(10): 1664–1676

[106] Li K, Yan B, Wang W, Gharavi H. An effective video synopsis approach with seam carving. IEEE Signal Processing Letters, 2016, 23(1): 11–14

[107] Lee D S. Effective Gaussian mixture learning for video background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(5): 827–832

[108] Li Z, Ishwar P, Konrad J. Video condensation by ribbon carving. IEEE Transactions on Image Processing, 2009, 18(11): 2572–2583

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag Berlin Heidelberg