Large-scale video compression: recent advances and challenges

Abstract

The evolution of social network and multimedia technologies encourage more and more people to generate and upload visual information, which leads to the generation of large-scale video data. Therefore, preeminent compression technologies are highly desired to facilitate the storage and transmission of these tremendous video data for a wide variety of applications. In this paper, a systematic review of the recent advances for large-scale video compression (LSVC) is presented. Specifically, fast video coding algorithms and effective models to improve video compression efficiency are introduced in detail, since coding complexity and compression efficiency are two important factors to evaluate video coding approaches. Finally, the challenges and future research trends for LSVC are discussed.

Keywords

large-scale video compression / fast video coding / compression efficiency

References

[1] Wiegand T, Sullivan G J, Bjøntegaard G, Luthra A. Overview of the H.264/AVC video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 2003, 13(7): 560–576

[2] Sullivan G J, Ohm J R, Han W J, Wiegand T. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1649–1668

[3] Cherubini M, Oliveira R D, Oliver N. Understanding near-duplicate videos: a user-centric approach. In: Proceedings of ACM International Conference on Multimedia. 2009, 35–44

[4] Zhao L, Fan X, Ma S, Zhao D. Fast intra-encoding algorithm for high efficiency video coding. Signal Processing: Image Communication, 2014, 29(9): 935–944

[5] Cho S, Kim M. Fast CU splitting and pruning for suboptimal CU partitioning in HEVC intra coding. IEEE Transactions on Circuits and Systems for Video Technology, 2013, 23(9): 1555–1564

[6] Min B, Cheung R C C. A fast CU size decision algorithm for the HEVC intra encoder. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(5): 892–896

[7] Zhang Q, Sun J, Duan Y, Guo Z. A two-stage fast CU size decision method for HEVC intracoding. In: Proceedings of International Workshop on Multimedia Signal Processing. 2015, 1–6

[8] Lee D, Jeong J. Fast intra coding unit decision for high efficiency video coding based on statistical information. Signal Processing: Image Communication. 2017, 55: 121–129

[9] Wang Y, Fan X, Zhao L, Ma S, Zhao D, Gao W. A fast intra coding algorithm for HEVC. In: Proceedings of IEEE International Conference on Image Processing. 2014, 4117–4121

[10] Wang Y, Takagi R, Yoshitake G. A simple and fast CU division algorithm for HEVC intra prediction. IEICE Transactions on Information and Systems, 2017, 100(5): 1140–1143

[11] Zhang Y, Kwong S, Zhang G, Pan Z, Yuan H, Jiang G. Low complexity HEVC intra coding for high-quality mobile video communication. IEEE Transactions on Industrial Informatics, 2015, 11(6): 1492–1504

[12] Liu Z, Yu X, Gao Y, Chen S, Ji X, Wang D. CU partition mode decision for HEVC hardwired intra encoder using convolution neural network. IEEE Transactions on Image Processing, 2016, 25(11): 5088–5103

[13] Lim K, Lee J, Kim S, Lee S. Fast PU skip and split termination algorithm for HEVC intra prediction. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(8): 1335–1346

[14] Hu N, Yang E H. Fast mode selection for HEVC intra-frame coding with entropy coding refinement based on a transparent composite model. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(9): 1521–1532

[15] Na S, Lee W, Yoo K. Edge-based fast mode decision algorithm for intra prediction in HEVC. In: Proceedings of IEEE International Conference on Consumer Electronics. 2014, 11–14

[16] Chen G, Liu Z, Ikenaga T, Wang D. Fast HEVC intra mode decision using matching edge detector and kernel density estimation alike histogram generation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2013, 53–56

[17] Yao Y, Li X J, Lu Y. Fast intra mode decision algorithm for HEVC based on dominant edge assent distribution. Multimedia Tools and Applications, 2015, 75(4): 1963–1981

[18] Shen L, Zhang Z, An P. Fast CU size decision and mode decision algorithm for HEVC intra coding. IEEE Transactions on Consumer Electronics, 2013, 59(1): 207–213

[19] Zhang T, Sun M T, Zhao D, Gao W. Fast intra mode and CU size decision for HEVC. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(8): 1714–1726

[20] Xiong J, Li H, Meng F, Wu Q, Ngan K N. Fast HEVC inter CU decision based on latent SAD estimation. IEEE Transactions on Multimedia, 2015, 17(12): 2147–2159

[21] Shen L, Liu Z, Zhang X, Zhao W, Zhang Z. An effective CU size decision method for HEVC encoders. IEEE Transactions on Multimedia, 2013, 15(2): 465–470

[22] Pan Z, Kwong S, Zhang Y, Lei J, Yuan H. Fast coding tree unit depth decision for high efficiency video coding. In: Proceedings of IEEE International Conference on Image Processing. 2014, 3214–3218

[23] Wang H, Heng Y, Dun H. Optimal stopping theory based algorithm for coding unit size decision in HEVC. In: Proceedings of Asia- Pacific Signal and Information Processing Association Annual Summit and Conference. 2014, 1–6

[24] Wu X, Wang H, Wei Z. Optimal stopping theory based fast coding tree unit decision for high efficiency video coding. In: Proceedings of Visual Communications and Image Processing. 2016, 1–4

[25] Li Y, Yang G, Zhu Y, Ding X, Sun X. Adaptive inter CU depth decision for HEVC using optimal selection model and encoding parameters. IEEE Transactions on Broadcasting, 2017, 63(3): 535–546

[26] Zupancic I, Blasi S G, Peixoto E, Izquierdo E. Inter-prediction optimizations for video coding using adaptive coding unit visiting order. IEEE Transactions on Multimedia, 2016, 18(9): 1677–1690

[27] Yang J, Kim J, Won K, Lee H, Jeon B. Early skip detection for HEVC. JCT-VC document, JCTVC-G543, 2011.

[28] Goswami K, Lee J H, Jang K S, Kim B G, Kwon K K. Entropy difference-based early skip detection technique for high-efficiency video coding. Journal of Real-Time Image Processing, 2016, 12(2): 237–245

[29] Lee H, Shim H J, Park Y, Jeon B. Early skip mode decision for HEVC encoder with emphasis on coding quality. IEEE Transactions on Broadcasting, 2015, 61(3): 388–397

[30] Li Y, Yang G, Zhu Y, Ding X, Sun X. Unimodal stopping model based early SKIP mode decision for high efficiency video coding. IEEE Transactions on Multimedia, 2017, 19(7): 1431–1441

[31] Shen L, Zhang Z, Liu Z. Adaptive inter-mode decision for HEVC jointly utilizing inter-level and spatiotemporal correlations. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(10): 1709–1722

[32] Zhang J, Li B, Li H. An efficient fast mode decision method for inter prediction in HEVC. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(8): 1502–1515

[33] Jung S H, Park H W. A fast mode decision method in HEVC using adaptive ordering of modes. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(10): 1846–1858

[34] Ahn S, Lee B, Kim M. A novel fast CU encoding scheme based on spatiotemporal encoding parameters for HEVC inter coding. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(3): 422–435

[35] Chen F, Li P, Peng Z, Jiang G, Yu M, Shao F. A fast inter coding algorithm for HEVC based on texture and motion quad-tree models. Signal Processing: Image Communication, 2016, 47: 271–279

[36] Kim H S, Park R H. Fast CU partitioning algorithm for HEVC using an online-learning-based bayesian decision rule. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(1): 130–138

[37] Correa G, Assuncao P A, Agostini L V, Silva Cruz L A. Fast HEVC encoding decisions using data mining. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(4): 660–673

[38] Zhang Y, Kwong S, Wang X, Yuan H, Pan Z, Xu L. Machine learningbased coding unit depth decisions for flexible complexity allocation in high efficiency video coding. IEEE Transactions on Image Processing, 2015, 24(7): 2225–2238

[39] Zhu L, Zhang Y, Pan Z, Wang R, Kwong S, Peng Z. Binary and multiclass learning based low complexity optimization for HEVC encoding. IEEE Transactions on Broadcasting, 2017, 63(3): 547–561

[40] Kim I K, McCann K, Sugimoto K, Han W J. High efficiency video coding (HEVC) test model 10 encoder description. JCT-VC, Doc. JCTVC-L1002, 2013

[41] Zhao L, Tian Y, Huang T. Background-foreground division based search for motion estimation in surveillance video coding. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2014, 1–6

[42] Zhu W, Ding W, Xu J, Shi Y, Yin B. Hash-based block matching for screen content coding. IEEE Transactions on Multimedia, 2015, 17(7): 935–944

[43] Gao L, Dong S, Wang W, Wang R, Gao W. A novel integer-pixel motion estimation algorithm based on quadratic prediction. In: Proceedings of IEEE International Conference on Image Processing. 2015, 2810–2814

[44] Chen K, Sun J, Guo Z, Zhao D. A novel two-step integer-pixel motion estimation algorithm for HEVC encoding on a GPU. In: Proceedings of International Conference on Multimedia Modeling. 2017, 28–36

[45] Liao Z T, Shen C A. A novel search window selection scheme for the motion estimation of HEVC systems. In: Proceedings of International SoC Design Conference. 2015, 267–268

[46] Li Y, Liu Y, Yang H, Yang D. An adaptive search range method for HEVC with the k-nearest neighbor algorithm. In: Proceedings of Visual Communications and Image Processing. 2015, 1–4

[47] Pan Z, Lei J, Zhang Y, Sun X, Kwong S. Fast motion estimation based on content property for low-complexity H.265/HEVC encoder. IEEE Transactions on Broadcasting, 2016, 62(3): 675–684

[48] Fan R, Zhang Y, Li B. Motion classification-based fast motion estimation for high-efficiency video coding. IEEE Transactions on Multimedia, 2017, 19(5): 893–907

[49] Lim D B, Choi Y K, Lee H J, Chae S I. A fast fractional motion estimation algorithm for high efficiency video coding. In: Proceedings of International Conference on Electronics, Information, and Communications. 2016, 1–4

[50] Jia S, Ding W, Shi Y, Yin B. A fast sub-pixel motion estimation algorithm for HEVC. IEEE International Symposium on Circuits and Systems. 2016, 566–569

[51] Zhang Y, Kwong S, Jiang G, Wang H. Efficient multi-reference frame selection algorithm for hierarchical B pictures inmultiview video coding. IEEE Transactions on Broadcasting, 2011, 57(1): 15–23

[52] Liu Z, Li L, Song Y, Li S, Goto S, Ikenaga T. Motion feature and hadamard coefficient-based fast multiple reference frame motion estimation for H.264. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(5): 620–632

[53] Wang S, Ma S, Wang S, Zhao D, Gao W. Fast multi reference frame motion estimation for high efficiency video coding. In: Proceedings of IEEE International Conference on Image Processing. 2013, 2005–2009

[54] Yang S H, Huang K S. HEVC fast reference picture selection. Electronics Letters, 2015, 51(25): 2109–2111

[55] Pan Z, Jin P, Lei J, Zhang Y, Sun X, Kwong S. Fast reference frame selection based on content similarity for low complexity HEVC encoder. Journal of Visual Communication and Image Representation, 2016, 40: 516–524

[56] Teng S W, Hang H M, Chen Y F. Fast mode decision algorithm for residual quadtree coding in HEVC. In: Proceedings of IEEE Visual Communications and Image Processing. 2011, 1–4

[57] Shen L, Zhang Z, Zhang X, An P, Liu Z. Fast TU size decision algorithm for HEVC encoders using Bayesian theorem detection. Signal Processing: Image Communication, 2015, 32: 121–128

[58] Wu X, Wang H, Wei Z. Bayesian rule based fast TU depth decision algorithm for high efficiency video coding. In: Proceedings of IEEE Visual Communications and Image Processing. 2016, 1–4

[59] Wang H, Kwong S. Prediction of zero quantized DCT coefficients in H.264/AVC using hadamard transformed information. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(4): 510–515

[60] Wang H, Kwong S. Hybrid model to detect zero quantized DCT coefficients in H.264. IEEE Transactions on Multimedia, 2007, 9(4): 728–735

[61] Wang H, Du H, Wu J. Predicting zero coefficients for high efficiency video coding. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2014, 1–6

[62] Wang H, Du H, Lin W, Kwong S, Au O C, Wu J, Wei Z. Early detection of all-zero 4 × 4 blocks in High Efficiency Video Coding. Journal of Visual Communication and Image Representation, 2014, 25(7): 1784–1790

[63] Lee B, Jung J, Kim M. An all-zero block detection scheme for low-complexity HEVC encoders. IEEE Transactions on Multimedia, 2016, 18(7): 1257–1268

[64] Au O C, Li S, Zou R, Dai W, Sun L. Digital photo album compression based on global motion compensation and intra/inter prediction. In: Proceedings of International Conference on Audio, Language and Image Processing. 2012, 84–90

[65] Zou R, Au O C, Zhou G, Dai W, Hu W, Wan P. Personal photo album compression and management. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2013, 1428–1431

[66] Ling Y, Au O C, Zou R, Pang J, Yang H, Zheng A. Photo album compression by leveraging temporal-spatial correlations and HEVC. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2014, 1917–1920

[67] Shi Z, Sun X, Wu F. Photo album compression for cloud storage using local features. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2014, 4(1): 17–28

[68] Wu H, Sun X, Yang J, Zeng W, Wu F. Lossless compression of JPEG coded photo collections. IEEE Transactions on Image Processing, 2016, 25(6): 2684–2696

[69] Vetro A, Wiegand T, Sullivan G J. Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard. Proceedings of the IEEE, 2011, 99(4): 626–642.

[70] Merkle P, Smolíc A, Müller K, Wiegand T. Efficient prediction structures for multiview video coding. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(11): 1461–1473

[71] Wang H, Ma M, Jiang Y G, Wei Z. A framework of video coding for compressing near-duplicate videos. In: Proceedings of International Conference on Multimedia Modeling. 2014, 518–528

[72] Wang H, Ma M, Tian T. Effectively compressing near-duplicate videos in a joint way. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2015, 1–6

[73] Bay H, Tuytelaars T, Gool L V. Surf: speeded up robust features. In: Proceedings of European Conference on Computer Vision. 2006, 404–417

[74] Muja M, Lowe D G. Scalable nearest neighbor algorithms for high dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(11): 2227–2240

[75] Fishler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395

[76] Wang H, Tian T, Ma M, Wu J. Joint compression of near-duplicate videos. IEEE Transactions on Multimedia, 2017, 19(5): 908–920

[77] Wu X, Ngo C W, Hauptmann A G, Tan H K. Real-time near-duplicate elimination for web video search with content and context. IEEE Transactions on Multimedia, 2009, 11(2): 196–207

[78] Wang H, Zhu F, Xiao B, Wang L, Jiang Y G. Gpu-based mapreduce for large-scale near-duplicate video retrieval. Multimedia Tools and Applications, 2015, 74(23): 10515–10534.

[79] Gao Y, Zhu C, Li S, Yang T. Temporal dependent rate-distortion optimization for low-delay hierarchical video coding. IEEE Transactions on Image Processing, 2017, 26(9): 4457–4470.

[80] Chen H, Zhang T, Sun M T, Saxena A, Budagavi M. Improving intra prediction in high-efficiency video coding. IEEE Transactions on Image Processing, 2016, 25(8): 3671–3682

[81] Lan C, Xu J, Shi G, Wu F. Variable block-sized signal dependent transform for video coding. IEEE Transactions on Circuits and Systems for Video Technology, 2017, DOI: 10.1109/TCSVT.2017.2689032

[82] Li L, Li H, Liu D, Li Z, Yang H, Lin S, Chen H, Wu F. An efficient four-parameter affine motion model for video coding. IEEE Transactions on Circuits and Systems for Video Technology, 2017, DOI: 10.1109/TCSVT.2017.2699919

[83] Ma S, Zhang X, Zhang J, Jia C, Wang S, Gao W. Nonlocal in-loop filter: the way toward next-generation video coding?. IEEE Multi Media, 2016, 23(2): 16–26

[84] Chen J, Chen Y, Karczewicz M, Li X, Liu H, Zhang L, Zhao X. Coding tools investigation for next generation video coding. ITU-T SG16 Doc. COM16-C806, 2015

[85] Karczewicz M, Chen J, Chien W J, Li X, Said A, Zhang L, Zhao X. Study of coding efficiency improvements beyond HEVC.MPEG Doc. m37102, 2015

[86] An J, Huang H, Zhang K. Quadtree plus binary tree structure integration with JEM tools. Joint Video Exploration Team, JVET-B0023, 2016

[87] Chen J, Chien W J, Karczewicz M, Li X, Liu H, Said A, Zhang L, Zhao X. Further improvements to HMKTA-1.0. ITU-T SG16/Q6 Doc. VCEG-AZ07, 2015

[88] Alshina E, Alshin A, Min J H, Choi K, Saxena A, Budagavi M. Known tools performance investigation for next generation video coding. ITU-T SG16/Q6 Doc. VCEG-AZ05, 2015

[89] Chien W J, Karczewicz M. Extension of advanced temporal motion vector predictor. ITU-T SG16/Q6 Doc. VCEG-AZ10, 2015

[90] Choi K, Alshina E, Alshin A, Kim C. Information on coding efficiency improvements over HEVC for 4K content. MPEG Doc. m37043, 2015

[91] Martin E. Saccadic suppression: a review and an analysis. Psychological Bulletin, 1974, 81(12): 899–917

[92] Itti L, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254–1259

[93] Gao D, Mahadevan V, Vasoncelos N. The discriminant centersurround hypothesis for bottom-up saliency. In: Proceedings of Advances in Neural Information Processing Systems. 2007, 497–504

[94] Goferman S, Zelnik-Manor L, Tal A. Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10): 1915–1926

[95] Imamoglu N, Lin W, Fang Y. A saliency detection model using lowlevel features based on wavelet transform. IEEE Transactions onMultimedia, 2013, 15(1): 96–105

[96] Hadizadeh H, Bajic I V. Saliency-aware video compression. IEEE Transactions on Image Processing, 2014, 23(1): 19–33

[97] Li Y, Liao W, Huang J, He D, Chen Z. Saliency based perceptual HEVC. In: Proceedings of IEEE International Conference on Multimedia and Expo Workshops. 2014, 1–5

[98] Doulamis N, Doulamis A, Kalogeras D, Kollias S. Low bit-rate coding of image sequences using adaptive regions of interest. IEEE Transactions on Circuits and Systems for Video Technology, 1998, 8(8): 928–934

[99] Xu M, Deng X, Li S, Wang Z. Region-of-interest based conversational HEVC coding with hierarchical perception model of face. IEEE Journal of Selected Topics in Signal Processing, 2014, 8(3): 475–489

[100] Yang X, Lin W, Lu Z, Ong E, Yao S. Motion-compensated residue preprocessing in video coding based on just-noticeable-distortion profile. IEEE Transactions on Circuits and Systems for Video Technology, 2005, 15(6): 742–752

[101] Liu A, Lin W, Paul M, Deng C, Zhang F. Just noticeable difference for images with decomposition model for separating edge and textured regions. IEEE Transactions on Circuits and Systems for Video Technology, 2010, 20(11): 1648–1652

[102] Wu J, Shi G, Lin W, Liu A, Qi F. Just noticeable difference estimation for images with free-energy principle. IEEE Transactions on Multimedia, 2013, 15(7): 1705–1710

[103] Wu J, Li L, Dong W, Shi G, Lin W, Kuo C C J. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing, 2017, 26(6): 2682–2693

[104] Ahumada A, Peterson H. Luminance-model-based DCT quantization for color image compression. Proceedings of the SPIE, 1992, 1666: 365–374

[105] Hontsch I, Karam L J. Adaptive image coding with perceptual distortion control. IEEE Transactions on Image Processing, 2002, 11(3): 213–222

[106] Wei Z, Ngan K N. Spatio-temporal just noticeable distortion profile for grey scale image/video in DCT domain. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(3): 337–346

[107] Hu S, Wang H, Kuo C C J. A GMM-based stair quality model for human perceived JPEG images. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. 2016, 1070–1074

[108] Jin L, Yuchieh L J, Hu S, Wang H, Wang P, Katsavounidis I, Aaron A, Kuo C C J. Statistical study on perceived JPEG image quality via MCL-JCI dataset construction and analysis. Electronic Imaging, 2016, 9: 1–9

[109] Chen Z, Guillemot C. Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model. IEEE Transactions on Circuits and Systems for Video Technology, 2010, 20(6): 806–819

[110] Luo Z, Song L, Zheng S, Ling N. H.264/advanced video control perceptual optimization coding based on JND-directed coefficient suppression. IEEE Transactions on Circuits and Systems for Video Technology, 2013, 23(6): 935–948

[111] Yang X K, Ling W S, Lu Z K, Ong E P, Yao S S. Just noticeable distortion model and its applications in video coding. Signal Processing: Image Communication, 2005, 20(7): 662–680

[112] Kim J, Bae S H, Kim M. An HEVC-compliant perceptual video coding scheme based on JND models for variable block-sized transform kernels. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(11): 1786–1800

[113] Abdoli M, Henry F, Brault P, Duhamel P, Dufaux F. Intra prediction using in-loop residual coding for the post-HEVC standard. In: Proceedings of IEEE International Workshop on Multimedia Signal Processing. 2017, 1–6

[114] Wang H, Fu J, Lin W, Hu S, Kuo C C J, Zuo L. Image quality assessment based on local linear lnformation and distortion-specific compensation. IEEE Transactions on Image Processing, 2017, 26(2): 915–926

[115] Wang T, Chen M, Chao H. A novel deep learning-based method of improving coding efficiency from the decoder-end for HEVC. In: Proceedings of Data Compression Conference. 2017, 410–419

[116] Li Y, Liu D, Li H, Li L, Wu F, Zhang H, Yang H. Convolutional neural network-based block up-sampling for intra frame coding. IEEE Transactions on Circuits and Systems for Video Technology, 2017, DOI: 10.1109/TCSVT.2017.2727682

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature