A survey of data-driven approach on multimedia QoE evaluation

Abstract

With the development of mobile communication technology and the growth of mobile device, the requirements for user quality of experience (QoE) become higher and higher. Network operators and content providers are interested in QoE evaluation for improving users’ QoE. However, multimedia QoE evaluation faces severe challenges due to the subjective properties of the QoE. In this paper, we provide a survey of the state of the art about applying data-driven approach on QoE evaluation. Firstly, we describe the way to choose factors influencing QoE. Then we investigate and discuss the strengths and shortcomings of existing machine learning algorithms for modeling and predicting users’ QoE. Finally, we describe our research work on how to evaluate QoE in imbalanced dataset.

Keywords

quality of experience / data-driven / machine learning / imbalanced dataset

References

[1] Cisco V N I. Global mobile data traffic forecast update, 2015–2020, White Paper, Document ID, 2016, 958959758

[2] Index C U N. Cisco visual networking index: global mobile data traffic forecast2014–2019. Technical Report, 2015

[3] ITU-T Recommendation P.10/G.100, Vocabulary for performance and quality of service. Amendment 2: New definitions for inclusion in Recommendation ITU-T P.10/G.100, Int. Telecomm. Union, Geneva 2008.

[4] ETSI Technical Report 102 643 V1.0.2, Human Factors (HF); Quality of Experience (QoE) requirements for real-time communication services, 2010

[5] Yamagishi K, Hayashi T. Parametric packet-layer model for monitoring video quality of IPTV services. In: Proceedings of 2008 IEEE International Conference on Communications. 2008, 110–114

[6] Belmudez B, Möller S. Extension of the G.1070 video quality function for the MPEG2 video codec. In: Proceedings of the 2nd International Workshop on Quality of Multimedia Experience. 2010, 7–10

[7] You F H, Zhang W, Xiao J. Packet loss pattern and parametric video quality model for IPTV. In: Proceedings of the 8th IEEE/ACIS International Conference on Computer and Information Science. 2009, 824–828

[8] Wang Z, Li L, Wang W, Wan Z, Fang Y, Cai C. A study on QoS/QoE correlation model in wireless-network. In: Proceedings of Signal and Information Processing Association Annual Summit and Conference. 2014, 1–6

[9] Kim H J, Lee D H, Lee J M, Lee K H, Lyu W, Choi S G. The QoE evaluation method through the QoS-QoE correlation model. In: Proceedings of Networked Computing and Advanced Information Management. 2008, 719–725

[10] Paudyal P, Battisti F, Carli M. Impact of video content and transmission impairments on quality of experience. Multimedia Tools and Applications, 2016, 75(23): 16461–16485

[11] Kim J, Um T W, Ryu W, Lee B S, Hahn M. IPTV systems, standards and architectures: part ii-heterogeneous networks and terminalaware QoS/QoE-guaranteed mobile IPTV service. IEEE Communications Magazine, 2008, 46(5): 110–117

[12] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612

[13] Khorsandroo S, Md Noor R, Khorsandroo S. A generic quantitative relationship to assess interdependency of QoE and QoS. KSII Transactions on Internet and Information Systems, 2013, 7(2): 327–346

[14] Reichl P, Egger S, Schatz R, D'alconzo A. The logarithmic nature of QoE and the role of the weber-fechner law in QoE assessment. In: Proceedings of 2010 IEEE International Conference on Communications. 2010, 1–5

[15] Reichl P, Tuffin B, Schatz R. Logarithmic laws in service quality perception: where microeconomics meets psychophysics and quality of experience. Telecommunication Systems, 2013, 52(2): 587–600

[16] Hannuksela M M. Does context matter in quality evaluation of mobile television? In: Proceedings of International Conference on Human Computer Interaction with Mobile Devices and Services. 2008, 63–72

[17] Kaikkonen A, Kekäläinen A, Cankar M, Kallio T, Kankainen A. Usability testing of mobile applications: a comparison between laboratory and field testing. Journal of Usability Studies, 2005, 1(1): 4–16

[18] Kjeldskov J, Stage J. New techniques for usability evaluation of mobile systems. International Journal of Human-Computer Studies, 2004, 60(5–6): 599–620

[19] Machajdik J, Hanbury A, Garz A, Sablatnig R. Affective computing for wearable diary and lifelogging systems: an overview. In: Proceedings of the Workshop of the Austrian Association for Pattern Recognition. 2011, 2447–2456

[20] Hamm J, Stone B, Belkin M, Dennis S. Automatic annotation of daily activity from smartphone-based multisensory streams. In: Proceedings of the International Conference on Mobile Computing, Applications, and Services. 2012, 328–342

[21] Jalal A, Uddin M Z, Kim T S. Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home. IEEE Transactions on Consumer Electronics, 2012, 58(3): 863–871

[22] Kamal S, Jalal A. A hybrid feature extraction approach for human detection, tracking and activity recognition using depth sensors. Arabian Journal for Science and Engineering, 2016, 41(3): 1043–1051

[23] Jalal A, Kamal S, Kim D. A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors, 2014, 14(7): 11735–11759

[24] Jalal A, Sarif N, Kim J T, Kim T S. Human activity recognition via recognized body parts of human depth silhouettes for residents monitoring services at smart home. Indoor and Built Environment, 2013, 22(1): 271–279

[25] Farooq A, Jalal A, Kamal S. Dense RGB-D map-based human tracking and activity recognition using skin joints features and self-organizing map. Ksii Transactions on Internet and Information Systems, 2015, 9(5): 1856–1869

[26] Jalal A, Kim Y, Kim D. Ridge body parts features for human pose estimation and recognition from RGB-D video data. In: Proceedings of International Conference on Computing, Communication and Networking Technologies. 2014, 1–6

[27] Jalal A, Kim J T, Kim T S. Development of a life logging system via depth imaging-based human activity recognition for smart homes. In: Proceedings of the International Symposium on Sustainable Healthy Buildings. 2012, 91–104

[28] Jalal A, Lee S, Kim J T, Kim T S. Human activity recognition via the features of labeled depth body parts. In: Proceedings of International Conference on Smart Homes and Health Telematics. 2012, 246–249

[29] Barakovi′c S, Skorin-Kapov L. Survey and challenges of QoE management issues in wireless networks. Journal of Computer Networks and Communications, 2013: 1–28

[30] He Z, Mao S, Jiang T. A survey of QoE-driven video streaming over cognitive radio networks. IEEE Network, 2015, 29(6): 20–25

[31] Tang W, Nguyen T D, Huh E N. A survey study on QoE perspective of mobile cloud computing. In: Proceedings of IEEE International Conference on Information Science and Applications. 2014, 1–4

[32] Mitra K, Zaslavsky A, hlund C. QoE modelling, measurement and prediction: a review. 2014, arXiv preprint arXiv:1410.6952

[33] Seufert M, Egger S, Slanina M, Zinner T, Hossfeld T, Tran-Gia P. A survey on quality of experience of http adaptive streaming. IEEE Communications Surveys and Tutorials, 2015, 17(1): 469–492

[34] Alreshoodi M, Woods J. Survey on QoE\QoS correlation models for multimedia services. International Journal of Distributed and Parallel Systems, 2013, 4(3): 53–72

[35] Yun L, Peng Z. An automatic hand gesture recognition system based on viola-jones method and SVMS. In: Proceedings of the 2nd International Workshop on Computer Science and Engineering. 2009, 72–76

[36] Kamal S, Jalal A, Kim D. Depth images-based human detection, tracking and activity recognition using spatiotemporal features and modified HMM. Journal of Electrical Engineering and Technology, 2016, 11(3): 1921–1926

[37] Yamauchi K, Chen W, Wei D. 3D mobile phone applications in telemedicine-a survey. In: Proceedings of the 5th International Conference on Computer and Information Technology. 2005, 956–960

[38] Jalal A, Uddin I. Security architecture for third generation (3g) using gmhs cellular network. In: Proceedings of International Conference on Emerging Technologies. 2007, 74–79

[39] Jalal A, Zeb M A. Security and QoS optimization for distributed real time environment. In: Proceedings of the IEEE International Conference on Computer and Information Technology. 2007, 369–374

[40] Jalal A, Rasheed Y A. Collaboration achievement along with performance maintenance in video streaming. In: Proceedings of the IEEE Conference on Interactive Computer Aided Learning. 2007, 1–8

[41] Jalal A, Zeb M A. Security enhancement for e-learning portal. International Journal of Computer Science and Network Security, 2008, 8(3): 41–45

[42] Jalal A, Kim S, Yun B. Assembled algorithm in the real-time H. 263 codec for advanced performance. In: Proceedings of the 7th InternationalWorkshop on Enterprise Networking and Computing in Healthcare Industry. 2005, 295–298

[43] Jalal A, Kim S. Advanced performance achievement using multialgorithmic approach of video transcoder for low bit rate wireless communication. International Journal on Graphics, Vision and Image Processing, 2005, 5(9): 27–32

[44] Jalal A, Sarif N, Kim J T, Kim T S. Human activity recognition via recognized body parts of human depth silhouettes for residents monitoring services at smart home. Indoor and Built Environment, 2013, 22(1): 271–279

[45] Jalal A, Kim S. A complexity removal in the floating point and rate control phenomenon. In: Proceedings of the Conference on Korea Multimedia Society. 2005, 48–51

[46] Jalal A, Kim S. Global security using human face understanding under vision ubiquitous architecture system. Enformatika, 2011, 13: 7–11

[47] Jalal A, Kim S. Algorithmic implementation and efficiency maintenance of real-time environment using low-bitrate wireless communication. In: Proceedings of IEEEWorkshop on Software Technologies for Future Embedded and Ubiquitous Systems, and the 2nd International Workshop on Collaborative Computing, Integration, and Assurance. 2006, 81–88

[48] Zhou L. QoE-driven delay announcement for cloud mobile media. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(1): 84–94

[49] Zhou L. On data-driven delay estimation for media cloud. IEEE Transactions on Multimedia, 2016, 18(5): 905–915

[50] Chen Y, Wu K, Zhang Q. From QoS to QoE: a tutorial on video quality assessment. IEEE Communications Surveys and Tutorials, 2015, 17(2): 1126–1165

[51] Barakovi′c S, Barakovi′c J, Bajri′c H. QoE dimensions and QoE measurement of NGN services. In: Proceedings of the 18th Telecommunications Forum. 2010, 15–18

[52] Skorin-Kapov L, Varela M. A multi-dimensional view of QoE: the arcu model. In: Proceedings of the 35th International Convention. 2012, 662–666

[53] Gill P, Arlitt M, Li Z, Mahanti A. Youtube traffic characterization: a view from the edge. In: Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement. 2007, 15–28

[54] Brunnström K, Beker S A, De Moor K, Dooms A, Egger S, Garcia M N, Hossfeld T, Jumisko-Pyykkö S, Keimel C, Larabi M C. Qualinet white paper on definitions of quality of experience. In: Proceedings of Qualinet White Paper on Definitions of Quality of Experience Output from the 5th Qualinet Meeting. 2013

[55] Piyathilaka L, Kodagoda S. Gaussian mixture based hmm for human daily activity recognition using 3D skeleton features. In: Proceedings of the 8th IEEE Conference on Industrial Electronics and Applications. 2013, 567–572

[56] Jalal A, Kamal S, Kim D. Depth map-based human activity tracking and recognition using body joints features and self-organized map. In: Proceedings of International Conference on Computing, Communication and Networking Technologies. 2014, 1–6

[57] Jalal A, Kim Y. Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data. In: Proceedings of the International Conference on Advanced Video and Signal Based Surveillance. 2014, 119–124

[58] Jalal A, Kim J T, Kim T S. Human activity recognition using the labeled depth body parts information of depth silhouettes. In: Proceedings of the 6th International Symposium on Sustainable Healthy Buildings. 2012, 1–8

[59] Arlot S, Celisse A. A survey of cross-validation procedures for model selection. Statistics Surveys, 2010, 4: 40–79

[60] Kim J H. Estimating classification error rate: repeated crossvalidation, repeated hold-out and bootstrap. Computational Statistics and Data Analysis, 2009, 53(11): 3735–3745

[61] Dobrian F, Awan A, Zhan J, Zhang H. Understanding the impact of video quality on user engagement. ACM SIGCOMMComputer Communication Review, 2011, 41(4): 362–373

[62] Balachandran A, Sekar V, Akella A, Seshan S, Stoica I, Zhang H. Developing a predictive model of quality of experience for internet video. ACM SIGCOMM Computer Communication Review, 2013, 43(4): 339–350

[63] Zhang Y, Yue T, Wang H, Wei A. Predicting the quality of experience for internet video with fuzzy decision tree. In: Proceedings of the 17th International Conference on Computational Science and Engineering. 2014, 1181–1187

[64] Menkovski V, Exarchakos G, Liotta A. Online QoE prediction. In: Proceedings of the 2nd International Workshop on Quality of Multimedia Experience. 2010, 118–123

[65] Menkovski V, Exarchakos G, Liotta A. Online learning for quality of experience management. In: Proceedings of the Annual Machine Learning Conference of Belgium and The Netherlands. 2010

[66] Zheng K, Zhang X, Zheng Q, Xiang W. Quality-of-experience assessment and its application to video services in LTE networks. IEEE Wireless Communications, 2015, 22(1): 70–78

[67] Paudel I, Pokhrel J, Wehbi B, Cavalli A, Jouaber B. Estimation of video QoE from mac parameters in wireless network: a random neural network approach. In: Proceedings of the International Symposium on Communications and Information Technologies. 2015, 51–55

[68] Kapa M, Happe L, Jakab F. Prediction of quality of user experience for video streaming over IP networks. Cyber Journals, 2012: 22–35

[69] Mushtaq M S, Augustin B, Mellouk A. Empirical study based on machine learning approach to assess the QoS/QoE correlation. In: Proceedings of the 17th European Conference on Networks and Optical Communications. 2012, 1–7

[70] Chen H, Yu X, Xie L. End-to-end quality adaptation scheme based on QoE prediction for video streaming service in lte networks. In: Proceedings of the International Symposium on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks. 2013, 627–633

[71] Qian L, Chen H, Xie L. SVM-based QoE estimation model for video streaming service over wireless networks. In: Proceedings of the International Conference on Wireless Communications and Signal Processing. 2015, 1–6

[72] Kang Y, Chen H, Xie L. An artificial-neural-network-based QoE estimation model for video streaming over wireless networks. In: Proceedings of the International Conference on Communications in China. 2013, 264–269

[73] De Pessemier T, De Moor K, Joseph W, De Marez L, Martens L. Quantifying the influence of rebuffering interruptions on the user’s quality of experience during mobile video watching. IEEE Transactions on Broadcasting, 2013, 59(1): 47–61

[74] Aguiar E, Riker A, Cerqueira E, Abelém A, Mu M, Braun T, Curado M, Zeadally S. A real-time video quality estimator for emerging wireless multimedia systems. Wireless Networks, 2014, 20(7): 1759–1776

[75] Chen Y, Chen Q, Zhang F, Zhang Q, Wu K, Huang R, Zhou L. Understanding viewer engagement of video service in wi-fi network. Computer Networks, 2015, 91: 101–116

[76] Krishnan S S, Sitaraman R K. Video stream quality impacts viewer behavior: inferring causality using quasi-experimental designs. IEEE/ACM Transactions on Networking, 2013, 21(6): 2001–2014

[77] Hsu C W, Lin C J. A comparison of methods for multiclass support vector machines. IEEE Transactions on Neural Networks, 2002, 13(2): 415–425

[78] Wang B, Zou D, Ding R. Support vector regression based video quality prediction. In: Proceedings of the International Symposium on Multimedia. 2011, 476–481

[79] Andrews R, Diederich J, Tickle A B. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-based Systems, 1995, 8(6): 373–389

[80] Marchette D J. Bayesian networks and decision graphs. Technometrics, 2008, 45(2): 178–179

[81] Mitra K, Zaslavsky A, Hlund C. Dynamic bayesian networks for sequential quality of experience modelling and measurement. In: Proceedings of International Conference on Smart Spaces and Next Generation Wired/wireless Networking. 2011, 135–146

[82] Mian A U, Hu Z, Tian H. Estimation of in-service quality of experience for peer-to-peer live video streaming systems using a usercentric and context-aware approach based on bayesian networks. Transactions on Emerging Telecommunications Technologies, 2013, 24(3): 280–287

[83] Rabiner L R. A tutorial on hidden markov models and selected applications in speech recognition. Readings in Speech Recognition, 1990, 77(2): 267–296

[84] Jalal A, Kim Y, Kamal S, Farooq A, Kim D. Human daily activity recognition with joints plus body features representation using kinect sensor. In: Proceedings of the International Conference on Informatics, Electronics and Vision. 2015, 1–6

[85] Jala l A, Kamal S, Kim D. Human depth sensors-based activity recognition using spatiotemporal features and hidden markov model for smart environments. Journal of Computer Networks and Communications, 2016: 1–11

[86] Jalal A, Kamal S, Kim D. Individual detection-tracking-recognition using depth activity images. In: Proceedings of the International Conference on Ubiquitous Robots and Ambient Intelligence. 2015, 450–455

[87] Jalal A, Kamal S. Real-time life logging via a depth silhouette-based human activity recognition system for smart home services. In: Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance. 2014, 74–80

[88] Jalal A, Kim Y H, Kim Y J, Kamal S, Kim D. Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recognition, 2016, 61: 295–308

[89] Jalal A, Kamal S, Kim D. Shape and motion features approach for activity tracking and recognition from kinect video camera. In: Proceedings of the IEEE International Conference on Advanced Information networking and Applications Workshops. 2015, 445–450

[90] Min C, Hao Y, Mao S, Di W, Yuan Z. User intent-oriented video QoE with emotion detection networking. In: Proceedings of Global Communications Conference. 2017, 1–6

[91] Sun Y, Yin X, Wang N, Jiang J, Sekar V, Jin Y, Sinopoli B. Analyzing TCP throughput stability and predictability with implications for adaptive video streaming. 2015, arXiv preprint arXiv: 1506.05541

[92] Mitra K, Hlund C, Zaslavsky A. QoE estimation and prediction using hidden markov models in heterogeneous access networks. In: Proceedings of Telecommunication Networks and Applications Conference. 2012, 1–5

[93] Hoßfeld T, Biedermann S, Schatz R, Platzer A, Egger S, Fiedler M. The memory effect and its implications on web QoE modeling. In: Proceedings of the 23rd International Teletraffic Congress. 2011, 103–110

[94] Tasaka S. A bayesian hierarchical model of QoE in interactive audiovisual communications. In: Proceedings of the IEEE International Conference on Communications. 2015, 6983–6989

[95] Charonyktakis P, Plakia M, Tsamardinos I, Papadopouli M. On usercentric modular QoE prediction for voip based on machine-learning algorithms. IEEE Transactions on Mobile Computing, 2016, 15(6): 1443–1456

[96] Venkataraman M, Chatterjee M, Chattopadhyay S. Evaluating quality of experience for streaming video in real time. In: Proceedings of IEEE Global Telecommunications Conference. 2009, 1–6

[97] Zhou Z H. When semi-supervised learning meets ensemble learning. Frontiers of Electrical and Electronic Engineering in China, 2011, 6(1): 6–16

[98] He H, Garcia E A. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 2009, 21(9): 1263–1284

[99] Kubat M, Holte R C, Matwin S. Machine learning for the detection of oil spills in satellite radar images. Machine Learning, 1998, 30(2-3): 195–215

[100] He H, Shen X. A ranked subspace learning method for gene expression data classification. In: Proceedings of the International Conference on Artificial Intelligence. 2007, 358–364

[101] Chawla N V, Bowyer K W, Hall L O, Kegelmeyer W P. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 2002, 16: 321–357

[102] López V, Fernández A, García S, Palade V, Herrera F. An insight into classification with imbalanced data: empirical results and current trends on using data intrinsic characteristics. Information Sciences, 2013, 250: 113–141

[103] Ramyachitra D, Manikandan P. Imbalanced dataset classification and solutions: a review. International Journal of Computing and Business Research, 2014, 5(4): 1–29

[104] Wang L, Jin J, Huang R, Wei X, Chen J. Unbiased decision tree model for user’s QoE in imbalanced dataset. In: Proceedings of the International Conference on Cloud Computing Research and Innovations. 2016, 114–119

[105] Huang R, Wei X, Lv C, L i X, Zhang S. Prediction model for user’s QoE in imbalanced dataset. In: Proceedings of the 1st International Conference on Computational Intelligence Theory, Systems and Applications. 2015, 41–45

[106] Liu R, Huang R, Qian Y, Wei X, Lu P. Improving user’s quality of experience in imbalanced dataset. In: Proceedings of the 2016 International Wireless Communications and Mobile Computing Conference. 2016, 644–649

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature