Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models
Abstract
:1. Introduction
- This paper uses a subjective approach to assessing the level of user experience, which is based on a unique questionnaire with 11 questions.
- Survey questions were formulated based on the selected original set of five factors affecting QoE: (1) legal–regulatory; (2) technological–process; (3) content-formatted and performative; (4) contextual–relational; (5) subjective–user.
- The subjects of user evaluation are all telecommunications services of the three largest mobile operators operating on the territory of the Republic of Srpska and Bosnia and Herzegovina. It is important to point out that there is no previously published research on this topic that is related to the mentioned geographical area, as well as the observed set of services, which also represents the great practical importance of this paper.
- This paper presents a unique methodology based on a combination of mathematical, statistical and machine learning methods in order to assess, classify and predict the quality of user experience at the level of an individual user, which is why a large number of models were created.
- The possibilities of synthetic data augmentation using the data augmentation (DA) method were demonstrated, as well as the way in which this method affects the performance improvements of machine learning models.
2. Review of Relevant Published Research
3. Materials and Research Methods
- In the first step, we performed the analysis of various factors that affect the quality of user experience and created an interactive QoEi model;
- In the next step, the research process was implemented in accordance with a subjective approach to the assessment of the level of user experience and the survey method, on the basis of which a research instrument was created—a QoE questionnaire. A correlation amongst the influencing factors on QoEi in formulated questions and research-independent, transition and dependent variables was established.
- The third step was the process of online surveying of users of network services and applications about the level of certain indicators of the quality of subjective–user experience in interactions with communication services performed by professional companies—providers of telecommunication services in certain locations;
- Data obtained by surveying users was prepared for processing in the fourth step;
- The statistical analysis of the research sample was performed in the fifth step, where basic statistical indicators related to the responses to individual questions and the structure of respondents were given;
- In the sixth step, a mathematical model was created to assess the subjective-user QoEi based on the responses to the questions from the QoE questionnaire as input variables;
- In the seventh step, a QoEi probability model was created;
- Correlation analysis of research variables was performed in the eighth step;
- The last step represents the special focus of the research and refers to the results of QoEi modeling. Within this step, the results of the QoEi prediction and classification model based on machine learning techniques are particularly important.
3.1. Analysis of Influencing Factors and Creation of an Interactive QoEi Model
- Subjective–user influencing factors: demographic and socio-economic background, physical and mental constitution or emotional state of the user.
- Technological–process influencing factors: transmission, encoding, storage, display and reproduction/media display, etc.
- Contextual–relational influencing factors: any property of the situation that describes the user environment, in terms of physical (location and space, activities, state-mobility and behavior), time, social (people who are present or involved in the experience), economic (costs, type of subscription or type of brand of service/system), and technical characteristics.
- Content-formatted and performative influencing factors, which in the case of videos are related to traffic class or streaming quality, encoding speed, resolution, duration, movement patterns, type and content structure of videos, etc.
- Legal–regulatory influencing factors in multidimensional space on the intuitive and systemic quality of the user experience. According to technical specification [5], in this paper, an expanded number, i.e., five multi-dimensional areas in which QoE influencing factors for a specific service/application are evident, namely: application robustness area, operator/provider network resource area, network traffic context area, subjective user area and legal–regulatory area. The given categorization of space in this research is synchronized with the categorized paired factors of influence on the overall level of QoE, i.e., legal–regulatory, technological–process, relational–contextual, content–performative and subjective–user.
3.2. QoE Questionnaire and Selection of Research Variables
- Service level measurements represent subjective measurements. They are most often carried out by agents accessing telecommunication services and responding to the created research questions at the end.
3.3. Survey Process
3.4. Preprocessing of Data Collected
3.5. Statistical Analysis of Data Collected
3.6. Assessment of User QoEi with a Mathematical Model
3.7. Creating a QoEi Probability Model
3.8. Correlation Analysis of Research Variables
3.9. Creating a Model for QoEi Prediction and Classification
Models for QoEi Prediction
4. Factors Affecting the Quality of User Experience
4.1. Legal–Regulatory Factors
4.2. Technological–Process Factors of Network Services/Applications
4.3. Content-Formatted and Performative Factors
4.4. Contextual–Relational Factors
4.5. Subjective–User Factors
4.6. An Interaction Model of Paired Factors Affecting QoEi
5. Results of QoEi Modeling and Discussion
5.1. Research Sample Statistics
5.2. QoEi Estimation Model
5.3. QoEi Probability Model
5.4. Correlation Analysis of Research Variables
5.5. Predictive Models of QoEi
5.5.1. Multiple Linear Regression Model
5.5.2. Boosted Decision Tree Model
5.5.3. Predictive Models Created by Using an Automatic Modeling Method
- Minimum records in parent branch—prevents splitting if the number of records in a node to be split (parent) is less than the set value—2% of the total dataset.
- Minimum records in child branch—prevents the split if the number of records in any branch created by the split (child node) would be less than the set value—1% of the total dataset.
5.6. Models for QoEi Classification
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Zhu, Y.; Heynderickx, I.; Redi, J.A. Understanding the role of social context and user factors in video quality of experience. Comput. Hum. Behav. 2015, 49, 412–426. [Google Scholar] [CrossRef]
- Geiser, M.; Panwar, D.; Tomar, P.; Harsh, H.; Zhang, X.; Solanki, A.; Nayyar, A.; Alzubi, J.A. An optimization model for software quality prediction with case study analysis using MATLAB. IEEE Access 2019, 7, 85123–85138. [Google Scholar] [CrossRef]
- Movassagh, A.A.; Alzubi, J.A.; Gheisari, M.; Rahimi, M.; Mohan, S.; Abbasi, A.A.; Nabipour, N. Artificial neural networks training algorithm integrating invasive weed optimization with differential evolutionary model. J. Ambient Intell. Humaniz. Comput. 2021, 12, 1–9. [Google Scholar] [CrossRef]
- Hansmann, R.; Mieg, H.A.; Frischknecht, P. Principal sustainability components: Empirical analysis of synergies between the three pillars of sustainability. Int. J. Sustain. Dev. World Ecol. 2012, 19, 451–459. [Google Scholar] [CrossRef]
- ETSI TS 103 294; Speech and Multimedia Transmission Quality (STQ); Quality of Experience; A Monitoring Architecture, Technical Specification, V1.1.1. European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2014. Available online: https://www.etsi.org/deliver/etsi_ts/103200_103299/103294/01.01.01_60/ts_103294v010101p.pdf (accessed on 30 March 2021).
- ETSI TR 102 643; Human Factors (HF); Quality of Experience (QoE) Requirements for Real-Time Communication Services, Technical Report, V1.0.1 (2009-12). European Telecommunications Standards Institute: Sophia Antipolis Cedex, France. 2009. Available online: https://www.etsi.org/deliver/etsi_tr/102600_102699/102643/01.00.01_60/tr_102643v010001p.pdf (accessed on 30 March 2021).
- Laghari, K. On Quality of Experience (QoE) for Multimedia Services in Communication Ecosystem. Ph.D. Thesis, Institut National des Telecommunictions, Télécom SudParis, Paris, France, 30 April 2012. Available online: https://tel.archives-ouvertes.fr/tel-00873612/document (accessed on 13 November 2022).
- ITU-T Recommendation P.10/G.100; Amendment 2: New Definitions for Inclusion in Recommendation ITU-T P.10/G.100. International Telecommunication Union: Geneva, Switzerland, 2008. Available online: https://www.itu.int/rec/T-REC-P.10-200807-S!Amd2/en (accessed on 8 November 2021).
- Vakili, A.; Grégoire, J.-C. QoE management in a video conferencing application. In Future Information Technology, Application and Service; Lecture Notes in Electrical Engineering; Park, J.J., Leung, V.C.M., Wang, C.L., Shon, T., Eds.; Springer: Dordrecht, The Netherlands, 2012; Volume 164, pp. 191–201. [Google Scholar] [CrossRef]
- ITU-T Recommendation P.10/G.100 (11/17); Vocabulary for Performance, Quality of Service and Quality of Experience. International Telecommunication Union: Geneva, Switzerland, 2017. Available online: https://www.itu.int/rec/T-REC-P.10-201711-I/en (accessed on 8 November 2022).
- Dai, Q. A Survey of Quality of Experience. In Energy-Aware Communications. EUNICE 2011; Lecture Notes in Computer Science; Lehnert, R., Ed.; Springer: Berlin, Heidelberg, 2011; Volume 6955, pp. 146–156. [Google Scholar] [CrossRef]
- Eswara, N.; Ashique, S.; Panchbhai, A.; Chakraborty, S.; Sethuram, H.P.; Kuchi, K.; Kumar, A.; Channappayya, S.S. Streaming video QoE modeling and prediction: A long short-term memory approach. IEEE Trans. Circuits Syst. Video Technol. 2019, 30, 661–673. [Google Scholar] [CrossRef] [Green Version]
- Barman, N.; Martini, M.G. Qoe modeling for HTTP adaptive video streaming–a survey and open challenges. IEEE Access 2019, 7, 30831–30859. [Google Scholar] [CrossRef]
- Ruan, J.; Xie, D. A survey on QoE-oriented VR video streaming: Some research issues and challenges. Electronics 2021, 10, 2155. [Google Scholar] [CrossRef]
- Banjanin, M.K.; Maričić, G.; Stojčić, M. Multifactor influences on the quality of experience service users of telecommunication providers in the Republic of Srpska, Bosnia and Herzegovina. Int. J. Qual. Res. 2022, 17. [Google Scholar] [CrossRef]
- Daengsi, T.; Wuttidittachotti, P. QoE Modeling for Voice over IP: Simplified E-model Enhancement Utilizing the Subjective MOS Prediction Model: A Case of G. 729 and Thai Users. J. Netw. Syst. Manag. 2019, 27, 837–859. [Google Scholar] [CrossRef]
- García-Pineda, M.; Segura-Garcia, J.; Felici-Castell, S. A holistic modeling for QoE estimation in live video streaming applications over LTE Advanced technologies with Full and Non Reference approaches. Comput. Commun. 2018, 117, 13–23. [Google Scholar] [CrossRef]
- Ickin, S.; Vandikas, K.; Fiedler, M. Privacy preserving qoe modeling using collaborative learning. In Proceedings of the 4th Internet-QoE Workshop on QoE-Based Analysis and Management of Data Communication Networks, Los Cabos, Mexico, 21 October 2019; pp. 13–18. [Google Scholar]
- Khokhar, M.J.; Saber, N.A.; Spetebroot, T.; Barakat, C. An intelligent sampling framework for controlled experimentation and QoE modeling. Comput. Netw. 2018, 147, 246–261. [Google Scholar] [CrossRef] [Green Version]
- Dasari, M.; Sanadhya, S.; Vlachou, C.; Kim, K.H.; Das, S.R. Scalable Ground-Truth Annotation for Video QoE Modeling in Enterprise WiFi. In Proceedings of the 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), Banff, AB, Canada, 4–6 June 2018; pp. 1–6. [Google Scholar]
- Veeraragavan, N.R.; Montecchi, L.; Nostro, N.; Vitenberg, R.; Meling, H.; Bondavalli, A. Modeling QoE in dependable tele-immersive applications: A case study of world opera. IEEE Trans. Parallel Distrib. Syst. 2015, 27, 2667–2681. [Google Scholar] [CrossRef]
- Hoßfeld, T.; Biedermann, S.; Schatz, R.; Platzer, A.; Egger, S.; Fiedler, M. The memory effect and its implications on Web QoE modeling. In Proceedings of the 2011 23rd International Teletraffic Congress (ITC), San Francisco, CA, USA, 6–9 September 2011; pp. 103–110. [Google Scholar]
- Lycett, M.; Radwan, O. Developing a quality of experience (QoE) model for web applications. Inf. Syst. J. 2019, 29, 175–199. [Google Scholar] [CrossRef] [Green Version]
- Banjanin, M.K.; Stojčić, M.; Drajić, D.; Ćurguz, Z.; Milanović, Z.; Stjepanović, A. Adaptive Modeling of Prediction of Telecommunications Network Throughput Performances in the Domain of Motorway Coverage. Appl. Sci. 2021, 11, 3559. [Google Scholar] [CrossRef]
- Bouraqia, K.; Sabir, E.; Sadik, M.; Ladid, L. Quality of experience for streaming services: Measurements, challenges and insights. IEEE Access 2020, 8, 13341–13361. [Google Scholar] [CrossRef]
- Hu, Z.; Yan, H.; Yan, T.; Geng, H.; Liu, G. Evaluating QoE in VoIP networks with QoS mapping and machine learning algorithms. Neurocomputing 2020, 386, 63–83. [Google Scholar] [CrossRef]
- Isak-Zatega, S.; Lipovac, A.; Lipovac, V. Logistic regression based in-service assessment of mobile web browsing service quality acceptability. EURASIP J. Wirel. Commun. Netw. 2020, 96, 1–21. [Google Scholar] [CrossRef]
- Mitra, K.; Zaslavsky, A.; Åhlund, C. QoE modelling, measurement and prediction: A review. arXiv 2014, arXiv:1410.6952. [Google Scholar] [CrossRef]
- Pal, D.; Triyason, T. A survey of standardized approaches towards the quality of experience evaluation for video services: An ITU perspective. Int. J. Digit. Multimed. Broadcast. 2018, 2018, 1724. [Google Scholar] [CrossRef] [Green Version]
- Juluri, P.; Tamarapalli, V.; Medhi, D. Measurement of quality of experience of video-on-demand services: A survey. IEEE Commun. Surv. Tutor. 2015, 18, 401–418. [Google Scholar] [CrossRef]
- Baraković Husić, J.; Baraković, S.; Cero, E.; Slamnik, N.; Oćuz, M.; Dedović, A.; Zupčić, O. Quality of experience for unified communications: A survey. Int. J. Netw. Manag. 2020, 30, e2083. [Google Scholar] [CrossRef]
- Banjanin, M.K.; Stojčić, M. Conceptual Model of the Cyber-physical System in the Space of the M9J Road Section. In Proceedings of the 15th International Conference on Advanced Technologies, Systems and Services in Telecommunications (TELSIKS), Niš, Serbia, 20–22 October 2021; pp. 299–302. [Google Scholar]
- Belmudez, B.; Möller, S. Audiovisual quality integration for interactive communications. EURASIP J. Audio Speech Music Process. 2013, 2013, 1–23. [Google Scholar] [CrossRef] [Green Version]
- Cavanaugh, J.E.; Neath, A.A. The Akaike information criterion: Background, derivation, properties, application, interpretation, and refinements. Wiley Interdiscip. Rev. Comput. Stat. 2019, 11, e1460. [Google Scholar] [CrossRef]
- Portet, S. A primer on model selection using the Akaike Information Criterion. Infect. Dis. Model. 2020, 5, 111–128. [Google Scholar] [CrossRef] [PubMed]
- Ćurguz, Z.; Banjanin, M.; Stojčić, M. Machine learning models for prediction of mobile network user throughput in the area of trunk road and motorway sections. In Proceedings of the First International Conference on Advances in Traffic and Communication Technologies, Sarajevo, Bosnia and Herzegovina, 26–27 May 2022; pp. 27–35. [Google Scholar]
- Ćurguz, Z.; Banjanin, M.; Stojčić, M. Prediction of user throughput in the mobile network along the motorway and trunk road. Sci. Eng. Technol. 2022, 2, 23–30. [Google Scholar] [CrossRef]
- Simakovic, M.; Cica, Z.; Drajic, D. Big-Data Platform for Performance Monitoring of Telecom-Service-Provider Networks. Electronics 2022, 11, 2224. [Google Scholar] [CrossRef]
- Stojčić, M.; Banjanin, M.K. Predictive Modeling of Telecommunications Traffic Performance Based on Machine Learning Techniques. In Proceedings of the VIII International Symposium NEW HORIZONS 2021 of Transport and Communications, Doboj, Bosnia and Herzegovina, 26–27 November 2021; pp. 378–385. [Google Scholar]
- Ivaniš, P.; Drajić, D. Information Theory and Coding-Solved Problems; Springer International Publishing: Cham, Switzerland, 2012; ISBN 978-3-319-49369-5. [Google Scholar] [CrossRef]
- Stojčić, M.; Banjanin, M.; Ćurguz, Z.; Stjepanović, A. Machine Learning Model of Communication of Physical and Virtual Sensors in the Mobile Network on the Motorway Section. In Proceedings of the 44th International Convention, CTI, MIPRO 2021, Opatija, Croatia, 27 September–1 October 2021; pp. 447–452. [Google Scholar]
- Tensorflow. Available online: https://www.tensorflow.org/tutorials/images/data_augmentation (accessed on 13 November 2022).
- ETSI TS 102 250-2; Speech and Multimedia Transmission Quality (STQ); QoS Aspects for Popular Services in Mobile Networks; Part 2: Definition of Quality of Service Parameters and Their Computation, Technical Specification, V2.4.1. European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2015. Available online: https://www.etsi.org/deliver/etsi_ts/102200_102299/10225002/02.04.01_60/ts_10225002v020401p.pdf (accessed on 2 April 2021).
- ETSI TS 102 250-1; Speech and Multimedia Transmission Quality (STQ); QoS Aspects for Popular Services in Mobile Networks; Part 1: Assessment of Quality of Service, Technical Specification, V2.2.1 (2011-04). European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2011. Available online: https://www.etsi.org/deliver/etsi_ts/102200_102299/10225001/02.02.01_60/ts_10225001v020201p.pdf (accessed on 2 April 2021).
- ETSI TR 103 488; Speech and Multimedia Transmission Quality (STQ); Guidelines on OTT Video Streaming; Service Quality Evaluation Procedures, Technical Specification, V1.1.1 (2019-01). European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2019. Available online: https://www.etsi.org/deliver/etsi_tr/103400_103499/103488/01.01.01_60/tr_103488v010101p.pdf (accessed on 11 April 2021).
- 3GPP TR 26.944; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; End-to-End Multimedia Services Performance Metrics (Release 10), Technical Report, V10.0.0 (2011-03). 3rd Generation Partnership Project: Sophia Antipolis, France, 2011. Available online: https://www.arib.or.jp/english/html/overview/doc/STDT63v9_10/5_Appendix/Rel10/26/26944-a00.pdf (accessed on 31 March 2021).
- ITU-T Recommendation G.1000; Communications Quality of Service: A Framework and Definitions. International Telecommunication Union: Geneva, Switzerland, 2002.
- Mtel. Opšti Uslovi za Pružanje Telekomunikacionih Usluga (Prečišćeni Tekst). Available online: https://mtel.ba/Binary/397/Opsti-uslovi-za-pruzanje-telekomunikacionih-usluganesluzbeni-precisceni-tekst.pdf (accessed on 8 October 2022).
- GSM Association. Definition of Quality of Service Parameters and Their Computation; Official Document IR.42, Version 9.0. Available online: https://www.gsma.com/newsroom/wp-content/uploads//IR.42-v9.0.pdf (accessed on 3 April 2021).
- BaBatunde, K.A.; Akinboboye, S. Corporate social responsibility effect on consumer patronage-management perspective: Case study of a telecommunication company in Nigeria. J. Komun. 2013, 29, 55–71. [Google Scholar]
- Maričić, G.; Banjanin, M.K.; Stojčić, M. Legal-Regulatory Paired Component in the QoE Model for Assessment of the Quality of Experience of Users of Services of Company. In Proceedings of the Materials of 1st International Scientific and Practical Internet Conference “The impact of COVID-19 Pandemic on development of modern world: Threats and opportunities”-WayScience, Dnipro, Ukraine, 9–10 September 2021; pp. 33–36. [Google Scholar]
- Banjanin, K.M. Komunikacioni Inženjering; Univerzitet u Istočnom Sarajevu, Saobraćajno-Tehnički Fakultet Doboj: Doboj, Bosnia and Herzegovina, 2007; ISBN 978-99938-859-4-8. [Google Scholar]
- Brunnström, K.; Beker, S.A.; De Moor, K.; Dooms, A.; Egger, S.; Garcia, M.-N.; Hossfeld, T.; Jumisko-Pyykkö, S.; Keimel, C.; Larabi, C.; et al. Qualinet white paper on definitions of quality of experience. In Proceedings of the Fifth Qualinet Meeting, Novi Sad, Serbia, 12 March 2013. [Google Scholar]
- Reiter, U.; Brunnström, K.; Moor, K.D.; Larabi, M.C.; Pereira, M.; Pinheiro, A.; Zgank, A. Factors influencing quality of experience. In Quality of Experience; Möller, S., Raake, A., Eds.; Springer: Cham, Switzerland, 2014; pp. 55–72. [Google Scholar] [CrossRef]
- Rahman, M.A.; El Saddik, A.; Gueaieb, W. Augmenting context awareness by combining body sensor networks and social networks. IEEE Trans. Instrum. Meas. 2010, 60, 345–353. [Google Scholar] [CrossRef]
- Su, J.H.; Yeh, H.H.; Yu, P.S.; Tseng, V.S. Music recommendation using content and context information mining. IEEE Intell Syst. 2010, 25, 1541–1672. [Google Scholar] [CrossRef]
- Naumann, A.B.; Wechsung, I.; Hurtienne, J. Multimodal interaction: A suitable strategy for including older users? Interact. Comput. 2010, 22, 465–474. [Google Scholar] [CrossRef]
- Hyder, M.; Laghari, K.u.R.; Crespi, N.; Haun, M.; Hoene, C. Are QoE Requirements for Multimedia Services Different for Men and Women? Analysis of Gender Differences in Forming QoE in Virtual Acoustic Environments. In Emerging Trends and Applications in Information Communication Technologies IMTIC 2012. Communications in Computer and Information Science; Chowdhry, B.S., Shaikh, F.K., Hussain, D.M.A., Uqaili, M.A., Eds.; Springer: Berlin/Heidelberg, Germany, 2012; Volume 281, pp. 200–209. [Google Scholar] [CrossRef]
- Jumisko-Pyykkö, S.; Häkkinen, J.; Nyman, G. Experienced quality factors: Qualitative evaluation approach to audiovisual quality. In Multimedia on Mobile Devices; SPIE: Bellingham, WA, USA, 2007; Volume 6507, pp. 169–180. [Google Scholar] [CrossRef]
- MathWorks. Available online: https://www.mathworks.com/products/demos/machine-learning/boosted-regression.html (accessed on 8 November 2022).
- IBM. Available online: https://www.ibm.com/docs/en/SS3RA7_18.3.0/pdf/ModelerModelingNodes.pdf (accessed on 8 November 2022).
- Selvanathan, M.; Jayabalan, N.; Saini, G.K.; Supramaniam, M.; Hussin, N. Employee Productivity in Malaysian Private Higher Educational Institutions. PalArch’s J. Archaeol. Egypt/Egyptol. 2020, 17, 66–79. [Google Scholar] [CrossRef]
- Stackoverflow. Available online: https://stackoverflow.com/questions/39265746/data-augmentation-techniques-forgeneral-datasets (accessed on 8 November 2022).
- Singla, A.; Rao, R.R.R.; Göring, S.; Raake, A. Assessing media qoe, simulator sickness and presence for omnidirectional videos with different test protocols. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 1163–1164. [Google Scholar] [CrossRef]
- Kara, P.A.; Bokor, L.; Sackl, A.; Mourão, M. What your phone makes you see: Investigation of the effect of end-user devices on the assessment of perceived multimedia quality. In Proceedings of the 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), Messinia, Greece, 26–29 May 2015; pp. 1–6. [Google Scholar] [CrossRef]
- IBM. Available online: https://www.ibm.com/docs/en/spss-modeler/saas?topic=models-how-svm-works (accessed on 8 November 2022).
- Scikit-Learn. Available online: https://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html (accessed on 8 November 2022).
Ord. Number | Title of Paper | Service/ Application Observed | Methods and Models Used | Observed Factors/Variables Affecting QoE | Comparative Improvements Presented in This Paper |
---|---|---|---|---|---|
[16] | QoE Modeling for Voice over IP: Simplified E-model Enhancement Utilizing the Subjective MOS Prediction Model: A Case of G.729 and Thai Users | VoIP | Objective simplified E-model; subjective MOS model for prediction | Delay, packet loss, jitter | (a), (b), (c), (d), (e), (f), (g), (h), (i), (j) |
[17] | A holistic modeling for QoE estimation in live video streaming applications over LTE Advanced technologies with Full and Non Reference approaches | Live video streaming | Statistical modeling—regression analysis for objective assessment of video quality; factor analysis | Variables related to QoS, bit stream and basic video quality metrics grouped into factors | (a), (b), (d), (e), (f), (g), (i), (j) |
[18] | Privacy Preserving QoE Modeling using Collaborative Learning | Applicable to all services | A machine learning model with data privacy protection—a collaborative machine learning model | Maximum bandwidth for downlink; search time; assessment time | (a), (b), (d), (e), (f), (g), (i), (j) |
[19] | An Intelligent Sampling Framework for Controlled Experimentation and QoE Modeling | YouTube video streaming | Machine learning models | QoS variables (delay, bandwidth...) | (a), (b), (c), (d), (e), (f), (g), (i), (j) |
[20] | Scalable Ground-Truth Annotation for Video QoE Modeling in Enterprise WiFi | Video telephony | Adaboosted decision trees | Perceptual bitrate (PBR), freeze ratio, freeze length and number of video freezes | (a), (b), (c), (d), (e), (f), (g), (i), (j) |
[21] | Modeling QoE in Dependable Tele-immersive Applications: A Case Study of World Opera | World Opera application | Subjective method based on perceived reliability; stochastic activity networks (SANs) | Human perception of video and audio, audience characteristics, performance elements and artistic content | (a), (b), (c), (d), (e), (f), (h), (i), (j) |
[22] | The Memory Effect and Its Implications on Web QoE Modeling | Interactive Web services | Support vector machines; iterative exponential regressions; two-dimensional hidden Markov models | Technical factors (scope, page load time, packet loss...); psychological factors (expectations, memory effects, user) | (a), (b), (c), (d), (e), (f), (g), (i), (j) |
[25] | Quality of Experience for Streaming Services: Measurements, Challenges and Insights | Streaming services | Subjective methods; objective methods; hybrid methods | Human-related influencing factors; system-related influencing factors; context-related influencing factors; content-related influencing factors | (a), (b), (c), (d), (e), (f), (h), (i), (j) |
[26] | Evaluating QoE in VoIP networks with QoS mapping and machine learning algorithms | VoIP services | MOS model; PESQ model; E-model; a single-layer artificial neural network model | Echo, packet loss, jitter, bandwidth, delay | (a), (b), (c), (d), (e), (f), (g), (i), (j) |
[23] | Developing a Quality of Experience (QoE) model for Web Applications | Web applications | Quality of experience of Web application (QoEWA) model | Objective factors (KPI); subjective factors (KQI). | (a), (b), (d), (e), (f), (g), (h), (i), (j) |
[27] | Logistic regression based in-service assessment of mobile web browsing service quality acceptability | Searching the Web | Binary logistic regression model | Average time-to-connect-TCP | (a), (b), (d), (e), (f), (g), (h), (i), (j) |
Mark | Mean (MOS) | Standard Deviation | Variance | Sum of Squares | Min | Median | Max | |
---|---|---|---|---|---|---|---|---|
Indicators of user satisfaction | D1 | 3.32 | 1.01 | 1.03 | 1889 | 1 | 3 | 5 |
D2 | 3.17 | 0.99 | 0.99 | 1727 | 1 | 3 | 5 | |
D3 | 3.71 | 0.96 | 0.91 | 2300 | 1 | 4 | 5 | |
D4 | 3.00 | 1.02 | 1.04 | 1575 | 1 | 3 | 5 | |
D5 | 2.92 | 1.00 | 1.01 | 1499 | 1 | 3 | 5 | |
D6 | 2.99 | 1.02 | 1.03 | 1568 | 1 | 3 | 5 | |
D7 | 3.04 | 1.01 | 1.03 | 1616 | 1 | 3 | 5 | |
D8 | 3.16 | 0.95 | 0.90 | 1697 | 1 | 3 | 5 | |
D9 | 2.96 | 1.11 | 1.23 | 1569 | 1 | 3 | 5 | |
D10 | 3.09 | 1.09 | 1.19 | 1674 | 1 | 3 | 5 | |
Indicators of user dissatisfaction (forms and measures of rigidity) | C1 | 2.83 | 1.05 | 1.09 | 1426 | 1 | 3 | 5 |
C2 | 2.88 | 1.02 | 1.03 | 1462 | 1 | 3 | 5 | |
C3 | 2.82 | 1.01 | 1.01 | 1402 | 1 | 3 | 5 | |
C4 | 2.80 | 1.06 | 1.12 | 1408 | 1 | 3 | 5 | |
C5 | 2.87 | 1.08 | 1.17 | 1472 | 1 | 3 | 5 |
Distribution | AD | p | AIC |
---|---|---|---|
LogNormal—three parameter | 13.47 | 0.000 | 26.61 |
LogLogistic—three parameter | 12.16 | <0.005 | 47.09 |
Exponential—two parameter | 37.41 | <0.001 | 102.3 |
Logistic | 9.604 | <0.005 | 292.0 |
Normal | 10.53 | 0.000 | 295.5 |
Smallest extreme value | −157.0 | >0.250 | 176443 |
Largest extreme value | −86.76 | >0.250 | 1,579,994,965 |
Created Model | Correlation | Relative Error |
---|---|---|
1. Regression | 0.127 | 1.070 |
2. k-nearest neighbors (k-NN) | 0.206 | 1.075 |
3. C&R tree | 0.000 | 1.147 |
Absolute Value of the Correlation Coefficient | Qualitative Assessment |
---|---|
0.19 | Very low correlation |
0.39 | Low correlation |
0.59 | Moderate correlation |
0.79 | High correlation |
1.00 | Very high correlation |
Model/Component Number | Prediction Accuracy (A) | Number of Inputs | Number of Nodes |
---|---|---|---|
1 | 69.7% | 9 | 23 |
2 | 52.3% | 9 | 19 |
3 | 47.0% | 10 | 17 |
4 | 39.2% | 10 | 25 |
8 | 34.6% | 10 | 35 |
5 | 30.4% | 10 | 25 |
9 | 27.8% | 10 | 29 |
6 | 25.5% | 10 | 29 |
10 | 13.7% | 10 | 25 |
7 | 10.0% | 10 | 21 |
Continuous Scale | ACR Scale |
---|---|
0 ≤ QoEi < 0.5 | 0 |
0.5 ≤ QoEi < 1.5 | 1 |
1.5 ≤ QoEi < 2.5 | 2 |
2.5 ≤ QoEi < 3.5 | 3 |
3.5 ≤ QoEi < 4.5 | 4 |
4.5 ≤ QoEi ≤ 5 | 5 |
Model Created | Total Classification Accuracy [%] |
---|---|
1. k-NN | 50.00 |
2. C&R tree | 46.15 |
3. Neural network | 42.31 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Banjanin, M.K.; Stojčić, M.; Danilović, D.; Ćurguz, Z.; Vasiljević, M.; Puzić, G. Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models. Sustainability 2022, 14, 17053. https://doi.org/10.3390/su142417053
Banjanin MK, Stojčić M, Danilović D, Ćurguz Z, Vasiljević M, Puzić G. Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models. Sustainability. 2022; 14(24):17053. https://doi.org/10.3390/su142417053
Chicago/Turabian StyleBanjanin, Milorad K., Mirko Stojčić, Dejan Danilović, Zoran Ćurguz, Milan Vasiljević, and Goran Puzić. 2022. "Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models" Sustainability 14, no. 24: 17053. https://doi.org/10.3390/su142417053