# Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

_{i}. The results show that the relative error (RE) of the predictive models, created over the collected dataset, is insufficiently low, so the improvement of the prediction performance was achieved via data augmentation (DA). In this way, the relative prediction error is reduced to a value of RE = 0.247. The DA method was also applied for the creating a classification model, which at best demonstrated an accuracy of 94.048%.

## 1. Introduction

_{i}is observed as a dependent/output variable, where i ranges within 1...n users. This means that the arithmetic mean of QoE

_{i}values corresponds to the total QoE for a group of users expressed through MOS.

_{i}can be a very important parameter for planning and optimizing network resources and realizing the competitive advantage of individual service providers in the future.

- This paper uses a subjective approach to assessing the level of user experience, which is based on a unique questionnaire with 11 questions.
- Survey questions were formulated based on the selected original set of five factors affecting QoE: (1) legal–regulatory; (2) technological–process; (3) content-formatted and performative; (4) contextual–relational; (5) subjective–user.
- The subjects of user evaluation are all telecommunications services of the three largest mobile operators operating on the territory of the Republic of Srpska and Bosnia and Herzegovina. It is important to point out that there is no previously published research on this topic that is related to the mentioned geographical area, as well as the observed set of services, which also represents the great practical importance of this paper.
- This paper presents a unique methodology based on a combination of mathematical, statistical and machine learning methods in order to assess, classify and predict the quality of user experience at the level of an individual user, which is why a large number of models were created.
- The possibilities of synthetic data augmentation using the data augmentation (DA) method were demonstrated, as well as the way in which this method affects the performance improvements of machine learning models.

_{i}modeling are presented with discussion. Section 6 refers to the conclusion, after which a list of references is given.

## 2. Review of Relevant Published Research

_{i}, not at the level of a group of users; (b) original questionnaire for subjective user assessment of QoE

_{i}level with a unique set of questions representing independent/input, transition and dependent/output variables; (c) services that are subject to evaluation: all services and classes of telecommunication traffic, not just individual services or applications; (d) operators that are subject to evaluation: all mobile operators on the territory of the Republic of Srpska and B&H; (e) estimation of QoE

_{i}based on transition variables that represent indicators of satisfaction and indicators of dissatisfaction of service users; (f) original classification of combinations of paired factors affecting QoE

_{i}into five groups; (g) creation of an interactive model of factors that affect QoE

_{i}; (h) creation of multiple models for QoE

_{i}prediction and classification based on machine learning techniques; (i) the IBM SPSS Modeler software platform was used for modeling; (j) the DA method was applied. The mark in parentheses in front of individual improvements (a),...,(j) is used in the sixth column of Table 1 under the title “comparative improvements presented in the paper”.

## 3. Materials and Research Methods

- In the first step, we performed the analysis of various factors that affect the quality of user experience and created an interactive QoE
_{i}model; - In the next step, the research process was implemented in accordance with a subjective approach to the assessment of the level of user experience and the survey method, on the basis of which a research instrument was created—a QoE questionnaire. A correlation amongst the influencing factors on QoE
_{i}in formulated questions and research-independent, transition and dependent variables was established. - The third step was the process of online surveying of users of network services and applications about the level of certain indicators of the quality of subjective–user experience in interactions with communication services performed by professional companies—providers of telecommunication services in certain locations;
- Data obtained by surveying users was prepared for processing in the fourth step;
- The statistical analysis of the research sample was performed in the fifth step, where basic statistical indicators related to the responses to individual questions and the structure of respondents were given;
- In the sixth step, a mathematical model was created to assess the subjective-user QoE
_{i}based on the responses to the questions from the QoE questionnaire as input variables; - In the seventh step, a QoE
_{i}probability model was created; - Correlation analysis of research variables was performed in the eighth step;
- The last step represents the special focus of the research and refers to the results of QoE
_{i}modeling. Within this step, the results of the QoE_{i}prediction and classification model based on machine learning techniques are particularly important.

#### 3.1. Analysis of Influencing Factors and Creation of an Interactive QoE_{i} Model

- Subjective–user influencing factors: demographic and socio-economic background, physical and mental constitution or emotional state of the user.
- Technological–process influencing factors: transmission, encoding, storage, display and reproduction/media display, etc.
- Contextual–relational influencing factors: any property of the situation that describes the user environment, in terms of physical (location and space, activities, state-mobility and behavior), time, social (people who are present or involved in the experience), economic (costs, type of subscription or type of brand of service/system), and technical characteristics.
- Content-formatted and performative influencing factors, which in the case of videos are related to traffic class or streaming quality, encoding speed, resolution, duration, movement patterns, type and content structure of videos, etc.
- Legal–regulatory influencing factors in multidimensional space on the intuitive and systemic quality of the user experience. According to technical specification [5], in this paper, an expanded number, i.e., five multi-dimensional areas in which QoE influencing factors for a specific service/application are evident, namely: application robustness area, operator/provider network resource area, network traffic context area, subjective user area and legal–regulatory area. The given categorization of space in this research is synchronized with the categorized paired factors of influence on the overall level of QoE, i.e., legal–regulatory, technological–process, relational–contextual, content–performative and subjective–user.

_{i}, in this step of the research modeling of their interactions was performed. Their interdependencies and correlations, as well as correlations with research variables, are shown graphically.

#### 3.2. QoE Questionnaire and Selection of Research Variables

- Service level measurements represent subjective measurements. They are most often carried out by agents accessing telecommunication services and responding to the created research questions at the end.

_{i}is carried out with a subjective approach and a survey method, based on which a research instrument was created—a QoE questionnaire with 11 questions [15], through which five reconstructed groups of paired factors affecting QoE

_{i}were presented.

_{i}, a selection of research variables was made on the basis of the questions in the questionnaire. Ten independent/input variables (X

_{1}...X

_{10}) were identified, which were defined by questions 1–9 of the questionnaire: age (X

_{1}); gender (X

_{2}); legal–regulatory affiliation of the organization/firm (X

_{3}); provider(s) with whom the user has experience (X

_{4}); qualitative level of experience—perceived level of QoS (X

_{5}); level of satisfaction with the price of the provider’s services (X

_{6}); evaluation of the user’s legal security in interactions with the service provider in the area of service provision on the basis of contracts and payment of bills by cost calculation (X

_{7}); user experience expressed by characteristics for four traffic classes (X

_{8}); user experience expressed by levels for four traffic classes (X

_{9}); length of user experience of the provider’s services (X

_{10}).

_{i}are represented by 11 questions in the questionnaire, based on which, in the next step, three classes of research variables are defined. It is noticeable that the input independent variables (X

_{1}....X

_{10}), transition variables (D

_{1}...D

_{10}; C

_{1}...C

_{5}) and an output dependent variable QoE

_{i}were identified.

_{1}), creativity (D

_{2}), communication (D

_{3}), personality (D

_{4}), courage (D

_{5}), confidence (D

_{6}), charisma (D

_{7}), competence (D

_{8}), common sense (D

_{9}), and memory (D

_{10}). Dissatisfaction indicators are defined by connection of online services with the following forms and measures of rigidity that services cause in users: interpersonal (C

_{1}), behavioral (C

_{2}), structural—longing for structure and coping with the lack of structure (C

_{3}), prospective anxiety (C

_{4}), inhibitory rigidity (C

_{5}). The two groups of indicators represent subjective user factors, which, based on a mathematical model, enable a direct assessment of the quality of user experience QoE

_{i}as a dependent variable in the evaluation model.

#### 3.3. Survey Process

#### 3.4. Preprocessing of Data Collected

#### 3.5. Statistical Analysis of Data Collected

#### 3.6. Assessment of User QoE_{i} with a Mathematical Model

_{i}, as the first step, implies the identification of KQI that directly affect the quality of user experience [11]. The following expression represents the generic relation between the QoE value and identified indicators:

_{i}, it is necessary to assign certain weighting factors to the identified indicators. Thus, most often, QoE can be expressed by a linear combination of weight indicators, which is shown by the following expression [11]:

_{j}represents the weighting factor assigned to the indicator ${I}_{E}^{j}$. In this research, both satisfaction indicators and dissatisfaction indicators were taken into account, and memorized in the subjective user experience for which each QoE

_{i}was determined when creating a linear mathematical model.

#### 3.7. Creating a QoE_{i} Probability Model

_{i}will take a value from a certain interval [a, b]. Using the SPC tool for Excel and the distribution fitting procedure, data fitting, i.e., QoE

_{i}values with several common distributions, was tested. As a result of this procedure, the Anderson–Darling (AD) statistic test, as well as corresponding p-values, were given for data fitting with a certain type of distribution, and the distributions were ranked according to the Akaike information criterion (AIC). AIC, as one of the most common tools in statistical modeling, is a means of selecting the best model in a set of models. Therefore, this criterion is an indicator of the degree of the statistical model fitting with real data (goodness of fit—GOF). Its mathematical formulation is an extension of the maximum likelihood principle [34]:

#### 3.8. Correlation Analysis of Research Variables

_{1}–X

_{10}), transitional (D, C) and a dependent variable (QoE

_{i}). One of the main reasons for using Spearman’s correlation coefficient in this research is that the values of most variables are measured on an ordinal scale, where categories are ranked from 1 to 5. As a means of visualizing correlation coefficients, a correlogram or correlation matrix is used [36,37]. A correlogram enables the analysis of the relationship between each pair of numerical variables through a scatterplot, or some other symbol that represents the correlation (in this case, color).

#### 3.9. Creating a Model for QoE_{i} Prediction and Classification

_{i}ratings, calculated on the basis of a mathematical model, are continuous in nature, and prediction models are created for their forecasting. Mapping these values into a discrete, ordinal scale with numeric levels 0, 1, 2, 3, 4 and 5 allows for the creation of classification models based on machine learning techniques.

#### Models for QoE_{i} Prediction

_{i}models based on multiple linear regression, decision trees and machine learning models were created in the IBM SPSS Modeler software environment using an automatic modeling method [40]. The data obtained by filling out the questionnaire along with the estimated QoE

_{i}values were structured in an Excel file into input/output vectors and divided into two parts. Ninety percent of the vectors were used to train the model and the remaining ten percent were used to test the prediction accuracy performance. As input to all created models, 10 independent variables (X

_{1}…X

_{10}) were used. All models were created according to the supervised learning paradigm [39].

_{i}and the sum of squared errors of the null model, or, given mathematically:

_{i}is the estimated value of user experience of the ith user, QoE

_{ip}is the prediction based on the actual value of QoE

_{i}, and QoE

_{im}is the arithmetic mean of the variable QoE

_{i}.

_{i}classification models created by using an automatic modeling method.

_{i}classification models, as well as predictive models, were created in the IBM SPSS Modeler software environment using the Auto Classifier option. Support techniques included neural networks, C&R trees; quick, unbiased, efficient statistical trees (QUESTs); CHAID; C5.0; logistic regression; decision lists; Bayesian networks; discriminants; nearest neighbors and SVM. The DA method was used to improve classification performance by artificially expanding the training and testing datasets. A set of 10 independent variables was used as an input to the classification models, and the dataset was divided in the ratio 90%:10%.

## 4. Factors Affecting the Quality of User Experience

#### 4.1. Legal–Regulatory Factors

#### 4.2. Technological–Process Factors of Network Services/Applications

#### 4.3. Content-Formatted and Performative Factors

#### 4.4. Contextual–Relational Factors

#### 4.5. Subjective–User Factors

#### 4.6. An Interaction Model of Paired Factors Affecting QoE_{i}

_{i}value. As can be concluded from the figure, there are two-way relations of influence according to the each with each principle of the paired factors and their corresponding input variables, including transitory subjective–user variables of satisfaction and/or dissatisfaction [15]. The value of QoE

_{i}as a dependent variable is obtained via a mathematical model, directly based on users’ ratings of satisfaction and dissatisfaction indicators, while independent variables serve as inputs to QoE

_{i}prediction and classification models.

## 5. Results of QoE_{i} Modeling and Discussion

_{i}values, the probability model of the dependent variable and the correlation analysis of the research variables. Based on the data collected and values calculated, the second part of the section describes how to create predictive and QoE

_{i}classification models.

#### 5.1. Research Sample Statistics

_{i}values are expressed through five indicators of dissatisfaction defined by question 11, out of which behavioral rigidity has the greatest impact, rated with MOS = 2.88, and the least influence is caused by prospective anxiety, with a value of MOS = 2.80 [15]. As can be seen on the basis of Table 2, the MOS values are lower for indicators of dissatisfaction compared to indicators of satisfaction.

#### 5.2. QoE_{i} Estimation Model

_{1},…,D

_{10}). These indicators affect the increase in the overall level of QoE

_{i}by multiplying their weights with a corresponding user rating. Negative indicators (C

_{1},…,C

_{5}) are associated with weight values equal to −0.2, which means that they affect the reduction of the QoE

_{i}level. Based on the above, a mathematical formulation can be derived for estimating the user QoE

_{i}:

_{j}denotes the user’s ratings of satisfaction indicators, and C

_{k}denotes the ratings of dissatisfaction indicators. Therefore, the total level of user QoE

_{i}is determined on a scale of real numbers from 0 to 5, noting that negative values of QoE

_{i}are mapped to the value of 0. The arithmetic mean of individual, subjective QoE

_{i}values calculated in this way, where i = 1...157, is equal to the QoE value for a group of 157 users and amounts to MOS = 0.503. If this MOS is rounded to an integer value, a score of 1 is obtained, which qualitatively represents a bad user QoE rating for the observed services and telecommunications operators.

#### 5.3. QoE_{i} Probability Model

_{i}. So, although a smaller value of the AD test indicates a better fit of the data with the observed type of distribution (logistic distribution has the smallest value of AD = 9.604), the choice was made according to AIC. Smaller p-values represent evidence against the null hypothesis of a statistical fit of the data with a particular distribution.

#### 5.4. Correlation Analysis of Research Variables

_{i}value, the strongest correlation (by absolute value) with user ratings of the quality of experience has transitional variables used to define satisfaction indicators D (0.41) and dissatisfaction indicators C (−0.55). According to the correlogram, the highest correlation exists between variables X

_{6}and X

_{7}and is 0.63, and then between X

_{5}and X

_{9}, with a coefficient of 0.43. Additionally, it can be noted that a correlation equal to zero exists between the independent variables X

_{1}and X

_{6}and between X

_{1}and the dissatisfaction indicator C.

#### 5.5. Predictive Models of QoE_{i}

_{i}prediction using a multiple linear regression model, decision trees and models based on machine learning created in the IBM SPSS Modeler software environment are presented.

#### 5.5.1. Multiple Linear Regression Model

_{i}= 0.343 + 0.0115 X

_{1}+ 0.104 X

_{2}− 0.0438 X

_{3}− 0.0081 X

_{4}− 0.0133 X

_{5}+ 0.0271 X

_{6}+ 0.0210 X

_{7}− 0.0085 X

_{8}− 0.0007 X

_{9}+ 0.0035 X

_{10}

_{i}prediction, it is observed that the mean square error (MSE) of the test in this case has a value of MSE = 0.625, while the coefficient of determination is R

^{2}= 2.5%. Therefore, it is a very low coefficient of determination, which confirms the shortcomings of the linear model.

#### 5.5.2. Boosted Decision Tree Model

_{8}) and the user experience expressed by the levels for the four traffic classes (X

_{9}), represent the variables with the greatest influence (predictor importance) on the prediction of QoE

_{i}. In last place in terms of influence is the variable representing the age (X

_{1}) of the respondent.

_{i}prediction error is then MSE = 0.16. As a result of training and testing the model, by executing the code [60], the MATLAB program package generates a graphic representation of the importance of the influence of each of the mentioned inputs on the prediction of QoE

_{i}(Figure 6a). By creating a decision tree with inputs that are exclusively satisfaction indicators, Figure 6b shows as a result the ranking of the importance of the influence on the prediction of QoE

_{i}for each of them. It can be concluded that the rating of the service impact on the user’s charisma has the greatest influence (D

_{7}). The prediction error of the quality of user experience in this case has a value of MSE = 0.45. Nevertheless, the smallest prediction error is achieved if only five dissatisfaction indicators denoted by C are considered as inputs (MSE = 0.09). This result is in accordance with the presented mathematical model for estimating QoE

_{i}(expression (5)) in which higher-weighting coefficients according to absolute value (0.2) are associated with indicators of dissatisfaction. Figure 6c graphically shows the importance of the influence of indicator C on the prediction of the target variable, from which it can be concluded that the rating of the connection of services with prospective anxiety (C

_{4}) is at the top of the ranking by the value of prediction importance. So, taking into account the MSE values, it is clear that the prediction using the boosted decision tree model shows significantly more accurate results compared to the linear model.

#### 5.5.3. Predictive Models Created by Using an Automatic Modeling Method

_{i}values and the QoE

_{i}values obtained by prediction, and that is for the model based on the k-NN machine learning technique (r = 0.206). The C&R tree model shows the worst prediction performance in terms of both correlation (0.000) and relative error (1.147) [62].

_{1}...X

_{10}and one dependent variable QoE

_{i}), 5 nodes in the hidden layer and 11 nodes in the output layer in which the input vectors (X

_{1}’...X

_{10}’, QoE

_{i}’) are reconstructed. The transfer functions of the neurons of the hidden and output layer have the form of a logistic sigmoid function (logsig):

_{i}values from the set for testing the model and the values obtained as a result of prediction have a very high level of correlation. In total, 2512 input–output vectors were used for training and testing the tree model, out of which 2260 were used for training and 252 for testing. The trend of decreasing relative error of all the models shown in Figure 9 can be represented by the following linear equation:

^{−4}is defined by default to create a new split in the tree. All divisions are binary, which means that each division generates two subgroups, each of which is then divided into another two and so on until one of the stopping criteria is reached. The following were set as stopping criteria in the created C&R tree model:

- Minimum records in parent branch—prevents splitting if the number of records in a node to be split (parent) is less than the set value—2% of the total dataset.
- Minimum records in child branch—prevents the split if the number of records in any branch created by the split (child node) would be less than the set value—1% of the total dataset.

_{ip}

_{max}—the maximum value of the prediction variable QoE

_{i}, QoE

_{ip}

_{min}—the minimum value of the prediction variable QoE

_{i}. In addition to the value of A, Table 6 shows the number of inputs for each component, and the size of the model expressed by the number of nodes.

#### 5.6. Models for QoE_{i} Classification

_{i}values obtained on the basis of the mathematical assessment model presented by Expression (5) are of a continuous nature, in the first step of the process of creating this type of model it is necessary to map a continuous into a discrete absolute category rating (ACR) scale as presented in Table 7 [64,65].

_{i}classification simultaneously. Similar to the Auto Numeric node, in only one pass through the modeling process, Auto Classifier examines different machine learning techniques with default option settings. Options in this case mean the number of neural network layers, the number of neurons in each layer, the shape and parameters of the classification function, the training algorithm, stopping criteria, tree size, etc. Based on the test results, Auto Classifier offers the most accurate solutions and ranks them according to the overall classification accuracy expressed in percentages. Based on the available set of 157 vectors, several models were created, and Table 8 shows the results of testing the three most accurate models.

_{i}prediction model. Four iterations of increasing the dataset were performed, according to the procedure previously carried out for predictive models (Figure 8), and the results of testing the three most accurate models after each iteration are presented in a diagram in Figure 11. As a criterion for selecting the final, most accurate model, the maximum total classification accuracy was observed, which was achieved after the fourth iteration, i.e., with a total set of 2512 input–output vectors, and it amounts to 94.048%. Increasing the overall classification accuracy by increasing the data set can be represented by a linear growth trend that has the following form:

^{−1}to 10

^{−6}, and according to default settings in the software, it is C = 10

^{−3}. A smaller criterion value results in a more accurate model, but it takes more time to train [67]. The RBF gamma parameter is 0.1 and can be considered as the “expansion” of the kernel and thus the decision region. When gamma is low, the decision area is very wide. In the case of a high value of gamma, the decision boundaries are very distorted and so-called decision boundary islands are created around the points belonging to the same category.

## 6. Conclusions

_{i}indicators. One of the important novelties presented in this paper compared to previous publications is the dependent/output variable QoE

_{i}, which indicates the sustainable quality of user experience at the level of an individual user. The arithmetic mean of QoE

_{i}for i = 1...157, actually represents the value of QoE for a specified group of users expressed through MOS. Given that in Section 5 (Section 5.2) the MOS value was calculated and is 0.503, it can be concluded that the overall quality of experience for the sample of 157 users is insufficiently acceptable.

_{i}predictive modeling, a multiple linear regression model, a machine learning model based on decision trees and predictive models based on an automatic modeling method were created. The results show that the accuracy of the model trained and tested on a set of collected data was insufficiently high. In order to improve the prediction accuracy performance, a synthetic expansion of the dataset was performed using the data augmentation method in four iterations, which increased the basic set from 157 to 2512 input–output vectors. The smallest relative prediction error was achieved with the C&R tree model and is RE = 0.274, and it was further reduced by 9.9% to the value of RE = 0.247 with the boosting method.

_{i}classification, the models trained and tested on the initial set of 157 vectors also demonstrated insufficiently high accuracy, 50% at best. However, by augmenting the dataset with the DA method in four iterations, it was shown that the model based on SVM achieves the highest accuracy of 94.048% among the created models. All created models for assessment, prediction and classification of sustainable quality of user experience can serve entities that provide telecommunication services as a tool to adapt them to users in the future and thus achieve long-term sustainability of quality with increased effectiveness and efficiency of business results.

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Zhu, Y.; Heynderickx, I.; Redi, J.A. Understanding the role of social context and user factors in video quality of experience. Comput. Hum. Behav.
**2015**, 49, 412–426. [Google Scholar] [CrossRef] - Geiser, M.; Panwar, D.; Tomar, P.; Harsh, H.; Zhang, X.; Solanki, A.; Nayyar, A.; Alzubi, J.A. An optimization model for software quality prediction with case study analysis using MATLAB. IEEE Access
**2019**, 7, 85123–85138. [Google Scholar] [CrossRef] - Movassagh, A.A.; Alzubi, J.A.; Gheisari, M.; Rahimi, M.; Mohan, S.; Abbasi, A.A.; Nabipour, N. Artificial neural networks training algorithm integrating invasive weed optimization with differential evolutionary model. J. Ambient Intell. Humaniz. Comput.
**2021**, 12, 1–9. [Google Scholar] [CrossRef] - Hansmann, R.; Mieg, H.A.; Frischknecht, P. Principal sustainability components: Empirical analysis of synergies between the three pillars of sustainability. Int. J. Sustain. Dev. World Ecol.
**2012**, 19, 451–459. [Google Scholar] [CrossRef] - ETSI TS 103 294; Speech and Multimedia Transmission Quality (STQ); Quality of Experience; A Monitoring Architecture, Technical Specification, V1.1.1. European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2014. Available online: https://www.etsi.org/deliver/etsi_ts/103200_103299/103294/01.01.01_60/ts_103294v010101p.pdf (accessed on 30 March 2021).
- ETSI TR 102 643; Human Factors (HF); Quality of Experience (QoE) Requirements for Real-Time Communication Services, Technical Report, V1.0.1 (2009-12). European Telecommunications Standards Institute: Sophia Antipolis Cedex, France. 2009. Available online: https://www.etsi.org/deliver/etsi_tr/102600_102699/102643/01.00.01_60/tr_102643v010001p.pdf (accessed on 30 March 2021).
- Laghari, K. On Quality of Experience (QoE) for Multimedia Services in Communication Ecosystem. Ph.D. Thesis, Institut National des Telecommunictions, Télécom SudParis, Paris, France, 30 April 2012. Available online: https://tel.archives-ouvertes.fr/tel-00873612/document (accessed on 13 November 2022).
- ITU-T Recommendation P.10/G.100; Amendment 2: New Definitions for Inclusion in Recommendation ITU-T P.10/G.100. International Telecommunication Union: Geneva, Switzerland, 2008. Available online: https://www.itu.int/rec/T-REC-P.10-200807-S!Amd2/en (accessed on 8 November 2021).
- Vakili, A.; Grégoire, J.-C. QoE management in a video conferencing application. In Future Information Technology, Application and Service; Lecture Notes in Electrical Engineering; Park, J.J., Leung, V.C.M., Wang, C.L., Shon, T., Eds.; Springer: Dordrecht, The Netherlands, 2012; Volume 164, pp. 191–201. [Google Scholar] [CrossRef]
- ITU-T Recommendation P.10/G.100 (11/17); Vocabulary for Performance, Quality of Service and Quality of Experience. International Telecommunication Union: Geneva, Switzerland, 2017. Available online: https://www.itu.int/rec/T-REC-P.10-201711-I/en (accessed on 8 November 2022).
- Dai, Q. A Survey of Quality of Experience. In Energy-Aware Communications. EUNICE 2011; Lecture Notes in Computer Science; Lehnert, R., Ed.; Springer: Berlin, Heidelberg, 2011; Volume 6955, pp. 146–156. [Google Scholar] [CrossRef]
- Eswara, N.; Ashique, S.; Panchbhai, A.; Chakraborty, S.; Sethuram, H.P.; Kuchi, K.; Kumar, A.; Channappayya, S.S. Streaming video QoE modeling and prediction: A long short-term memory approach. IEEE Trans. Circuits Syst. Video Technol.
**2019**, 30, 661–673. [Google Scholar] [CrossRef][Green Version] - Barman, N.; Martini, M.G. Qoe modeling for HTTP adaptive video streaming–a survey and open challenges. IEEE Access
**2019**, 7, 30831–30859. [Google Scholar] [CrossRef] - Ruan, J.; Xie, D. A survey on QoE-oriented VR video streaming: Some research issues and challenges. Electronics
**2021**, 10, 2155. [Google Scholar] [CrossRef] - Banjanin, M.K.; Maričić, G.; Stojčić, M. Multifactor influences on the quality of experience service users of telecommunication providers in the Republic of Srpska, Bosnia and Herzegovina. Int. J. Qual. Res.
**2022**, 17. [Google Scholar] [CrossRef] - Daengsi, T.; Wuttidittachotti, P. QoE Modeling for Voice over IP: Simplified E-model Enhancement Utilizing the Subjective MOS Prediction Model: A Case of G. 729 and Thai Users. J. Netw. Syst. Manag.
**2019**, 27, 837–859. [Google Scholar] [CrossRef] - García-Pineda, M.; Segura-Garcia, J.; Felici-Castell, S. A holistic modeling for QoE estimation in live video streaming applications over LTE Advanced technologies with Full and Non Reference approaches. Comput. Commun.
**2018**, 117, 13–23. [Google Scholar] [CrossRef] - Ickin, S.; Vandikas, K.; Fiedler, M. Privacy preserving qoe modeling using collaborative learning. In Proceedings of the 4th Internet-QoE Workshop on QoE-Based Analysis and Management of Data Communication Networks, Los Cabos, Mexico, 21 October 2019; pp. 13–18. [Google Scholar]
- Khokhar, M.J.; Saber, N.A.; Spetebroot, T.; Barakat, C. An intelligent sampling framework for controlled experimentation and QoE modeling. Comput. Netw.
**2018**, 147, 246–261. [Google Scholar] [CrossRef][Green Version] - Dasari, M.; Sanadhya, S.; Vlachou, C.; Kim, K.H.; Das, S.R. Scalable Ground-Truth Annotation for Video QoE Modeling in Enterprise WiFi. In Proceedings of the 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), Banff, AB, Canada, 4–6 June 2018; pp. 1–6. [Google Scholar]
- Veeraragavan, N.R.; Montecchi, L.; Nostro, N.; Vitenberg, R.; Meling, H.; Bondavalli, A. Modeling QoE in dependable tele-immersive applications: A case study of world opera. IEEE Trans. Parallel Distrib. Syst.
**2015**, 27, 2667–2681. [Google Scholar] [CrossRef] - Hoßfeld, T.; Biedermann, S.; Schatz, R.; Platzer, A.; Egger, S.; Fiedler, M. The memory effect and its implications on Web QoE modeling. In Proceedings of the 2011 23rd International Teletraffic Congress (ITC), San Francisco, CA, USA, 6–9 September 2011; pp. 103–110. [Google Scholar]
- Lycett, M.; Radwan, O. Developing a quality of experience (QoE) model for web applications. Inf. Syst. J.
**2019**, 29, 175–199. [Google Scholar] [CrossRef][Green Version] - Banjanin, M.K.; Stojčić, M.; Drajić, D.; Ćurguz, Z.; Milanović, Z.; Stjepanović, A. Adaptive Modeling of Prediction of Telecommunications Network Throughput Performances in the Domain of Motorway Coverage. Appl. Sci.
**2021**, 11, 3559. [Google Scholar] [CrossRef] - Bouraqia, K.; Sabir, E.; Sadik, M.; Ladid, L. Quality of experience for streaming services: Measurements, challenges and insights. IEEE Access
**2020**, 8, 13341–13361. [Google Scholar] [CrossRef] - Hu, Z.; Yan, H.; Yan, T.; Geng, H.; Liu, G. Evaluating QoE in VoIP networks with QoS mapping and machine learning algorithms. Neurocomputing
**2020**, 386, 63–83. [Google Scholar] [CrossRef] - Isak-Zatega, S.; Lipovac, A.; Lipovac, V. Logistic regression based in-service assessment of mobile web browsing service quality acceptability. EURASIP J. Wirel. Commun. Netw.
**2020**, 96, 1–21. [Google Scholar] [CrossRef] - Mitra, K.; Zaslavsky, A.; Åhlund, C. QoE modelling, measurement and prediction: A review. arXiv
**2014**, arXiv:1410.6952. [Google Scholar] [CrossRef] - Pal, D.; Triyason, T. A survey of standardized approaches towards the quality of experience evaluation for video services: An ITU perspective. Int. J. Digit. Multimed. Broadcast.
**2018**, 2018, 1724. [Google Scholar] [CrossRef][Green Version] - Juluri, P.; Tamarapalli, V.; Medhi, D. Measurement of quality of experience of video-on-demand services: A survey. IEEE Commun. Surv. Tutor.
**2015**, 18, 401–418. [Google Scholar] [CrossRef] - Baraković Husić, J.; Baraković, S.; Cero, E.; Slamnik, N.; Oćuz, M.; Dedović, A.; Zupčić, O. Quality of experience for unified communications: A survey. Int. J. Netw. Manag.
**2020**, 30, e2083. [Google Scholar] [CrossRef] - Banjanin, M.K.; Stojčić, M. Conceptual Model of the Cyber-physical System in the Space of the M9J Road Section. In Proceedings of the 15th International Conference on Advanced Technologies, Systems and Services in Telecommunications (TELSIKS), Niš, Serbia, 20–22 October 2021; pp. 299–302. [Google Scholar]
- Belmudez, B.; Möller, S. Audiovisual quality integration for interactive communications. EURASIP J. Audio Speech Music Process.
**2013**, 2013, 1–23. [Google Scholar] [CrossRef][Green Version] - Cavanaugh, J.E.; Neath, A.A. The Akaike information criterion: Background, derivation, properties, application, interpretation, and refinements. Wiley Interdiscip. Rev. Comput. Stat.
**2019**, 11, e1460. [Google Scholar] [CrossRef] - Portet, S. A primer on model selection using the Akaike Information Criterion. Infect. Dis. Model.
**2020**, 5, 111–128. [Google Scholar] [CrossRef] [PubMed] - Ćurguz, Z.; Banjanin, M.; Stojčić, M. Machine learning models for prediction of mobile network user throughput in the area of trunk road and motorway sections. In Proceedings of the First International Conference on Advances in Traffic and Communication Technologies, Sarajevo, Bosnia and Herzegovina, 26–27 May 2022; pp. 27–35. [Google Scholar]
- Ćurguz, Z.; Banjanin, M.; Stojčić, M. Prediction of user throughput in the mobile network along the motorway and trunk road. Sci. Eng. Technol.
**2022**, 2, 23–30. [Google Scholar] [CrossRef] - Simakovic, M.; Cica, Z.; Drajic, D. Big-Data Platform for Performance Monitoring of Telecom-Service-Provider Networks. Electronics
**2022**, 11, 2224. [Google Scholar] [CrossRef] - Stojčić, M.; Banjanin, M.K. Predictive Modeling of Telecommunications Traffic Performance Based on Machine Learning Techniques. In Proceedings of the VIII International Symposium NEW HORIZONS 2021 of Transport and Communications, Doboj, Bosnia and Herzegovina, 26–27 November 2021; pp. 378–385. [Google Scholar]
- Ivaniš, P.; Drajić, D. Information Theory and Coding-Solved Problems; Springer International Publishing: Cham, Switzerland, 2012; ISBN 978-3-319-49369-5. [Google Scholar] [CrossRef]
- Stojčić, M.; Banjanin, M.; Ćurguz, Z.; Stjepanović, A. Machine Learning Model of Communication of Physical and Virtual Sensors in the Mobile Network on the Motorway Section. In Proceedings of the 44th International Convention, CTI, MIPRO 2021, Opatija, Croatia, 27 September–1 October 2021; pp. 447–452. [Google Scholar]
- Tensorflow. Available online: https://www.tensorflow.org/tutorials/images/data_augmentation (accessed on 13 November 2022).
- ETSI TS 102 250-2; Speech and Multimedia Transmission Quality (STQ); QoS Aspects for Popular Services in Mobile Networks; Part 2: Definition of Quality of Service Parameters and Their Computation, Technical Specification, V2.4.1. European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2015. Available online: https://www.etsi.org/deliver/etsi_ts/102200_102299/10225002/02.04.01_60/ts_10225002v020401p.pdf (accessed on 2 April 2021).
- ETSI TS 102 250-1; Speech and Multimedia Transmission Quality (STQ); QoS Aspects for Popular Services in Mobile Networks; Part 1: Assessment of Quality of Service, Technical Specification, V2.2.1 (2011-04). European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2011. Available online: https://www.etsi.org/deliver/etsi_ts/102200_102299/10225001/02.02.01_60/ts_10225001v020201p.pdf (accessed on 2 April 2021).
- ETSI TR 103 488; Speech and Multimedia Transmission Quality (STQ); Guidelines on OTT Video Streaming; Service Quality Evaluation Procedures, Technical Specification, V1.1.1 (2019-01). European Telecommunications Standards Institute: Sophia Antipolis Cedex, France, 2019. Available online: https://www.etsi.org/deliver/etsi_tr/103400_103499/103488/01.01.01_60/tr_103488v010101p.pdf (accessed on 11 April 2021).
- 3GPP TR 26.944; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; End-to-End Multimedia Services Performance Metrics (Release 10), Technical Report, V10.0.0 (2011-03). 3rd Generation Partnership Project: Sophia Antipolis, France, 2011. Available online: https://www.arib.or.jp/english/html/overview/doc/STDT63v9_10/5_Appendix/Rel10/26/26944-a00.pdf (accessed on 31 March 2021).
- ITU-T Recommendation G.1000; Communications Quality of Service: A Framework and Definitions. International Telecommunication Union: Geneva, Switzerland, 2002.
- Mtel. Opšti Uslovi za Pružanje Telekomunikacionih Usluga (Prečišćeni Tekst). Available online: https://mtel.ba/Binary/397/Opsti-uslovi-za-pruzanje-telekomunikacionih-usluganesluzbeni-precisceni-tekst.pdf (accessed on 8 October 2022).
- GSM Association. Definition of Quality of Service Parameters and Their Computation; Official Document IR.42, Version 9.0. Available online: https://www.gsma.com/newsroom/wp-content/uploads//IR.42-v9.0.pdf (accessed on 3 April 2021).
- BaBatunde, K.A.; Akinboboye, S. Corporate social responsibility effect on consumer patronage-management perspective: Case study of a telecommunication company in Nigeria. J. Komun.
**2013**, 29, 55–71. [Google Scholar] - Maričić, G.; Banjanin, M.K.; Stojčić, M. Legal-Regulatory Paired Component in the QoE Model for Assessment of the Quality of Experience of Users of Services of Company. In Proceedings of the Materials of 1st International Scientific and Practical Internet Conference “The impact of COVID-19 Pandemic on development of modern world: Threats and opportunities”-WayScience, Dnipro, Ukraine, 9–10 September 2021; pp. 33–36. [Google Scholar]
- Banjanin, K.M. Komunikacioni Inženjering; Univerzitet u Istočnom Sarajevu, Saobraćajno-Tehnički Fakultet Doboj: Doboj, Bosnia and Herzegovina, 2007; ISBN 978-99938-859-4-8. [Google Scholar]
- Brunnström, K.; Beker, S.A.; De Moor, K.; Dooms, A.; Egger, S.; Garcia, M.-N.; Hossfeld, T.; Jumisko-Pyykkö, S.; Keimel, C.; Larabi, C.; et al. Qualinet white paper on definitions of quality of experience. In Proceedings of the Fifth Qualinet Meeting, Novi Sad, Serbia, 12 March 2013. [Google Scholar]
- Reiter, U.; Brunnström, K.; Moor, K.D.; Larabi, M.C.; Pereira, M.; Pinheiro, A.; Zgank, A. Factors influencing quality of experience. In Quality of Experience; Möller, S., Raake, A., Eds.; Springer: Cham, Switzerland, 2014; pp. 55–72. [Google Scholar] [CrossRef]
- Rahman, M.A.; El Saddik, A.; Gueaieb, W. Augmenting context awareness by combining body sensor networks and social networks. IEEE Trans. Instrum. Meas.
**2010**, 60, 345–353. [Google Scholar] [CrossRef] - Su, J.H.; Yeh, H.H.; Yu, P.S.; Tseng, V.S. Music recommendation using content and context information mining. IEEE Intell Syst.
**2010**, 25, 1541–1672. [Google Scholar] [CrossRef] - Naumann, A.B.; Wechsung, I.; Hurtienne, J. Multimodal interaction: A suitable strategy for including older users? Interact. Comput.
**2010**, 22, 465–474. [Google Scholar] [CrossRef] - Hyder, M.; Laghari, K.u.R.; Crespi, N.; Haun, M.; Hoene, C. Are QoE Requirements for Multimedia Services Different for Men and Women? Analysis of Gender Differences in Forming QoE in Virtual Acoustic Environments. In Emerging Trends and Applications in Information Communication Technologies IMTIC 2012. Communications in Computer and Information Science; Chowdhry, B.S., Shaikh, F.K., Hussain, D.M.A., Uqaili, M.A., Eds.; Springer: Berlin/Heidelberg, Germany, 2012; Volume 281, pp. 200–209. [Google Scholar] [CrossRef]
- Jumisko-Pyykkö, S.; Häkkinen, J.; Nyman, G. Experienced quality factors: Qualitative evaluation approach to audiovisual quality. In Multimedia on Mobile Devices; SPIE: Bellingham, WA, USA, 2007; Volume 6507, pp. 169–180. [Google Scholar] [CrossRef]
- MathWorks. Available online: https://www.mathworks.com/products/demos/machine-learning/boosted-regression.html (accessed on 8 November 2022).
- IBM. Available online: https://www.ibm.com/docs/en/SS3RA7_18.3.0/pdf/ModelerModelingNodes.pdf (accessed on 8 November 2022).
- Selvanathan, M.; Jayabalan, N.; Saini, G.K.; Supramaniam, M.; Hussin, N. Employee Productivity in Malaysian Private Higher Educational Institutions. PalArch’s J. Archaeol. Egypt/Egyptol.
**2020**, 17, 66–79. [Google Scholar] [CrossRef] - Stackoverflow. Available online: https://stackoverflow.com/questions/39265746/data-augmentation-techniques-forgeneral-datasets (accessed on 8 November 2022).
- Singla, A.; Rao, R.R.R.; Göring, S.; Raake, A. Assessing media qoe, simulator sickness and presence for omnidirectional videos with different test protocols. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 1163–1164. [Google Scholar] [CrossRef]
- Kara, P.A.; Bokor, L.; Sackl, A.; Mourão, M. What your phone makes you see: Investigation of the effect of end-user devices on the assessment of perceived multimedia quality. In Proceedings of the 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), Messinia, Greece, 26–29 May 2015; pp. 1–6. [Google Scholar] [CrossRef]
- IBM. Available online: https://www.ibm.com/docs/en/spss-modeler/saas?topic=models-how-svm-works (accessed on 8 November 2022).
- Scikit-Learn. Available online: https://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html (accessed on 8 November 2022).

**Figure 1.**Correlation amongst QoEi influencing factors, questionnaire questions and research variables.

**Figure 5.**The importance of the influence of certain independent variables on the prediction of QoE

_{i}.

**Figure 6.**The importance of the influence of indicators on the prediction of QoE

_{i}: (

**a**) all indicators; (

**b**) satisfaction indicators; (

**c**) indicators of dissatisfaction (adapted with permission from Ref. [15]. 2023, International Journal for Quality Research ).

**Figure 9.**Results of testing predictive models of QoE

_{i}in four iterations of increasing the dataset using the DA method.

**Figure 11.**Results of testing the QoEi classification model in four iterations of increasing the dataset using the DA method.

**Table 1.**Comparative overview of QoE modeling in previous research with QoE

_{i}modeling in this paper.

Ord. Number | Title of Paper | Service/ Application Observed | Methods and Models Used | Observed Factors/Variables Affecting QoE | Comparative Improvements Presented in This Paper |
---|---|---|---|---|---|

[16] | QoE Modeling for Voice over IP: Simplified E-model Enhancement Utilizing the Subjective MOS Prediction Model: A Case of G.729 and Thai Users | VoIP | Objective simplified E-model; subjective MOS model for prediction | Delay, packet loss, jitter | (a), (b), (c), (d), (e), (f), (g), (h), (i), (j) |

[17] | A holistic modeling for QoE estimation in live video streaming applications over LTE Advanced technologies with Full and Non Reference approaches | Live video streaming | Statistical modeling—regression analysis for objective assessment of video quality; factor analysis | Variables related to QoS, bit stream and basic video quality metrics grouped into factors | (a), (b), (d), (e), (f), (g), (i), (j) |

[18] | Privacy Preserving QoE Modeling using Collaborative Learning | Applicable to all services | A machine learning model with data privacy protection—a collaborative machine learning model | Maximum bandwidth for downlink; search time; assessment time | (a), (b), (d), (e), (f), (g), (i), (j) |

[19] | An Intelligent Sampling Framework for Controlled Experimentation and QoE Modeling | YouTube video streaming | Machine learning models | QoS variables (delay, bandwidth...) | (a), (b), (c), (d), (e), (f), (g), (i), (j) |

[20] | Scalable Ground-Truth Annotation for Video QoE Modeling in Enterprise WiFi | Video telephony | Adaboosted decision trees | Perceptual bitrate (PBR), freeze ratio, freeze length and number of video freezes | (a), (b), (c), (d), (e), (f), (g), (i), (j) |

[21] | Modeling QoE in Dependable Tele-immersive Applications: A Case Study of World Opera | World Opera application | Subjective method based on perceived reliability; stochastic activity networks (SANs) | Human perception of video and audio, audience characteristics, performance elements and artistic content | (a), (b), (c), (d), (e), (f), (h), (i), (j) |

[22] | The Memory Effect and Its Implications on Web QoE Modeling | Interactive Web services | Support vector machines; iterative exponential regressions; two-dimensional hidden Markov models | Technical factors (scope, page load time, packet loss...); psychological factors (expectations, memory effects, user) | (a), (b), (c), (d), (e), (f), (g), (i), (j) |

[25] | Quality of Experience for Streaming Services: Measurements, Challenges and Insights | Streaming services | Subjective methods; objective methods; hybrid methods | Human-related influencing factors; system-related influencing factors; context-related influencing factors; content-related influencing factors | (a), (b), (c), (d), (e), (f), (h), (i), (j) |

[26] | Evaluating QoE in VoIP networks with QoS mapping and machine learning algorithms | VoIP services | MOS model; PESQ model; E-model; a single-layer artificial neural network model | Echo, packet loss, jitter, bandwidth, delay | (a), (b), (c), (d), (e), (f), (g), (i), (j) |

[23] | Developing a Quality of Experience (QoE) model for Web Applications | Web applications | Quality of experience of Web application (QoEWA) model | Objective factors (KPI); subjective factors (KQI). | (a), (b), (d), (e), (f), (g), (h), (i), (j) |

[27] | Logistic regression based in-service assessment of mobile web browsing service quality acceptability | Searching the Web | Binary logistic regression model | Average time-to-connect-TCP | (a), (b), (d), (e), (f), (g), (h), (i), (j) |

Mark | Mean (MOS) | Standard Deviation | Variance | Sum of Squares | Min | Median | Max | |
---|---|---|---|---|---|---|---|---|

Indicators of user satisfaction | D_{1} | 3.32 | 1.01 | 1.03 | 1889 | 1 | 3 | 5 |

D_{2} | 3.17 | 0.99 | 0.99 | 1727 | 1 | 3 | 5 | |

D_{3} | 3.71 | 0.96 | 0.91 | 2300 | 1 | 4 | 5 | |

D_{4} | 3.00 | 1.02 | 1.04 | 1575 | 1 | 3 | 5 | |

D_{5} | 2.92 | 1.00 | 1.01 | 1499 | 1 | 3 | 5 | |

D_{6} | 2.99 | 1.02 | 1.03 | 1568 | 1 | 3 | 5 | |

D_{7} | 3.04 | 1.01 | 1.03 | 1616 | 1 | 3 | 5 | |

D_{8} | 3.16 | 0.95 | 0.90 | 1697 | 1 | 3 | 5 | |

D_{9} | 2.96 | 1.11 | 1.23 | 1569 | 1 | 3 | 5 | |

D_{10} | 3.09 | 1.09 | 1.19 | 1674 | 1 | 3 | 5 | |

Indicators of user dissatisfaction (forms and measures of rigidity) | C_{1} | 2.83 | 1.05 | 1.09 | 1426 | 1 | 3 | 5 |

C_{2} | 2.88 | 1.02 | 1.03 | 1462 | 1 | 3 | 5 | |

C_{3} | 2.82 | 1.01 | 1.01 | 1402 | 1 | 3 | 5 | |

C_{4} | 2.80 | 1.06 | 1.12 | 1408 | 1 | 3 | 5 | |

C_{5} | 2.87 | 1.08 | 1.17 | 1472 | 1 | 3 | 5 |

Distribution | AD | p | AIC |
---|---|---|---|

LogNormal—three parameter | 13.47 | 0.000 | 26.61 |

LogLogistic—three parameter | 12.16 | <0.005 | 47.09 |

Exponential—two parameter | 37.41 | <0.001 | 102.3 |

Logistic | 9.604 | <0.005 | 292.0 |

Normal | 10.53 | 0.000 | 295.5 |

Smallest extreme value | −157.0 | >0.250 | 176443 |

Largest extreme value | −86.76 | >0.250 | 1,579,994,965 |

Created Model | Correlation | Relative Error |
---|---|---|

1. Regression | 0.127 | 1.070 |

2. k-nearest neighbors (k-NN) | 0.206 | 1.075 |

3. C&R tree | 0.000 | 1.147 |

Absolute Value of the Correlation Coefficient | Qualitative Assessment |
---|---|

$0.00<\left|r\right|\le \text{}$0.19 | Very low correlation |

$0.20\le \left|r\right|\le \text{}$0.39 | Low correlation |

$0.40\le \left|r\right|\le \text{}$0.59 | Moderate correlation |

$0.60\le \left|r\right|\le \text{}$0.79 | High correlation |

$0.80\le \left|r\right|\le \text{}$1.00 | Very high correlation |

Model/Component Number | Prediction Accuracy (A) | Number of Inputs | Number of Nodes |
---|---|---|---|

1 | 69.7% | 9 | 23 |

2 | 52.3% | 9 | 19 |

3 | 47.0% | 10 | 17 |

4 | 39.2% | 10 | 25 |

8 | 34.6% | 10 | 35 |

5 | 30.4% | 10 | 25 |

9 | 27.8% | 10 | 29 |

6 | 25.5% | 10 | 29 |

10 | 13.7% | 10 | 25 |

7 | 10.0% | 10 | 21 |

Continuous Scale | ACR Scale |
---|---|

0 ≤ QoE_{i} < 0.5 | 0 |

0.5 ≤ QoE_{i} < 1.5 | 1 |

1.5 ≤ QoE_{i} < 2.5 | 2 |

2.5 ≤ QoE_{i} < 3.5 | 3 |

3.5 ≤ QoE_{i} < 4.5 | 4 |

4.5 ≤ QoE_{i} ≤ 5 | 5 |

Model Created | Total Classification Accuracy [%] |
---|---|

1. k-NN | 50.00 |

2. C&R tree | 46.15 |

3. Neural network | 42.31 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Banjanin, M.K.; Stojčić, M.; Danilović, D.; Ćurguz, Z.; Vasiljević, M.; Puzić, G.
Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models. *Sustainability* **2022**, *14*, 17053.
https://doi.org/10.3390/su142417053

**AMA Style**

Banjanin MK, Stojčić M, Danilović D, Ćurguz Z, Vasiljević M, Puzić G.
Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models. *Sustainability*. 2022; 14(24):17053.
https://doi.org/10.3390/su142417053

**Chicago/Turabian Style**

Banjanin, Milorad K., Mirko Stojčić, Dejan Danilović, Zoran Ćurguz, Milan Vasiljević, and Goran Puzić.
2022. "Classification and Prediction of Sustainable Quality of Experience of Telecommunication Service Users Using Machine Learning Models" *Sustainability* 14, no. 24: 17053.
https://doi.org/10.3390/su142417053