Forecasting Customer Complaints in the Mobile Telecommunication Sector Using Supervised Machine Learning ^†

Abstract

The telecommunications sector continues to experience exponential growth in demand, accompanied by a corresponding increase in customer complaints regarding service quality. To effectively address these challenges, many telecom companies rely on customer feedback to assess and improve their network and services. This case study focused on a Lebanese telecom company, investigating the application of machine learning algorithms, particularly Artificial Neural Networks. The analysis performed compares the effectiveness of various optimizers and activation functions to identify the most suitable approach for our specific context. Utilizing a sample database comprising 10,000 mobile market subscribers, this study incorporates variables such as gender, age, device manufacturer, service quality, and complaint status. The results of this case study emphasize that, across various metrics, and despite its complexity, Artificial Neural Networks outperform other algorithms in terms of prediction performance. Additionally, we propose a segmented prediction model based on time intervals and customer groups to enhance prediction accuracy and practical utility. The segmentation will involve examining customer groups based on their characteristics.

1. Introduction

Over the course of numerous years, customer feedback has served as a fundamental element in assessing the performance of companies [1]. Consequently, the call center, acting as an intermediary between the company and its clientele, assumed a pivotal role in the collection, processing, and oversight of customer feedback, encompassing both positive and negative opinions, facilitated by automated control systems [2].

Despite the significant advancements in technology leading to heightened data processing capabilities, instances of customer complaints remain unavoidable. The subsequent

Table 1. Complaint distribution among users based on reason categories [3].

	Coverage Problem	Difficulty Obtaining Information	Incorrect Billing	Incorrect Charges	Breach of Contract	Difficulty Unsubscribing	Delayed Service
Number of Users	536	73	37	34	27	13	9
Number of Complainers	153	63	35	31	23	12	8
Percentage of Complainers (%)	28.5	86.3	94.6	91.2	85.2	92.3	88.9

delineates customer complaint reasons and their corresponding likelihood to complain relative to service problem types.

This table illustrates the diverse array of factors contributing to customer dissatisfaction, including issues such as coverage discrepancies, difficulties in accessing information, inaccuracies in billing, erroneous charges, breaches of contractual agreements, and delays in service provision [3].

Our primary focus is on the aspect of coverage, particularly in the realm of telecommunications. Coverage pertains to the geographical expanse within which a base station signal can effectively reach [4]. It is contingent upon various factors such as terrain features (e.g., mountains), building structures, and the technology employed (e.g., LTE). The efficacy of mobile device connectivity to a base station hinges upon the strength of the signal it emits. However, obstacles such as buildings, particularly those constructed with metallic materials, can impede signal propagation. Additionally, signal penetration underground is limited, further exacerbating coverage issues in certain locales [5].

Hence, the problem of inadequate coverage can be construed as a manifestation of service quality deficiencies, given that suboptimal coverage often underpins most service-related shortcomings.

2. Overview and Related Work

The detection of customer service quality issues often remains elusive until customers formally voice their complaints. To mitigate latent customer grievances, it becomes imperative to proactively identify and address instances of poor service quality before they escalate into formal complaints [6]. Establishing systems capable of discerning nascent shifts in trends is paramount. These early warning mechanisms hold significant utility across various domains, including financial securities trading, sales performance forecasting, and churn analysis [7].

In his work published in 2018, Choi defined customer complaints as requests or demands for adjustments in products or services. This definition draws upon research findings [8]. For instance, Peek (2023) noted that personal characteristics and internal influences exert the greatest influence on customer complaints [9]. Building upon this, Nimako (2012) characterized customer complaints as negative communications about products or services directed towards the firm, manufacturer, or third-party entities [10]. Kim et al. (2003) emphasized the prevalence of complaints among dissatisfied users and identified various contributing factors, including customer personality, attitude, motives, perceived value of time, information level, and socio-demographic characteristics [11]. Furthermore, others categorized customer responses into three types: direct communication of complaints to the company, product rejections accompanied by warnings to acquaintances, and legal action against the company or governmental complaints [12].

These research insights underscore the multifaceted nature of customer complaints and the diverse responses they elicit from consumers, as shown in Figure 1.

Additional factors affecting customer complaints were discussed in the literature, including customer experiences, socio-cultural values, psychological attributes, emotional dispositions, and personality traits [13,14,15]. Further, Garin-Munoz study has focused on customer complaint behavior within the mobile telecommunications sector. The study delineated six stages, where initially, in stage 1, 28.5% of users declared service problems. Subsequent stages unveiled only 57.9% of these customers who faced service problems actually lodged complaints, with the final stage indicating that 94% of those who did complain opted to directly contact the company [3].

Through a simple calculation in Figure 2, it becomes evident that only 54.42% of clients who encounter service issues proactively provide negative feedback to the company, leaving 45.58% of such clients undocumented in the company’s complaint database. This calculation is derived by multiplying the percentage of clients who voice complaints about services (57.9%) by the proportion of those who directly communicate their grievances to the company (94%):

$$\begin{gathered} Percentage of clients providing negative feedback to the company \left(\%\right) \\ =0.579 x 0.94=0.54426=54.42\% \end{gathered}$$

(1)

Subtracting this figure from 100%, representing the total client’s facing service problems, reveals that 45.58% of clients opt not to voice their grievances directly to the company. Instead, they may silently switch to another service provider, potentially disseminating negative perceptions about the product and the telecom company, which could manifest in a decreased statistical matrix for the company overall.

Further analysis reveals that only 15.51% of clients who encounter service problems choose to express their complaints directly to the company. This is calculated by multiplying the percentage of clients experiencing service issues by the proportion of clients who express complaints in any form, then by the percentage of clients who opt to communicate their complaints directly to the company:

$$\begin{gathered} Percentage of clients encoutering service problems and reporting \\ complaints directly to the company \left(\%\right) \\ =0.285 \times 0.579 \times 0.94=0.1551=15.51\% \end{gathered}$$

(2)

This leaves us to ponder the fate of the remaining 84.48% of clients. Are they content with the service, or are they contemplating switching providers in the near future?

Numerous studies have explored customer behavior in response to products or services that fail to meet their expectations; however, these studies have predominantly relied on survey and interview data, often lagging comprehensive integration of all client data into a unified model first. Additionally, they often struggle to predict whether a client will indeed lodge a complaint about a particular service or product.

This paper shifts its focus towards harnessing the capabilities of machine learning (ML) modeling to address these limitations. Specifically, we propose a modern strategy or novel approach leveraging various attributes extracted from a Lebanese telecom company’s database pertaining to customer data. Our approach utilizes ML Artificial Neural Networks (ANNs) to analyze and forecast customer complaints. First, through experimentation we conducted comparisons across multiple activation functions, loss functions, and learning rates to identify the most effective and accurate model for predicting client complaint behavior.

3. Research Objective and Process

As demonstrated earlier, a significant portion (45.58%) of unsatisfied clients refrain from lodging complaints with the company, while only 15.51% express negative feedback directly. Consequently, according to Garin-Munoz’s study, there arises a necessity for the company to devise a method to anticipate the satisfaction levels of the remaining 84.49% of clients [3].

This paper proposes the implementation of an Artificial Neural Network (ANN) model utilizing logistic regression (LR) to forecast whether a client encounters service issues. Throughout the project we delve into discussions and comparisons between various activation functions and parameter optimizers to identify the most suitable ones yielding the highest accuracy.

Subsequently, we employ a Recurrent Neural Network (RNN) to assess the company’s services, particularly focusing on network coverage and data consumption. This enables us to forecast whether service quality is expected to improve or deteriorate.

4. Data Processing

The study uses information from the Touch Demo database between 2016 and 2017. We obtain a sample of call records from Touch (Beirut, Lebanon) clients, during a six-month period, which are combined and aggregated into one dataset.

The dataset is evenly balanced between complaining customers and non-complainers, with each group comprising 50% of the data. To facilitate model development and evaluation, I partitioned the learning dataset into three distinct groups: a 60% training set, a 20% validation set, and another 20% test set.

Table 2. Description of the learning dataset.

Learning Dataset	Description
Training Set	Used to train the machine-learning model by learning the network parameters (e.g., weights and biases). One epoch represents a complete pass through the entire training dataset.
Validation Set	Used to tune the model’s hyper-parameters (e.g., network architecture, number of hidden layers, learning rate) and to optimize model performance during training.
Test Set	Used exclusively to evaluate the final model’s predictive performance on unseen data after training and hyper-parameter tuning have been completed.

provides a summary of the definitions of each dataset, delineating their roles in the modeling process.

In the training set, we build the machine learning model using a selected optimizer based on the validation data values; as shown in the Figure 3, this phase will be repeated until we have an accurate result with an accurate model in the testing phase; we use the generated model to predict potential complaints and tack the related action on the spot.

The customer complaint prediction analysis focuses on the below main points:

• Correlation between the dependent and independent variables (Device Module, Network Area, age, service kind and network coverage);
• Decision making enhancement and new service launches (new bundles with customized apps, videos, TV channel…);
• New marketing and broadcast promotions: dedicated offers for subscribers predicted to complain or experiencing service failure.

5. Modeling

5.1. Definition of Logistic Regression

Logistic regression was proposed in the late 1960s and early 1970s (Cabrera, 1994), and it became routinely available in statistical packages in the early 1980s [16].

Logistic regression is a statistical method for binary classification. It extends the idea of linear regression to scenarios where the dependent variable is categorical, not continuous. Typically, linear regression model is used when the outcome to be predicted belongs to one of two possible categories, such as “yes” or “no”, “success” or “failure”, “win” or “lose”, etc.

Logistic regression, on the other hand, is used to predict binary outcomes, such as whether a patient has a disease or not, or whether a customer will churn or leave a complaint.

According to Peng et al. (2002), logistic regression relies on the mathematical concept of logarithms, particularly the natural logarithm of coefficients [17]. Figure 4 presents the conceptual architecture of the Logistic Regression model, where multiple input features are weighted and combined to estimate the probability of class membership, enabling binary classification of customer complaints.

To fit a logistic regression model, you need to have a dataset with one or more independent variables (Figure 5) and a binary dependent variable (0, 1 or yes, no). The independent variables can be either continuous or categorical, but the dependent variable must be binary. Once you have your dataset, you can fit the logistic regression model by using an optimization algorithm (Adadelta, Adamax, Adam, etc.) or specific activation functions (Sigmoid, Softmax, ReLU) to maximize the likelihood of the observed data [18].

5.2. Activation Functions

Activation functions play a critical role in shaping the architecture of ANNs, providing them with non-linear characteristics essential for learning complex representations. In the early stages of ANN development, the Sigmoid function emerged as the predominant choice for activation [20]. However, as ANN applications evolved across various domains, the shortcomings of Sigmoid, particularly its susceptibility to the vanishing gradient problem due to its small derivative, became apparent. Consequently, alternative activation functions like Softmax and ReLU gained prominence. Notably, ReLU offers a derivative of one for positive inputs, addressing some of the limitations encountered with Sigmoid [21].

• Sigmoid
  The Sigmoid activation function serves to transform input values from the range (−∞; +∞) to a bounded range between [0;1]. It exhibits inherent non-linearity and possesses a smooth derivative, as depicted in Figure 6. However, due to its output being constrained within the [0;1] range, the output of each neuron is compressed, potentially leading to vanishing gradients, particularly in deep networks [22].

  $$\sigma \left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$$

  (3)
• ReLU
  ReLU, a linear activation function, effectively nullifies any negative input, preserving positive values unchanged. This simplicity is a notable advantage; however, the drawback lies in its tendency to transform all negative values to zero. This issue, known as “Dying ReLU” Figure 7, is a specific instance of the vanishing gradient problem. Once a neuron becomes negative, it tends to remain inactive, hindering its ability to recover and contribute to the network’s learning process [23].

  $$ReLu \left(x\right)=\max (0,x)$$

  (4)
• Softmax
  The Softmax activation function is primarily employed in the output layer, especially in neural networks designed for classification tasks. In this configuration in Figure 8, each Softmax unit corresponds to a specific class in a multi-class classification problem. For instance, in a K-class classification scenario, such as the MNIST dataset where digits are classified, K represents the number of possible classes (e.g., K = 10 for digits 0 through 9). Each Softmax unit “i” computes the probability that the input belongs to class-i, providing a probability distribution over all possible classes [24].

  $$\sigma \left(z\right)j={\displaystyle \frac{e^{zj}}{{\displaystyle {\sum }_{i=1}^K}{e}^{zi}}}$$

  (5)

6. Methodology

6.1. Dataset Sample

This database was employed to collect information required for further analysis.

Table 3. Dataset sample.

Device_Model	Area	PSTN_ID	Age	Title	Local Call	Roaming Call	AVG Voice Call Duration/Cycle	AVG Data Call Duration/Cycle	Nbr Voice Call/Cycle	Nbr Data Call/Cycle	Service Type	Speed	Network Coverage Traffic Failure	Data Consumption Failure
Apple A1981/Apple iPhone 14 Pro Max	BEIRUT	340	C	Mr.	YES	NO	66	985	32	32	HS5	4G	1	0
Apple A1577/Apple iPhone 12 Pro	BEIRUT	61	B	Mr.	YES	YES	14	563	11	28	HS4	4G	0	0
Samsung SM-G920F/Samsung Galaxy S6	BEIRUT	72	B	Eng.	YES	NO	18	789	23	59	HS3	4G	0	0
Samsung SM-G610FDS/Samsung Galaxy J7 Prime	BEIRUT	590	A	Mr.	NO	NO	99	365	32	22	HS4	3G	0	0
Apple A1708/Apple iPhone SE	BEIRUT	341	B	MS	NO	YES	89	899	25	12	HS3	4G	0	0
Samsung SM-J200X/Samsung Galaxy J2	BEIRUT	577	B	Eng.	YES	NO	48	1025	36	31	HS3	4G	0	1
Samsung SM-J730FDS/Samsung Galaxy J7 Prime	BEIRUT	345	A	Mr.	YES	NO	361	1702	22	29	HS1	4G	1	0
Apple A1758/Apple iPhone 11	BEIRUT	501	C	Eng.	YES	NO	287	965	33	39	HS1	4G	0	0

presents a sample of this dataset.

To strengthen its capabilities, we predominantly employed ReLU as the activation function and the input and hidden layers, while Softmax was utilized in the output layer. We conducted a comparative analysis across various optimizers paired with different learning grades to identify the most suitable optimizer for our project.

6.2. Analysis of ADADELTA Optimizer Learning Rate = 0.1, Epochs = 1000

As shown in the Figure 9, the prediction accuracy for data consumption complaints surpasses 90% when utilizing a high learning rate of 1.0 and a normal number of training epochs, specifically 1000. This approach minimizes training time and CPU resource consumption, making it suitable for environments with limited server hardware capabilities. On the other hand, the prediction accuracy for network failure exhibits an ascending trend, also reaching above 90%.

6.3. Analysis of ADADELTA Optimizer Learning Rate = 0.001, Epochs = 1000

As shown in the Figure 10 conversely, the graphs indicate that for data consumption complaint prediction, using a high number of training epochs (Epochs 1000) results in lower accuracy rates (<90%) and a relatively straight-line pattern. This suggests that a longer training process is required, demanding significant hardware resources (CPU, Memory, etc.), yet yielding diminished accuracy.

6.4. Analysis of ADAGRAD Optimizer Learning Rate = 0.001, Epochs = 1000

The graphs in Figure 11 indicate the prediction accuracy for data consumption complaints falls below 90% when utilizing a low learning rate of 0.001 and a high number of training epochs (1000). This results in extended training times and increased CPU resource consumption, necessitating substantial server hardware capabilities. Conversely, the prediction graph (Figure 12) for network failure achieves an accuracy of 85% with a consistent, stable line pattern when trained with the maximum number of epochs, namely 1000.

Evidence of overfitting in the neural network is apparent in the observed curve, which exhibits a plateau with minimal variation. This plateau indicates that the accuracy levels have stabilized at approximately 0.866 for “y accuracy”, around 0.84 for “x accuracy”, and approximately 0.834 for “z accuracy”.

6.5. Analysis of ADAGRAD Optimizer Learning Rate = 0.01, Epochs = 1000

On the contrary, the graph depicting the performance of the ADAGRAD Optimizer (Figure 12) illustrates that using a low number of training epochs (100) for data consumption complaint prediction results in an accuracy rate below 85%. This suggests that additional training iterations are necessary to achieve higher accuracy, demanding significant hardware resources (CPU and memory).

It is worth mentioning that this curve (Figure 12) does not explicitly suggest overfitting of the network. However, it does indicate that while the results are not as precise as those obtained with ADADELTA, there may be some overfitting occurring in the trained set. Furthermore, the results indicate that the ADADELTA Optimizer stands out as a reliable and effective choice compared to other optimizers.

7. Conclusions

In the field of mobile telecommunication technologies and data sciences, mobile operators have access to a wealth of data regarding customer’s service experiences. This case study leverages these datasets to propose a novel methodology for predicting customer complaints using machine learning techniques. By utilizing a combination of real industry mobile operator datasets as learning input data, we introduce a new research approach that achieves a prediction accuracy exceeding 92%. While this accuracy is commendable, the best model parameters include two levels of Dense with “RELU” Activation and one Dense with “Softmax” activation using ADADELTA Optimizer with a small learning rate 0.1. Further enhancements could be attained through the utilization of larger training datasets and possibly by optimizing the combination of activation functions and their associated learning rates. Across various metrics, it becomes evident that despite their complexity, Artificial Neural Networks outperform other algorithms in terms of prediction performance.

Moving forwards, our future work will delve into segmented prediction techniques to explore the influence of demographic variables and device manufacturers on customer complaints. Specifically, we will be aiming to examine the relationship between gender, age, device manufacturers, and their impacts on predicting customer complaints.