WES-BTM: A Short Text-Based Topic Clustering Model

Abstract

User comments often contain their most practical requirements. Using topic modeling of user comments, it is possible to classify and downscale text data, mine the information in user comments, and understand users’ requirements and preferences. However, user comment texts are usually short and lack rich word frequency and contextual information with sparsity. The traditional topic model cannot model and analyze these short texts well. The biterm topic model (BTM), while solving the sparsity problem, suffers from accuracy and noise problems. In order to eliminate information barriers and further ensure information symmetry, a new topic clustering model, termed the word-embedding similarity-based BTM (WES-BTM), is proposed in this paper. The WES-BTM builds on the BTM by converting word pairs into word vectors and calculating their similarity to perform word pair filtering, which in turn improves clustering accuracy. Based on the experimental results using actual data, the WES-BTM outperforms the BTM, LDA, and NMF models in terms of topic coherence, perplexity, and Jensen–Shannon divergence. It is verified that the WES-BTM can effectively reduce noise and improve the quality of topic clustering. In this way, the information in user comments can be better mined.

1. Introduction

Topic modeling is one of the most important methods in the field of text mining today and is important for understanding user needs and preferences [1]. The topic model analyses the structural and semantic information of text data by classifying and downgrading the text data to achieve a deep summary of the text [2]. The traditional LDA topic model is an unsupervised learning approach based on a probabilistic graphical model that generates polynomial distributions of topics a priori using Dirichlet [3]. However, when extracting topics from short texts such as user comments, the direct application of the LDA topic model will face the problem of severe sparsity. The BTM focuses on topic modeling for short texts and solves the problems of short text length and semantic sparsity by realizing word pair co-occurrence on the whole dataset [4]. Thus, the mining of hidden information in the short text corpus is realized. However, the BTM does not take into account the similarity between word pairs, which can lead to problems with accuracy and noise in the clustering results.

To solve the above problems, this paper introduces a WES-BTM (biterm topic model based on word embedding similarity). The WES-BTM uses word embeddings to convert word pairs into vector representations and computes the similarity between them for word pair filtering. The WES-BTM generates biterms from preprocessed data and generates word vectors with the help of pre-trained models. Cosine similarity is used to compare and calculate the similarity between two words to obtain the semantic relevance between them [5]. The parameter value γ was set and the screening range of similarity was obtained experimentally.

We compared the WES-BTM model with the BTM using user reviews from a typical manufacturing company, GREE, as a data source. We observed that the subject consistency scores of the WES-BTM are significantly higher compared with the BTM. It is verified that the WES-BTM can effectively reduce noise and improve the quality of topic clustering. Finally, we use the WES-BTM to mine the potential information in the text to construct thematic indicators for the analysis and identification of user needs, which enables enterprises to better understand user needs and further ensure the symmetry of information.

2. Literature Review

With the advent of the big data era, text data have exploded [6]. This has led to difficulty in using traditional data processing and analysis means to meet the current needs. In the face of this challenge, text mining has gradually become a mainstream method for text information analysis, as it is capable of extracting key information from text datasets [7]. Among the many techniques of text mining, topic modeling is particularly remarkable [8]. It is capable of detecting and outlining the intrinsic themes or topics of documents from large-scale documents. Topic modeling dissects the structural and semantic information in text data by classifying and downgrading the text data to achieve a deep summary overview of the text [2]. Various modeling techniques based on mathematical frameworks have been proposed in the research field of topic mining.

Deerwester et al. proposed the latent semantic analysis (LSA) model of LSA based on singular value decomposition (SVD), which discovers topic-based semantic relationships between text and words using matrix decomposition [9]. However, LSA lacks explicit probabilistic interpretation and requires large-scale matrix operations. For this reason, Hofmann proposed the probabilistic latent semantic analysis (PLSA) model, the core idea of PLSA is to build a probabilistic model for document–word item co-occurrence data [10]. Although PLSA solves some of the problems of LSA, the number of its parameters increases linearly with the number of documents, which may lead to overfitting. To solve this problem, Blei et al. [3] proposed a three-level probabilistic topic model LDA (latent Dirichlet allocation) based on semantic analysis research and then constructed a probabilistic generative model that generates polynomial distributions of topics with a Dirichlet prior. His proposed LDA topic model is widely used to mine hidden topics in text corpora.

LDA can handle large text data very well [11]. However, texts such as comments and tweets are usually short and contain only a few words [12]. This makes it difficult for LDA to estimate reliable topic distributions from short texts, and direct application will face the problem of serious sparsity. Many scholars have modeled topics by splicing short texts into long texts. Hong et al. [13] demonstrated that the length of a document can have a large impact on the effectiveness of a trained topic model and that model texts trained on summarized long texts have better performance. Zhao et al. [14] developed a Twitter-LDA model designed for short tweets, which finds topics from a representative sample of the entire Twitter sample to discover themes. Balikas et al. [15] found that grouping words in a sentence with the same theme can better handle the sparsity of short texts. In addition to these methods, some new technologies have emerged in the field of thematic modeling in recent years. Top2Vec is a novel method that utilizes vector embedding and clustering to discover topics from short texts [16]. BER Topic is another method that utilizes the powerful capabilities of the BERT language model for encoding and representing text [17]. By using pre-trained BERT models, BER Topic can capture rich semantic information from short texts and achieve more accurate topic modeling. Yan et al. [4] proposed a generative word pair theme model (BTM) to learn themes in short texts by directly modeling the generation of word pairs across the entire corpus. The BTM, by observing the co-occurrence patterns of word pairs across the entire dataset, can effectively provide a better model for short texts to infer topics. The BTM is also widely used in text clustering [18], topic mining [19], and hot topic detection [20], among others. In order to achieve better results, scholars have improved the BTM. Hu et al. selected words with specific parts of speech to form binomials for the topic model, thereby improving the clustering effect of the model [21]. Huang et al. improved the dual-term topic model by introducing an embedded topic model to distinguish between noisy topics and potential topics [22]. Wu et al. proposed a short text clustering algorithm that combines the BTM and GloVe similarity linearly, thereby improving clustering performance [18].

In summary, the BTM is more suitable for short text topic mining, and more and more scholars process and analyze short text on the web using the BTM [23]. However, the traditional BTM does not take into account the correlation between word pairs while solving the sparsity problem of short text, which may add non-semantic heterogeneity and noise into the modeling process, thus affecting the effect of topic modeling. Therefore, in this paper, we propose a WES-BTM that uses the word2vec pre-training model for word vector processing and applies cosine similarity to calculate the similarity to realize word pair filtering, in order to reduce the noise and improve the quality of topic clustering, so as to be able to more accurately extract the topic content of short text.

3. Research Design

User reviews are typically short text data that contain valuable information about users’ needs and attitudes [24]. In order to effectively extract topic information and potential information from text data, this paper proposes a WES-BTM for text mining, which can be summarized in four steps: (1) collecting product reviews from two e-commerce platforms, Tmall and JD.com. The data are then cleaned and word splitting is performed using Jieba, which is one of the most commonly used Chinese word splitting tools [25]; (2) creating biterms from the preprocessed data, vectorizing them using pre-trained models, and filtering out the biterms with low similarity by calculating their similarity (3) determining the optimal number of topics, modeling, and training to obtain topic clustering results; and (4) analyzing topic clustering results and visualizing them for further analysis. The specific process is shown in Figure 1.

3.1. Data Collection

With the development of e-commerce enterprises on the Internet, people’s shopping habits have undergone tremendous changes, and online shopping has become the main way to purchase goods [26]. JD.com is China’s largest self-operated e-commerce enterprise [27]. Tmall is the largest comprehensive e-commerce platform in terms of size in China [28]. The product reviews on these platforms can be used to obtain feedback from consumers.

GREE Electric Appliances is a prominent home appliance company in China, known for its large user base. GREE air conditioners are highly regarded for their excellent service and high user satisfaction [29]. Analyzing the reviews of GREE enterprise can provide valuable insights into the actual feedback and opinions of users regarding a product. Hence, this research focuses on GREE Air Conditioning as the subject of study and collects user comments using a Python crawler, which will serve as the basis for further research.

3.2. WES-BTM

3.2.1. Biterms Screening

Product reviews are often in the form of short texts. Compared with ordinary documents, short texts lack word frequency and contextual information. Therefore, there is a serious sparsity problem when modeling with traditional topic models. The BTM represents topics as groups of related words, and their relevance is inferred from co-occurrence patterns [4]. By extracting all biterms from the entire corpus to construct training data, the sparsity problem of traditional topic models is solved. While modeling using the BTM, if the similarity between word pairs is low, it may lead to assigning word pairs belonging to the same topic to different topics. Then, the accuracy of the model is reduced. In addition, word pairs with low similarity may also increase the noise level of the model, thus making it more difficult for the model to distinguish topics. To compute the similarity between biterms, a pre-trained Word2Vec word embedding model is used [30]. Word2Vec is a method for capturing semantic information in natural language by learning word vectors from large-scale corpora [31]. This pre-trained model provides high-quality word embedding vectors. Then, the cosine similarity between biterms is calculated using the formula shown in Equation (1). Cosine similarity calculates word vector similarity, independent of vector length and direction, and is a simple and effective measure of data similarity. This method requires inputting two vectors representing biterms and returns a score between 0 and 1, where 1 indicates the highest similarity.

$$sim(v_i,v_j)=\frac{{\overrightarrow{v}}_i\cdot {\overrightarrow{v}}_j}{\left|{\overrightarrow{v}}_i\right|\cdot \left|{\overrightarrow{v}}_j\right|}$$

(1)

3.2.2. Model Building

This article builds a WES-BTM based on the characteristics of user reviews for manufactured products. The WES-BTM probabilistic graphical model is shown specifically in Figure 2. The meanings of each parameter in the model are shown in

Table 1. Meaning of each parameter in the WES-BTM.

Parameter	Meaning
$\alpha$	Hyperparameter of Dirichlet distribution for θ
$\beta$	Hyperparameter of Dirichlet distribution for $\varphi$
$\varphi$	Topic-specific word distribution
$\theta$	Topic distribution of the whole corpus
$w_i,w_j$	Biterms from the multinomial distribution $\varphi$ of words
$\left|B\right|$	All biterms in the short text corpus
K	Dimensionality of topics
$z$	Topic selected from multinomial distribution $\theta$ of topics
$r$	Similarity threshold

.

The process of WES-BTM topic generation consists of three steps:

① For each specific topic z, calculate the topic distribution $\varphi$~Dir ($\beta$).

② Calculate the topic distribution $\theta$ across the entire corpus as $\theta ~\mathrm{D}\mathrm{i}\mathrm{r}\left(\alpha \right)$.

③ Calculate the similarity between biterms and select them for screening.

For each selected word pair b in set B, perform the following operations: Suppose the selected biterm $b=\left(w_i,w_j\right)$: (1) Randomly draw a topic z from the entire collection of $\theta$, that is, z~Muli ($\theta$). (2) Randomly draw a pair of words $w_i$ and $w_j$ from topic z, that is, $w_i$, $w_j$~Muli ($\varphi$). Based on the above steps, the joint distribution probability of word pair b can be obtained, as shown in Equation (2), and the probability of the entire corpus can be obtained, as shown in Equation (3).

$$P(B)={\displaystyle \sum _{z}P(z)}P(w_i|z)P(w_j|z)={\displaystyle \sum _z{\theta }_z{\varphi }_{i|z}}{\varphi }_{j|z}$$

(2)

$$P(B)={\displaystyle \prod _{(\mathrm{i},j)}{\displaystyle \sum _z{\theta }_z{\varphi }_{i|z}}}{\varphi }_{j|z}$$

(3)

Gibbs sampling simplifies the sampling process by decomposing the joint distribution into a series of conditional distributions, allowing us to sample one variable at a time. This sequential sampling method has high computational efficiency and is easy to converge. When modeling with the WES-BTM, using Gibbs sampling for global parameters $\theta$ and $\varphi$ performs approximate inference as follows: Firstly, randomly select the initial state of a Markov chain [32]. Then, calculate the conditional distribution for each b = $\left(w_i,w_j\right)$, where $z_{-b}$ represents the topic allocation of all biterms except for b. By applying the chain rule to the joint probability of the entire data, the conditional probability can be obtained, as shown in Equation (4). Estimate the topic–word distribution $\varphi$ and global topic distribution $\theta$ using word co-occurrences and counts of word–topic assignments as, shown in Equations (5) and (6).

$$P(z|z_{-b},B,\alpha ,\beta )\propto (n_z+\alpha )\frac{(n_{w|z}+\beta )(n_{w_j|z}+\beta )}{{({\displaystyle {\sum }_w{n}_{w|z}+M\beta })}^2}$$

(4)

$${\theta }_z=\frac{n_z+\alpha }{\left|B\right|+K_{\alpha }}$$

(5)

$${\varphi }_{\omega |\mathrm{z}}=\frac{n_{\omega |z}+\beta }{({\displaystyle {\sum }_{\omega }{n}_{\omega |z}+M_{\beta }})}$$

(6)

In Equation (4), $n_z$ denotes the number of occurrences of topic z in the corpus; $n_{\left.w\right|z}$ denotes the number of occurrences of ω under the topic word z; and M is the number of distinct words in the corpus without considering repetition. In Equation (5), ${\theta }_z$ denotes the probability distribution of topic z in the corpus; K denotes the number of topics, and $K_{\alpha }$ denotes the hyperparameter $\alpha$ multiplied by the total number of topics. In Equation (6), ${\varphi }_{\left.\mathsf{\omega }\right|z}$ denotes the probability distribution of words ω under a given topic z and $M_{\beta }$ denotes the hyperparameter $\beta$ multiplied by the size of the vocabulary list.

3.3. Contrast Experiment

To verify the effectiveness of the model, we compared the WES-BTM with the following baseline models: the Nmf model, the LDA model, and the BTM. The Nmf model (non-negative matrix factorization) is a topic model method based on matrix decomposition. It assumes that the data in the text matrix are non-negative and decomposes the text matrix into two non-negative matrices, representing the relationship between document topic and topic word, respectively. The LDA model is a probability graph model used to learn hidden topic structures from text. It views documents as a mixture of topics and models the generation process of each word as sampling from the topic. The BTM is a topic model method for short text data. It learns topic distribution based on the co-occurrence relationship of all word pairs in the document.

This article uses topic consistency, perplexity, and Jensen–Shannon divergence to evaluate the effectiveness of the topic model.

Topic coherence is an indicator used to measure the consistency and interpretability of the topics generated using the topic model. It is based on the principle of word co-occurrence and evaluates the cohesion of a topic by calculating the similarity between different words. A higher degree of topic consistency means that the correlation between topic words is stronger and more explanatory. The formula is shown in Equation (7).

$$TC=\frac{2}{\left|T\right|(|T|-1)}{\displaystyle {\sum }_{i=1}^{\left|T\right|}{\displaystyle {\sum }_{j=i+1}^{\left|T\right|}sim(T_i,T_j)}}$$

(7)

where $\left|T\right|$ represents the number of topics and $sim\left(T_i,T_j\right)$ represents the similarity value obtained by calculating the correlation between the i-th and j-th topics. The specific formula for $sim\left(T_i,T_j\right)$ is shown in Equation (8):

$$sim(T_i,T_j)=\frac{{\displaystyle {\sum }_{k,l\in Z}PMI(w_k,w_l)}}{N}$$

(8)

where Z represents the word set composed of two topics, $T_i\mathrm{a}\mathrm{n}\mathrm{d}{T}_j$, and N is the number of biterms. The specific formula for $PMI\left(w_k,w_l\right)$ is shown in Equation (9):

$$PMI(w_k,w_l)=\log \frac{P(w_k,w_l)}{P(w_k)P(w_l)}$$

(9)

where $w_k$ and $w_l$ stand for word or combination of words. $k$ and $l$ represent different indexes or numbers, which are used to identify different words. $P\left(w_k\right)$ and $P\left(w_l\right)$ represent the probability of words appearing in documents and $P\left(w_k,w_l\right)$ represents the probability of the two words appearing at the same time.

Perplexity is one of the commonly used evaluation indicators in topic models, which is used to measure the model’s ability to predict new documents. The lower the perplexity, the stronger the model’s ability to predict new documents. Perplexity can be understood as the uncertainty in the model’s prediction of the next vocabulary for a given set of documents. Generally speaking, models with lower levels of perplexity are more able to accurately predict the next word.

$$P\mathrm{erplexity}=\exp (-\frac{{\displaystyle {\sum }_{d=1}^M\log p(w_d)}}{{\displaystyle {\sum }_{d=1}^M{N}_d}})$$

(10)

where W is a set of product review texts, M is the total number of review texts, $p\left(w_d\right)$ is the probability of a word in a review text, and $N_d$ is the number of words in the review text.

Jensen–Shannon divergence is a measure of the difference between two probability distributions. In the topic model, we can use Jensen–Shannon divergence to measure the similarity between the generated topic distribution and the real topic distribution. A lower Jensen–Shannon divergence means that the generated topic distribution is closer to the real distribution, indicating that the model performs better.

$$JSD(P||Q)=\frac{1}{2}{D}_{KL}(P||M)+\frac{1}{2}{D}_{KL}(Q||M)$$

(11)

where $D_{KL}$ represents KL divergence, P and Q are two probability distributions, and M is the mean distribution of P and Q.

4. Result and Discussion

4.1. Data Collection and Preprocessing

We collected comments on a specific air conditioning product of GREE from the JD and Tmall platforms, obtaining 1000 comments from JD and 1980 comments from Tmall, totaling 2980 comments. We removed duplicate values from the crawled data, merged the initial and additional comments from users to ensure the authenticity and integrity of the comments., and eliminated phrases with a character length of less than four. The Harbin Institute of Technology Stop Words List is currently the mainstream stop word list [33]. In this paper, we use the Harbin Institute of Technology stop word list as the base stop word list, and add “GREE”, “air conditioner”, and other words that are not meaningful to the topic analysis. The stop words are excluded from the comments. Since a word is the smallest unit of writing in Chinese text, there is no obvious punctuation to distinguish between words. In the information processing of text, words are the components that can minimally represent the meaning of text. Jieba word segmentation tools are one of the most common Chinese word segmentation tools that provide various functions such as Chinese word segmentation and keyword extraction [34]. In this article, we use Jieba word segmentation tools to split the obtained product reviews for subsequent experiments.

4.2. Biterm Similarity Calculation

Word2Vec is a method for capturing semantic information in natural language by learning word vectors from large-scale corpora [31]. With the help of word vectors generated using the pre-trained model, we use the cosine similarity method to calculate the similarity between two words to obtain the semantic relevance between them. Some of the results obtained from the calculation are shown in

Table 2. Partial biterm similarity.

Word1	Word2	Similarity
Rest	Online shopping	0.099184
One-stop	Full process	0.46240
Door-to-door	Delivered to home	0.44340
Reliable	Cute	0.29023
Price Comparison	Cheap	0.59638
Impact	Repair	0.20116

. When “Door-to-Door” and “Delivered to home” appear together, it can be inferred that the topic category is more related to on-site maintenance. However, “Rest” and “Online shopping” cannot provide clear information about the topic. Therefore, it is necessary to remove biterms with low similarity using screening.

4.3. Parameter Setting

To investigate the impact of parameter γ on model performance, this paper measures the clustering topic quality with TC value [35].

$$C(z;V^{(z)})={\displaystyle \sum _{t=2}^T{\displaystyle \sum _{l=1}^t\log \frac{D(v_m{}^{(z)},v_l{}^{(z)})+1}{D(v_l{}^{(z)})}}}$$

(12)

where $D\left(v\right)$ is the document frequency of word type $v$, $D\left(v,v^{\prime }\right)$ is the common document frequency of word type $v$, and $V^{\left(t\right)}=\left(V_1^{\left(t\right)}\cdots V_m^{\left(t\right)}\right)$ is the possible word in the list topic T with the most M.

By calculating the semantic relevance of the topics, we evaluate the effectiveness of the topic model and improve the performance of the topic model. In order to select a suitable threshold value γ, the TC values of the BTM with γ = 0.1, 0.2, 0.3, and 0.4 and no improvement were respectively calculated under different numbers of topics.

Before training, various parameters were set. After conducting experiments and evaluating performance, we selected the following parameter values: $\alpha$ = 50/K, $\beta$ = 0.01, and iterations = 200. Regarding the distribution of topics in documents, $\alpha$ determines the sparsity of topics. A smaller $\alpha$ value results in a sparser topic distribution, where the document focuses more on a few topics. Conversely, a larger $\alpha$ value leads to a more uniform topic distribution, with the document covering multiple topics more evenly. By setting $\alpha$ to 50/K, we can achieve a sparser topic distribution in large-scale problems, enabling more topics to concentrate on a few words and enhancing the model’s ability to recognize keywords. $\beta$ serves as a hyperparameter controlling the sparsity of the topic–word distribution. A smaller $\beta$ value helps the model to better discover and distinguish keywords across different topics, thus improving the accuracy of the topic model. Conversely, a larger $\beta$ value helps capture more general topics and common vocabulary. The experiment found that $\beta$ = 0.01 yields better results. Regarding the selection of the iteration frequency, conducting 200 iterations allows us to fully leverage the corpus information in most scenarios. This process gradually stabilizes the model and yields a reliable topic distribution. Thus, considering the experimental results and performance evaluation, we chose the parameter values $\alpha$ = 50/K and $\beta$ = 0.01 and iterations = 200 to obtain a model with a sparse topic distribution, high accuracy, and stability for large-scale problems. The results are shown in

Table 3. TC values under different numbers of topics.

Threshold γ	K = 5	K = 10	K = 15	K = 20	K = 25
0	−19.2738	−107.27	−268.705	−493.265	−810.569
0.1	−20.4315	−109.156	−261.272	−479.804	−797.129
0.2	−19.1881	−104.833	−265.059	−488.35	−775.682
0.3	−21.9134	−110.502	−275.766	−490.324	−779.5
0.4	−22.2633	−112.257	−266.203	−483.858	−744.105

. The higher the TC value, the stronger the correlation between topics, which means a better topic model. When γ is set to 0.2, the results are better at multiple numbers of topics, and all experimental results are better than those without setting thresholds. So, in this paper, the γ value is set to 0.2.

4.4. Optimal Number of Topics

As the BTM belongs to unsupervised learning, the number of topics needs to be determined before modeling. Perplexity is an important indicator based on information entropy to determine the optimal number of topics [36]. Topic perplexity is an effective method to measure whether the number of topics is reasonable. Therefore, to select an appropriate number of topics, this study calculates perplexity according to Equation (10).

We calculated the perplexity score. The results are shown in Figure 3. According to the perplexity calculation results, as the number of topics increases, the perplexity continuously decreases. When the number of topics is 10, the rate of decrease in perplexity changes. So, 10 is chosen as the optimal number of topics.

4.5. Topic Model Training

This study selected GREE enterprise product reviews as data acquisition channels on JD and Tmall platforms. After processing the comment data, a WES-BTM was constructed. Based on the calculation of perplexity, the number of topics is set to 10. The parameters are set to the following values: α = 50/K and β = 0.01. After training the topic model for product comment text, we obtained the generated keywords and topic distribution, as shown in

Table 4. Topic keywords and distribution.

Topic	Keywords
Topic 1	logistics	contact	delivery	arrival	shipment
Topic 2	return	charge	logistics	reply	refund
Topic 3	mode	function	brand	sleep	sleeping
Topic 4	quality	packaging	appearance	attitude	return
Topic 5	abnormal noise	solve	charge	repair	use
Topic 6	phone	reservation	contact	call	complaint
Topic 7	brand	quality	purchase	trust	like
Topic 8	personnel	brand	repair	old machine	solve
Topic 9	energy saving	quiet	energy efficiency	brand	heating efficiency
Topic 10	online	explanation	feedback	contact	handle

.

4.6. Results Analysis

4.6.1. Compared algorithms

Table 5. Results of comparative experiments.

	Metric	Jensen–Shannon Divergence	Topic Coherence	Perplexity
Model
NMF		0.732260575	69.29741217	1192.004004
LDA		0.726564254	69.99972076	1225.850567
BTM		0.679978854	72.62718216	816.7186215
WES-BTM		0.679495459	72.6664798	810.5492609

shows the comparison results of the WES-BTM and other baseline models. By computing coherence measures, we find that the WES-BTM achieves higher coherence scores, indicating its ability to generate more consistent and meaningful topics. Perplexity is an important metric for assessing model fit. We observe that the WES-BTM achieves significantly lower perplexity than the BTM, LDA, and NMF models, indicating its advantage in capturing patterns and structures within the data. We further evaluate the differences in topic distributions among the different methods by computing Jensen–Shannon. The results reveal that the Jensen–Shannon values between the topic distributions of the WES-BTM and the true topic distributions are smaller, indicating that the WES-BTM better captures the underlying topic structure in the data.

4.6.2. Topic Strength Analysis

The topic distribution of the user comments is shown in

Table 6. Distribution of document topics.

	Topic	1	2	3	4	5	6	7	8	9	10
Number
1		0.02224	0	0.00205	0.00403	0.00487	0.0105	0.17302	0.00519	0.03042	0.74766
2		0.10231	0.00803	0.09877	0.01188	0.00761	0.06512	0.4701	0.00673	0.20856	0.02087
3		0.06096	0.00648	0.01792	0.00541	0.01029	0.01544	0.72332	0.00081	0.15876	0.0006
…
2898		0.11855	0.01644	0.03072	0.00149	0.02322	0.05286	0.10575	0.0031	0.64223	0.00565
2899		0.04238	0.04128	0.03361	0.0012	0.00272	0.01534	0.02914	0.00877	0.75504	0.07053

. In Table 6, the header row represents the topic, and the header column represents the comment number. The table displays the probability values of each comment’s topic. The first comment discusses the excellent post-purchase services and the enthusiastic and meticulous repair personnel. Its topic is consistent with the semantics “on-line”, “contact”, and “processing” in Topic 10 and matches the probability calculation. In the second comment, the user describes the reliable air conditioning products and fast delivery, which is consistent with the semantics “brand”, “quality”, and “purchase” in Topic 7. In the third comment, the user praises the finely crafted and beautiful product with good quality, which is consistent with the semantics “brand”, “quality”, and “purchase” in Topic 7. The 2898th and 2899th comments describe the simple and good-looking appearance of the air conditioning products, good cooling effect, and energy-saving features, which are consistent with the semantics “energy-saving”, “energy efficiency”, and “brand” in Topic 9. Therefore, the probability of the comments belonging to Topic 9 is higher. We conducted document probability distribution statistics, and the number of comments belonging to each document topic is shown in Figure 4. Topic 9 has the most comments (1025), followed by Topic 7 (577), Topic 6 (277), Topic 3 (240), Topic 1 (220), Topic 8 (143), Topic 5 (129), Topic 2 (125), Topic 10 (100), and Topic 4 (63).

4.6.3. Topic Visualization

Using the WES-BTM, we obtained the clustering results of 10 topics. Due to the high similarity between some topics, the clustering topics were further manually divided. The results were divided into four categories: logistics services, after-sales services, product quality and efficiency, and management and maintenance services, as shown in Figure 5.

The visualization of topic words generated using the WES-BTM is shown in Figure 6. Topic 1 is about logistics services. From the word cloud, we can see that the topic words for logistics services include “delivery”, “logistics”, “contact”, and “arrival”. Logistics is part of product services, and in order to accomplish logistics distribution, most companies cooperate with logistics companies. However, negative reviews from consumers about logistics services can also affect the products and the manufacturing companies themselves. Therefore, companies can improve consumer satisfaction with manufacturing industry products and services by integrating with the logistics industry.

Topic 2 is about after-sales services, and its topic words include “contact”, “phone”, “handle”, “return”, and “charge”. The primary goal of companies in promoting the service-oriented process is to increase customer satisfaction, thus enhancing the company’s competitiveness and increasing its income. Contact refers to the interaction between the customer and the service team or support personnel. The customer can seek help by telephone, email, or online chat. Customer satisfaction is partly determined by the service of the service personnel. Improving the service awareness of service personnel and increasing service training is crucial to enhancing customer satisfaction with after-sales service.

Topic 3 is about product quality and efficiency. The topic keywords for product quality and efficiency include “brand”, “quality”, “mode”, and “use”. These keywords reflect customers’ level of attention to the product itself. Customers’ demands for products mainly include product quality and product efficacy. Manufacturing companies rely on the sale of high-quality and high-efficiency products. Customers attach great importance to the product itself, and for enterprises, focusing on product quality, improving product cost-effectiveness, and meeting customers’ purchasing needs are the basis for enhancing their competitiveness. Products are the basis for customer consumption and the basis for the sustainable development of manufacturing enterprises. Because customers attach great importance to product quality at present, enterprises should pay attention to the products themselves while following the trend and reforming service-oriented development.

Topic 4 is about maintenance services. The topic keywords for maintenance services include “personnel”, “solve”, “machine”, “contact”, and “brand”. After a manufacturing product is put into use, consumers often have a need for maintenance. The customer needs to communicate and contact the maintenance technician or maintenance department. De-signing a consumer-preferred repair strategy is beneficial for manufacturers to consolidate their market position and enhance their market competitiveness. Product repair services determine to some extent the user’s impression of the company and whether they will continue to choose to buy the brand’s products.

5. Conclusions

A WES-BTM is proposed to address the shortcomings of short text with short length and sparse semantics, as well as existing models. This model obtains word vectors by loading a Chinese Word2vec pre-training model. Next, the cosine similarity between word vectors is calculated and pairs of words with low similarity are filtered out. Finally, Gibbs sampling is performed on the selected word pairs to complete the training of the model. We use product reviews from JD and Tmall as data sources for the topic model. The results indicate that the WES-BTM performs well in multiple evaluation indicators such as topic coherence, perplexity, and Jensen–Shannon divergence. Finally, the results were further organized and visualized using comment data as an example.

Future research can further improve the WES-BTM to cope with more complex and diverse user comment data. In addition, this model can be applied to other fields, such as social media analysis, market research, etc., to explore more valuable user needs and attitude information, and provide a more accurate basis for enterprise decision-making.