Marketing Insights from Reviews Using Topic Modeling with BERTopic and Deep Clustering Network

Abstract

The feedback shared by consumers on e-commerce platforms holds immense value in marketing, as it offers insights into their opinions and preferences, which are readily accessible. However, analyzing a large volume of reviews manually is impractical. Therefore, automating the extraction of essential insights from these data can provide more comprehensive and efficient information. This research focuses on leveraging clustering algorithms to automate the extraction of consumer intentions, related products, and the pros and cons of products from review data. To achieve this, a review dataset was created by performing web crawling on the Naver Shopping platform. The findings are expected to contribute to a more precise understanding of consumer sentiments, enabling marketers to make informed decisions across a wide range of products and services.

1. Introduction

In the digital age, consumers have increasingly been leaving and sharing reviews about various products and services through online platforms. These review data have become a valuable asset for companies as the data can be used to understand consumer opinions and preferences, thereby improving marketing strategies. However, effectively analyzing a large volume of review data and extracting useful information from it is a challenging task.

This research proposes a method of extracting and analyzing marketing-related information based on review data using clustering-based topic modeling as the core technique. A clustering algorithm is a computational method used in data analysis and machine learning to group a set of data points into clusters based on their similarities. Topic modeling is a natural language processing technique used to extract hidden topics from text data. Text data can contain various topics, and each document may be related to multiple topics. Topic modeling automatically identifies these topics and estimates which topics each document is composed of.

Topic modeling and clustering algorithms are a methodology widely used in marketing, such as for customer segmentation and recommendation systems, as they group data with similar characteristics. In a study [1], product description data were collected from an e-commerce platform that sells mobile phones, and topic modeling was conducted using LDA, a traditional topic modeling method. The study then extracted the concept of products and the relationships between each feature using keywords extracted from each topic, constructing a knowledge graph to find semantic knowledge about the products. LDA has also been utilized for topic modeling of consumer-related topics in various domains, such as food distribution, food tourism, apartment interior design, and airline services in Korea, to analyze consumer opinions [2,3,4,5].

In this study, we introduce a process that utilizes clustering algorithms based on review data that are accessible from external sources to analyze consumer intent and extract the relationship between products. By clustering review data left by consumers, groups with similar consumer opinions can be formed, enabling the identification of consumer purchase intent and the relevance between products. This provides important information for companies to understand the association between specific products and other products, allowing them to develop sales strategies. Additionally, we present a process to extract the advantages and disadvantages of products from review data. Using topic modeling, we classify review data into respective topics and extract the advantages and disadvantages of products from each topic. This allows companies to identify which products consumers evaluate most positively and areas that need improvement. It can serve as valuable reference material for product development and marketing strategy formulation.

There are a variety of topic modeling algorithms, but this study focuses on BERTopic [6]. BERTopic demonstrates a better performance as it leverages BERT [7] for vector embedding and considers the semantics and contextual information of the text. BERTopic is a topic modeling approach that utilizes BERT’s embedding ability and demonstrates that the task can be addressed simply by employing a clustering-based approach. In this study, other clustering algorithms with good performance, such as the DCN, have been used as alternatives to generate results and compare their performances. By incorporating different clustering algorithms into the basic structure of BERTopic, this research aims to explore and compare the results obtained. This will provide insights into the effectiveness and performance of various clustering algorithms in the context of topic modeling.

2. Background Algorithms

2.1. K-Means Algorithms

One of the most representative clustering algorithms is the K-means algorithm [8]. The K-means algorithm selects k cluster centroids and iteratively moves them to optimize the following cost function during training:

$$\begin{array}{c}argmin\\ S\end{array} \sum _{i=1}^k\sum _{x\in S_i}{\left|\left|x-{\mathsf{\mu }}_i\right|\right|}^2$$

where k is the number of clusters, S is the set of clusters, $S_1,S_2,\dots ,S_k$, and ${\mathsf{\mu }}_i$ is the centroid of $S_i$.

2.2. HDBSCAN

There is another algorithm called hierarchical clustering. It is a bottom-up approach that creates a hierarchy of clusters by iteratively merging or splitting data points based on their similarities. The basic idea behind hierarchical clustering is to start with each data point as its own cluster and then iteratively merge or split clusters until a stopping criterion is met. The result is a hierarchical structure that is often represented as a dendrogram, which shows the relationships between clusters at different levels of similarity. In this study, we will use the HDBSCAN algorithm [9] among various hierarchical clustering algorithms. This algorithm is a modification of the DBSCAN algorithm [10] that is a part of the hierarchical algorithm paradigm. It calculates the mutual reachability between each pair of data points and constructs a minimum spanning tree based on these points to transform it into a hierarchical structure for clustering. This approach has the advantage of being able to find clusters of various shapes better than the K-means algorithm. One important factor to consider when performing clustering is the dimensionality of the data. If the dimensionality of the data is too high, it becomes difficult to achieve good clustering results due to the curse of dimensionality. Therefore, in cases where the dimensionality of the data is excessively high, it is common to incorporate dimensionality reduction algorithms such as UMAP into the clustering process.

2.3. Deep Clustering Network (DCN)

One of the other methods for reducing the dimensionality of the data is to utilize autoencoders to create latent vectors of the data. The algorithm initially proposed for this purpose is called Deep Embedded Clustering (DEC) [11]. This algorithm utilizes stacked autoencoders to extract lower-dimensional latent vectors for the input data, which are then used for clustering, demonstrating good performance. The Deep Clustering Network (DCN) improves upon DEC by modifying the loss function to achieve slightly better performance. This method also relies on autoencoders for dimensionality reduction and utilizes them for clustering [12]. The novelty of this research lies in the definition of a new loss function by combining the loss functions of autoencoders and the K-means algorithm. This new loss function is used during the dimensionality reduction process to generate vectors that are more suitable for the K-means algorithm, resulting in enhanced performance for clustering. Below is the expression for the loss function used in this research.

$$min\sum _{i=1}^N\left(\mathcal{l}\left(g\left(f\left(x_i\right)\right),x_i\right)+\frac{\mathsf{\lambda }}{2}{\left|\left|f\left(x_i\right)-M{s}_i\right|\right|}_2^2\right)$$

The term on the left within the sigma represents the loss function of the autoencoder, which calculates the error between the input data and the reconstructed output data. In this research, the Mean Square Error (MSE) was used, but other functions, such as L1-norm or KL-divergence, can also be used. On the right is the form of the loss function for the K-means algorithm. By attaching a coefficient lambda to it, the degree of adjustment and influence can be controlled.

2.4. Latent Dirichlet Allocation (LDA)

The most widely used technique among various topic modeling methods is Latent Dirichlet Allocation (LDA) [13]. LDA assumes that documents are composed of multiple topics, each document is a mixture of topics, and topics are represented by word distributions. It iteratively infers the topic distribution within documents and the word distribution within topics. While LDA performs well and is widely used, it has limitations in capturing semantic relationships between words.

2.5. BERTopic

One topic modeling method that overcomes these limitations is BERTopic, which, as the name suggests, utilizes BERT [7]. It first embeds the text data into vectors using Sentence BERT [14] and then performs clustering on these vectors to conduct topic modeling. Clustering is carried out by reducing the dimensionality of the vectors using UMAP, as mentioned before, and by applying HDBSCAN. BERTopic demonstrates a better performance as it leverages BERT for vector embedding and considers the semantics and contextual information of the text. BERTopic is a topic modeling approach that utilizes BERT’s embedding ability and demonstrates that the task can be addressed simply by employing a clustering-based approach. In this study, in addition to HDSCAN, which is the default clustering algorithm in BERTopic, other clustering algorithms with a good performance, such as the DCN, have been used as alternatives to generate results and compare their performances. By incorporating different clustering algorithms into the basic structure of BERTopic, this research aims to explore and compare the results obtained. This will provide insights into the effectiveness and performance of various clustering algorithms in the context of topic modeling.

3. Literature Review

Efforts to uncover latent insights through the utilization of topic modeling have been present across various fields, including marketing. In a previous study [15], the factors that impacted the user experience on mobile health apps were analyzed using BERTopic with registered user review data, providing insights into the direction of service development. Additionally, basic topic modeling methods have been expanded through the incorporation of new models or fusion with other machine learning models. Study [16] introduced a supervised topic modeling approach, Hierarchical Dirichlet Process-based Inverse Regression (HDP-IR), which is based on LDA and Inverse Regression, to extend to the approach’s use in predictive models for variables such as customer sentiment, product quality, and affect. In study [17], BERTopic was used to extract topic probability distributions from topic modeling results from a Twitter dataset. These distributions were then combined with vector representations of the original data for use in a classification model, which demonstrated a better performance than a basic classification model. Study [18] proposed a novel robust risk maximum expert consensus model by combining the mean-variance theory with the traditional maximum expert consensus model. This model enables the derivation of consensus decisions for the entire group from individual opinions, and it can also be employed to analyze consumer opinions embedded within review data.

Furthermore, there have been variations in the configuration of BERTopic. Study [19] compared the performance of BERTopic’s clustering module by switching from HDBSCAN to the K-means algorithm. The results measured using the representative evaluation metrics of topic modeling, such as NPMI and Topic Diversity, showed that the model incorporating the K-means algorithm did not surpass the performance of the baseline model in several benchmarks and datasets.

4. Methodology

In this section, based on the consumer review data collected from an actual e-commerce platform, we will explain the method of (1) the process of extracting the advantages and disadvantages of each product and (2) the process of finding the relationships between the products. These processes will utilize clustering algorithms and topic modeling techniques such as BERTopic and the DCN, and we will also compare their performance with traditional methods such as LDA and K-means algorithms. The data crawling, implementation, and experimentation of all models were carried out using Python. The basic BERTopic was utilized using the provided library, while the DCN-based model was adapted from the official implementation code by the DCN researchers to suit the data of this study. The K-means algorithm employed the code from the sklearn library [20], and LDA as well as the evaluation metric calculations, were performed using OCTIS [21].

4.1. Data Collection and Preprocessing

We chose ‘Naver Shopping’, one of the largest e-commerce platforms in Korea, as the platform to collect consumer reviews. We selected 17 product keywords based on sales scale and crawled the review content and ratings for each product, resulting in a total of approximately 500,000 reviews. The collected reviews were saved and managed in a csv file format. Figure 1a illustrates the distribution of star ratings for the crawled reviews. It is evident that the majority of reviews are assigned 5 stars, and as the star ratings decrease, the number of reviews also decreases. Due to significant variations in the number of reviews between star ratings, a logarithmic scale was employed on the y-axis for visualization purposes.

The collected data were preprocessed using the Korean corpus-trained morphological analyzer Mecab-ko, which decomposed the data into morphemes. Based on our intuition and observations, we determined that important information about the products is concentrated in nouns, verbs, and adjectives in the review text. Therefore, we kept only these three parts of speech as morphemes and removed all others. We also removed additional Korean stopwords. Figure 1b presents the frequencies of the top 10 most frequently occurring parts of speech (POS) after morphological analysis of all reviews. Generally, nouns, verbs, adjectives, and adverbs are observed to occur frequently. This observation serves as another basis for extracting and utilizing only these four POS categories in the preprocessing of the study.

4.2. Extracting Advantages and Disadvantages

To extract the advantages and disadvantages, we divided the review data based on the star ratings. We used only the reviews that received a 5-star rating for extracting advantages and reviews that received a 1-star or 2-star rating for extracting disadvantages. We then proceeded with topic modeling using these segmented review data.

Next, we embedded the review data into vectors. In this process, we utilized KcBERT [22], which is a BERT model trained using comments collected from Korean online news. KcBERT performs well in natural language processing tasks for the informal text that often contains colloquial language, slang, and typographical errors. We employed mean pooling with KcBERT to embed the review data into vectors.

After embedding the data, we conducted topic modeling. We hypothesized that the keywords extracted from the topics would effectively represent the characteristics (advantages and disadvantages) of the products. We then explored the BERTopic method and a new approach using the DCN module as a replacement for BERTopic’s clustering module. The overall process is depicted in Figure 2.

4.3. Analysis of Related Products

The analysis of related products follows a process similar to the one described for extracting advantages and disadvantages. The main difference is that while extracting advantages and disadvantages, we performed clustering-based topic modeling for individual product reviews. In the analysis of related products, we conducted topic modeling using the reviews of multiple products. We then examined which product reviews appear together within each topic. If two products are frequently co-located within the same topic, this suggests that they share certain characteristics and indicates a relationship between the two products. Therefore, when extracting product features, we focused on the frequency of co-occurrence of the product by using TF alone instead of using c-TF-IDF to extract products from clustered topics. The overall process is depicted in Figure 3.

4.4. Hyperparameter Optimization

Both clustering algorithms and topic modeling are highly influenced by an important initial hyperparameter: the number of clusters or topics. One widely known method, especially used in K-means-based clustering algorithms, is the elbow method [23]. This method involves calculating the Within Clusters Sum of Squares (WCSS) metric for different numbers of clusters and selecting the point on the WCSS cluster number graph where the curve resembles an elbow shape as the optimal number of clusters. Another widely used method is the silhouette score, which measures how similar an object is to its own cluster compared to other clusters [24].

However, in this study, as clustering is used as part of the topic modeling process, we aimed to optimize the hyperparameters based on the results of the topic modeling. There are various metrics available to evaluate the results of topic modeling [25]. In this study, we utilized the coherence of topics metric based on word embeddings [26]. The formula for this metric is as follows:

$$W_{CE}=\frac{1}{k}\sum _{t=1}^k\sum _{i=1}^{N-1}\sum _{j=i+1}^{N}cosine\_similarity\left(w_{ti},w_{tj}\right)$$

where k is the number of topics, N is the number of keywords extracted from topic clusters, and $w_{ti},w_{tj}$ is each keyword’s embedding vector within topic t. Our goal is to extract product characteristics from the keywords extracted from each topic. Thus, the more consistent the extracted keywords are within each topic, the more they align with our objective. Therefore, the hyperparameter optimization process was conducted by changing the number of topics (clusters) and selecting the number that resulted in the highest value of the word embedding metric. We used the Korean pretrained FastText model [27]. In the authors’ original paper, GloVe was used, but due to the nature of the Korean language, errors in the morphological analysis are not negligible, and Out of Vocabulary (OOV) issues can occur frequently. FastText, on the other hand, considers the subwords of words and generates embedding vectors, making it capable of addressing OOV problems, unlike other word embedding methods. Therefore, considering the characteristics of our current task and the potential occurrence of OOV problems, we chose FastText as it is more suitable.

5. Experiments

In this section, we will examine the process of extracting product advantages, disadvantages, and related products using real data. Then, we will compare the performance of BERTopic and the DCN, which were used in this study, with traditional methods such as LDA and K-means algorithms using various metrics. By analyzing the extracted information and evaluating the performance of different algorithms, we can gain insights into the strengths and weaknesses of each method in terms of accurately identifying product characteristics and discovering related products.

All models were trained using the optimized number of topics determined through the hyperparameter optimization process mentioned earlier. For the DCN, the training process involved two stages: pretraining using stacked autoencoders and actual DCN training. Both stages were trained for 50 epochs. However, we encountered a problem with the trivial solution due to a setting in the original research when using the MNIST benchmark dataset, where all data were assigned to a single label. To address this issue, they reduced the size of the hidden layers in the autoencoder. Instead of using (500-500-2000) parameters, they used (192-192-768) hidden layer sizes because the original value appeared excessively large in comparison to the embedding vector dimension of 768 employed in this experiment. This discrepancy raised concerns akin to the curse of dimensionality, suggesting that the characteristics of each vector were being overlooked, resulting in potential issues. The dimension of the generated latent vectors was set to 10.

In the task of extracting the pros and cons of products, we conducted experiments using traditional topic modeling methods such as LDA, the default configuration of BERTopic, and modified models of BERTopic with its clustering module replaced by the K-means algorithm and DCN. Additionally, for the task of extracting related products, since the clustering process is essential, we utilized K-means, the DCN, and the modified BERTopic model but excluded LDA.

6. Results

6.1. Extracting Features

The performance of the models was evaluated using three metrics. Topic coherence is an evaluation metric derived from the assumption that better-performing topic models will result in extracted keywords for each topic that have semantically similar meanings. In this experiment, we used NPMI (Normalized Pointwise Mutual Information) [28] and Word Embedding-based Topic Coherence. NPMI quantifies how often two words appear together in a text. It ranges from −1 to 1, with higher values indicating better coherence. Word Embedding Coherence (WE), previously mentioned in Section 4.4, is an evaluation metric that involves embedding the extracted keywords as vectors and calculating the similarity between words. A higher value indicates a higher semantic similarity among keywords within the topic. Topic Diversity (TD) is an evaluation metric that assesses how distinct the extracted topics are from each other in terms of meaning [29]. It measures how different semantically meaningful keywords have been extracted among different topics. It is calculated as the proportion of unique words across all topics. It ranges from 0 to 1, with values closer to 1 indicating a higher diversity. The results of the topic modeling for a randomly selected product, which is detergent, are presented in Figure 4.

The left side of each graph represents the result evaluation metric for extracting advantages, while the right side represents the evaluation metric for extracting disadvantages.

The extraction of disadvantages received lower scores compared to the extraction of advantages in terms of performance. This is likely due to the difference in dataset sizes. Generally, the number of reviews with low ratings is significantly lower than those with high ratings in consumer-generated reviews. Therefore, the amount of data used for extracting disadvantages was much smaller in the collected dataset, which is reflected in the results.

When comparing the performance among the models, the more recent algorithms using the DCN and BERTopic generally outperformed traditional algorithms. Between the DCN and BERTopic, BERTopic showed slightly higher scores. Particularly, BERTopic exhibited the most outstanding performance in terms of Topic Diversity among the four models. However, for the word embedding metric, which was emphasized in this study, the DCN also demonstrated a good level of performance comparable to BERTopic. An example of extracted features of a product is shown in

Table 1. Results of extracting features of a product.

Pros		Cons
Keyword	Models	Keyword	Models
mold	All	odor	All
removal	All	bleach	All
cleaning	All	intense	DCN BERTopic
toilet	All	same	K-means BERTopic
spray	All
bathroom	K-means DCN BERTopic

. The keywords in the table below were extracted from two or more of the various models, and the ones that have marketing significance were selected. The original topic modeling results can be found in Appendix A.

The selected product is a mold remover detergent. We can see that keywords related to the product include ‘mold’, ‘cleaning’, ‘odor’, and ‘removal’. From the table, it can be inferred that the product has a good effect on mold removal but has an intense odor, which seems to be its major drawback. If devising a sales strategy for this product or a similar one, efforts could be directed towards mitigating the odor issue or emphasizing its effective mold removal capabilities by crafting advertisements.

Furthermore, in the case of the LDA and K-means methods, which exhibited relatively low topic diversity values, the occurrence of words such as ‘mold’, ‘spray’, ‘use’, and ‘removal’ across multiple topics with overlaps can be observed. On the contrary, for the DCN-based model and BERTopic, which had relatively high topic diversity values, there were fewer overlapping words between different topics and a greater diversity of extracted keywords. This could potentially help reduce overlooked information in consumer reviews and yield more insights. In terms of topic coherence, higher values facilitated the comprehension of what content described the entire topic. For instance, in the case of LDA with the lowest WE value, words such as ‘mold’, ‘delivery’, ‘product’, and ‘price’, which are semantically distant, appeared together within one topic, making it challenging to grasp the overarching theme. Conversely, the DCN-based model and BERTopic exhibited favorable values, enabling easier inference of the predominant content within each topic. As an illustrative example from the advantages extracted by BERTopic, the first topic predicted the dominance of ‘mold removal effectiveness’, the second topic represented ‘bathroom/cleaning’, and the third topic pertained to ‘wall cleaning/price discounts’. Therefore, we believe that topic coherence, when used alongside the size information of categorized topic clusters, could serve as a significant metric when analyzing prevalent opinions and more in-depth insights.

6.2. Extracting Related Products

In the process of extracting related products, we used clustering-based topic modeling algorithms. Therefore, we examined the results obtained using three models: K-means, the DCN, and BERTopic. In this experiment, we unified the number of topics to be 5. The DCN was trained with the same settings as the previous experiment but with a reduced number of epochs (25) to prevent trivial solution issues. The extracted results are presented in

Table 2. Results of extracting related products.

	Topic 1	Topic 2	Topic 3	Topic 4	Topic 5
K-means	Men’s underwear	Clothes dryer	Interior items	Chicken breast	Detergent
	Jeans	Slip-on shoes	Key case	Crocs	Jeans
	Detergent	Treadmill	Crocs	Slip-on shoes	Key case
	Umbrella	Travel suitcase	Men’s underwear	Anti-reflux cushion	Paraffin machine
	Anti-reflux cushion	Anti-reflux cushion	Jeans	Travel suitcase	Anti-reflux cushion
DCN	Men’s underwear Jeans	Chicken breast	Detergent	Jeans	Clothes dryer
	Detergent	Key case	Jeans	Interior items	Slip-on shoes
	Umbrella	Anti-reflux cushion	Key case	Chicken breast	Treadmill
	Anti-reflux cushion	Travel suitcase	Paraffin machine Men’s underwear	Detergent	Interior items
		Paraffin machine		Clothes hanger	Yoga mat
UMAP + HDBSCAN (BERTopic)	Interior items	Detergent	Men’s underwear	Men’s underwear	Raincoat
	Chicken breast	Jeans	Clothes hanger	Paraffin machine Clothes hanger	Men’s underwear
	Men’s underwear	Raincoat	Jeans	Travel suitcase	Jeans
	Crocs	Key case	Travel suitcase	Jeans	Umbrella
	Jeans	Clothes hanger	Paraffin machine		Crocs

. It is anticipated that by training the models with a larger set of products and refined data and interpreting the generated topics together, more valuable information can be obtained.

7. Discussion

In this study, topic modeling was employed to extract product characteristics and related products from consumer reviews for potential utilization in marketing insights. During this process, the clustering module of BERTopic was replaced with a DCN, a recently created neural network-based algorithm, to examine the outcomes of the task. Furthermore, a performance comparison was conducted between the DCN-based model, the baseline BERTopic, and other alternative algorithms. Despite the various clustering algorithms that can be applied to BERTopic’s clustering module, instances of utilizing recently proposed neural network-based clustering algorithms were limited. The endeavor to incorporate the DCN carries significance from the perspective of applying unexplored clustering algorithms to BERTopic and comparing their performances.

The model employing the DCN exhibited a favorable performance; however, in comparison to the baseline BERTopic, it showed similar or slightly inferior results. This disparity might be attributed to the fundamental paradigm differences between the DCN and HDBSCAN. The DCN relies on the K-means algorithm, which utilizes distances between data points, while HDBSCAN employs data density for hierarchical clustering. The baseline BERTopic benefits from excluding reviews identified as noise during the clustering process, which is anticipated to positively impact its performance. Nevertheless, reviews categorized as noise could potentially be covered to some extent during data preprocessing, and there is a possibility of the loss of valuable information if informative reviews are mistakenly categorized as noise. Therefore, further investigation is warranted in this regard.

The extracted product characteristics and related products obtained by utilizing the proposed method in this study can potentially aid in discovering marketing insights and devising strategies. Observing the extracted product characteristics can provide insights into consumer reactions and purchasing intentions, informing strategies for product improvement, advertising emphasis, and more. The insights gained can be utilized as reference materials for formulating marketing strategies. Concerning the extraction of related products, clustering products with similar content mentioned in reviews allows for the identification of products that consumers purchase with similar intentions or features, even if they do not belong to the same category. This information can aid in decisions about which additional products to offer alongside high-selling items or in exposing related products to consumers purchasing a specific item.

8. Conclusions

Demonstrating the potential utility of identifying topic-modeling-derived pros and cons of products and related products, this study underscores the capacity of using these data to facilitate the formulation of marketing strategies for product sales. Moreover, the comparison of performance results by substituting BERTopic’s clustering module with the DCN, a recently created neural network algorithm, exhibits the favorable performance of the DCN-based model and the baseline BERTopic. Furthermore, the exploration of various clustering algorithms suggests the possibility of enhancing the performance of baseline BERTopic across multiple tasks. However, it is noted that models targeting low-rated reviews, which commonly exhibit a limited data volume, exhibited a comparatively lower performance, necessitating further investigations to address this limitation.