TR-GPT-CF: A Topic Refinement Method Using GPT and Coherence Filtering
Round 1
Reviewer 1 Report
Comments and Suggestions for Authors How do you determine the threshold for z-score misalignment detection, and how does it impact the overall results?
What are the specific advantages of combining WordNet with GPT for word replacement, compared to using just one of them?
Can you provide examples of how the hybrid approach improves topic coherence in comparison to traditional methods?
How does your method handle topics with highly domain-specific vocabulary or jargon?
Are there any limitations or trade-offs in terms of computational cost when applying this approach to larger datasets?
How do you measure the interpretability of the topics produced by your method, and what metrics are used in the evaluation?
Connect your work to security in one paragraph:
"Systematic Poisoning Attacks on and Defenses for Machine Learning in Healthcare", "Energy-Efficient Long-term Continuous Personal Health Monitoring", "Artificial intelligence security: Threats and countermeasures", "AI-empowered IoT security for smart cities"
Comments on the Quality of English Language
NA
Author Response
Thank you very much for taking the time to review this manuscript. I really appreciate your comments. Below, I have addressed each of your questions and hope my responses are satisfactory.
Comments 1: How do you determine the threshold for Z-score misalignment detection, and how does it impact the overall results?
Response 1: Thank you for your insightful question. To determine the threshold for detecting misaligned words, we initially explore Z-score thresholds ranging from ±1.5 to ±3.0 and evaluate their impact on the topic coherence metric (C_V). Selecting an appropriate Z-score threshold remains a challenge, as it significantly influences the identification of misaligned words. An improperly chosen threshold may result in correctly associated topic words being erroneously classified as misaligned word. After multiple attempts, we decided to use a Z-score threshold of -1.5 and applied it to all scenarios across all datasets and models.
We recognize that the selection of the Z-score threshold is a limitation of our study. To clarify this, I have added one sentence in section 2.1 line 305 – 307. I also address this limitation in Section 3.4
Comments 2: What are the specific advantages of combining WordNet with GPT for word replacement, compared to using just one of them?
Response 2: Thank you for your insightful question. By combining WordNet and GPT, we leverage their complementary strengths to overcome the limitations of using each approach independently. Although this combination improves coherence, as discussed in Section 3.2, not all datasets benefit equally from this approach.
Comments 3: Can you provide examples of how the hybrid approach improves topic coherence in comparison to traditional methods?
Response 3: Thank you for your valuable question. To demonstrate how our hybrid approach (WordNet + GPT-based refinement) improves topic coherence compared to traditional methods, we provide examples of topics generated before and after refinement. Traditional topic modeling methods often produce topics containing semantically weak or misaligned words, reducing interpretability. Our hybrid approach refines these topics by replacing misaligned words, leading to more semantically coherent topics.
In Section 3.1, we evaluate topic coherence across datasets. We conduct experiments using our topic refinement method, which employs a hybrid approach for word replacement. For each dataset presented in the table, we compare topic coherence before and after refinement. The ‘before refinement’ condition involves extracting topics using only traditional methods without refinement, while the ‘after refinement’ condition involves refining the topics extracted by traditional topic modeling methods using our proposed topic refinement mechanism.
Comments 4: How does your method handle topics with highly domain-specific vocabulary or jargon?
Response 4: Thank you for your valuable question. Our approach is designed to efficiently manage specialized terminology by leveraging the lexical structure of WordNet and the contextual understanding of GPT in a complementary manner. In our misaligned word replacement mechanism, explained in Section 2.2, WordNet and GPT generate alternative words based on the centroid word of the topic.
When handling datasets with highly domain-specific vocabulary, candidate words produced by GPT tend to have higher coherence compared to those generated by WordNet. This is because GPT derives words from contextual usage and is trained on diverse text sources, including scientific literature and medical research, among others.
As an example, in Section 3.3, we provide a qualitative comparison across datasets, including the Science Article dataset.
Comments 5: Are there any limitations or trade-offs in terms of computational cost when applying this approach to larger datasets?
Response 5: Thank you for your insightful question. We acknowledge that our refinement method incurs a certain computational cost, particularly when scaling to large text corpora, despite its effectiveness in enhancing topic coherence. The primary trade-offs involve increased processing time due to Z-score computation, GPT-based replacements, and embedding similarity calculations.
I have added an additional Section 3.4, ‘Limitations,’ to the manuscript to discuss the constraints of our approach.
Comments 6: How do you measure the interpretability of the topics produced by your method, and what metrics are used in the evaluation?
Response 6: Thank you for your valuable question. In our study, we employ only the topic coherence metric C_V for evaluation, along with manual inspection to assess the interpretability of the topics. This is one of the limitations of our study, and I have added an additional Section 3.4, ‘Limitations,’ to the manuscript.
In future work, we will incorporate additional metrics to enhance the evaluation.
Comments 7: Connect your work to security in one paragraph:
"Systematic Poisoning Attacks on and Defenses for Machine Learning in Healthcare", "Energy-Efficient Long-term Continuous Personal Health Monitoring", "Artificial intelligence security: Threats and countermeasures", "AI-empowered IoT security for smart cities"
Response 7: I try to connect our work to "Energy-Efficient Long-term Continuous Personal Health Monitoring”:
In such applications, accurate and interpretable topic extraction is essential for processing medical records, patient data, and real-time health monitoring reports. Traditional topic models often generate noisy or incoherent topics, which can lead to misinterpretations in clinical decision-making. By enhancing topic coherence and removing misaligned words, our method ensures that AI-driven health monitoring system extract more meaningful insights from medical text data. This improvement is particularly beneficial in resource-constrained environments, where computational efficiency is critical for long-term health monitoring. The ability to filter and refine topic-related information contributes to more reliable AI models, supporting healthcare professionals in early disease detection, risk assessment, and personalized health recommendations.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsThis manuscript represents a report of an innovative and exciting development of a post-extracted topic refinement approach using z-score centroid-based misaligned word detection with hybrid semantic-contextual word replacement regarding WordNet and GPT to replace words misaligned within topics.
The authors have explained their method well with helpful and well-designed tables and figures. This topic is appropriate for the Special Issue to which it is submitted. The weakness of this study is that there are too few references, and some of those provided are outdated. The gold standard for scientific research is that citations reference work published in the previous five years. This attention to current citations is particularly relevant for such a new and constantly evolving topic. Below are the suggested edits—primarily to improve the citations.
Line by line suggested edits
19-20 Please cite research published since 2021 to support this claim.
20-22 Please cite research published since 2021 to support this claim.
22-24 Please cite research published since 2021 to support this claim.
36-38 Please cite research published since 2021 to support this claim.
38-39 Please cite research published since 2021 to support this claim.
42-43 Please cite research published since 2021 to support this claim.
43-45 Please cite research published since 2021 to support this claim.
47 Please find a supporting citation for [8] of research published since 2021.
49 Please find a supporting citation for [9] of research published since 2021.
51 Please find a supporting citation for [10] of research published since 2021.
58 Please find a supporting citation for [11] of research published since 2021.
59-60 Please cite research published since 2021 to support this claim.
60-62 Please cite research published since 2021 to support this claim.
63 Please find a supporting citation for [12] of research published since 2021.
71-73 Please cite research published since 2021 to support this claim.
74-76 Please cite research published since 2021 to support this claim.
76-78 Please cite research published since 2021 to support this claim.
79-82 Please cite research published since 2021 to support this claim.
82-84 Please cite research published since 2021 to support this claim.
86 Please find a supporting citation for [16] of research published since 2021.
87-89 Please cite these studies.
89 Please cite the majority.
93 Please start a new paragraph after “studies,” ending with a period rather than a comma.
116 Please start a new paragraph after “space.”
163-175 Please state if this manuscript is the first publication of this innovative topic refinement framework. If there are earlier publications regarding this model, please cite them.
194-195 Please cite research published since 2021 to support this claim.
197 Please explain in the text why [25] is an inspiration and cite other studies published since 2021 using [25] as an inspiration.
214 Please find supporting citations for [9] and [26] of research published since 2021.
224 Please find a supporting citation for [29] of research published since 2021.
256 Please cite a reference to the Hugging Face transformer library.
283 Please cite a reference published since 2021 that uses [34] similarly to this study.
320-321 Please cite research published since 2021 defining IDF.
331-332 Please cite research published since 2021 to support this claim.
342 Please find a supporting citation for [29] of research published since 2021.
345 Change “general term” to “general”.
361 Please cite research published since 2021 to define Google Colab’s T4 GPU and explain, in the text, the reason for its selection.
362 Please cite research published since 2021 to define OpenAI’s API version 1.55.3 and explain, in the text, the reason for its selection.
372 Please cite a reference published since 2021 for Gensim’s CoherenceModel.
377 Please cite a reference published since 2021 for Pointwise Mutual Information.
608 Please provide a limitations section regarding the innovation.
Author Response
Thank you very much for taking the time to review this manuscript. I appreciate your comments and advice. Below, I have addressed each of your comments in a table format to make them more concise and easier to read. I hope my responses are satisfactory.
Please find my detailed responses in the attached file, where I have provided a table format for clarity. Kindly refer to the response letter for a structured explanation addressing each comment.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Comments and Suggestions for AuthorsThank you to the authors for the changes made. All have improved the manuscript in the manner expected. A few remain.
Citation [45] is a 2003 publication—it’s too outdated. Here is a Google Scholar search of the topic for research published since 2021: https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&as_ylo=2021&q=%22Non-negative+Matrix+Factorization%22&btnG=. Note that there are “About 19,100 results”. Please find a current reference to NMF among these to cite to support [45].
Citation [54] is missing. That it is missing is fine since it is outdated. However, delete it from the reference list and the citations. Then, reorder the references after [53].
Citation [56] is outdated. Here is a Google Scholar search of the topic of research published since 2021: https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&as_ylo=2021&q=%22BERT-base-uncased+model%22+%22BERT+embeddings%22&btnG=. There are “About 369 results”. Please find a current reference to the BERT-base-uncased model of BERT embeddings to support citation [56].
The authors stated in their cover letter that the citation [57] has current research support. This citation isn’t evident. Here is a Google Scholar search of the topic for research published since 2021: https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&as_ylo=2021&q=%22Hugging+Face+transformer+library%22+%22bert-base-uncased+version%22+%22BERT+model%22&btnG=. There are two returns. Please find a supporting citation among them.
Citation [63] requires support to demonstrate its current relevance. Here is a Google Scholar search of the topic for research published since 2021: https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=select+the+candidate+with+the+highest+coherence+score+to+replace+the+misaligned+word%2C+using+the+topic+coherence+metric+C+_v%2C&btnG=. Note that there are “About 15,500 results”. Please use the most relevant of these to support [63].
Author Response
Thank you very much for taking the time to review the revised manuscript. I really appreciate your comments. Below, I have addressed each of your questions and hope my responses are satisfactory. Kindly refer to the attached file for our point by point responses.
Author Response File: Author Response.pdf
