Review Reports - Enriching Language Models with Graph-Based Context Information to Better Understand Textual Data

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper proposes a graph-based approach to linking texts, with a particular focus on studying in depth a mechanism for integrating contextual information with the structured information provided by graphs. The study empirically demonstrates that incorporating graph-based contextualization into the BERT model improves its performance on a classification task example. Specifically, on the Pubmed dataset, a reduction in error of about 9% to 8% was observed while simultaneously increasing the number of parameters by only 1.6%.

Strengths:

-The authors provide a public repository of their contribution, which is highly appreciated.

-The explanation of vector construction is very detailed and easy to understand, and the reading is smooth.

-The experiments justify the authors' claims, and therefore, the contribution is scientifically sound.

Weaknesses, Despite the solid contribution, there are several points to improve:

-Results should be emphasized; Table 1 and Figure 5 attempt to communicate results but must be more effective.

-The authors should improve the way they describe their findings, perhaps by improving the quality of the plot and making the table more appealing. The authors highlight limitations in Section 4. However, they should be addressed better in a more descriptive subsection.

In conclusion, I believe the contribution is valid; however, I recommend that the authors include contributions in their background that address the actual ability to represent things epistemologically in transformer-based models, such as "Knowing knowledge: Epistemological study of knowledge in transformers." Otherwise, I believe the contribution is valid.

Author Response

We are truly grateful for your invaluable feedback and appreciative words. We highlight the weaknesses below and answer how we rewrote our paper to remedy them.

Ad 1) We emphasised the results by changing Table 1 and Figure 5; now, they are more precise and related.

Ad 2) We added a new section on limitations and rewrote the conclusions.

Ad 3) We relate our approach to the proposed paper - our description is in the Related Work Section. Thank you for showing more research efforts related to our approach and another interesting perspective on enriching language models.

In our paper's original main text file, we make blue fonts for the most important additions to the text. We did not emphasise most minor changes.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper "Enriching language models with graph-based context information to better understand textual data" by Roethel, Ganzha, and Wróblewska proposes an innovative approach to enhancing the capabilities of language models by incorporating graph-based context. This study is particularly timely and relevant given the growing complexity of data and the need for more sophisticated tools in natural language processing (NLP).

The authors convincingly argue that many texts (like Wikipedia articles, tweets, and scientific papers) are interconnected, which can be represented through graph-like structures. They suggest that integrating this contextual information into language models could improve their ability to understand and process text more effectively. The core of their experimental approach involves the integration of a Graph Neural Network (GNN) with the BERT model, which they refer to as GCBERT.

One of the most significant contributions of this paper is the demonstration that incorporating graph-based contextualization into the BERT model not only improves its performance on a classification task but does so with a minimal increase in computational complexity. Specifically, the error reduction achieved on the Pubmed dataset (from 8.51% to 7.96%) with only a 1.6% increase in model parameters is both impressive and promising.

The methodology section is well-structured, detailing various experimental setups and modifications such as late fusion, early fusion (GCBERT), and looped GCBERT. Each of these methods offers different advantages and potential use cases, thus broadening the impact of the research. Furthermore, the experimental results are robust, supported by clear statistical evidence and a thorough analysis of the outcomes.

However, the study is not without limitations. The primary dataset used (Pubmed) is somewhat niche, focusing on diabetes-related topics. This may limit the generalizability of the findings to other fields or types of textual data. Also, the complexity introduced by the graph-based models could potentially lead to issues of scalability and efficiency when applied to very large datasets or real-time applications.

In conclusion, this paper makes a significant contribution to the field of natural language processing by demonstrating the utility of graph-based contextual information in enhancing the performance of language models. The innovative approach of GCBERT could lead to more accurate and efficient models, which is crucial for handling the increasing amount of textual data in various domains. Further research could explore the application of these techniques across more diverse datasets and in real-world scenarios to fully harness the potential of graph-enhanced language models.

The paper by Roethel, Ganzha, and Wróblewska offers significant contributions to integrating graph-based context into language models, particularly with the innovative GCBERT model. However, to further enhance the study's impact and address some of the limitations, the following recommendations could be considered:

1. **Expanding Dataset Variety:**

- **Diverse Domains:** Extend experiments beyond the Pubmed dataset to include texts from various fields such as finance, law, and literature. This would help in understanding the applicability and robustness of the proposed models across different contexts.

- **Larger and More Varied Graphs:** Test the model on datasets with larger and more complex graph structures, such as social media networks or large-scale bibliographic databases, to evaluate the scalability and effectiveness in more dynamic environments.

2. **Enhanced Model Evaluation:**

- **Real-Time Processing:** Assess the performance of the graph-enhanced models in real-time processing scenarios to understand their practical applicability in live systems.

- **Comparative Analysis:** Conduct detailed comparisons with more recent and advanced graph neural network models or other state-of-the-art NLP models to benchmark GCBERT's performance comprehensively.

3. **Model Optimization and Efficiency:**

- **Optimization Techniques:** Explore advanced model optimization techniques to reduce computational overhead without compromising performance. Techniques such as pruning, quantization, and knowledge distillation could be beneficial.

- **Hybrid Architectures:** Investigate hybrid approaches that combine the strengths of different types of neural networks, such as integrating convolutional neural networks (CNNs) for feature extraction with GNNs for context processing.

4. **In-depth Analysis of Graph Utilization:**

- **Feature Engineering:** Experiment with different types of features extracted from the graph, such as node centrality or community structures, which might provide additional predictive power.

- **Graph Dynamics:** Study the impact of evolving graph structures over time on model performance, especially relevant in social media or citation networks where relationships can change dynamically.

5. **Further Theoretical Exploration:**

- **Theoretical Insights:** Develop more theoretical insights into why and how graph context improves language model performance. Understanding the underlying mechanisms can lead to more targeted improvements and new model architectures.

- **Error Analysis:** Conduct a thorough error analysis to identify specific instances where GCBERT underperforms or fails, leading to targeted improvements in model design or training procedures.

6. **Broader Implications and Applications:**

- **Cross-Lingual and Multimodal Applications:** Test the effectiveness of GCBERT in cross-lingual settings or integrate multimodal data (e.g., text with images or videos) to enhance understanding and performance in complex informational environments.

- **Ethical Considerations:** Address potential biases introduced by graph structures, especially in sensitive applications like recommendation systems or content moderation.

By pursuing these recommendations, the research could significantly advance the integration of graph-based context into language models, improving both their theoretical foundations and practical applications.

Comments on the Quality of English Language

Refinement of Sentence Structure: Some sentences may be slightly complex or long, which could be simplified for better readability without losing meaning.

Consistency in Terminology: Ensuring that terminology is used consistently throughout the paper can help avoid confusion, especially in a field that involves complex concepts like graph neural networks and natural language processing.

Transitional Phrases: Improving transitions between sections and paragraphs to enhance the logical flow of arguments and maintain reader engagement throughout the paper.

Author Response

We are grateful for your feedback and appreciative words.

We explained and revised the important issues related to the selection of data in the Dataset Selection Section. Moreover, we added a few more words on this topic in the limitations section. We also underlined that preparing reliable datasets combining graph and text information is a very important research direction for further research.

We also analysed your other recommendations and incorporated them into concluding remarks and limitation sections at the end of the paper.

Additionally, we revised our text, especially concentrating on sentence structures, consistency of the terminology, and transition phrases, as suggested.

In our paper's original main text file, we make blue fonts for the most important additions to the text. We did not emphasise most minor changes.