Enhanced Feature Selection via Hierarchical Concept Modeling

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Author:

The following comments and suggestions have been made regarding your manuscript.

1. The motivation explains the need for hierarchical models, and it could more clearly justify why hierarchical feature selection (using FCA and DT specifically) is better suited to certain tasks than non-hierarchical or flat feature selection methods. 

2.  The literature review should be more in-depth in the sense of establishing methodologies that are based on hierarchical feature selection. I believe the authors should compare the methods (for example, FCA and DT are mentioned, but these could be compared with other methods, such as fuzzy rough sets or multi-granularity clustering structures) to provide additional context. Also, discussing recent advancements in hybrid methods that combine different feature selection techniques could enrich the state-of-the-art review.

3. Clarify the Comparative Advantage of Hierarchical Methods. The paper could benefit from a more detailed discussion of why the hierarchical concept model outperforms traditional feature selection methods in specific cases.

4. The explanation of FCA in the manuscript is somewhat brief. Expanding on how FCA specifically reduces feature dimensionality while retaining accuracy could enhance the readers' understanding of its impact on classification outcomes, particularly in applications that require robust feature reduction.

5. The authors have not defined the assumptions used in the case study; it is necessary to clarify this point to produce reproducible results.

6. In the discussion of the results, the authors should state under what conditions the results could be improved and the next step to improve the proposed method. This includes a discussion of practical use cases where this hierarchical feature selection model could be beneficial. It also discusses the model's adaptability to other classification algorithms beyond the DT, SVM, and ANN tested.

7. The tables and figures showing accuracy comparisons could be further optimized. For instance, using color-coded heat maps to indicate classification performance across methods or visualizing the importance of features by dataset could make data interpretation more intuitive.

8. Regarding the metrics that the authors implement for optimization, they are advised to include more statistical methods to validate the selection of features. (e.g., permutation testing or cross-validation metrics) could add robustness to the conclusions.

9. The authors should include in the conclusions possible future work involving the potential challenges in implementing hierarchical schemes with large datasets and possible solutions or alternatives that could be explored.

 

Comments on the Quality of English Language

none

Author Response

Thank you very much for reviewing our manuscript.  We also greatly appreciate the reviewers for their complimentary comments and suggestions. We have carried out more descriptions and the experiments that the reviewers suggested and revised the manuscript accordingly.

We are grateful for the reviewer's feedback. Below are our responses addressing each point:

1. The motivation explains the need for hierarchical models, and it could more clearly justify why hierarchical feature selection (using FCA and DT specifically) is better suited to certain tasks than non-hierarchical or flat feature selection methods.

Response. Thank you for your insightful feedback. We appreciate your suggestion to clarify the specific advantages of hierarchical feature selection, particularly with FCA and DT, compared to non-hierarchical or flat methods. In response, we have expanded the motivation section to more explicitly address the benefits of using hierarchical models for feature selection in section 1 (Introduction-lines 56-77). We emphasize that hierarchical approaches, such as FCA and DT, offer unique advantages in managing multi-level relationships within complex datasets. These hierarchical methods capture layered dependencies between features, allowing for a more structured reduction of the feature space.

 

2. The literature review should be more in-depth in the sense of establishing methodologies that are based on hierarchical feature selection. I believe the authors should compare the methods (for example, FCA and DT are mentioned, but these could be compared with other methods, such as fuzzy rough sets or multi-granularity clustering structures) to provide additional context. Also, discussing recent advancements in hybrid methods that combine different feature selection techniques could enrich the state-of-the-art review.

Response. Thank you for your valuable feedback. We appreciate your suggestion to provide a more comprehensive discussion of hierarchical feature selection methodologies, including comparisons with other methods and recent advancements in hybrid approaches. In response, we have expanded the literature review to provide a deeper analysis of methodologies based on hierarchical feature selection in  section 2 (Related Works-lines 146-176).

 

3. Clarify the Comparative Advantage of Hierarchical Methods. The paper could benefit from a more detailed discussion of why the hierarchical concept model outperforms traditional feature selection methods in specific cases.

Response. Thank you for your valuable feedback. We attempt to describe more details in section 3, 5, and 6. We have expanded the section 5 (Results and Discussion-lines 506-520) to provide a more detailed analysis of why the hierarchical concept model, using FCA and DT, offers distinct advantages in specific cases.  Moreover, we briefly mention these comparative advantages in the section 6 (Conclusions-lines 599-606) to reinforce the strengths of hierarchical methods over traditional methods for certain tasks, summarizing key insights from the expanded discussion and in section 3  (The Proposed Models for Feature Selection-lines 208-213) we briefly add to the explanation of why FCA and DT as hierarchical models were chosen over flat feature selection methods.

4. The explanation of FCA in the manuscript is somewhat brief. Expanding on how FCA specifically reduces feature dimensionality while retaining accuracy could enhance the readers' understanding of its impact on classification outcomes, particularly in applications that require robust feature reduction.

Response. Thank you for helpful comments and suggestions. We attempt to elaborate on FCA as a method for feature selection, emphasizing its role in reducing feature dimensionality while retaining classification accuracy in section 3 (The Proposed Models for Feature Selection-lines 194-204). Moreover, we explain its application in generating a minimal feature set while retaining high classification performance in section 5.2.2 (Feature Selection Using FCA-346-357) and in this section 5.4 (lines 532-542), we highlight FCA’s capability to reduce the computational load of classification models and improve interpretability without losing predictive accuracy, which is crucial for complex or large datasets.

 

5. The authors have not defined the assumptions used in the case study; it is necessary to clarify this point to produce reproducible results.

Response. Thank you for helpful comments and suggestions. We have added clarification on the assumptions made in the case study to enhance reproducibility in section 3 (The Proposed Models for Feature Selection – lines 215-226).

 

6. In the discussion of the results, the authors should state under what conditions the results could be improved and the next step to improve the proposed method. This includes a discussion of practical use cases where this hierarchical feature selection model could be beneficial. It also discusses the model's adaptability to other classification algorithms beyond the DT, SVM, and ANN tested.

Response.   Thank you for your valuable feedback. We attempt to describe more details in section 5.4 and 6. We have expanded the discussion to outline conditions under which the results of the hierarchical feature selection model could be enhanced, as well as potential next steps for improvement in section 5.4 (Comparison of Performance – lines 543-551) and the next step to improve the proposed method in section 6 (lines 607-613).

 

7. The tables and figures showing accuracy comparisons could be further optimized. For instance, using color-coded heat maps to indicate classification performance across methods or visualizing the importance of features by dataset could make data interpretation more intuitive.

Response.   Thank you for your valuable feedback. We have enhanced the tables and figures displaying accuracy comparisons by incorporating color-coded heat maps. These heat maps visually represent classification performance across various methods and classifiers, with color intensity indicating accuracy levels. This improvement enables quick and intuitive comparisons across datasets and classifiers, as demonstrated in Figures 3, 5, and 10, with corresponding details provided in lines 318-323, 361-392, and 451-480, respectively. Furthermore, we have included visualizations that highlight feature importance by dataset, offering a clear representation of which features contribute most significantly to classification outcomes. This is presented in Figure 6, with details outlined in lines 408-419. These updates make data interpretation more accessible and align with the reviewer’s suggestion for enhanced clarity and visual appeal.

8. Regarding the metrics that the authors implement for optimization, they are advised to include more statistical methods to validate the selection of features. (e.g., permutation testing or cross-validation metrics) could add robustness to the conclusions.

Response.   Thank you for your valuable suggestion. We acknowledge the importance of incorporating additional statistical methods, such as permutation testing or cross-validation metrics, to enhance the robustness of the feature selection process. However, conducting further experimentation and analyzing the results would require additional time, as the experiments would need to be accompanied by explanations, potentially in the form of new graphs or tables. Given the multiple datasets used in this study, this would result in a significant number of additional visualizations and analyses, which must be completed within the 7-day revision timeline. Therefore, we have provided a rationale and explanation for our chosen approach in lines 310-315 of the revised manuscript.

9. The authors should include in the conclusions possible future work involving the potential challenges in implementing hierarchical schemes with large datasets and possible solutions or alternatives that could be explored.

Response.   Thank you for your valuable feedback. We attempt to describe more details in section 5.4 (lines 553-574) and section 6 (lines 607-613).

 

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript presents an approach to feature selection utilizing hierarchical concept models derived from FCA and DT. The authors evaluate the performance of their method on 10 datasets from the UCI Machine Learning Repository.

Pros:

1. The authors have conducted a thorough evaluation of their method across multiple datasets and classification algorithms, providing a robust analysis of the approach's effectiveness. 2. The proposed method successfully reduces the dimensionality of the feature space

Cons:

1. Could the authors elaborate on how the hierarchical structure is formed and how it influences the selection of features at each level? 2. Is there a sensitivity analysis conducted to understand how changes in the number of levels or the choice of attributes at each level affect the outcome? 3. It is recommended to introduce related methods: When does maml work the best? an empirical study on model-agnostic meta-learning in nlp applications; Poisoning medical knowledge using large language models; Hypergraph-enhanced Dual Semi-supervised Graph Classification. 4. Is the performance of the feature selection method affected by class imbalance in the datasets, and if so, how is this addressed? 

Comments on the Quality of English Language

The English could be improved to more clearly express the research.

Author Response

Thank you very much for reviewing our manuscript.  We also greatly appreciate the reviewers for their complimentary comments and suggestions. We have carried out more descriptions and the experiments that the reviewers suggested and revised the manuscript accordingly. We are grateful for the reviewer's feedback. Below are our responses addressing each point:

Pros:

1. The authors have conducted a thorough evaluation of their method across multiple datasets and classification algorithms, providing a robust analysis of the approach's effectiveness. 2. The proposed method successfully reduces the dimensionality of the feature space

Response. We sincerely thank the reviewers for their time and effort in thoroughly evaluating our work and for recognizing the strengths of our proposed method. Your positive feedback is greatly appreciated and encourages us to continue refining and advancing our research.

 

Cons:

1. Could the authors elaborate on how the hierarchical structure is formed and how it influences the selection of features at each level? 2. Is there a sensitivity analysis conducted to understand how changes in the number of levels or the choice of attributes at each level affect the outcome? 3. It is recommended to introduce related methods: When does maml work the best? an empirical study on model-agnostic meta-learning in nlp applications; Poisoning medical knowledge using large language models; Hypergraph-enhanced Dual Semi-supervised Graph Classification. 4. Is the performance of the feature selection method affected by class imbalance in the datasets, and if so, how is this addressed?

 

Response. Thank you for your useful feedback. We appreciate your suggestions for each of the following:

1. Could the authors elaborate on how the hierarchical structure is formed and how it influences the selection of features at each level?

Response. We have expanded our explanation of FCA as a method for hierarchical feature selection, particularly focusing on its role in reducing feature dimensionality while maintaining classification accuracy. This elaboration can be found in Section 3 (The Proposed Models for Feature Selection, lines 194-204). Additionally, we explain its application in generating a minimal yet effective feature set in Section 5.2.2 (Feature Selection Using FCA, lines 346-357). Furthermore, in Section 5.4 (lines 532-542), we highlight FCA's capability to reduce the computational load and improve interpretability, which is particularly important for handling complex or large datasets.

2. Is there a sensitivity analysis conducted to understand how changes in the number of levels or the choice of attributes at each level affect the outcome?

Response. We recognize the importance of sensitivity analysis in evaluating the impact of hierarchical levels and attribute choices. However, conducting such additional experimentation within the 7-day revision timeline would require significant resources and additional visualizations, given the multiple datasets used. To address this, we have included an explanation of our chosen approach and its rationale in Section 5.4 (lines 310-315) of the revised manuscript.

3. It is recommended to introduce related methods: When does maml work the best? an empirical study on model-agnostic meta-learning in nlp applications; Poisoning medical knowledge using large language models; Hypergraph-enhanced Dual Semi-supervised Graph Classification.

Response.  We have expanded the Related Works section to include a brief discussion of these recommended studies. These details can be found in Section 2 (Related Works, lines 117-125), where we provide an overview of these methods and position our work relative to them.

 

4. Is the performance of the feature selection method affected by class imbalance in the datasets, and if so, how is this addressed?

Response.  We have included a discussion addressing the impact of class imbalance and our approach to handling it in Section 5.4 (lines 561–574). This includes techniques to mitigate imbalance effects, ensuring the robustness of our feature selection method in datasets with skewed class distributions.

Reviewer 3 Report

Comments and Suggestions for Authors

 

the proposed method was compared with filter methods (Information Gain and Chi-Square), and forward and backward selection methods. Why did you only compare the proposed method with these approaches? Consider including a broader range of feature selection techniques for a more comprehensive evaluation.

You used only accuracy as the classification performance evaluation metric. It is strongly recommended to include additional metrics such as F1-score, Cohen’s Kappa, sensitivity, and specificity to provide a more robust evaluation of the proposed method

A simulation study should be conducted under various scenarios under different levels of multicollinearity, different dataset dimensions (high-dimensional and low-dimensional data), and scenarios involving grouping effects. The proposed method’s performance on simulated data across these scenarios should be analyzed.

Multicollinearity is an important issue in feature selection. It is recommended to examine the performance of the proposed method regarding multicollinearity.

The conclusion does not fully reflect the findings of the study. It should be revised to better summarize the results, contributions, and implications of the research.

 

The study lacks a discussion on its limitations and directions for future research. These should be specified.

Author Response

Thank you very much for reviewing our manuscript.  We also greatly appreciate the reviewers for their complimentary comments and suggestions. We have carried out more descriptions and the experiments that the reviewers suggested and revised the manuscript accordingly. We are grateful for the reviewer's feedback. Below are our responses addressing each point:

1. the proposed method was compared with filter methods (Information Gain and Chi-Square), and forward and backward selection methods. Why did you only compare the proposed method with these approaches? Consider including a broader range of feature selection techniques for a more comprehensive evaluation.

Response.  Thank you for your valuable feedback. We acknowledge the importance of comparing the proposed method with a broader range of feature selection techniques. Due to time and resource constraints, we initially selected commonly used filter methods (Information Gain and Chi-Square) and wrapper methods (Forward and Backward Selection) as baselines. However, we agree that additional methods such as embedded techniques and advanced hybrid models could provide a more comprehensive evaluation. Future work will include these comparisons to further validate and strengthen our conclusions (lines 607-613).

 

2. You used only accuracy as the classification performance evaluation metric. It is strongly recommended to include additional metrics such as F1-score, Cohen’s Kappa, sensitivity, and specificity to provide a more robust evaluation of the proposed method

Response.  We recognize the importance of sensitivity analysis in evaluating the impact of hierarchical levels and attribute choices. However, conducting such additional experimentation within the 7-day revision timeline would require significant resources and additional visualizations, given the multiple datasets used. To address this, we have included an explanation of our chosen approach and its rationale in Section 5.4 (lines 310-315) of the revised manuscript.

 

3. A simulation study should be conducted under various scenarios under different levels of multicollinearity, different dataset dimensions (high-dimensional and low-dimensional data), and scenarios involving grouping effects. The proposed method’s performance on simulated data across these scenarios should be analyzed.

Response.  Thank you for your insightful recommendation. We agree that a simulation study under diverse scenarios, including varying levels of multicollinearity, dataset dimensions (high and low), and grouping effects, would greatly enhance the evaluation of our proposed method. Such simulations would provide a deeper understanding of the method’s performance across a range of conditions. However, due to the limited revision timeline of less than 7 days, we are unable to conduct additional trials currently. We will consider incorporating these simulations in future work to further validate our approach (lines 561-574, 607-613).

4. Multicollinearity is an important issue in feature selection. It is recommended to examine the performance of the proposed method regarding multicollinearity.

Response.  Multicollinearity is indeed a critical issue in feature selection. While the hierarchical feature selection methods (FCA and DT) inherently reduce redundancy, we agree that a more explicit analysis of multicollinearity’s impact is necessary. In future work, we plan to include diagnostic measures (e.g., Variance Inflation Factor) and adjust the selection criteria to mitigate multicollinearity’s effects, ensuring that selected features are independent and impactful (lines 561-574).

 

5. The conclusion does not fully reflect the findings of the study. It should be revised to better summarize the results, contributions, and implications of the research.

Response.  Thank you for highlighting the need for a more comprehensive conclusion. We have revised the conclusion to more effectively summarize the study’s findings, focusing on the contributions of hierarchical feature selection in reducing dimensionality without compromising accuracy. The revised conclusion emphasizes the comparative advantages of hierarchical approaches over flat methods and discusses the practical implications for real-world applications. Additionally, it outlines potential future directions for improving the proposed approach (lines 599–606). Furthermore, we have incorporated details reflecting the study’s findings and summarizing the results, contributions, and implications in lines 367–392, 460–480, 506–520, 532–542, and 553–560 of the manuscript. These additions ensure that the key insights and impacts of the research are clearly communicated.

 

6. The study lacks a discussion on its limitations and directions for future research. These should be specified.

Response. We appreciate your suggestion to include a discussion on the study’s limitations and future research directions. In the revised manuscript, we have added a section to address the following limitations: (1) The limited scope of feature selection methods used for comparison and future directions (in lines 607-613), (2) The absence of additional performance metrics beyond accuracy (lines 310-315), and (3) The need for a detailed analysis of multicollinearity and imbalanced datasets (lines 561-574).

 

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Author:

I appreciate the responses sent to my questions and suggestions. I consider this new version suitable for publication.

Author Response

Response:
Thank you for your kind feedback and for taking the time to review our revised manuscript. We are delighted to hear that you find the updated version suitable for publication. Your insightful questions and suggestions have significantly contributed to enhancing the quality of our work. We deeply appreciate your efforts throughout the review process.

Reviewer 3 Report

Comments and Suggestions for Authors

Dear Authors,

I realized that some of my comments has not been fully implemented. Specifically, my comment: about the recommendation to include additional metrics such as F1-score, Cohen’s Kappa, sensitivity, and specificity to provide a more robust evaluation of the proposed method has not been adequately addressed. I kindly recommend you to include this suggestion to ensure a more comprehensive t evaluation of your proposed method.

Author Response

Response: Thank you for your valuable suggestion. We acknowledge the importance of incorporating additional performance metrics such as F1-score, Cohen’s Kappa, sensitivity, and specificity to provide a more robust evaluation of the proposed method. We appreciate your feedback and fully understand the value of these metrics in presenting a comprehensive evaluation.

However, conducting further experimentation and analyzing the results would require additional time, as these experiments would necessitate detailed explanations, potentially involving new graphs or tables. Given the multiple datasets used in this study, including these metrics would result in a significant increase in visualizations and analyses, which must be completed within the 3-day revision timeline.

In our current experiments, we focused on thoroughly analyzing and explaining accuracy in Tables 3, 4, 5, and 7. Adding additional metrics to these tables would require extensive explanations and analyses similar to those provided for accuracy, necessitating the inclusion of more images and graphs. As the number of evaluators increases, the number of visualizations and analyses would exponentially grow, making it challenging to address within the given time constraints.

Therefore, we prioritized using accuracy as the primary performance metric, as it is a widely recognized and critical measure for evaluating classification performance. Accuracy serves as a clear representative metric to demonstrate the method's effectiveness and ensures the clarity and focus of our analyses. We appreciate your understanding and will consider incorporating additional metrics in future work to enhance the evaluation.