Identifying New Promising Research Directions with Open Peer Reviews and Contextual Top2Vec

Abstract

The reliable and early detection of promising research directions is of great practical importance, especially in cases of limited resources. It enables researchers, funding experts, and science authorities to focus their efforts effectively. Although citation analysis has been commonly considered the primary tool to detect directions for a long time, it lacks responsiveness, as it requires time for citations to emerge. In this paper, we propose a conceptual framework that detects new research directions with a contextual Top2Vec model, collects and analyzes reviews for those directions via Transformer-based classifiers, ranks them, and generates short summaries for the highest-scoring ones with a BART model. Averaging review scores for a whole topic helps mitigate the review bias problem. Experiments on past ICLR open reviews show that the highly ranked directions detected are significantly better cited; additionally, in most cases, they exhibit better publication dynamics.

1. Introduction

Researchers use terms “promising research directions”, “research front”, and “frontier” as synonyms to define the latest and most advanced knowledge and research directions in a field, and they mark the forefront of scientific and technological development [1]. Identifying the directions is essential for guiding scientific progress and allocating resources effectively. A common way to reveal promising research directions is to quantitatively analyze recent citation and publication trends with modern text mining and statistical tools in large research paper and patent databases [2]. Although this approach is very objective and free from expert bias, it has several disadvantages.

• Scientometric analysis relies on accurate, manually curated international citation databases, which can be costly to maintain and use. However, these databases often fail to capture local trends and needs. For instance, research in region-specific agriculture may hold significant practical value, yet these related papers are frequently not indexed or well-cited in international databases.
• There is a substantial gap between publishing the first innovative results and presenting the first robust signals of a new frontier in citation databases. For example, the preprint of the Vaswani et al. “Attention is all you need” paper [3] was published first in June 2017 (Figure 1), but even in 2018, citation databases could not reliably detect this new frontier (because all citation information for 2018 was available only in 2019).

We tackle these problems by collecting and analyzing open reviews. The main contributions of this paper are the following.

• We have proposed and tested an end-to-end conceptual framework that combines several ideas in review-based ranking and text mining: use topic-level assessment to tackle reviewing bias [4], apply pre-trained classifiers to extract relevant fragments from the reviews, and employ the classifiers to obtain unified review scores [5]; therefore, scores from different sources can be considered together. This framework builds and detects novel research directions on preprint databases with Contextual Top2Vec [6], summarizes them with a Transformer-based sequence-to-sequence model, and automatically ranks them based on the related open reviews with a Transformer-based encoder model. This integrated framework enables the early identification and ranking of novel research directions without relying on citation databases, reducing bias and accelerating discovery.
• We have validated this end-to-end framework through experiments on retrospective data from the International Conference on Learning Representations (ICLR 2017–2019) [4] and arXiv.org (2017–2021) [7]. Results show that high-ranked research directions identified by our method achieve significantly better citation performance than others.

The paper is organized as follows. Section 2 provides a review of the detection of research directions, topic modeling, and summarization. Section 3 describes the proposed framework and the method we use to summarize the directions. Section 4 briefly discusses the dataset we utilized in the experiments. Section 5 presents the results of the experiments on the ICLR datasets. Finally, Section 6 and Section 7 provide a discussion and conclusion regarding the results obtained.

2. Related Work

2.1. Detection of Promising Research Directions

Most research in promising research direction detection reveals outbursts in publication or citation activity, or the growing usage of particular keywords, and considers related papers as frontiers. For example, paper [8] utilizes the high-ranked conference papers provided by Google Scholar Metrics. They first investigate the annual publication dynamics in a particular area, and then they use the paper’s keywords to detect the promising directions. They study frequency, betweenness centrality, clustering of keywords, and detect rapidly growing keywords. Ye G. and colleagues highlight the problem that the utilization of paper or patent databases as the initial data to reveal frontiers may lead to outdated results [9]. They propose an adapted method that uses research grant data to identify research front topics and forecast trends. First, they apply a topic model to detect research directions, and then relate them to funded projects and cross-domain categories. Finally, they study the evolution of the obtained directions to predict development trends. Although experiments show that the method is effective in research direction identification, many research funds still do not publish project reviews and do not provide references to related papers. Quantitative publication activity analysis is also a frequently used technique to detect promising directions in particular areas. For example, paper [10] uses Top2Vec to build topics from arXiv.org and analyze their dynamics. Another way to tackle the gap when using the databases is to create predictive models. Paper [11] proposes a global graph ranking model that utilizes node embedding. They propose a cost function that solves the ranking problem with a probabilistic regression. The proposed model has been tested on an arXiv paper citation network using standard information retrieval-based metrics. The results show that the proposed model outperforms, on average, other state-of-the-art static models as well as dynamic node ranking models.

We rely on review scores to identify promising directions. The natural question that arises, though, is whether there is any reliable relationship between the review score and novelty and demand of the reviewed work [4]. Paper [12] studies relationships between novelty and the acceptance of manuscripts submitted to scientific journals. They use the unusualness of reference combinations in their citation lists as a measure of novelty and conventionality. Experiments show that both novelty and conventionality are related to acceptance. They also show that papers with novel results are cited better. Paper [13] concludes that peer review in journals with higher impact factors tends to be more thorough. Similarly, paper [14] uses data from the National Institutes of Health fund to study how peer reviews can predict the future quality of proposed research. They show that higher peer-review scores are associated with better research citations. Wang and colleagues [4] apply the Ward clustering algorithm to obtain journal submission clusters. They show that the clusters with higher review scores are cited better. It is worth noting that they could not detect a similar pattern for individual papers, because many rejected papers may be accepted later at other conferences or journals and gain many citations. Paper [15] proposed an approach to predict a paper’s citations. The approach utilizes a heterogeneous publication network of nodes, including papers, authors, venues, and terms. Then, it applies a graph neural network to jointly embed all types of nodes and links to model the paper’s impact. The experiments confirm that the proposed method overcomes others in citation prediction with publication networks.

Despite the recent rise in open-review resources, obtaining labeled corpora for different tasks remains a problem [16]. Study [17] describes the review score prediction and the paper decision prediction on two datasets with ICLR submissions; a similar dataset is also proposed in the paper [18]. Paper [5] proposes the PEERAssist framework to predict review scores. This framework outperforms similar approaches, which utilize sentiment analysis. Ribeiro and colleagues propose a dataset to perform review sentiment analysis and investigate polarities [19]. The CiteTracked [20] dataset combines reviews from the NeurIPS conference with the paper’s citations, which helps study paper citations based on the reviews. Paper [21] introduces the Open Review-Based dataset that includes a list of more than 36 K papers, and more than 89 K reviews, together with decisions gathered from OpenReview.net and SciPost.org. However, in this work, we still rely on a smaller ICLR dataset from [4] because it also provides citation information about reviewed papers.

2.2. Topical Vector Models

Many recent studies in topic modeling are devoted to marrying topic models and dense word/sentence embeddings. The Embedded Topic Model [22] is one of the first approaches that combine a probabilistic topic model with word embeddings. This approach uses word embeddings to represent lexis in a document to tackle the drawbacks of the bag-of-words models. However, it still ignores the syntax of words and requires the number of topics to be determined. The paper [23] proposes Top2Vec that utilizes joint document and word embeddings to find topic vectors. Top2Vec combines syntactic features from document embeddings with the semantic ones from word embeddings. It also automatically estimates the number of topics. The BERTopic approach [24] utilizes contextual token-level embeddings from BERT [25], which combine syntax and semantic features out of the box. However, this approach does not assign topics to all documents, and it only sets one topic for a document. Paper [6] extends the Top2Vec approach with contextual document embeddings from SentenceBERT. They show that the approach outperforms others on a set of topic model evaluation metrics and supports hierarchical topic reduction. Therefore, we shall use ContextualTop2Vec in our experiments.

2.3. Topic Summarization

A common approach in text summarization is to use text-to-text Transformer models like T5 [26] or BART [27]. For example, paper [28] uses T5 trained on the UCI drug reviews dataset. Similarly, paper [29] studies Transformer-based models for summarization, with a focus on BART. They evaluate the variants of BART with the BERTScore [30]. They also claim that BART’s encoder captures richer contextual relationships, which makes it well-suited for abstractive summarization. In contrast, T5 follows a standard text-to-text approach, which simplifies adaptation to multiple tasks but complicates training for summarization. These differences in the models allow one to pick between processing speed, memory consumption, and summary quality. There are also open pre-trained models for scientific text summarization, for example, T5 trained on arXiv.org [31].

Topic summarization brings additional questions on how to extract and combine information-rich and topic-related fragments from the topic’s papers. Onan et al. [32] propose the FuzzyTP-BERT framework for extractive text summarization. It combines Fuzzy Topic Modeling with BERT. FuzzyTP-BERT integrates fuzzy logic to refine topic modeling, enhancing the semantic sensitivity of summaries by allowing a more nuanced representation of word–topic relationships. Paper [33] proposes a topic summarization framework that is compatible with various Transformers.

3. Materials and Methods

3.1. General Framework

Figure 2 shows the general schema of our framework. First, it collects open reviews from high-ranked conferences (CORE A/A* [34]) and open research journals from a particular research field for a given year y (in this research, we limited ourselves to ICLR reviews only). It collects the full texts of relevant research papers from preprint repositories as well. Second, it extracts the fragments from the reviews, which are potentially related to the prospects of the analyzed piece of research, neglecting other parts related to the quality of submission text, number of experiments, etc. Namely, it extracts the fragments related to motivation, novelty of work, and the description of the proposed method/approach. We tackle the fragment extraction problem as a sequence labeling task and solve it at the sentence level with a Transformer-based encoder model (xlm-roberta [35]). Further, we shall refer to this subtask as “Review segmentation”.

Third, it uses the collected preprint full texts and a contextual Top2Vec model to build topics (research directions). We use SentenceBERT’s all-MiniLM-L6-v2 model as the encoder to generate phrase and document embeddings for the Top2Vec because it works fast on CPU, keeping competitive performance.

Fourth, it detects novel research directions. Top2Vec cannot provide probability estimations for the topic terms; therefore, straightforward approaches to compare topics, like Kullback–Leibler divergence, cannot be applied. We decided to neglect topic term weights and to use Spearman’s correlation between term ranks. Thus, a research direction is considered to be novel if its ranked term list has a negative correlation with the previous year’s topics’ ranked term lists, and this correlation is statistically significant (p-value < 0.05).

Fifth, it applies another Transformer-based encoder to predict the reviewer’s score of the papers in the novel research directions. We refer to this subtask as “Review classification”. We utilize a simple ternary classification setting, which distinguishes the “accept” overall score from “borderline” and “decline”. The motivation here is two-fold:

–

We cannot rely on the explicit score given by the reviewer, because different conferences, journals, and research funds use different scoring systems.

–

This rough (“accept”/”borderline”/”decline”) scoring leads to lower classification error; in addition, combining these scores for a whole topic would provide more versatile information anyway.

In addition to the pure text of reviews, we tested a couple of heuristic features to gain classification scores. The first one is whether a review has more than one question. Too many questions in a review might be a sign that a reviewer perceives the submission skeptically. The second one is whether the reviewer explicitly declares that the submission results are novel (we shall further refer to this subtask as “Review classification”). We trained and applied an additional Transformer-based classifier to detect such reviews and split them from those in which “novelty” words are not related to the results.

Finally, we utilize a lightweight text generation model to build descriptions for the novel high-rated research directions (we shall refer to this subtask as “Topic summarization”). In short, the description approach has the following steps: 1. Extract top-n documents that mostly relate to the direction. 2. Extract sentences that contain the directions’ terms from the documents. 3. Use the generative model (BART) to form a fluent, coherent description of this set of sentences. Section 3.3 describes these steps in more detail.

In this research, we have also applied the following additional validation steps to be sure the extracted topics actually reflect the perspective research directions.

1. Check on retrospective data if there is more significant paper publication growth for the high-ranked research directions than for the low-ranked ones. The motivation behind this is that writing publications requires organizational and financial efforts that are unlikely to be spent on dead-end projects. We use the following score to estimate the growth (Equation (1)).

$$GR=1-{\displaystyle \frac{S_0}{{\sum }_{i=0..N}{s}_i}},$$

(1)

where $S_0$ is a number of the direction’s publications at starting year y, and $s_i$ is number of the direction’s publications at year y + i.

2. Check on retrospective data if there are more citations for the high-ranked directions than for the low-ranked ones.

All the sources implementing those steps are openly available in the COntext TOp2vec Frontiers (COTOF) repository [36].

3.2. Phrase Generation for the Contextual Top2Vec

Contextual Top2Vec uses standard statistical approaches to build topic phrases. Namely, they use pointwise mutual information (PMI) [37] to detect word collocations in texts and utilize them to build the model’s dictionary. However, this approach may not perform well on small texts like reviews or submission abstracts. In this study, we tested several alternatives to this approach to tackle this.

In the first approach, we use a linguistic parser [38] to explicitly extract syntactically related word pairs. The second approach is to consider internal attention from the encoder model used in the Contextual Top2Vec. In this approach, we just find word pairs with high absolute attention values (see Equation (2)).

$$AttnRelScore(i,j)={\sum }_{layer=1..N}{\sum }_{head=1..H}\left|{ATTN}_{layer,head}(i,j)\right|,$$

(2)

where N is the number of layers in the encoder, H is the number of heads in the encoder, ${ATTN}_{layer,head}(i,j)$ is the attention weights between words i and j in the layer layer and the head head. In our experiments, we have empirically derived the following rule: we leave the word pairs if their AttnRelScore is higher than the mean AttnRelScore for all the pairs in the text. An advantage of this approach is that it does not need any third-party tool to extract phrases.

In all the approaches, we filter out non-frequent pairs, which appear less than four times in the text corpus.

3.3. Research Direction Summarization

Our framework uses a two-stage summarization procedure that allows us to use small generative models with limited context size (Figure 3). In the first stage, we summarize each direction’s paper independently, and then we summarize all the paper summarizations. Namely, the procedure includes the following steps.

• Get the most relevant direction’s documents with the Contextual Top2vec model. In our experiments, we obtained the 10 most relevant papers for each direction.
• From each document’s text extract title, select the first M = 5 sentences (they are likely to be from the document’s abstract), and top K = 5 (as they follow in the paper’s text) sentences containing top keywords or key phrases from the topic.
• Use the extracted sentences as a prompt and summarize each paper with an LLM.
• Concatenate all the paper summarizations and use it as a prompt to summarize the whole direction.

Figure 3. Schema of the research direction summarization.

Figure 3. Schema of the research direction summarization.

4. Datasets

We have picked the ICLR dataset from [4] as the primary one for three reasons. First, preprints of the most reviewed papers from this dataset are openly available on arXiv.org. Second, the provided reviews are pretty historical, covering conferences from 2017 to 2019; therefore, there is room to track the detected frontiers on retrospective data. Third, it contains Google Scholar citations of the reviewed papers, which are essential to check the success of the detected directions. We also collected all the arXiv.org preprints from 2017 to 2021 via the provided open API (778.5 K preprints) to track publication dynamics GR for the directions. The major limitation of the ICLR dataset is that it contains data from a single scientific area. Although our methods do not use any area-specific dictionaries or thesauri, the results still might differ for other fields of research. In the future, we shall extend the datasets with examples from various scientific areas to tackle this problem.

In this study, we also created a tiny dataset (82 samples) for the “Novelty detection” task [39]. It provides sentences from various reviews; half of them claim that the overall reviewed research results are new, and other half just contain a novelty-related lexis (“new”, “novel”, etc.), which is not related to the submission results. This dataset is used to generate an additional feature (IS NEW) for review scoring.

We have also tested several techniques to build phrases for Top2Vec. Two commonly used datasets have been utilized to perform this task: 20 news groups (20NG) and Yahoo, which were highlighted as the most suitable ones in paper [6]. We were also curious how the techniques would help in datasets with short texts. To this end, we utilized a part of the ICLR dataset, containing only review texts from ICLR 2017, which are considerably shorter than the texts from 20NG or Yahoo.

Finally, we have created another dataset for scientific text summarization [40]. The dataset provides about 1K snippets of research papers for 99 search queries with their short summaries of the snippets for each query generated with ChatGPT-4o mini. After the generation, we manually filtered out all the samples that seemed to contain hallucinations.

5. Experiment Results

First, we tested the techniques for building phrases for the Contextual Top2Vec. In our experiments, we use standard hyperparameters for the Top2Vec: embedding size 384 (as all-MiniLM-L6-v2 provides), UMAP’s number of neighbors 50, UMAP reduction to 5, and Hdbscan’s min_cluster_size 15. We use standard coherence scores and tools to obtain them [41] but estimate them separately at the phrase and word levels.

Table 1. Topic coherence scores for different approaches to build phrases.

	PMI			SYNTAX			ATTENTION
Dataset	20NG	Yahoo	ICLR	20NG	Yahoo	ICLR	20NG	Yahoo	ICLR
Cnpmi_phrase	−0.311	−0.263	−0.492	−0.304	−0.259	−0.487	−0.307	−0.259	−0.492
Cnpmi_words	0.074	0.073	−0.025	0.094	0.066	0.005	0.084	0.070	−0.008
Cv_phrase	0.389	0.318	0.564	0.374	0.319	0.563	0.385	0.316	0.564
Cv_words	0.601	0.560	0.421	0.616	0.546	0.457	0.606	0.561	0.443
Cwe_words	0.107	0.079	0.071	0.108	0.074	0.078	0.110	0.073	0.072
CSBERT	0.385	0.365	0.413	0.392	0.360	0.408	0.385	0.364	0.402
CBERTScore	0.650	0.585	0.638	0.649	0.589	0.638	0.650	0.589	0.640

The best results for each dataset are marked in bold.

shows that all the approaches show very close performance in terms of topic coherence. Phrasal-based scores seem to be stricter than the word-based ones in almost all cases. The syntax-based approach is slightly better on the ICLR dataset. This result can be related to shorter texts in this corpus compared to others. The attention-based approach achieves better CBERTScore in all the datasets.

The next section of the experiments is devoted to testing Transformer-based models for various text classification subtasks. We utilized the Huggingface Transformers library to perform the experiments. For all the subtasks, we applied similar training hyperparameters: learning rate 2 × 10⁻⁵, batch size 8, weight_decay 0.01, and warmup_ratio 0.15. We used standard optimizers and losses provided by Transformer’s Trainer for the sequence classification class. We repeated all the classification experiments three times with different random seeds and train/test splits and obtained mean scores with their standard deviations.

We started with the review segmentation classifiers. These sentence-level models filter out all review fragments that are not directly related to the novelty, motivation, or technical details of the analyzed research. According to

Table 2. Classification metrics for the ‘’Review segmentation”.

Model	P	R	F1
xlm-roberta-base	0.74 ± 0.05	0.77 ± 0.04	0.74 ± 0.05
xlm-roberta-large	0.74 ± 0.02	0.78 ± 0.02	0.74 ± 0.03
bert-base-multilingual-uncased	0.70 ± 0.11	0.73 ± 0.11	0.71 ± 0.11

The best results are marked in bold.

, xlm-roberta shows the best results for this task; therefore, we selected this model for further experiments. The dataset we used for this experiment contains non-open-access reviews. Thus, we keep it private and provide access to the best model [42].

For the novelty detection, we use the distilbert-base-uncased model (see

Table 3. Classification metrics for the ‘’Novelty detection”.

Model	P	R	F1
bert-base-uncased	0.93 ± 0.12	0.93 ± 0.13	0.93 ± 0.12
bert-large-uncased	0.93 ± 0.12	0.93 ± 0.12	0.93 ± 0.12
distilbert-base-uncased	0.91 ± 0.07	0.90 ± 0.06	0.90 ± 0.06
distilroberta-base	0.73 ± 0.06	0.74 ± 0.05	0.73 ± 0.06
roberta-base	0.81 ± 0.03	0.79 ± 0.02	0.80 ± 0.02
roberta-large	0.90 ± 0.10	0.87 ± 0.10	0.87 ± 0.12
xlm-roberta-base	0.71 ± 0.36	0.74 ± 0.21	0.70 ± 0.29
xlm-roberta-large	0.64 ± 0.31	0.63 ± 0.16	0.56 ± 0.23

The best results are marked in bold.

), which is one of the smallest models. It achieves the second-best result with a reasonable standard deviation, since the size of the dataset for this task is small. The relatively high standard deviation in the classification scores indicates that this dataset is too small to support robust training; therefore, it should be extended in the future.

Finally, we have tested the models to classify reviews into three classes (reject, borderline, accept).

Table 4. Classification metrics for the “Review classification”.

Model	TEXT			TEXT + QUESTION TAG			TEXT + QUESTION TAG + IS NEW
	P	R	F1	P	R	F1	P	R	F1
bert-base- uncased	0.66 ± 0.03	0.64 ± 0.02	0.64 ± 0.03	0.59 ± 0.03	0.61 ± 0.03	0.59 ± 0.03	0.60 ± 0.05	0.62 ± 0.04	0.60 ± 0.05
bert-large- uncased	0.62 ± 0.04	0.62 ± 0.03	0.61 ± 0.04	0.60 ± 0.03	0.61 ± 0.02	0.59 ± 0.04	0.54 ± 0.14	0.54 ± 0.13	0.53 ± 0.15
distilbert-base- uncased	0.56 ± 0.01	0.58 ± 0.01	0.57 ± 0.01	0.62 ± 0.00	0.62 ± 0.00	0.62 ± 0.00	0.37 ± 0.01	0.46 ± 0.01	0.40 ± 0.01
distilroberta-base	0.57 ± 0.00	0.60 ± 0.00	0.57 ± 0.00	0.60 ± 0.04	0.60 ± 0.01	0.59 ± 0.04	0.41 ± 0.17	0.48 ± 0.11	0.44 ± 0.14
roberta-base	0.63 ± 0.01	0.62 ± 0.03	0.62 ± 0.02	0.66 ± 0.02	0.65 ± 0.02	0.65 ± 0.02	0.54 ± 0.15	0.53 ± 0.08	0.51 ± 0.11
roberta-large	0.63 ± 0.02	0.62 ± 0.02	0.62 ± 0.01	0.64 ± 0.02	0.58 ± 0.08	0.58 ± 0.07	0.61 ± 0.04	0.63 ± 0.02	0.61 ± 0.05
xlm-roberta-base	0.61 ± 0.02	0.61 ± 0.01	0.60 ± 0.01	0.28 ± 0.24	0.42 ± 0.15	0.32 ± 0.20	0.36 ± 0.01	0.46 ± 0.01	0.40 ± 0.01
xlm-roberta-large	0.42 ± 0.20	0.50 ± 0.10	0.44 ± 0.15	0.32 ± 0.10	0.37 ± 0.04	0.28 ± 0.06	0.49 ± 0.12	0.50 ± 0.05	0.47 ± 0.09

The best results are marked in bold.

shows the obtained scores. The “TEXT” column provides results for the case where we classify a pure review of text fragments. In the “TEXT + QUESTION TAG” case, we added a special tag word if there is more than one question in the review. Column “TEXT + QUESTION TAG + IS NEW” provides results for the case when we also add another special tag if the novelty classifier concludes that the research is novel. According to Table 4, the roberta-base model shows the best results in the case “TEXT + QUESTION TAG”. This means that the novelty classifier does not provide helpful information for the task, but the question tags do.

After the high-ranked research directions are revealed, we then build short text annotations for them. For this task, we have tested several models with full-Transformer (T5 and BART) and decoder-only (GPT-based) architectures. Although the mistralai/Mistral-7B-Instruct-v0.2 shows the best scores on this task, the much smaller and faster BART-based model philschmid/bart-large-cnn-samsum obtains results which are nearly as good, so we choose it for the following experiments (

Table 5. Metrics for the topic summarization.

Model_Name	Avg Ime per Batch	Rouge1	Rouge2	Rougel	Bleu4	Bertscore
philschmid/bart-large-cnn-samsum	2.43 ± 0.16	0.33 ± 0.02	0.08 ± 0.00	0.19 ± 0.00	0.02 ± 0.01	0.60 ± 0.00
Talina06/arxiv-summarization	3.03 ± 0.51	0.26 ± 0.03	0.05 ± 0.00	0.15 ± 0.01	0.02 ± 0.01	0.56 ± 0.00
facebook/bart-large-cnn	2.88 ± 0.66	0.33 ± 0.02	0.08 ± 0.00	0.18 ± 0.00	0.03 ± 0.01	0.60 ± 0.00
google/flan-t5-large	9.72 ± 1.46	0.22 ± 0.02	0.04 ± 0.00	0.13 ± 0.01	0.01 ± 0.00	0.52 ± 0.01
google/pegasus-cnn_dailymail	3.09 ± 0.30	0.31 ± 0.03	0.06 ± 0.00	0.17 ± 0.01	0.02 ± 0.01	0.60 ± 0.00
google/pegasus-xsum	2.55 ± 1.00	0.23 ± 0.09	0.04 ± 0.02	0.14 ± 0.03	0.01 ± 0.01	0.54 ± 0.01
mistralai/Mistral-7B-Instruct-v0.2	36.77 ± 3.93	0.39 ± 0.03	0.10 ± 0.02	0.19 ± 0.02	0.04 ± 0.01	0.61 ± 0.02
openlm-research/open_llama_13b	45.20 ± 12.64	0.25 ± 0.03	0.03 ± 0.00	0.14 ± 0.01	0.01 ± 0.00	0.54 ± 0.01
Falconsai/text_summarization	3.71 ± 0.62	0.28 ± 0.05	0.05 ± 0.01	0.15 ± 0.01	0.02 ± 0.01	0.54 ± 0.01
NousResearch/Nous-Hermes-13b	30.32 ± 19.02	0.22 ± 0.04	0.03 ± 0.01	0.12 ± 0.02	0.01 ± 0.00	0.54 ± 0.01
Qwen/Qwen1.5-7B-Chat	15.60 ± 11.63	0.20 ± 0.06	0.02 ± 0.01	0.11 ± 0.02	0.01 ± 0.01	0.53 ± 0.04

The best results are marked in bold.

). We used the same strict hyperparameters for all the generative models: temperature 0.2, top_p 0.3, top_k 3, repetition_penalty 1.5 to reduce hallucinations.

We also limit input text (left context size) to 2048 tokens, document-level annotations to 64 tokens, and overall annotations to 300 tokens; therefore, even models with small context windows, such as T5, can operate with these settings.

We have validated our framework on the open reviews from ICLR 2017–2019. Specifically, we detected the 10 best- and 10 worst-ranked research directions, and then we used retrospective arXiv.org data and citation information to compare these two groups. We collected related preprints for each direction with the same SentenceBERT model (all-MiniLM-L6-v2) as used in the Contextual Top2Vec, estimating cosine similarity between topic and preprint vectors. We selected a strict similarity threshold of 0.6 to ensure that the preprints were very close to the topics. After obtaining the topic’s GR scores for the 2017–2021 interval and their Google Scholar citations for 2021 (Figure 4), we estimated Mann–Whitney statistics for the best- and worst-ranked topics.

Table 6. Mann–Whitney U statistics and p-values to show contrast between top- and low-ranked directions in GR and average citation number.

Conference	Original Scores		Predicted Scores
	GR	Cites	GR	Cites
ICLR 2017	86.0 (p-value 0.002)	95.0 (p-value 0.0001)	82.0 (p-value 0.007)	85.0 (p-value 0.003)
ICLR 2018	45.0 (p-value 0.658)	93.0 (p-value 0.0002)	47.0 (p-value 0.602)	75.0 (p-value 0.031)
ICLR 2019	80.0 (p-value 0.011)	89.0 (p-value 0.001)	79.0 (p-value 0.014)	75.5 (p-value 0.031)

shows the obtained results for two cases: (1) we use the original reviewer’s scores to neglect potential review classification errors when we rank topics, and (2) we use the predicted classification results to rank topics. According to Table 6, there is no qualitative difference between predicted and original review scores, i.e., our classifier is suitable for the promising direction detection task. Citation scores for the best-ranked direction are statistically higher than those of the worst-ranked ones in all cases.

Regarding the GR score, the same results are obtained for two of the three conferences. We believe that this is because of a limitation of the GR score, which is insensitive to the research direction size (i.e., it produces the same score for growth from 1 paper to 10 as for growth from 100 papers to 1000).

6. Discussion

The experiment results show that the proposed framework can detect promising research directions based on open peer reviews. Our results support previous theoretical [43] and experimental findings [4], showing that diversifying peers by adding independent concurrent panels (which we simulate by averaging review scores for a whole direction) leads to an improvement in the reproducibility of analysis. Automatic peer-review score assignment makes it possible to apply the framework to any conference or journal regardless of their internal scoring system. Of course, the ICLR dataset is helpful only to illustrate the framework’s idea, because it is narrow-focused and (despite the high rank of the conference) may still be biased. In order to make a working framework, we need to collect and combine all the reviews from different sources. Moreover, since there is a very limited number of high-ranked venues, we need to test less-ranked conferences and journals to check if our findings hold for them. Another problem is that we use arXiv.org as the single source of preprints to build topics. However, it is a limited source in terms of research area variety and number of preprints; therefore, we need to broaden the list of preprints and open-access paper databases.

We also found that although the applied cumulative growth ratio (GR) score can be estimated without any citation database, it may not be the best option to validate the success of a direction because it is insensitive to the direction’s size. In the future, we will test other publication activity-based scores, which would also consider the size of the topic.

7. Conclusions

The proposed conceptual framework uses open reviews to detect novel research directions, rank them, and describe them. This framework mitigates the time gap between the appearance of new promising directions and their detection; in addition, it does not need any additional citation databases. The cornerstone of the framework is the contextual Top2Vec model, which we preferred over others, since it automatically adjusts the number of topics, supports phrases to describe the topics, and builds topics’ centroids in the document’s vector space, which makes it possible to track the revealed research directions on open research paper databases. The framework generates review scores based on the review texts and averages the scores for the whole topic, which helps tackle reviewing bias. The statistical analysis on the retrospective data from ICLR 2017–2019 conferences has shown that the detected high-ranked directions outperform the low-ranked ones in citations. The high-ranked topics also achieved up to twice the citation growth as the low-ranked ones.

In the future, we shall broaden the list of considered open reviews by adding alphaXiv, PCI Peer Community, SciPost.org, openly available reviews from research journals, and funds. This would further smooth out the reviewing bias and make the ranks more balanced from different points of view. We will also keep testing our framework at other conferences to strengthen the statistical results obtained. Ultimately, the proposed framework would enable scientific authorities, funding agencies, and research institutions to create analytical tools for planning or adjusting their activity structures, thereby ensuring that promising directions receive sufficient organizational and budgetary support.