Contrastive Refinement for Dense Retrieval Inference in the Open-Domain Question Answering Task

Round 1

Reviewer 1 Report

In this paper, the authors propose and discuss a contrastive relevance dataset centered on passages and queries separately by using an improved pseudo relevance feedback (PRF) algorithm.

The article is interesting and very topical. There are some areas that should be improved and/or further discussed. Namely,

1. There are some typos and at times the overall description is slightly awkward. I suggest having the manuscript proofread to address all these issues

2. Some of the equations need more explanation, despite being introduced in existing literature. For example, (1) and (2) should be better described mathematically. I personally don't understand (4). What is [1:k]? Equation 7 is a recursive equation, and it would be interesting to see if it has any specific dynamical property. I am not sure whether  this would be relevant, but recursiveness might lead to interesting (as well as problematic) asymptotic behaviours.

3. The experimental discussion should be further expanded to provide some more general details. For example, a better description of the dataset would help the reader to fully understand the context.

Author Response

Thank you for your valuable feedback on our manuscript. We appreciate your suggestions for improvement, and we have taken them into account in our revisions.

1. We have thoroughly proofread our manuscript to address any typos and awkward phrasing issues.

2. In response to your comments on the equations, we have reviewed them and verified their accuracy by referencing the previous paper. To enhance understanding, we have included additional details in our equations.

   1. For Eq(1) and (2), we should note that they are both hidden states of neural networks, which are effective in NLP but not mathematically interpretable.

   2. For Eq(4), [1:k] refers to the indices from 1 to k, which is used to get the top 1 to k retrieval results from the sorted list.
   3. For Eq(7), it is a simple and classic linear weight calculation equation used to update mathematical vector, so i don't think it may have any specific dynamical property.
3. We agree that expanding the experimental discussion would be beneficial. To provide a better understanding of the context, we have included a more detailed description of our experiments settings.

Reviewer 2 Report

The paper presents novel learning approach for the OpenDomain QA. This approach is based on some modifications of the Dense Passage Retrieval Model which include modified learning function and new negative sampling technique. The paper has a number of major drawbacks that have to be addressed before the acceptance.

1. The paper is not self-contained. Despite it's heavily based on the DPR paper a lot of important detail should be repeated here, e.g., the encoder description.

2. Evaluation seems to be very premature at the moment:

2.1 Only one dataset is used. Why other datasets from the DPR paper are not explored here?

2.2 The initial DPR settings were modified (for FAISS) which made the results not comparable with the original paper. This should be either well motivated or changed.

2.3 All baselines (including sparse models) are missing.

2.4 Two main tables with the results are confusing since they are not corresponding with the main results from the DPR paper in terms of metrics. For the pipeline "Exact match" metrics is somehow used with the Top-N setting which is not the case for the DPR paper and also not reasonable.

Author Response

Thank you for your valuable feedback on our paper. We have carefully considered your comments and revised our paper accordingly.

1. We have included more details about the encoder and other important information from the DPR paper to make our paper more self-contained. 

2. We appreciate your comment on the evaluation. To address your concerns:

   1. We have added another large dataset TriviaQA in the evaluation, which helps to provide a more comprehensive evaluation of our approach.

   2. We have modified our experimental settings to be the same as the original paper and rerun all experiments, ensuring that our results are directly comparable with those reported in the DPR paper.

   3. We have included classic and popular baselines such as BM25, ANCE, GAR and etc in our evaluation, which provides a fair comparison of our approach against existing state-of-the-art techniques.

   4. We apologize for any confusion caused and have now updated the tables to correspond with the main results from the DPR paper in terms of metrics, and have made sure to use appropriate metrics for each experiment.

Round 2

Reviewer 2 Report

The authors presented a modified version of the paper. The paper is substantially improved in many aspects: writing, logical structure, presentation of the novel ideas. Authors provided additional experimental results that helped to increase an overall merit of the paper. In my opinion, the paper is ready for the publication.