Using Conceptual Recurrence and Consistency Metrics for Topic Segmentation in Debate

Round 1

Reviewer 1 Report

The paper is an interesting paper about a topic segmentation model for debates based on conceptual recurrence and debate consistency metrics. The authors research the conceptual similarity of conceptual recurrence and debate consistency metrics relate to topic segmentation.

Some questions:

Whether in an informal situation or in more formal settings such as a political debate or business meeting, a conversation is often not about just one thing: topics evolve and are replaced as the conversation unfolds. Can you please increase the explanations about the data properties?

Could your model be used in Asynchronous Conversations? Discuss this topic.

What about monologue vs. dialogue?

Please discuss further the evaluation metrics in use.

Some of the first successful approaches to segmentation focus on changes in lexical distribution, and this still forms the core of many current algorithms. Have you considered the idea that the essential insight is that a topic shifts tend to be marked by a change in the vocabulary used, which can be detected by looking for minima in some lexical cohesion metric?

A rather different approach is to look for the characteristic features of the boundaries themselves: the cue phrases people use to signal topic change, the prosodic features often exhibited in speech at the beginnings and ends of topic discussions, or the introduction of new referents. Please talk further about this idea.

How does changes in the language used by participants affect your model?

Author Response

Thank you very much for your interest in and review for our research and academic community, even in the midst of your work. We would like to do our best to take the comments of reviewers into account. We don't know where the reviewers live, but we wish people in Europe and the world safety and peace.

The response letter is attached as a pdf file.

Author Response File: Author Response.pdf

Reviewer 2 Report

This article presents a work focusing on topic segmentation in debate/conversation transcripts. In order to detect the borders of particular conversation segments and also to estimate the characteristics of such segments (in particular, consistency), the authors propose a technique titled CSseg that makes use of conceptual recurrence plots and corresponding matrix calculations. The authors contribute the description of (1) several approaches for estimating topic segment similarity based on conceptual recurrence within particular utterances as well as (2) several approaches for estimating debate consistency for debate transcripts in particular. The interactive interface of CSseg provides the user with options to adjust particular parameters and weights used for calculations in order to arrive at better results and to explore the identified topics and transcript texts. The validation of the work described by the authors includes (1) a quantitative comparison between CSseg and a previous approach titled LCseg based on ground truth data collected from several transcript annotators, and (2) a use case scenario for the interactive approach based on particular debate transcript from Korean TV.

Overall, I think that this is an interesting work, and the task of analysis of conversational data, especially involving (potentially high-profile) public debates, can be indeed facilitated by the means of visual analytics, as relying on computational means exclusively might not provide the users with appropriate results or—even if the results themselves are quite accurate—might not be valuable to the users due to the lack of understanding, overwhelming level of detail, and so on. And while this work does not provide much novelty from the point of view of visualization itself (the authors directly discuss the prior works by Angus et al. and other authors with a focus on recurrence plots and visual support for topic segmentation in text data), the combination of the proposed computational methods with an interactive visual approach is a valuable contribution itself, in my opinion.

I also think that the overall level of presentation in this manuscript is high (some particular issues are discussed below, though), the related work is covered sufficiently, and I especially like the abundance of figures demonstrating the outcomes of alternative computational approaches and techniques. The discussed validation strategies are also valid, in my opinion.

The issues that I have with the manuscript are as follows:

1) One issue which might seem minor, but could lead to rather severe consequences and need for major revisions of the results, figures, etc. is related to the described methods of calculating counts for similarity cohesion computation.

- Section 6.2.2 states that for the Plane method, the count value is (N*(N-1)/2) for N utterances, while according to Algorithm 6 in Appendix A, it should be (N*(N+1)/2). Based on Figure 11 and further considerations, I tend to believe the version from Algorithm 6; the version with (N-1) would not lead to a proper count value for N=1, for instance.

- Section 6.2.2 states that for the Line method, the count value is (2N-3) for N utterances, while according to Algorithm 6 in Appendix A, it should be (2N-1). With similar considerations as above (also see Figure 12), I tend to believe the version from Algorithm 6.

Thus, the information about the formulae is contradictory, and while it might be simple typos in the main part of the text, the real question is which formulate were actually used for the authors’ implementation. This issue might require fixing the implementation and re-running the numerical comparisons with LCseg and creating updated figures for the manuscript.

2) I miss the discussion of the authors’ design process, and in particular, I am trying to understand whether any collaborating domain experts in discourse analysis were involved in the process of requirement formulation, design, and validation of the CSseg computational and interactive visual aspects.

3) In relation to the previous note, I would like to see further discussion of the (potential) applications of the proposed approach. Can CSseg be used by, e.g., journalists or social scientists on their own? Have the authors considered applying it to other sorts of conversational texts beyond debates? And so on..

4) I am also missing a number of details regarding visualization design and implementation. While the authors have discussed the motivation for using a conceptual recurrence plot (I believe so), how and why was a particular colormap used for the various utterance cells and also the similarity cells (green—white) in the matrix view? What are the interactions available to the user when working with CSseg — e.g., are brushing & linking supported for the matrix view vs the text view? Are there any additional details on demand, e.g., the particular quantitative values of concept similarity, etc. to be displayed when hovering or clicking on the matrix cells? A number of such details should be added to Section 7, for instance, which is very brief and lacking detail compared to several other sections. Furthermore, it is not clear how the interactive visual approach fits the framework illustrated in Figure 6.

5) I have several notes regarding the structure and contents that are more substantial then typos or other minor language issues:

- With regard to the discussion of main concepts and background work, I think the authors should introduce their operationalized definition of “discourse” directly at the beginning of Section 1.

- Research questions in Section 1 are not formulated as actual questions, which is somewhat annoying. I prefer the formulation in the caption of Figure 1, for instance, and would suggest that the authors update the RQs in Section 1 in a similar fashion.

- The background work on discourse and debate analysis in Section 2.4 is very important for the understanding of other related work on computational and visual approaches for the same/related problems, and thus, in my opinion, the authors should move Section 2.4 to the beginning of Section 2.

- The discussion of related work in the current Sections 2.1 and 2.2 is quite descriptive and it lacks a critical perspective—more specifically, the authors should state more clearly what the differences are between the discussed approaches and the proposed CSseg approach.

- The authors have used a lot of example figures from prior papers in Section 2, namely within Figures 2–5. While it is good to see the examples, I would urge the authors to consider the issue of obtaining reprint permissions from the respective copyright holders / publishers — perhaps, it is a better idea to approach the authors of several such tools and ask them to create new screenshots that do not fall under the copyright of previously published papers. In the worst case, even removing these figures from the manuscript altogether would be fine, in my opinion.

- When discussing the evaluation metrics P_k and WD in Section 8.2, the authors introduce them as “degree of agreement”, but it seems to me that smaller values of these metrics actually denote higher agreement/quality (the metrics thus act more as distance rather than similarity metrics). Please clarify this point in the manuscript.

Minor issues:

- Line 107: “it identifies” — it is not clear what “it” refers to in that sentence

- Lines 114–118: duplicate sentences starting with “They suggested...”!

- Line 166: “coherent166 segments” — this is an issue that also occurs several further times throughout the manuscript, probably caused by copy&paste issues, perhaps during the copy editing process

- Line 224: “thesemantic”

- Line 257: “analyzes 257 human”

- Line 316: “each speaker’s 316 utterance unit”

- Line 328: “the 328 main”

- Line 356: “Figure ??” !!!

- Figure 9 caption: “these represents”

- The plane(i,j) formula in Section 6.2.1.1: “, where ∀i, ∀j” — it seems that the ending of the formula might be missing, otherwise, I do not think these quantifiers are necessary to mention here…

- I believe Figures 11–20 are not called out from the text at all!

- Figure 14 caption: “used in the Plane method” — I believe this figure provides the results of the combined method (Plane + Line + Point) instead

- Line 450: “the combined “Plane + Line + Plane” method ” — “Plane + Line + Point”?

- Line 460: “In the case of the Line method, the count of conceptual similarities is 1” — I believe the Point method is discussed in this sentence, not Line

- Lines 636–638: “(a-2) can adjust the weights of the similarity cohesion calculation methods. In (a-2), the user can adjust the weights described in Section 6.2.1.” — repetitive…

- Lines 662–665: “662 The debates...” and several similar issues

- Line 680: “consists as shown”

- Line 685: “Figures 29, 30, 31” — should be Figures 23, 24, and 25!

- Line 812: “are applied, A controller”

- Lines 900–912: the two respective paragraphs are repetitive to a certain extent

- Line 952: “she introduce us her lab’s researches”

- Line 958: I believe there is no actual Appendix B in the manuscript, and the boilerplate instructions should be removed…

- With regard to bibliography, please check the references carefully — it is very weird that references 5 and 8 look different with regard to the font and formatting of journal details, for instance (this might have to do with Bibtex entry contents or errors…).

- [24]: “Technical report, Citeseer” — any further details about the original publication place / institution?

- [33]: “textiling”

- [40]: publisher information missing

- [54]: the paper is listed an Arxiv preprint (the first option on Google Scholar), but it was actually published back in 1995 at ACL, see https://aclanthology.org/P95-1015/ . Please check the other Arxiv references, especially older ones, and see if the respective papers were published...

To summarize, I think this work has merits, and I would recommend to consider it after a revision — and the scope of such a revision in my opinion is somewhere between a minor and a major one. If the authors recognize that my first point is unfortunately correct (and the formulae used for count calculations in the implementation contain errors), this would definitely mean a major revision with updates for multiple figures, tables, etc.

Author Response

Thank you for your interest in and review for our research and academic community, even in the midst of your work. In particular, we can feel that the review was conducted with affectionate interest. We would like to do our best to take the comments of reviewers into account. We don't know where the reviewers live, but we wish people in Europe and the world safety and peace.

The response letter is attached as a pdf file.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have justified/added my suggestions from the first round. In my opinion the quality of the paper has increased.

Author Response

We are delighted for your positive review and thank you for your dedication to our research and academic community until the very end.

Author Response File: Author Response.pdf

Reviewer 2 Report

I would like to thank the authors for their work on improving the manuscript and for a detailed cover letter. In the following, I will focus on my previous critical notes and requests and will comment whether the authors have addressed the respective issues, in my opinion.

1) Issue with the confusing formulae regarding count calculations: I believe the authors have resolved the confusion with the addition of the note for Algorithm 6.

2) Design process discussion: the authors have added a statement about potentially involving domain experts for future collaborations to Section 10, but in my opinion, a clear statement about the lack of involvement of such domain experts (if that is the case) during the design and implementation stages of this project should be added, e.g., to Section 1 or Section 3.

3) Further discussion of potential applications: I am satisfied with the additional clarifications made by the authors in Section 10.

4) Missing details about the visualization design and implementation: the authors have partially addressed my comment, but I am still missing further details in Section 7, e.g., whether the color map used in the implementation was chosen arbitrary or the choice was motivated by the previous work, for instance.

5) Further notes regarding the structure and contents of the manuscript: I am satisfied with the revisions conducted by the authors in this regard (or their well-motivated comments on declining my suggestions for the respective points).

6) Minor presentation issues: I am satisfied with the revisions made by the authors.

- However, note that line 477 in the revised manuscript still mentions “(Plane + Line + Plane)”! Should it not be “(Plane + Line + ***Point***)”?

Overall, I believe the quality of the manuscript has improved as the result of the revision. I will suggest a minor revision decision at this point so that the authors could fix some of the minor remaining issues and also consider adding further details, as mentioned in my notes above; but even the current version of the manuscript could potentially be published without major issues, in my opinion.

Author Response

We have appreciated your dedication to our research and academic community by carefully reviewing the paper until the very end. We have checked all the reviewer's comments and wrote revisions as follows.

Author Response File: Author Response.pdf