Extracting Proceedings Data from Court Cases with Machine Learning
Round 1
Reviewer 1 Report
1) The literature review conducted in Section 2 is very terse. More discussions of recent articles should be added. I suggest creating a separate section called 'Related Works'
2) A Table should be added to Section 2 to summarize and compare existing studies in terms of different aspects, etc. used machine learning models, used datasets, pros, and cons, etc.
3) A flowchart that clearly explains the methodology proposed in the manuscript should be added in the 'Methodology section'
4) The limitation of the proposed method should be described in the Conclusion before deriving future work.
5) The authors are invited to carefully proofread their articles to correct typos and grammatical issues.
Author Response
Please see the attachment
Author Response File: 
Author Response.docx
Reviewer 2 Report
In general, this is an interesting manuscript, though I encourage the editor to find additional reviewers with expertise specifically in NLP methods like NER, since that is not my area of expertise. In general, my ability to evaluate the manuscript was also somewhat hampered by linguistic editing issues beyond the scope of what I can correct in a review of this type. Especially since the topic involves a number of technical terms, incorrect translations, ambiguous word use, and especially the frequent choice of antiquated or extremely uncommon English words which happen to have French cognates hampers understanding of the methods applied. Section 2.3 is a good example of this problem, with many components being very difficult to parse.
Section 2.4 - if the sentence is the basic learning unit, then is care taken to make sure the training and test data sets are further divided by document/case/participant? It seems that failure to do so would lead to over-optimistic performance characterization, particularly if the algorithms can simply learn case-specific outcomes.
More should also be said here about the actual methods applied, since the authors use a proprietary tool. It would be difficult for someone to replicate this analysis from what is written alone without using the same Kairntech tool.
Author Response
Please see the attachment
Author Response File: Author Response.docx
Reviewer 3 Report
My largest point of feedback deals with the presentation of the methodology and results. There's a lot going on and it is difficult to track it all. Here are some suggestions:
* more descriptive captions of tables and figures
* some footnotes to tables were missing (see Table 5 - asterisks on "seconds to annotate one decision *"
* More use of footnotes to explain details would help non-legal readers. For example:
- Table 2, does a check mark indicate that the label is required to be included in that type of court proceeding or that it was found? If "found", then is Table 3 the number of times it was found. And does that imply that there were proceedings from those courts where the label was not found?
- Table 3: are those numbers the number of times that label was found in those types of proceedings? Would percentages make sense in that table?
I'm not familiar with legal proceedings. It would be helpful to clarifying a few points. 1) Is a "decision" equivalent to a case document (I'm referring to p. 9 Table 3 last row)? What's a segment (Table 3 last row)? Maybe an image describing the structure of a legal proceeding document or the process would be helpful for non-legal readers.
I'm not sure how helpful Figure 2 is...could you put numbers in a table? Is the purpose of Figure 2 to highlight the labels that didn't perform well?
Author Response
Please see the attachment
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The authors have addressed all my comments, I have no further recommendations. The paper is ready for publication.
Reviewer 2 Report
My questions have been addressed and the language has been greatly improved.
