Review Reports

Round 1

Reviewer 1 Report

This article discusses some of the (statistical) errors made in two serial killer nurses. It is meant for a broad audience (at least that is what I assume).

my comments: 

1. The article is indeed accessible for a broad audience - and in this sense it really does a good job in explaining the role of statistics and the risks of flawed reasoning in legal cases - except for the use of a few terms that are not explicated. I marked them in the text. 

2. abstract: in Dutch law there is no such thing as declaring innocence. Lucia de Berk was acquitted, no more and no less. 

3. as the authors no doubt know, there is no such thing as 'direct' evidence and it is vague what they mean by hard evidence. similarly, there is no such thing as neutral evidence (only in relation to hypotheses compared). This should be reformulated. see my comments in the manuscript. 

4. The main issue I have with the text is that the analysis is presented in less neutral terms than is fitting for an article in a refereed academic journal. At many points it shows too much that one of the authors worked for the defence of both nurses. I marked some sentences in the manuscript (among others the way Elffers is presented). This should be reformulated. 

Comments for author File: Comments.pdf

Author Response

Main issue was lack of neutrality in the paper. We have rewritten the paper in as neutral (factual) tone as possible; in particular: we have removed ad hominem remarks as much as possible.

 

Explanation of technical (statistical) terms has been improved.

 

The reviewer states “ there is no such thing as 'direct' evidence and it is vague what they mean by hard evidence. similarly, there is no such thing as neutral evidence (only in relation to hypotheses compared)”. Of course we agree that these terms must be relative to a particular context and depend on many assumptions about ``how things work’’ in the real world in general, as well as how they possibly worked in the case under study. We do not wish to suggest that they have some absolute meaning. We do think that though these words do not correspond to formal and precisely defined concepts in legal scholarship or in rules of admissibility of evidence in any particular jurisdiction, they do correspond to much used concepts both in science, in society, in common speech. The modern theory of causality, moreover, formalises the concept of direct versus indirect evidence. If one interprets a Bayes net as a causal model then the causal structure expressed by the graph, together with the sizes of the conditional probabilities entered into the probability tables, allow one to label evidence as indirect or indirect, (relatively) hard or soft, (relatively) overwhelming or weak or neutral. The causal model expressed by a DAG is a story of “how things work”. It has to be supported by argument and facts; both common sense, and scientific. The conditional probabilities used in calculations based on that model similarly have to be supported by arguments.

 

We do not argue that a causal model must be interpreted in a causal way. However, as a model, it can be interpreted in a causal way.

 

We are also not saying that fact-finders must use Bayesian reasoning. We do believe that it is very interesting if a fact-finder’s conclusion contradicts Bayesian reasoning, if that Bayesian reasoning is based on arguments and assumptions and facts which the fact-finder cannot dispute. An advocate (for either side of a criminal case) can use a hypothetical causal model either to support or to fight the reasoning proposed by the other side, and we pray that the laws of admissibility of evidence in any particular jurisdiction allow “meta evidence”: evidence showing that certain reasoning is invalid because inconsistent whether with logic or with indisputable facts.

 

According to the subjective interpretation of probability, the conditional probabilities of one proposition given some others are numerical expressions of how much one should believe one thing, given whether or not some other things are true. “Fact finders” in criminal cases are actually decision makers: judges and juries decide whether or not a person is to be punished for an alleged crime. They make their personal decisions after hearing evidence. Juries do not justify their conclusions. Juries are swayed by evidence and by their feelings, conscious or unconscious, of whether evidence is hard or soft, direct or indirect, strong or weak or neutral. We hope it is not forbidden for legal scholars to use these words.

Reviewer 2 Report

Comments on the blinded manuscript “Statistical Issues in Serial Killer Nurse Cases”

General comments:

The paper describes some cases of nurses accused of having killed patients during their duties. In my opinion this is really interesting and, as the authors claim within the introduction, “ is vitally important that the academic legal community learns about these cases”. Personally I think that the manuscript should be improved in order to be published. First I think that the authors should concentrate a little bit more on the statistical approach by providing some additional details regarding the statistical reasoning used within the different cases and keeping in their mind that the potential readers of the contribution may not be confident with methodological statistics. Secondly, the structure of the paper is a little bit confusing. E.g. Lucia de Berk’s case is introduced in Section 3 and then concluded in Section 5. Similarly, Ben Geen’s case is introduced in Section 4 and then concluded in Section 8. I think the authors should think about a different structure which would make the article easier to be read.

ˆ Page 4, Figure 1. The authors should explain the type of computation that a statistician usually implements in presence of such type of table. Additionally, the should explain that proving that there is an association among the variables “Lucia on duty” and “Incident in shift” does not allow to claim the existence of a causal effect among those variables.

ˆ Section 4: In my opinion this section is way too long. The authors should concentrate more on the statistical details instead of providing such long description of the story (which for sure is interesting but a little bit out of topic considering that the article is supposed to be focussed on the statistical issues.). Furthermore I don’t understand why the authors first introduce Ben Geen case, then, in the further section they talk about the hindsight on Lucia de B. and then, by the end of the paper, they talk once more about Ben Geen. I think the authors should structure the paper in a more consistent way, talking separately about the different cases and then providing the methodological insights as the Bayesian Analysis and the Confirmation bias. 1

ˆ Page 10: Within the section regarding the Bayesian analysis the authors should explain, briefly, something more about Bayesian statistics and provide some additional information regarding the DAG representation. Secondly the authors should explain how Bayesian analysis could have been used for analysing the data of each of the presented cases

Comments for author File: Comments.pdf

Author Response

The reviewer wanted a different structure to the paper and more details on statistical methodology. We have completely restructured the paper by separating the two cases (Lucia de Berk, Ben Geen) and by making it clear that actually we are not comparing statistical analyses performed in those two cases, but comparing the manner in which underlying evidence was gathered. Our paper is not about the statistical study of the numbers of deaths or other incidents when a particular nurse is present. It is about deciding whether or not deaths or other incidents are suspicious — whether or not they should be counted at all.

 

The reviewer wanted the Bayesian analysis better explained, and we have done that. The reviewer also wanted the authors to explain how the analysis could have been used to analyse each presented case. Actually, we do not know how to do that. It would be a matter for further research to develop meaningful Bayesian analyses of either case, and in fact, we strongly feel that this should be attempted as soon as possible, for the Ben Geen case. However, it is an extremely daunting task, not least because so much data is not available. In as far as we are advocates of Ben Geen’s innocence, the aim of our paper is to make it plausible that he was not proven guilty beyond reasonable doubt;. We argue that the manner in which evidence was gathered against him was possibly illegal. Certainly, it was objectively very seriously flawed.

 

Our Bayesian analysis had a pedagogical aim: namely, to show that absence of evidence can well be evidence of absence. Another aim was to show the importance of the base rate, and to exemplify the adage “extraordinary claims require extraordinary evidence”.

 

We hope that the purpose of our paper is now much more clear. 

Reviewer 3 Report

This article is clearly fits the aims of Laws, as it bridges legal and statistical views of evidence and challenges injustices arising from poor evidence. There has been considerably improvement in collaboration between statistical, legal and law enforcement agencies. This article should contribute to further improvements.

Page 3, lines 102-106. I see this time, Fenton only credits himself. I wonder whether there is recognition of my work, which has been used by Gill.

Page 11, lines 466-473. This should be re-written, particularly as Lagnado's 2021 work is refered to with respect to confirmation bias.  It is not a matter for hope, but for proper professional conduct. First, discussions in a multi-disciplinary team to agree a list of questions. Then ensure that  good statistical methods are used to obtain data and assess its quality.  The assumptions to be evaluated must be stated explicitly, before the relevant statistics, whether p-values, Bayes factors or others, are calculated.

Page 11, line 480. It would be sensible to provide a reference to an accessible description of DAGs, and to give the name in full.

I wonder how comprehensible the discussion of prior and posterior probabilities is to non-statistical readers.

Page 14, lines 580-584. Some discussion of whether a professional statistician should simply answer the questions asked should be included. If a question is ambiguous, or badly posed, it ought to be challenged.

Page 17, section 'Confirmation bias'. Include some discussion of the many other biases which are relevant. I recommend referencing David Sackett's work, as a pioneer in evidence-based medicine.  author="DL Sackett", title="Bias in analytical research", journal="J Chronic Dis", year="1979", volume="32", pages="51-63".  

Page 19, line 783 'pro deo' or 'pro bono'. God knows!

Author Response

The reviewer makes a number of very good comments. We have addressed some of them in our revision, others are answered by the Royal Statistical Society booklet on statistical issues in serial killer nurse cases, which will be published very shortly; and our paper refers to that document for practical recommendations to investigators and lawyers involved in such cases in the future. Our paper is a case-study which criticises the way in which a notable past case was investigated and prosecuted. We would like other legal scholars to follow up our work with more study of other aspects of the Ben Geen case (and we intend in the future to further work on “opening it up” for further academic study). Regarding lessons for the future, the RSS hand-book contains both a comprehensive survey of dangers as well as advice for all parties involved in such cases in general, of how to avoid the dangers. Our revised paper refers to the RSS hand-book, and contains just a few specific hints and references.

 

The referee is not sure whether “pro deo” or “pro bono” is the correct term. Apparently both phrases mean the same thing, one being used more commonly in Anglo Saxon jurisdictions, the other on the continent of Europe where a Napoleonic system has been adopted.

Round 2

Reviewer 2 Report

The structure has been modified as I suggested. In  my opinion the manuscript can be published in the present form

Reviewer 3 Report

Thanks for an interesting read.