Testing for Level–Degree Interaction Effects in Two-Factor Fixed-Effects ANOVA When the Levels of Only One Factor Are Ordered

Round 1

Reviewer 1 Report

This paper contributes to the field of statistical analysis, particularly in the design and analysis of experiments involving ANOVA. Their innovative approach to dissecting interaction effects provides a powerful tool for statisticians dealing with complex experimental designs. One thing is that authors can add more references regarding the topic. 

Author Response

Thank you for reviewing our work. However the advice to authors is to only include what is relevant. In the first paragraph of the introduction we mention Box, Hunter and Hunter (2005), Kuehl (2000), Montgomery (1984) and others. Isn’t that enough?

Reviewer 2 Report

The abstract should be revised, as it should contain the object of study, objective, methodology and results/conclusion. As it stands, these aspects are neither clear nor explicit.

In the introduction, the authors cite an example taken from Wikipedia, which I find totally unacceptable for a scientific article of this nature.

In the introduction, the authors state (lines 31 to 33): “However we have found in the literature that it is unusual to consider unbalanced designs where there are different numbers of observations in the levels of the treatments, and rare to give any detail when the treatment levels are ordered, or to discuss contrasts for interactions.”. However, they make no reference to the literature mentioned, so this should be reviewed.

Also, in line 39, authors state: “In a sequence of papers the authors have endeavoured to give data analysts better tools for examining data sets from multifactor ANOVA models.”. However, they make no reference to the literature mentioned, so this should be reviewed.

In tables 2, 3, 5 and 6, the p-value should not be 0.000 (since the p-value is not actually zero, but very low). In scientific language, what should be put in these cases is <0.0001.

Also in the same table, there is no indication of what the 3 asterisks mean.

The state: “Those of degrees one, two and three have p-values 0.0000, 0.0000” since the p-value is not actually zero, but very low. The same happens on lines 216 and 224, which should be reviewed.

The paper does not have a "Conclusions" section.

The sentence construction, in general, should be improved.

Author Response

Response for the Second Reviewer

Thank you for the time and care you have taken to review our work. Your comments have been positive and helpful, and definitely improved the paper. In the following we make some general points, mainly for the editor, and then respond to your specific comments.

General comments

• The manuscript is titled an Article. Shouldn’t that be Case Report?
• Rayner’s affiliations are 1 and 2 while Livingston’s affiliation is 2.
• At line 128 and subsequently we have changed m_s to m_t, as s denotes the number of observations in each cell.
• At line 159 we have inserted the sentence

Throughout this article we use 100,000 permutations.

Subsequent statements about the number of permutations have been removed.

• The tables and figures have font sizes and types not consistent with the rest of the text and do not keep within the margins. We have not attempted to rectify these issues: your team is better at doing that than we are.
• We found that we had omitted

RAYNER, J.C.W. and LIVINGSTON Jr, G.C. (2023). An Introduction to Cochran–Mantel–Haenszel Testing and Nonparametric ANOVA. Chichester: Wiley. ISBN: 978-1-119-83200-3

from the list of references. This has been added as Rayner and Livingston Jr (2023b) and the other Rayner and Livingston Jr (2023) reference changed to (2023a) in the text and references.

The second reviewer makes several comments.

Reviewer’s comment. The abstract should be revised, as it should contain the object of study, objective, methodology and results/conclusion. As it stands, these aspects are neither clear nor explicit.

The current abstract follows.

In the two factor fixed effects ANOVA with the levels of one factor ordered and one not ordered, often there are no indications of any relationships between the levels of the unordered factor. Testing for main effects and their orthogonal contrasts is informative. For interaction effects, elsewhere we consider testing pairwise contrasts. Here in decomposing the interaction sum of squares for the balanced model, we find what we call level-degree interaction coefficients that may be used to test for the corresponding level-degree interaction effects. These give clear insights into what may be causing a significant interaction, even for the unbalanced model.

Revision. Since it isn’t a focus, the mention of relationships between the levels of the unordered factor can go. We suggest the following.

In testing for main effects, the use of orthogonal contrasts for balanced designs with the factor levels not ordered is well-known. Here we consider two factor fixed effects ANOVA with the levels of one factor ordered and one not ordered. The objective is to extend the idea of decomposing the main effect to decomposing the interaction. This is achieved by defining level-degree coefficients and testing if they are zero using permutation testing. These tests give clear insights into what may be causing a significant interaction, even for the unbalanced model.

Reviewer’s comment. In the introduction, the authors cite an example taken from Wikipedia, which I find totally unacceptable for a scientific article of this nature.

Response. If the editor wishes to do so, we have no objections to the sentence

See, for example, https://en.wikipedia.org/wiki/Contrast_(statistics)

being removed. The reference is given as evidence for the preceding statement. When we googled this topic, we found a number of accounts consistent with our claim; however we prefer to not point to such material since we are making a negative point: suggesting it could go further. Of course, a taste only may well be appropriate in many of these settings. Personally, we have no problem pointing to Wikipedia as we suspect many users of statistics use it in the course of their investigations.

Incidentally, in the version downloaded the url font type and size were not consistent with the rest of the text.

Reviewer’s Comment. In the introduction, the authors state (lines 31 to 33): “However … interactions.”. However, they make no reference to the literature mentioned, so this should be reviewed.

Response. We have already given several references in the first paragraph and we have no wish to criticise their work explicitly. In general what they did was appropriate for their objectives. We are mainly pointing out that what is in this paper is more than what is in well-respected sources.

Reviewer’s Comment. Also, in line 39, authors state: “In a sequence of papers the authors have endeavoured to give data analysts better tools for examining data sets from multifactor ANOVA models.” However, they make no reference to the literature mentioned, so this should be reviewed.

Response. The next three paragraphs discuss that literature. The first of these starts with “Thus …”.

Reviewer’s Comment. In tables 2, 3, 5 and 6, the p-value should not be 0.000 (since the p-value is not actually zero, but very low). In scientific language, what should be put in these cases is <0.0001. Also in the same table, there is no indication of what the 3 asterisks mean.

Response. Thank you, we have attended to those p-values and asterisks.

Reviewer’s Comment. The paper does not have a "Conclusions" section.

Response. We have added a new section 6.

6. Conclusion

Orthogonal contrasts have been used for some time to decompose an omnibus test statistic into focused components. This approach enables a significant effect to be better understood, and, occasionally, finds a significant effect that has been masked by a non-significant one. In the ANOVA contrasts are usually met for balanced fixed effects models in which the levels of the factor of interest are not ordered. To construct appropriate contrasts requires knowledge of which levels are most usefully compared with which. Usually contrasts are considered for main effects; interactions seem to have been rarely considered.

For two factor interactions in which the levels of both factors are ordered Rayner and Livingston (2023a) found orthogonal contrasts than can be used to test for focused interaction effects. When the levels of at least one factor are not ordered, Livingston Jr and Rayner (2024) address the issue of not knowing which levels should be compared with which by considering pairwise contrasts that compare every level with every other level.

Here we have considered two-factor fixed effects ANOVAs with the levels of one factor ordered and one not ordered. Level-degree coefficients have been defined and are such that if all such coefficients are zero then so is the interaction. Testing for such coefficients to be zero using permutation tests gives nonparametric focused tests. While these coefficients are not contrasts, they do enable a significant interaction effect to be better understood, and they may find significant effects that have been masked by non-significant ones.

Round 2

Reviewer 2 Report

I believe that all considerations have been taken into account, so I have nothing more to add. Well done!

Only minor linguistic details should be reviewed when producing the final document.