BetaBayes—A Bayesian Approach for Comparing Ecological Communities

Round 1

Reviewer 1 Report (Previous Reviewer 1)

Thank you for this update and improved version of the manuscript. I think you have fulfilled most of my concerns properly. However, I still find some weaknesses in your manuscript that should be fixed before acceptance for publication.

L57: “anticorrelated” sounds weird, could you mean “negatively correlated”?

L68: Remove one of the closings squared brackets “]”.

L166: “σs”, the ending “s” should be lowercase.

L178-179: Here you explain the difference between Bradley-Terry models and your approach, specifically about the sign. Is there any reason or implication for such change when studying biological communities?

Figure2: Grid facets should be identified with the letters in the legend (a, b, c, and d).

L273-279: There is some confusion on the actual numbers and units in the transformation. Please double check. Furthermore, it is unclear why these numbers. It would be good to know why you chose 10 km and 100 m.

L286-294: Slope numbers reported here doesn’t match those reported in figure 2(3). Furthermore, given the fact that the variables were transformed, I think your interpretation of changes of bray-curtis through the environmental gradients is not entirely correct. Please double check. In fact, I think the change should be about 36% every 10 km. 36 % every 1 m sounds unreasonable. Some for elevation.

Figure “2”. That would be the second “figure 2” of the document. I think it is actually “Figure 3”. However, this figure is not cited in the text. Change it to make sure the figure is referenced in the text. Furthermore, the figure legend has capital letters after commas and should be lowercase. For instance, “, B) Density…” should be “, B) density…”

L331-336: However, GDM uses non-negative regression to ensure always increasing parameters. However, here you specify “w1” and “w2” as Normal(0,10), which could potentially draw negative coefficients. Isn’t it? How could this affect the model specification?

L346: Unclear what “generative models” means. Please clarify and/or provide an example.

Author Response

Rev1 - L57: “anticorrelated” sounds weird, could you mean “negatively correlated”?

We replaced “anticorrelated” with “negatively correlated”(L89).

Rev1 - L68: Remove one of the closings squared brackets “]”.

Done.

Rev1 - L166: “σs”, the ending “s” should be lowercase.

Done.

Rev1 - L178-179: Here you explain the difference between Bradley-Terry models and your approach, specifically about the sign. Is there any reason or implication for such change when studying biological communities?

We significantly expanded the section in which we explain the connection between BetaBayes and Bradley-Terry models (L217-246). As for the sign, we chose to use the sum because it lead to better inferential performance. We added this explanation to the text (L237-238).

Rev1 - Figure2: Grid facets should be identified with the letters in the legend (a, b, c, and d).

Done.

Rev1 - L273-279: There is some confusion on the actual numbers and units in the transformation. Please double check. Furthermore, it is unclear why these numbers. It would be good to know why you chose 10 km and 100 m.

We thank Reviewers 1, 2 and 3 for pointing out this section was unclear and, in some places, wrong. We chose these two values because 10 and 100 are round numbers that make it easier for the reader to interpret the corresponding coefficients. We improved the explanation of how the data transformations facilitate the interpretation of the model’s coefficients (L363-369).

Rev1 - L286-294: Slope numbers reported here doesn’t match those reported in figure 2(3). Furthermore, given the fact that the variables were transformed, I think your interpretation of changes of bray-curtis through the environmental gradients is not entirely correct. Please double check. In fact, I think the change should be about 36% every 10 km. 36 % every 1 m sounds unreasonable. Some for elevation.

We thank the reviewer for pointing this out. Panel C from Figure 2 was wrong. We also corrected the interpretation for the coefficients (L363-369).

Rev1 - Figure “2”. That would be the second “figure 2” of the document. I think it is actually “Figure 3”. However, this figure is not cited in the text. Change it to make sure the figure is referenced in the text. Furthermore, the figure legend has capital letters after commas and should be lowercase. For instance, “, B) Density…” should be “, B) density…”

Done.

Rev1 - L331-336: However, GDM uses non-negative regression to ensure always increasing parameters. However, here you specify “w1” and “w2” as Normal(0,10), which could potentially draw negative coefficients. Isn’t it? How could this affect the model specification?

GDM only works if dissimilarity increases monotonically with environmental distance. When using B-splines, BetaBayes does not have this limitation, as it can model highly non-linear relationships. Therefore, it makes sense to choose a prior distribution that is compatible with this flexibility.

Rev1 - L346: Unclear what “generative models” means. Please clarify and/or provide an example.

We re-wrote the conclusion and now provide a better explanation of what is a generative model. In this context, it means we can use the model to generate data from the posterior distribution that we can compare with the observed data to assess model fit (L467-470).

Reviewer 2 Report (New Reviewer)

This manuscript describes BetaBayes, a method for relating changes in beta diversity to environmental factors while taking account of non-independence of the calculation of beta diversity among communities. The authors compare BetaBayes to two other popular methods, Mantel tests and generalized dissimilarity modeling (GDM). The paper begins with a nice review of Mantel tests and GDM, along with strengths and weaknesses of each. Overall I find the paper to be well written and easy to read, including sufficient information in the methods and results. I provide below a few major and minor comments which I hope will improve the manuscript.

Major comments

Introduction - It's not quite clear from the introduction why non-independence of beta diversity indices among communities would be a problem for relating changes in community composition to environmental factors. A few more sentences here to justify and provide rationale for the new method would be useful. For example, the rationale for BetaBayes on L137-141 makes a lot of sense to me, could this be summarized here to help scaffold the expectation of the reader?

Simulation study - When new methods are proposed I find it much easier to evaluate their performance when a simulation study is presented. This allows to control for the _known_ structure of the data and evaluate the performance of the model given perfect information about how it _should_ perform. This could be quite a bit more work, so I will leave it to the editor and the authors whether this is something worth doing.

Direct comparison of methods - For the empirical example, Mantel test, GDM, and BetaBayes all identify strong effects of both geographic distance and elevation. If all these methods arrive at more or less the same conclusion it's hard for a reader to understand how BetaBayes "improve(s) upon GDM and Mantel tests" (L344). Unless I missed it, I do not find any discussion of the advantage of BetaBayes in terms of its performance on the real data.

Minor comments

L32 - "... BetaBayes provides a step towards consistently modelling community composition changes…" <- This is a little vague as a conclusion. Could the nature of the improvement BetaBayes makes be made more clear?

L199-205 - It's mentioned here that the model validation will involve 'checking for patterns' but this seems a little informal. It would be nice to have more explicit detail about how the differences between these distributions will be quantified.

L229 - First occurrence of 'Bray-Curtis' dissimilarity. This could use a reference. It also might be nice to clarify the difference between BC and Sorensen, for unfamiliar readers.

L275 - The nature of the transformation of the covariates is not quite clear.

L281 - "retrodictive"? I don't know this term, and it's not used otherwise in the manuscript.

L290 - "This result means that when geographical distance increases by 1 meter beyond a threshold of 10 km, the Bray-Curtis index increases by 36.6%." Where does the '1 meter' increment come from here?

L299 - The caption says 'precipitation difference', but the figure shows 'Elevation slope.' I believe elevation is what is tested in the main text.

Author Response

Rev2 - “Introduction - It's not quite clear from the introduction why non-independence of beta diversity indices among communities would be a problem for relating changes in community composition to environmental factors. A few more sentences here to justify and provide rationale for the new method would be useful. For example, the rationale for BetaBayes on L137-141 makes a lot of sense to me, could this be summarized here to help scaffold the expectation of the reader?”

We improved the section in which we explain why it is important to consider the pairwise dependence (L65-72).

Rev2 - Simulation study - When new methods are proposed I find it much easier to evaluate their performance when a simulation study is presented. This allows to control for the _known_ structure of the data and evaluate the performance of the model given perfect information about how it _should_ perform. This could be quite a bit more work, so I will leave it to the editor and the authors whether this is something worth doing.

BetaBayes is based on the Bradley-Terry model, which is a popular and time proven framework for modelling paired comparisons. The Bradley-Terry model is supported by multiple simulation studies that extensively tested its performance under multiple dependence scenarios. In the previous manuscript, we did not make this clear. In section 3.1 we added a section where we explain that Bradley-Terry models are widely used outside the ecological literature L217-222 and in the Conclusion we now mention that the Bradley-Terry model has been extensively tested in simulation studies (L457-459).

Rev2 - Direct comparison of methods - For the empirical example, Mantel test, GDM, and BetaBayes all identify strong effects of both geographic distance and elevation. If all these methods arrive at more or less the same conclusion it's hard for a reader to understand how BetaBayes "improve(s) upon GDM and Mantel tests" (L344). Unless I missed it, I do not find any discussion of the advantage of BetaBayes in terms of its performance on the real data.

We rewrote the Conclusion and no provide a more in depth explanation of what sets BetaBayes apart from alternative methods (L464-483).

Rev2 - L32 - "... BetaBayes provides a step towards consistently modelling community composition changes…" <- This is a little vague as a conclusion. Could the nature of the improvement BetaBayes makes be made more clear?

We re-wrote the final section of the manuscript and now include a “Conclusion” where we elaborate on the unique features and advantages of BetaBayes compared to alternative methods (L457-483).

Rev2 - L199-205 - It's mentioned here that the model validation will involve 'checking for patterns' but this seems a little informal. It would be nice to have more explicit detail about how the differences between these distributions will be quantified.

We now explain which types of patterns we expect see in plots of “observed values vs predicted values “ and “residuals vs covariates” if the model fits the data well (L271-278).

Rev2 - L229 - First occurrence of 'Bray-Curtis' dissimilarity. This could use a reference. It also might be nice to clarify the difference between BC and Sorensen, for unfamiliar readers.

Done.

Rev2 - L275 - The nature of the transformation of the covariates is not quite clear.

Following a similar comment by Reviewer 1, we corrected the section in which we explain how the data transformation improves the interpretation of the model’s coefficient’s (L366-373).

Rev2 - L281 - "retrodictive"? I don't know this term, and it's not used otherwise in the manuscript.

By retrodictive, we meant the distribution of predicted indices according to the model. We removed the term because it may be unfamiliar to the readers.

Rev2 - L290 - "This result means that when geographical distance increases by 1 meter beyond a threshold of 10 km, the Bray-Curtis index increases by 36.6%." Where does the '1 meter' increment come from here?

We thank the reviewer for pointing this out, as this sentence was wrong. We corrected the sentence according to this comment and to a similar one made by Reviewer 1 (L366-373).

Rev2 - L299 - The caption says 'precipitation difference', but the figure shows 'Elevation slope.' I believe elevation is what is tested in the main text.

Done.

Reviewer 3 Report (New Reviewer)

This article proposes a new method to model beta diversity - BetaBayes - that improves existing models aiming to understand the drivers of changes in species composition through dissimilarity modelling. Modelling dissimilarities has been a long-standing interest in ecology but ecologists were faced with many statistical difficulties, mainly due to the nature of dissimilarities and of distance-distance relationships. Here, the authors extend an existing method (namely Generalised dissimilarity models) with the aim of directly and explicitly accounting for the non-independence of community dissimilarities in their modelling framework, which is one of the main issues in this kind of approaches. Overall, the article is well-written and structured, the methods are clearly presented with an accessible language for non-specialists and I also thank the authors for providing all the code for their analyses in the Appendix. However, I think that the necessity and value of the improvements described here are not sufficiently highlighted, which left the reader with many questions regarding the use and relevance of this new method. I highlight below my main comments in which I suggest that (1) an important part of the literature regarding beta diversity modelling is left apart in the context provided to the reader, which questions the relevance of the benchmarks used here to demonstrate the efficiency of the new approach proposed by the authors, (2) the way the comparison to existing methods is performed and the absence of simulation tests weaken the overall demonstration, despite the good description of why theoretically BetaBayes should be better, and (3) the discussion does not help the readers to make sense of the results presented here and of their whole significance in terms of future modelling strategies, despite some interesting perspectives. Overall, more justification and tests appear needed from my perspective for this new method to be published and most of all, for it to make a successful entrance in the toolbox of ecologists.

## Major comments

1. The lack of context on the whole breadth of methods available to model beta diversity could limit the uptake of the approach by ecologists

The context/intro is only focused on dissimilarities modelling techniques, but forget many other methods perhaps even better suited for beta diversity modelling (d'Amen et al. 2017, Pollock et al. 2020 TREE, or Viana et al. 2022 for reviews and categorization of such methods). While this is surely a step forward compared to GDM, it does not deal with all the issues associated with modelling dissimilarities (e.g. Legendre et al. 2015). Given the difficulties of modelling dissimilarities (non-independence of the dissimilarities, handling autocorrelation, dispersion issues in distance-distance relationships), this lacks of context makes it unclear why we should use this approach compared to using modelling techniques based on raw data, that directly model species composition (as suggested by Legendre et al. 2015). Directly using RDA with appropriate data transformation (Legendre and Gallagher 2001), or jSDM (Warton et al. 2015) could seem better approaches to distinguish natural from anthropogenic-driven biodiversity changes than dissimilarity-based approaches. For example, methods such as redundancy analysis (RDA) or distance-based RDA (using PCOA axes as responses variables) have been shown to have greater power than Mantel-based approaches (Legendre et al. 2015, Guillot et al. 2013), especially when coupled with MEM (Moran's eigenvector maps, Dray et al. 2006) for spatial analysis. Similarly, jSDM have been advocated compared to dissimilarity-based approaches (Warton et al. 2011, 2015). From my perspective (which might be biased) these approaches have several advantages and few drawbacks when comparing with dissimilarity-based approaches, so why a reader like me should switch to BetaBayes? I can see the interest of modelling beta diversity components (turnover and nestedness) explicitly, but otherwise, methods giving access to the species contributing most to beta diversity and directly predicting species composition rather than dissimilarities seem more interesting for management and conservation purposes. I would need more arguments about the pros and cons of this approach to be convinced to use it. 

2. The pros and cons of BetaBayes are not clearly highlighted by the methodological comparison provided, which appears insufficient to support the values of this new approach.

I feel that the comparison provided here is not enough to valorise BetaBayes. I question (i) the choice of the selected benchmarks, (ii) the way this methodological comparison is performed and discussed, and (iii) the absence of thorough simulation tests to complement that case study.

(i) My first major comment above suggests that Mantel test and GDM alone may not be sufficient benchmarks if the aim is to provide a new tool for beta diversity modelling. But if appropriate context is provided to justify that choice, it may be enough to focus on dissimilarity-based approaches. I question though, the choice of the Mantel test as the first benchmark, given the criticism around it (e.g. Legendre et al. 2015., Guillot et al. 2013), the latter being recognized in Section 4.1. (l. 71-72). As the test dataset concerns in part spatial distances, I suggest at least to control for autocorrelation in that approach using the latest permutation test developped : Crabot, J, Clappe, S, Dray, S, Datry, T. Testing the Mantel statistic with a spatially-constrained permutation procedure. Methods Ecol Evol. 2019; 10: 532– 540. https://doi.org/10.1111/2041-210X.13141

Even when compared to GDM, it may not be crystal clear for an average reader why it is that important to directly model the non-independence (compared to controlling it by randomization as in GDM). What influences does it have on parameter estimation and on the conclusions one would draw from these two methods? While the differences of the BetaBayes approach compared to classical GDM are clearly presented, the expected advantages in terms of parameter estimation, model fit, violation of assumptions, etc... are not that clear, and the examples does not help filling that void. In terms of discussion, the section on the possible extensions is interesting and well-written, but lack some perspectives on the comparison, helping the readers to make sense of the different results and of what it means in terms of the capacity of the BetaBayes approach compared to the other ones tested. 

(ii) Indeed, I find that the way the test dataset is analysed, and how the results are discussed, does not justify enough the value of the method proposed here. The lack of a discussion section summarizing and comparing the results of the different approach is limiting in that sense, because it is up to the reader to do that analysis, and to understand on its own the consequences of the differences between the approaches. 

Moreover, the only thing the reader can really compare across the methods is the significance and relative importance of each predictor. R2 or pseudo-R2 are not presented for all methods, observed vs predicted plots can be compared by the reader when looking at the appendix for BetaBayes, but the differences between them are not even mentioned in the text, and there is nothing about the type 1 error of this new approach compared to the original GDM (see below for that point). Based on that, and without any real discussion on it, it is hard for an outside reader to say if this approach is really better, except for the theoretical arguments presented in the intro. Also, it is hard to compare BetaBayes and GDM since you present a linear version of BetaBayes and not a non-linear one. I understand why starting with a simpler model is important for the reader but if you want to really compare what BetaBayes brings compared to GDM, I think you need to compute the most similar models possible. Appendix p5 you say about GDM that "The model explains 52.51% of the deviance" How much of the deviance does the BetaBayes explains with this linear formulation? How much would it explain with splines? Where do we expect a difference with GDM given the addition of these intercepts in BetaBates: in the pseudo-R2 ? In the response curves? In the significance? In the shape of the residuals? This is not clear from that comparison.

Also, this example does not show how other flaws of dissimilarities-based approaches affect BetaBayes: autocorrelation, heteroscedasticity... Autocorrelation is an important issue, especially as you study the importance of spatial distances. Hence, as suggested above, the extension of Mantel tests should be used instead, and a proper section should be dedicated to how betabayes responds to autocorrelation. In terms of fit and heteroscedasticity, the plot given in the appendix does not suggest any improvement compared to GDM (perhaps because of the linear constraints you chose here, but perhaps not only...). You say l. 282 "The posterior retrodictive distribution of Bray-Curtis indices closely matched the observed distribution, except for values below 0.39, which are slightly overestimated (Fig 2B, and Appendix 1)." Given the figures in Appendix p 11 (top right panel and middle left), the overestimation is not that little. The model clearly underestimates the extent of the turnover gradient. There is also from the middle right panel and the bottom left panel strong indications of heteroscedasticity in the residuals that show that the unique funnel-shape of distance-distance relationship is not adequately modelled here. Indeed, it is typical of distance-distance relationship to show different variances along the plot (high variances near 0 distances, and lower variance towards 1 distance, as observed in your case in appendix p6 or in Fig 4 of Wetzel et al. 2012 for example, or the other way around, see for e.g. Fig. 4 from Mclean et al. 2021). If the variance structure is not captured properly, the model cannot be adequate as it mixes up dispersion and location effects potentially (see for e.g. Warton et al. 2012 for similar arguments on categorical explanatory variables).

Warton, D.I., Wright, S.T. and Wang, Y. (2012), Distance-based multivariate analyses confound location and dispersion effects. Methods in Ecology and Evolution, 3: 89-101. https://doi.org/10.1111/j.2041-210X.2011.00127.x

Wetzel CE, Bicudo DdC, Ector L, Lobo EA, Soininen J, et al. (2012) Distance Decay of Similarity in Neotropical Diatom Communities. PLOS ONE 7(9): e45071. https://doi.org/10.1371/journal.pone.0045071

McLean, M., Stuart-Smith, R. D., Villéger, S., Auber, A., Edgar, G. J., MacNeil, M. A., ... & Mouillot, D. (2021). Trait similarity in reef fish faunas across the world’s oceans. Proceedings of the National Academy of Sciences, 118(12), e2012318118.

Thus, another section should be dedicated to how betabayes deal with issues of dispersion and heteroscedasticity in the residuals.

(iii) Beyond a more thorough analysis and discussion of the example provided, all these points show, in my opinion, the need for a proper simulation study to really test the value of BetaBayes. For example, quantifying the Type 1 error with or without spatial autocorrelation seems a prerequisite before publishing a new method to study distance-distance relationships e.g. Franckowiak RP, Panasci M, Jarvis KJ, Acuña-Rodriguez IS, Landguth EL, et al. (2017) Model selection with multiple regression on distance matrices leads to incorrect inferences. PLOS ONE 12(4): e0175194. https://doi.org/10.1371/journal.pone.0175194

## Minor comments

l. 43 - Citation of ref 7 does not seem appropriate here: variation partitioning using RDA or dbRDA is not quite like mantel test or GDM. Although the purpose is also to model beta diversity, it does not suffer from the same issue than distance-distance relationships (see Legendre et al. 2015). A more suitable ref to convey the idea of the sentence would be something like : Graco-Roza, C., Aarnio, S., Abrego, N., Acosta, A. T. R., Alahuhta, J., Altman, J., Angiolini, C., Aroviita, J., Attorre, F., Baastrup-Spohr, L., Barrera-Alba, J. J., Belmaker, J., Biurrun, I., Bonari, G., Bruelheide, H., Burrascano, S., Carboni, M., Cardoso, P., Carvalho, J. C., … Soininen, J. (2022). Distance decay 2.0 – A global synthesis of taxonomic and functional turnover in ecological communities. Global Ecology and Biogeography, 31, 1399– 1421. https://doi.org/10.1111/geb.13513

l. 44 - "For instance, if we calculate a Beta diversity index between three communities, A, B and C, the index corresponding to the comparison between A and B is not independent of the index corresponding to B and C because they share community B." This needs a bit of introduction and context before being introduced like that. Also, it is hard to understand from the introduction the objectives and why this study was performed. A bit more context is needed here

l. 49 - "The most popular statistical technique for analysing community similarity and dissimilarity is the Mantel test [8]" Although I agree it is a popular approach, still in use today (e.g. Graco-Roza et al. 2022), I am not sure it is the most popular currently (e.g. not even cited in Pollock et al. 2020 TREE, d'Amen et al. 2017, or Viana et al. 2022). it is perhaps more popular in genetics (e.g. testing for IBD) than in ecology

l. 53 - Please define $X_{ij}$ and $\Beta_{ij}$ in the formula. Also, $X_{ij}$ often refers to raw values while here, it refers to a dissimilarity. For consistency, I suggest to use the same formulation as in Box 10.2 of Legendre and Legendre 2012 with : $m = \sum_{i=1}^{n-1} \sum_{j=i+1}^{n} D_{Y_{ij}} D_{X_{ij}}$

l. 66 - Please add a reference for the last sentence of the paragraph

l. 71 - Please clarify what you mean by "underperform". Inflated type I error is not the same thing as type II error.

> l. 94 - "[...] constrained to increase monotonically. This constraint underlies a fundamental assumption of GDM that dissimilarity can grow only as sites become more different concerning predictor variables." This is not always the case, especially when there is spatial autocorrelation that allows sinusoidal signals for e.g. (Dray et al. 2006). So how is the extension of GDM you propose better suited to model spatial variation and how does it deal when there is autocorrelation?

l. 131 - Why should the variance parameter follow an exponential distribution? Why not half Cauchy as suggested by Gelman 2006 for example?

Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian analysis, 1(3), 515-534.

l. 149 - "we need to ensure that the order in which the communities appear in the Beta diversity indices does not matter (i.e., symmetry of contributions)." Note that there are asymmetric dissimilarity index, which can be used to compared to reference communities for e.g. Ruggiero et al. 1998. I would just add here that such symmetry is often a desirable property, that is fulfilled by most beta diversity dissimilarity measures (Koleff et al. 2003)

Koleff, P.; Gaston, K.J.; Lennon, J.J. Measuring beta diversity for presence-absence data. J. Anim. Ecol. 2003, 72, 367–382. 

Ruggiero, A.; Lawton, J.H.; Blackburn, T.M. The geographic ranges of mammalian species in South America: Spatial patterns in

environmental resistance and anisotropy. J. Biogeogr. 1998, 25, 1093–1103.

l. 153 - "Second, we need to assume that the contributions from individual communities are independent." This is not the case when there is autocorrelation. In most ecological cases, this assumption would therefore be violated. How does it influence the estimation and type 1 error of this approach?

l. 170 to 179 - I don't understand the analogy and equivalence. It seems to me that "the only critical difference" of sign is still a big one, but I'm not sure to follow through this part sorry.

l. 218 - "For this analysis, we selected 43 plots located in Panama at least 400 meters apart but no more than 60 km" Are they the same than in Ferrier et al. that also used 43 plots?

Figure 2 - Could you put the x and y axis of Fig2b on the same scale, as it is rather confusing to plot the predicted and observed values on different scales. Could you also add the unit for Geographic distance and Elevation on the x-axis text?

l. 267 - Why choose linear functions to compare with the non-linear GDM?

l. 274 - "We chose a weakly informative prior model that assigns a slightly higher probability to distributions of Bray-Curtis indices centred around 0.5, but that also assigns considerably probability to more humped distributions (Fig 2A)." Wrong fig label, and wrong figure (see below). Why do you choose a prior centred around 0.5, why do you expect that this would be uninformative? It really depends on the ecosystems and on the observed distribution of dissimilarities in the study, no? What is the observed distribution of dissimilarities in this example? Also, this prior includes zero dissimilarities while you excluded them from your data. Also, the prior likelihood given to 1 dissimilarity (and in general to high ones) really depends on the gradient covered in the study.

Figure 3 - Wrong label (label Figure 2 instead of Figure 3) and wrong plots compared to the appendix e.g. Fig A doesn't match with fig in Appendix p 9, and the scales of Fig C and D doesn't match with fig in Appendix p12. The text seems to refer to the figure in appendix but I’m not sure which one is the right one.

l. 277 - "We transformed geographical distance values by subtracting 10 km m from observed values and dividing the resulting value by 10 km. As for the difference in elevation, we subtracted 100 meters and then divided the result by 100 meters." I don't understand this transformation that looks like a standardization with mean == sd. Could you provide more justification, especially for how you choose these values? It says in the Appendix p8 that covariates are rescaled between 0 and 1 but this is not what that transformation does (or at least I don't see how it could rescale those variables given the distances observed e.g. elevation differences > to 800 means that at least there are elevation values above 800m, which would not be rescaled to 1 with this transformation). 

l. 277 - Typo in "10 km m", remove the extra "m"

l. 280 - "well mixing with Rhat values of ~ 1" You could refer to the appendix where these values are shown.

l. 284. "This result is probably caused by the regularizing prior we introduced for αs that pulls more extreme predicted values towards the mean." If this leads to overestimation, this means that the regularizing prior affect more lower values and that high ones, right? Why so?

l. 289 - "The geographical distance slope had the highest mean value, 0.366, with a 95% credibility interval of [0.323, 0.381], which indicates it is the strongest predictor" Numbers do not match with the main text figure but match with the Appendix. Are these numbers back-transformed from logit scale? It seems so from the appendix, but worth precising in the text. Also, how does this match with your results obtained with GDM, it is hard for the reader to link and compare these results.

l. 303 - " when the measurements are clustered, we can immediately extend BetaBayes to allow the relationship between Beta diversity indices and covariates to vary across those clusters. This feature may be helpful if we think the relationship between the indices and some environmental covariates may vary across different regions or study areas." This is extremely interesting but note that this can be done in hierarchical jSDMs too, and that it also exists more or less similar things with RDA (Vieira et al. 2019)

Vieira, DC, Brustolin, MC, Ferreira, FC, Fonseca, G. segRDA: An r package for performing piecewise redundancy analysis. Methods Ecol Evol. 2019; 10: 2189– 2194. https://doi.org/10.1111/2041-210X.13300

l. 322 - "We assumed linear relationships between the covariates and the Beta diversity indices in the above example and the previous sections." Not true because of the logit. 

l. 349 - "When we work with pairwise similarities or dissimilarities, we lose information on the identity of species shared across more than two sites." One of main thing here is that you lose relevant species-specific information, which seems to me as a drawback compared to transformation-based RDA (Legendre and Gallagher 2002) or to jSDMs (Warton et al. 2015, d'Amen et al. 2017, Pollock et al. 2020 TREE)

l. 350 - "we lose information on the identity of species shared across more than two sites" This also echoes the concept of Zeta diversity

Hui, C., & McGeoch, M. A. (2014). Zeta diversity as a concept and metric that unifies incidence-based biodiversity patterns. The American Naturalist, 184(5), 684-694.

l-347 to 352 - Suggested ref : Baselga, A. (2013). Multiple site dissimilarity quantifies compositional heterogeneity among several sites, while average pairwise dissimilarity may be misleading. Ecography, 36(2), 124-128.

l. 351 - "this approach does not consider the covariance problem between similarities since some pairs must share the same site" Could you be more explicit, I am not sure to properly understand the meaning behind this sentence

Appendix p.6  - Typo in the x axis of the fig "Elevantion" instead of "Elevation"

Appendix p.7  - Typo in "We can usese ..."

# References

Dray, S., Legendre, P., and Peres-Neto, P. R. (2006). Spatial modelling: a comprehen- sive framework for principal coordinate analysis of neighbour matrices (pcnm). Ecological Modelling, 196(3):483–493.

Warton, D. I., Blanchet, F. G., O’Hara, R. B., Ovaskainen, O., Taskinen, S., Walker, S. C., and Hui, F. K. (2015). So many variables: joint modeling in community ecology. Trends in Ecology & Evolution, 30(12):766–779.

Warton, D. I., Wright, S. T., and Wang, Y. (2011). Distance-based multivariate anal- yses confound location and dispersion effects. Methods in Ecology and Evolution, 3(1):89–101.

Pollock, L. J., O’Connor, L. M., Mokany, K., Rosauer, D. F., Talluto, M. V., and Thuiller, W. (2020). Protecting biodiversity (in all its complexity): New models and methods. Trends in Ecology & Evolution.

D’Amen, M., Rahbek, C., Zimmermann, N. E., and Guisan, A. (2017). Spatial pre- dictions at the community level: from current approaches to future frameworks. Biological Reviews, 92(1):169–187.

Legendre, P. and Gallagher, E. D. (2001). Ecologically meaningful transfor- mations for ordination of species data. Oecologia, 129(2):271–280.

Legendre, P., Fortin, M.-J. and Borcard, D. (2015), Should the Mantel test be used in spatial analysis?. Methods Ecol Evol, 6: 1239-1247. https://doi.org/10.1111/2041-210X.12425

Guillot, G. and Rousset, F. (2013), Dismantling the Mantel tests. Methods Ecol Evol, 4: 336-344. https://doi.org/10.1111/2041-210x.12018

Viana, D. S., Keil, P., and Jeliazkov, A. (2022). Disentangling spatial and environmental effects: Flexible methods for community ecology and macroecology. Ecosphere, 13(4):e4028.

Author Response

Rev3 - The lack of context on the whole breadth of methods available to model beta diversity could limit the uptake of the approach by ecologists.

Following a similar comment by the Editor, we re-wrote the introduction and now introduce dissimilarity modelling as one of several modelling strategies for analysing changes in community similarity following Guisan et al. and D’Amen et al.’s classification scheme. We stress that each modelling strategy is best suited for different ecological questions and for dealing with different data. We explain we are interested in improving this type of modelling because quantitative data gathered by Mokany et al. indicate dissimilarity modelling’s popularity is high and has grown between 2007 and 2021 (L44-74).

Rev3 - 2. The pros and cons of BetaBayes are not clearly highlighted by the methodological comparison provided, which appears insufficient to support the values of this new approach.

We rewrote the Conclusion and no provide a more in depth explanation of what sets BetaBayes apart from alternative methods (L464-483).

Rev3 - I question though, the choice of the Mantel test as the first benchmark, given the criticism around it (e.g. Legendre et al. 2015., Guillot et al. 2013), the latter being recognized in Section 4.1. (l. 71-72). As the test dataset concerns in part spatial distances, I suggest at least to control for autocorrelation in that approach using the latest permutation test developped : Crabot, J, Clappe, S, Dray, S, Datry, T. Testing the Mantel statistic with a spatially-constrained permutation procedure. Methods Ecol Evol. 2019; 10: 532– 540. https://doi.org/10.1111/2041-210X.13141

We agree with the criticism of Mantel’s test. We used it as a benchmark, because it remains widely used in the literature. Still, we complemented the comparison with GDM, which is a far more robust method.

The variant of the Mantel test presented in Crabot et al. considers spatial autocorrelation by “correcting” the Mantel’s statistic by the Euclidean distances separating the communities. In our example, we already use the Euclidean distance as a covariate, therefore, using this variant of the test would be the same as using the covariate “Euclidean distance” twice.

Following this comment, we decided to add a section explaining how to model spatial autocorrelation in BetaBayes without adding the distance between communities as a covariate (406-433).

Rev3 - Even when compared to GDM, it may not be crystal clear for an average reader why it is that important to directly model the non-independence (compared to controlling it by randomization as in GDM).

Following a similar comments by Reviewer 2, we improved the section in which we explain why it is important to consider the pairwise dependence (L65-72).

Rev3 - In terms of discussion, the section on the possible extensions is interesting and well-written, but lack some perspectives on the comparison, helping the readers to make sense of the different results and of what it means in terms of the capacity of the BetaBayes approach compared to the other ones tested.

We rewrote the Conclusion and no provide a more in depth explanation of what sets BetaBayes apart from alternative methods (L464-483).

Rev3 - Moreover, the only thing the reader can really compare across the methods is the significance and relative importance of each predictor. R2 or pseudo-R2 are not presented for all methods, observed vs predicted plots can be compared by the reader when looking at the appendix for BetaBayes, but the differences between them are not even mentioned in the text, and there is nothing about the type 1 ebrrror of this new approach compared to the original GDM (see below for that point). Based on that, and without any real discussion on it, it is hard for an outside reader to say if this approach is really better, except for the theoretical arguments presented in the intro.

BetaBayes is a Bayesian approach while Mantel tests and GDM are frequentist methods. There is no single valid metric (equivalent to the R2) that we can use to compare these three methods. Moreover, Mantel tests does not provide validation tools. We chose to use residual analysis to validate the models because it is a widely accepted model validation procedure in both frequentist and Bayesin statistics.

Following this comment, we rewrote the Conclusion to provide a more in depth explanation of what sets BetaBayes apart from alternative methods (L464-483).

Rev3 - Also, it is hard to compare BetaBayes and GDM since you present a linear version of BetaBayes and not a non-linear one. I understand why starting with a simpler model is important for the reader but if you want to really compare what BetaBayes brings compared to GDM, I think you need to compute the most similar models possible.

We disagree with the referee. The validation plots show the BetaBayes model fits the Panama data well, therefore we see no reason for adding B-splines to the model. Following this comment, we decided to partially rewrite the section on “splines”. We now state that “vanilla” BetaBayes can model moderately non-linear relationships, but that B-splines are required for more modelling more complex relationships (L437-455) .

Rev3 - Appendix p5 you say about GDM that "The model explains 52.51% of the deviance" How much of the deviance does the BetaBayes explains with this linear formulation? How much would it explain with splines? Where do we expect a difference with GDM given the addition of these intercepts in BetaBates: in the pseudo-R2 ? In the response curves? In the significance? In the shape of the residuals? This is not clear from that comparison.

See previous reply.

Rev3 - In terms of fit and heteroscedasticity, the plot given in the appendix does not suggest any improvement compared to GDM (perhaps because of the linear constraints you chose here, but perhaps not only...). You say l. 282 "The posterior retrodictive distribution of Bray-Curtis indices closely matched the observed distribution, except for values below 0.39, which are slightly overestimated (Fig 2B, and Appendix 1)." Given the figures in Appendix p 11 (top right panel and middle left), the overestimation is not that little. The model clearly underestimates the extent of the turnover gradient. There is also from the middle right panel and the bottom left panel strong indications of heteroscedasticity in the residuals that show that the unique funnel-shape of distance-distance relationship is not adequately modelled here. Indeed, it is typical of distance-distance relationship to show different variances along the plot (high variances near 0 distances, and lower variance towards 1 distance, as observed in your case in appendix p6 or in Fig 4 of Wetzel et al. 2012 for example, or the other way around, see for e.g. Fig. 4 from Mclean et al. 2021). If the variance structure is not captured properly, the model cannot be adequate as it mixes up dispersion and location effects potentially (see for e.g. Warton et al. 2012 for similar arguments on categorical explanatory variables).

Our prior model regularizes inferences of Bray-Curtis indices towards 0.5, which can introduce an apparent bias when there are only a small number of observations. That said, the observed bias is typically within the posterior uncertainties and so is not practically significant. Moreover, as more data are introduced the likelihood function dominates the structure of the posterior distribution and this prior bias weakens automatically. We added this explanation to the text (L356-364).

Rev3 -(iii) Beyond a more thorough analysis and discussion of the example provided, all these points show, in my opinion, the need for a proper simulation study to really test the value of BetaBayes. For example, quantifying the Type 1 error with or without spatial autocorrelation seems a prerequisite before publishing a new method to study distance-distance relationships e.g. Franckowiak RP, Panasci M, Jarvis KJ, Acuña-Rodriguez IS, Landguth EL, et al. (2017) Model selection with multiple regression on distance matrices leads to incorrect inferences. PLOS ONE 12(4): e0175194. https://doi.org/10.1371/journal.pone.0175194

BetaBayes is based on the Bradley-Terry model, which is a popular and time proven framework for modelling paired comparisons. The Bradley-Terry model is supported by multiple simulation studies that extensively tested its performance under multiple dependence scenarios. In the previous manuscript, we did not make this clear. In section 3.1 we added a section where we explain that Bradley-Terry models are widely used outside the ecological literature L217-222 and in the Conclusion we now mention that the Bradley-Terry model has been extensively tested in simulation studies (L457-459).

Rev3 - l. 43 - Citation of ref 7 does not seem appropriate here: variation partitioning using RDA or dbRDA is not quite like mantel test or GDM. Although the purpose is also to model beta diversity, it does not suffer from the same issue than distance-distance relationships (see Legendre et al. 2015). A more suitable ref to convey the idea of the sentence would be something like : Graco-Roza, C., Aarnio, S., Abrego, N., Acosta, A. T. R., Alahuhta, J., Altman, J., Angiolini, C., Aroviita, J., Attorre, F., Baastrup-Spohr, L., Barrera-Alba, J. J., Belmaker, J., Biurrun, I., Bonari, G., Bruelheide, H., Burrascano, S., Carboni, M., Cardoso, P., Carvalho, J. C., … Soininen, J. (2022). Distance decay 2.0 – A global synthesis of taxonomic and functional turnover in ecological communities. Global Ecology and Biogeography, 31, 1399– 1421. https://doi.org/10.1111/geb.13513

Done

Rev3 - l. 44 - "For instance, if we calculate a Beta diversity index between three communities, A, B and C, the index corresponding to the comparison between A and B is not independent of the index corresponding to B and C because they share community B." This needs a bit of introduction and context before being introduced like that. Also, it is hard to understand from the introduction the objectives and why this study was performed. A bit more context is needed here

We improved the section in which we explain why it is important to consider the pairwise dependence (L65-72).

Rev3 - l. 49 - "The most popular statistical technique for analysing community similarity and dissimilarity is the Mantel test [8]" Although I agree it is a popular approach, still in use today (e.g. Graco-Roza et al. 2022), I am not sure it is the most popular currently (e.g. not even cited in Pollock et al. 2020 TREE, d'Amen et al. 2017, or Viana et al. 2022). it is perhaps more popular in genetics (e.g. testing for IBD) than in ecology

We replaced “the most popular” with “one of the most popular”. Data from Mokany et al 2022 show that between 2007 and 2021, the cumulative number of published papers that use the words “generalised dissimilarity model” grew rapidly.

Rev3 - l. 53 - Please define $X_{ij}$ and $\Beta_{ij}$ in the formula. Also, $X_{ij}$ often refers to raw values while here, it refers to a dissimilarity. For consistency, I suggest to use the same formulation as in Box 10.2 of Legendre and Legendre 2012 with : $m = \sum_{i=1}^{n-1} \sum_{j=i+1}^{n} D_{Y_{ij}} D_{X_{ij}}$

Done.

Rev3 - l. 66 - Please add a reference for the last sentence of the paragraph

Done.

Rev3 - l. 71 - Please clarify what you mean by "underperform". Inflated type I error is not the same thing as type II error.

We replaced this sentence with: “Some studies have found that both Mantel and partial Mantel tests the acceptable rates of type-I errors when the data are spatially structured [11–13].” (L106).

Rev3 -l. 94 - "[...] constrained to increase monotonically. This constraint underlies a fundamental assumption of GDM that dissimilarity can grow only as sites become more different concerning predictor variables." This is not always the case, especially when there is spatial autocorrelation that allows sinusoidal signals for e.g. (Dray et al. 2006). So how is the extension of GDM you propose better suited to model spatial variation and how does it deal when there is autocorrelation?

This sentence is a direct quote from Ferrier et al. 2007, the paper that introduces GDM. As for how BetaBayes can handle spatial autocorrelation, we added a section in which we explain we can add a Gaussian process to BetaBayes to consider spatial autocorrelation (L406-433).

Rev3 - l. 131 - Why should the variance parameter follow an exponential distribution? Why not half Cauchy as suggested by Gelman 2006 for example?

The exponential prior, along with the half-Cauchy prior, is widely used in Bayesian statistics books for positive parameters (see McElreath 2020 and Lambert 2018).

McElreath, Richard. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. 2 edition. Boca Raton: Chapman and Hall/CRC, 2020.

Lambert, Ben. A Student’s Guide to Bayesian Statistics. 1 edition. Los Angeles: SAGE Publications Ltd, 2018.

Rev3 - l. 149 - "we need to ensure that the order in which the communities appear in the Beta diversity indices does not matter (i.e., symmetry of contributions)." Note that there are asymmetric dissimilarity index, which can be used to compared to reference communities for e.g. Ruggiero et al. 1998. I would just add here that such symmetry is often a desirable property, that is fulfilled by most beta diversity dissimilarity measures (Koleff et al. 2003).

We now mention this approach is valid only for symmetric Beta diversity indices (L191).

Rev3 - l. 153 - "Second, we need to assume that the contributions from individual communities are independent." This is not the case when there is autocorrelation. In most ecological cases, this assumption would therefore be violated. How does it influence the estimation and type 1 error of this approach?

We added a section explaining how BetaBayes can be modified to handle spatial autocorrelation (L406-433).

Rev3 - l. 170 to 179 - I don't understand the analogy and equivalence. It seems to me that "the only critical difference" of sign is still a big one, but I'm not sure to follow through this part sorry.

We agree this section was unclear. We partially rewrote this section and added text detailing how BetaBayes and the Bradley-Terry model are connected (L225-248).

Rev3 - l. 218 - "For this analysis, we selected 43 plots located in Panama at least 400 meters apart but no more than 60 km" Are they the same than in Ferrier et al. that also used 43 plots?

Ferrier et al. 2002 do not mention the selection criteria for the plots.

Rev3 - l. 274 - "We chose a weakly informative prior model that assigns a slightly higher probability to distributions of Bray-Curtis indices centred around 0.5, but that also assigns considerably probability to more humped distributions (Fig 2A)." Wrong fig label, and wrong figure (see below). Why do you choose a prior centred around 0.5, why do you expect that this would be uninformative? It really depends on the ecosystems and on the observed distribution of dissimilarities in the study, no? What is the observed distribution of dissimilarities in this example? Also, this prior includes zero dissimilarities while you excluded them from your data. Also, the prior likelihood given to 1 dissimilarity (and in general to high ones) really depends on the gradient covered in the study.

We corrected the Figure number and labels. We chose a weakly informative prior. While it assigns the highest probabilities to values around 0.5, the difference compared to lower and higer values is fairly small. Figure 3A shows that the prior model is in fact fairly flat. We rewrote this sentence to make this point clear (L348-354).

Rev3 - Figure 3 - Wrong label (label Figure 2 instead of Figure 3) and wrong plots compared to the appendix e.g. Fig A doesn't match with fig in Appendix p 9, and the scales of Fig C and D doesn't match with fig in Appendix p12. The text seems to refer to the figure in appendix but I’m not sure which one is the right one.

Done.

Rev3 - l. 277 - "We transformed geographical distance values by subtracting 10 km m from observed values and dividing the resulting value by 10 km. As for the difference in elevation, we subtracted 100 meters and then divided the result by 100 meters." I don't understand this transformation that looks like a standardization with mean == sd. Could you provide more justification, especially for how you choose these values? It says in the Appendix p8 that covariates are rescaled between 0 and 1 but this is not what that transformation does (or at least I don't see how it could rescale those variables given the distances observed e.g. elevation differences > to 800 means that at least there are elevation values above 800m, which would not be rescaled to 1 with this transformation).

Following a similar comment by Reviewer 1 and 2, we corrected the section in which we explain how the data transformation works and how it improves the interpretation of the model’s coefficient’s (L366-373).

We thank the reviewer for detecting the error in the Appendix. We corrected it.

l. 277 - Typo in "10 km m", remove the extra "m"

Done.

l. 280 - "well mixing with Rhat values of ~ 1" You could refer to the appendix where these values are shown.

Done.

l. 289 - "The geographical distance slope had the highest mean value, 0.366, with a 95% credibility interval of [0.323, 0.381], which indicates it is the strongest predictor" Numbers do not match with the main text figure but match with the Appendix. Are these numbers back-transformed from logit scale? It seems so from the appendix, but worth precising in the text. Also, how does this match with your results obtained with GDM, it is hard for the reader to link and compare these results.

We thank the reviewer for detecting this mistake. We added the correct Figure 2 to the manuscript.

Appendix p.6 - Typo in the x axis of the fig "Elevantion" instead of "Elevation"

Done.

Appendix p.7 - Typo in "We can usese ..."

Done.

Round 2

Reviewer 3 Report (New Reviewer)

The authors took my main comments into account and provide a throroughly improved version of the manuscript.

Although I have a few remaining suggestions (see below) in order to help the reader fully understand the functioning and power of the approach, I consider the manuscript publishable as it is.

I also thank the author for their detailed response, for adding the section on autocorrelation, and again, for providing code.

Further suggestions/comments:

• “we replaced the minus sign with the plus sign because it lead to higher inferential performance.” remains unclear to me. How could this change of sign influence inferential performance. It may be clear for specialist, but as outsider of this field of BT models, it remains a bit cryptic.
• “That said the observed bias is typically within the posterior uncertainties and so is not practically significant.” I’m not fully convinced if I look at the residuals and at predicted versus observed plot in supplementary p12, but that’s pretty minor. It is just that, as a newbie, I would be worry about the adequacy of my models if looking at such plot. Perhaps adding a quick discussion of how you interpret this in the supplementary could help people understand how to fit and assess the validity of these models? And when/how you consider that such biais are minor or not.
• “BetaBayes avoids hypothesis testing entirely and instead focuses on collecting information into inferences about the observed data.” That’s proper to Bayesian inference and could be said to Bayesian jSDM for exemple. While it is clearer to me how flexible the framework is, and how it can improve beta diversity modelling with the added material, I think the discussion lacks some key elements that you now provide in the results/discussion. For exemple, the fact that you can control for autocorrelation, perhaps better than in existing models, is something to put forward in my opinion.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

In this manuscript you identify an important aspect of betadiversity metrics that can affect betadiversity modeling approaches. Building from such characteristic, you propose a novel structure (i.e., formula) to fit a Bayesian regression model accounting for such aspects of the data and circumvent the limitation it imposes to traditional regression analysis. Although the basis of the manuscript seems reasonable, you fail to provide evidence for such limitation in real datasets and whether the method improves or changes interpretation of traditional regression analysis. This is because methodologically there are important limitations: you have created an artificial dataset imposing the same structure assumed by your modelling approach. Although this might be valid to illustrate the issue you are trying to fix and how the method works to fix the issue, it is not enough to demonstrate until what extent the issue is real and affect real datasets. A paper like this would need additional analysis to better illustrate and demonstrate the validity and usability of the approach. The manuscript is written in a correct language, and it reads fluently. However, the manuscript is missing important sections, like Results, Discussion, and Conclusions. Furthermore, the Introduction is only one paragraph long, failing to provide a deep and thoroughly review of the state of the question (beta diversity modelling), which might be extensive (there is a considerable amount of literature about betadiversity modeling), and even mentioning important contributions, like Generalized Dissimilarity Models (GDMs). Furthermore, there are a) errors numbering sections (there are 2.5.1 and 2.5.3, but not 2.5.2), b) some “paragraphs” that seems comments from co-authors (see last paragraph in section 2.5.1 in page 6), and c) some repetitions of multiple sentences (e.g., descriptions in the “simulation results” section and the figure legends). All this made me think that the manuscript was not ready for submission but an early draft.

I think your manuscript have potential to be interesting. However, it would require extensive work on the analysis and the manuscript before being ready for publication. Taken all the previous into account, I recommend your manuscript to be rejected for publication.

Abstract

Improve the link between sentences 2 and 3 of the abstract. For instance, you can explain that one way to disentangling drivers of change between ecological communities is by modeling betadiversity and study variables contribution…

Introduction

Why is the introduction so short? There is a big literature body on betadiversity modeling. This introduction fails to provide an overview of the state of the art of betadiversity modeling and where your new approach places and where the issue applies to all modelling approaches. More importantly, there are multiple aspects of betadiversity modeling that have been addressed in previous approaches and, at least to my understanding, you are not considering in your approach, which could limit the validity and usability of your approach. I am talking about a) non-linearity of betadiversity to changes in environmental drivers, b) non-negative relationship between betadiversity and changes in environmental drivers, and c) the saturating response of betadiversity (because it is bounded by the value 1) to certain degree of environmental change. All these casualties affect regression analysis of betadiversity. In fact, Generalized Dissimilarity Models (GDMs) have been designed and proposed to solve these problems. Do you think GDM is also affected by the problem you are addressing in your new approach? Is your approach affected by these limitations? In a manuscript like this I would expect to, at least, acknowledge all these aspects and discuss until what point you have taken them into account. Furthermore, this would allow to introduce a discussion on how these could affect you approach and whether it could be possible to expand your approach in the future to address them.

L42-44: Betadiversity itself is a distance metric, so not clear how this would be fixing the issue. Do you mean permutating distance matrices to perform the Mantel test and the partial Mantel test?

Methods

L68-69: A reference wouldn’t harm here. Especially because it has not been demonstrated anyhow in the manuscript.

L69-71: Consider removing all this sentence because it has already been said.

L87: “S” in the formula should be lowercase.

L91: “S” in the formula should be lowercase.

L92-95: All this is a bit unclear. I think part of the problem is that is not clear how s[i] is different from s1. I had to read the appendix and your code to understand the differences between the two ways to include the alpha terms. Please clarifies the differences between the two approaches in the manuscript so it is crystal clear upfront without having to read the supplementary information.

L99: Remove one dot.

Section 2.1 Simulations: The section should be 2.2 and not 2.1, which has been used in “General overview”.

L110-117: This methodological approach (simulate an artificial dataset with the same structure as assumed by your approach) could be valid to illustrate the issue and until what extent your approach capture the imposed relationship. However, this doesn’t provide evidence on how this issue is important in real datasets and until what point it changes interpretation of previous analysis that doesn’t account for lack of independency between betadiversity metrics.

L145: Is this formula missing “+ beta Xij”

L154-155: Precisely! (See my previous comment). It won’t be surprising if this model structure fits better the data and outperform model1 and model2.

Results

There is no Results section in your manuscript.

Section 2.2.2 Simulation results

I think this could be the first section in the Results section of your manuscript.

L170: Change “compare” to “compared”. Try to be consistent in your language. Methods and Results section describe what you did and what you obtained from your analysis. Hence, they are most typically written in past tense.

L174: I see and understand your point. However, I don’t think it is intuitive and obvious. Consider removing “unsurprising”.

L187-196: All these lines are almost a perfect duplicate of the figure 1 legend and, hence, it is unnecessary. Consider removing it and limit this part to a description of the results rather than a description of the figure. Same issue in other parts of manuscript. Review it carefully to avoid redundancy as much as possible.

L220: This reference to the supplementary information would fit better in a “Methods” section than in a “Results” section.

L223-227: Something weird here with font-size or font-style. Please review.

L231: Remove a backspace between closing parenthesis and dot.

Section 2.5 Model extension: This could be a “Further work” subsection in “Discussion”.

L273-275: Remove this paragraph. It looks like this was a comment from one of the co-authors. Given the structure problems of the manuscript and this comment. Is this an early draft of the manuscript rather than a final version?

L303: At this point I am wondering how many independent variables can be included in the analysis with your approach.

Figure1: Change “Tha)” to “a)”. Although I would suggest not start directly with “labels” but with a wide description of the figure.

Figure 4: “c) and d)”??? This figure is missing the description of facets c and d.

L388-395: All this is repetitive from the abstract and could be replaced by a more interesting and thorough discussion of the results, the approach, and its relationship with previous approaches to model betadiversity.

References: The length of this section (only 19 references) illustrates my point about the weak review of the beta-diversity modeling literature performed for this manuscript.

Appendix 1:

4.2 Model 2: “interceptsone” to “intercepts, one”??

4.4 Fit the models: Check if model$sample can be configured to be less verbose. It is not necessary in this tutorial to have such long output for the MCMC chains. Reducing function verbosity would reduce the length of the appendix in, at least, 2-3 pages.

Page 11: First sentence of the page seems weird (missing verb?). Furthermore, there are a format mistake with asterisk for bold fonts.

Page 16: Mismatch between the title of the section and the first sentence of the section. Change “Model_2” to “Model_3”.

Page 20: Cool figure, but this figure is not in the main manuscript. Clarifies where this fit in the model evaluation and the main manuscript.

Reviewer 2 Report

Summary

The authors are interested in using beta diversity indices to better understand ecological community changes due to human impact or otherwise. The central problem is that when comparing many communities, beta indices are all pairwise and relative to one another, and importantly, are not independent when the same community is used in multiple calculations. Thus to account for this lack of independence between pairwise measures that use a common community, the authors have developed an alternative Bayesian approach.

Comments

Generally, I think there is a lot of background information lacking about the use of Beta indices in general and most importantly, why not accounting for this lack of independence is so important. Why should I care about this issue, or what could I misinterpret if I did not account for this dependence between beta indices due to a common community? The problem that you are solving with your new model is not clearly defined. 

Additionally, I think much of the paper lacks context for the purpose of what is being modeled and how it will aid in our understanding of how we compare beta indices. I think using an empirical system with measured beta indices to also validate your model fit is necessary. 

The model designs are very reasonable and it is interesting to see how when accounting for the dependence between some indices, the models (2 and 3) fit much better to the simulated data. I think there is novelty and value to these new models (2 and 3, and the varying extensions). However, for an ecologist, the importance of this new model is not well highlighted.  I think, as written, this paper does not appeal to a general ecological or biodiversity-based audience, as again there is little context to the use of this model and its applicability to real, empirical systems. 

I also think the model space explored is fairly limited, and the simulations are done with non-extreme values for the parameters. It would be interesting to one, try to use simulated values that are based on observed values from real communities and see how the models behave and how the data fit them. 

The importance of each figure is not clear, as their relevance is not well explained in the text alongside the results. Figure 4 and 5 are not referenced or discussed in the text at all. 

I think if the context of the models and their application are improved, along with the general narrative of this manuscript, this will make a great paper, as the models are useful and improve how we model dependence across beta indices.

Reviewer 3 Report

Dear Dr. Dias et al,

Thank you for the opportunity to review your manuscript titled “BetaBayes – A Bayesian approach for comparing ecological communities”.

I really admire the scope of what you are attempting here.  A generative model for pairwise dissimilarity metrics would represent a major advance for our field.  However, I do not see why the model that you have proposed is a reasonable generative model for pairwise dissimilarity metrics.  If you can argue convincingly why this model is approximately correct (or can accurately and convincingly circumscribe the cases where we should expect the model to be approximately correct), then I would be happy to accept the paper. Detailed comments follow.

Best

Jacob Socolar

(I sign all my reviews except when double blind)

First, I’ll note that ecologists to date have not really succeeded in thinking generatively about compositional dissimilarities directly.  Instead, we tend to think generatively about the processes by which species sort into sites (i.e. species distribution models), and then we view dissimilarities as an emergent feature of our generative model for the distributions of species.  This way of thinking has led to two main strands in the literature on modeling and comparing pairwise dissimilarities. 

The first is so-called “Generalized dissimilarity modeling”, which essentially means fitting GAMs to the dissimilarities via maximum likelihood methods (see Ferrier et al 2007 “Using generalized dissimilarity modelling to analyse and predict patterns of beta diversity in regional biodiversity assessment” Diversity and Distributions).  These methods do not come with principled uncertainty quantification, especially in light of the nonindependence of the pairwise comparisons.  Note, however, that it has been claimed that the Bayesian bootstrap can provide adequate uncertainty quantification in these cases; I am unqualified to comment on whether this approach adequately deals with the non-independence issue (see Woolley et al 2016 “Characterising uncertainty in generalized dissimilarity models” Methods in Ecology and Evolution).

            The second is a variety of matrix regressions, Mantel tests, and null-model permutation tests designed to compare distributions of pairwise dissimilarities under some null assumption.  Like the previous approach, this approach is explicitly non-generative.  Moreover, it creates a great deal of controversy regarding what null assumptions are appropriate (in terms of whether and how to fix row or column sums).   

It would be AMAZING if we could move beyond this way of thinking—if we could directly write down generative models for the dissimilarities themselves.  I am fully on board with your views on the limitations of current methods and the advantages of replacing them with a Bayesian pipeline based on a principled generative model! 

A crucial requirement for successfully doing this is that the generative model for the dissimilarities (where it is more difficult to bring domain expertise to bear) must be consistent with our relatively well-developed generative ideas about how species distribute themselves across space or through time. My concern with the present manuscript is that it appears to fail this crucial requirement, and also that it appears to fail to adequately capture the dependency structure of the dissimilarity metrics.  That is, while I am fully convinced that you have adequately modeled the data that you simulated, I am unconvinced that the data that you simulated correspond to the actual structure of real-world dissimilarity data.

Suppose we have three sites A, B, and C, with A highly similar to B, and C different from both.  Your model can deal with this well; we assign low random intercepts to A and B, and a high random intercept to C, and in this way we capture the fact that a high dissimilarity between C and A also implies a high dissimilarity between C and B.  In this sense, I think that your model is adequate to capture triplet-wise nonindependence that arises due to the structure of the data.  However, it is well known that, just as the dependency structure of such data is not fully captured at the pairwise level, it also is not fully captured at the triplet-wise level (see Diserud and Odegaard 2007 “A multiple-site similarity measure”, Biology Letters).  Thus, consider sites A, B, and C as before, but now add site D, which is highly similar to site C.  There is no configuration of the random intercepts that can simultaneously capture these relationships.

Let’s back up and think about the generative model for species distributions in this case.  In general, we can imagine that species are distributed stochastically along some set of environmental gradients.  When these gradients are known, we can represent the model as a logistic GLM or GAM with Bernoulli error.  When the gradients are unknown, we can represent the model as a similar latent factor model.  When species interact, we can also represent the model as a latent factor model where the latent factors and their species-specific loadings represent a reduced-rank approximation to the full residual covariance matrix (see Warton et al 2015 “So many variables: joint modeling in community ecology” Trends in Ecology and Evolution).

Thus, I almost always think about species distributions as happening along some kind of gradient (either a measured gradient, an unmeasured gradient, or an approximation to the covariance that can be represented mathematically as a gradient).  For species distributions along a gradient, I think it quickly becomes clear that your random intercepts specification is inadequate, because this specification has no way to represent that sites can be spaced along a gradient, and has no way to represent the dependency structure that arises from this assumption.  Indeed it might be more adequate to represent the structure by differencing (rather than adding) the random intercept terms, which could give some picture of how far apart sites are along the gradient.  Proceeding from the Joint SDM of Warton et al (cited above), we could potentially envision a dissimilarity metric defined by the distance in latent environmental space.  I’m not saying this is a great metric, I’m just saying it has the flavor of a dissimiarlity metric.  I would think that a generative model for dissimilarities should be able to approximately capture and reproduce the resulting structure.  As far as I can see, your model cannot.

Additionally, I am skeptical of your approach in that it does not seem to pay close attention to the choice or computation of the dissimilarity metric.  Yet the precise form of the dependency structure in the pairwise dissimilarities depends in its details on the choice of the dissimilarity metric.  You mention the Sorenson index in the manuscript, but nowhere do I see that your derivation of your generative model is tailored to the behavior of that metric.