BayCoN Plots and Systematic Errors in Single-Crystal Diffraction Experiments

Abstract

Bayesian Conditional Probability (BayCoN) plots provide an empirical means to classify systematic errors in single-crystal diffraction experiments based on weighted residuals. Using a set of 314 structures from IUCrData, four recurring error types are identified and illustrated as follows: (i) incorrect standard uncertainties of observed intensities, (ii) intensity and significance cut-offs, (iii) cases where weak observed intensities exceed calculated ones, like with incomplete absorption correction, and (iv) cases where they are systematically smaller. Only eleven data sets showed uniform BayCoN plots. The analysis focuses on how the residual distribution reveals and categorizes systematic errors; the impact on model parameters and their uncertainties is not addressed in this work. The proposed classification is intended to assist crystallographers in improving experimental accuracy by recognizing characteristic residual patterns.

1. Introduction

The Bayesian Conditional Probability (BayCoN) plots were introduced in 2014 as a diagnostic and visualization tool for systematic errors in X-ray and neutron diffraction data as well as for providing a metric for quantification of systematic errors via the associated ${\chi }^2$ values [1]. Although the concept was acknowledged and successfully applied by other researchers for visualizing and quantifying systematic errors [2,3,4,5] and software for generating BayCoN plots was freely available, its use in the literature remains limited. This study demonstrates the diagnostic power of BayCon plots by applying them to 314 data sets published with IUCrData. The focus is on the appearance of systematic errors in order to develop the BayCoN plots into a mature tool for diagnosing the root causes of systematic errors. In some cases, those root causes are easy to assign, for example, systematically flawed standard uncertainties of the observed intensities lead to a characteristic pattern in the BayCoN plots. BayCoN plots effectively illustrate the disruptive effects of intensity and significance cut-offs. They clearly show en passant that once such cutoffs are applied, the weighted residuals are no longer identically distributed. Additionally, the corresponding BayCoN plots visualize how the application of an intensity or significance cut-off artificially limits the sum of squared residuals, the entity that is minimized in the least squares process and enters the Goodness of Fit and weighted agreement factors.

This study aims to establish practical, empirical categories of systematic errors by analysing recurring patterns in BayCoN plots from published data sets. This empirical approach fits within a broader framework that describes systematic errors on their residual patterns rather than their sources. A source-centered description of systematic errors in diffraction experiments categorizes errors by the source, e.g., “not modelled disorder”, “not modelled extinction”, etc., and is, of course, of great importance. Yet it is also very limited, as the source is not always known. A residual-based approach describes errors by their effects on residuals rather than their causes. This has several advantages. First, (i), it allows characterization even when the source is unknown. Second, (ii), it supports categorization. Third, (iii), it encourages open discussion of systematic errors within the crystallographic community. The descriptive categories help chart the landscape of systematic errors and can even guide the discovery of previously unrecognized ones. (iv), describing and categorizing systematic errors fosters an error-aware working environment, where such errors are recognized as valuable information for genuine progress. In a source-centered approach, lack of knowledge about the error source can become a dead end, potentially encouraging the concealment of systematic errors (error-aversive environment) rather than their appreciation and constructive use. (v) Descriptive methods can be continuously refined and expanded, creating a detailed network of descriptors capable of capturing even the most subtle systematic error. (vi) Descriptive methods offer a shared language and framework for investigating systematic errors and exchanging insights across the crystallographic community. This helps avoid redundant efforts by different research groups addressing the same errors repeatedly and encourages collaboration across the crystallographic community. (vii) Finally, descriptive methods can lead to identifying error sources and can enable the addition of precisely defined categories to the source-centered classifications.

The empirical categories discussed here are (i) errors in the $s.u.\left(I_{obs}\right)$ (Section 5.1), (ii) significance cut-off (Section 5.2), (iii) intensity cut-off (Section 5.3), (iv) weak observed intensities systematically stronger than weak calculated intensities, (Section 5.5), (v) weak observed intensities systematically weaker than weak calculated intensities (Section 5.5), and (vi) uniform BayCoN plots (Section 5.7). It should be noted that the categorization into patterns is a profiling task and may not immediately lead to identification of the root cause. As an example, there may be many different root causes for the weak observed intensities being larger than the corresponding calculated intensities: data integration problems, thermal diffuse scattering, fluorescence, twinning, disorder, or incomplete absorption correction may all play a role here. However, it is not the task of this work to identify the root causes in every individual case, but rather to establish categories that can later be further differentiated into subcategories.

Before discussing the categories (i)–(v) mentioned above, some background knowledge about BayCoN plots is briefly summarized in Section 3, where also the four basic categories Y are given for which the BayCoN plots $(\zeta ,Y)$ and $({\zeta }^2,Y)$ are all uniform in the absence of systematic errors. These are the calculated intensity, $I_{calc}$, the standard uncertainty of the observed intensity $\sigma \left(I_{obs}\right)$, the significance based on the calculated intensity, $I_{calc}/\sigma \left(I_{obs}\right)$, and the resolution $\sin \theta /\lambda$, in short: $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda \}$.

A very brief characterization of the 314 crystallographic data sets in the sample follows in Section 4. The distribution of the ${\chi }^2$ values for the BayCoN plots of the weighted residuals $(\zeta ,Y)$ and for the squared weighted residuals $({\zeta }^2,Y)$ are shown.

2. Methods

BayCoN plots and their associated ${\chi }^2$ values are employed to visualize recurring systematic errors in the sample. Complementing metrics are used for a more detailed characterization and for the quantification of systematic errors.

3. The BayCoN Plots and Complementing Metrics

BayCoN plot are constructed from information that needs to be either given or computed as follows: observed ($I_{obs}$) and corresponding calculated intensity ($I_{calc}$), standard uncertainty of the observed intensity, ($s.u.\left(I_{obs}\right)$), weights ${\sigma }^2\left(I_{obs}\right)$ or parameter to calculate weights from $s.u.\left(I_{obs}\right)$, and resolution. This information is typically available for single-crystal X-ray, neutron, and electron diffraction data. Powder diffraction data can be used also when the reflections are indexed and a structure model is refined.

3.1. Construction of the BayCoN Plots

The basic entities for the x-axis of the BayCoN plots are the weighted residuals $\zeta =(I_{obs}-I_{calc})/\sigma \left(I_{obs}\right)$ or their square, ${\zeta }^2$. For the y-axis, there are 4 different categories Y. These are the calculated intensity $Y=I_{calc}$, the standard uncertainty of the observed intensity $Y=\sigma \left(I_{obs}\right)$, the significance based on the calculated intensity, $Y=I_{calc}/\sigma \left(I_{obs}\right)$, and the resolution, $Y=\sin \theta /\lambda$, in total $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda \}$. For both axis, the normalized rank of the respective entity is used.

Throughout this work, ${\sigma }^2\left(I_{obs}\right)$ is defined to be the inverse weight w. For a weighting scheme as applied in SHELXL, the definition is given by

$${\sigma }^2\left(I_{obs}\right)=s.u{.}^2\left(I_{obs}\right)+X^2$$

(1)

with $X^2={\left(aP\right)}^2+bP$ and $P=(\max (0,I_{obs})+2I_{calc})/3$. The weighting scheme implemented in SHELXL allows for further modifications, but these are not of practical relevance for the data presented here. Note that applying a weighting implies the presence of a systematic error. In the absence of such errors, weighting would be unnecessary. It leaves, however, the important question open where the error emerges from. It may emerge from inadequate standard uncertainties as given in the reflection file (the $s.u.s\left(I_{obs}\right)$), or it may emerge from other model errors. This fundamental distinction is trivial but important. Consideration of this distinction in day to day applications may help to find reliable $s.u.s\left(I_{obs}\right)$.

The BayCoN plots are constructed to be uniform when no systematic errors are present. For this reason the calculated intensity $I_{calc}$ and the significance based on the calculated intensity $I_{calc}/\sigma \left(I_{obs}\right)$ are included in the set of properties Y, rather than the observed intensity and the significance based on the observed intensity. The residuals are expected to be symmetric with respect to the calculated intensity but not with respect to the observed intensity, because negative intensity observations result, by definition, in negative residuals.

In the BayCoN plots, the normalized rank of X, $X\in \{\zeta ,{\zeta }^2\}$ is plotted against the normalized rank of Y, $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda \}$. Normalization means that the rank of the residual is divided by the included number of reflections in the data set, such that the whole plot always appears in the unit square. Normalization is not a necessary condition but merely a convention. As a consequence of this construction scheme, every horizontal or vertical stripe of the same size contains the same number of reflections. When the density of dots is reduced in any area, there is necessarily another area in which the density of points is increased. The BayCoN plot is therefore uniform in the absence of systematic errors and the appearance and degree of non-uniformity can help to identify the error or at least to verbalize its effect on the uniformity of the residuals. Examples will be given later. See also the Appendix A for more information.

The BayCoN plots $(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))$ seem also particularly sensitive to some forms of disorder and twinning. The $(X,I_{calc}/\sigma \left(I_{obs}\right))$ plots with $X\in \{\zeta ,{\zeta }^2\}$ were not included in the freely available software packages for the calculation of BayCoN plots. How disorder and twinning affect the residual distribution as depicted by the BayCoN plots and quantified by the associated ${\chi }^2$ values will be addressed in a follow-up publication.

3.2. Limitations of the BayCoN Plots

As already mentioned, BayCoN plots are for appropriately chosen $Y\in \{I_{calc},\sigma \left(I_{obs}\right)$, $I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda \}$, uniform in the absence of systematic errors. Consequently a non-uniform BayCoN plot produces proof of a systematic connection between the residuals and the entity Y and it additionally visualizes this connection at the same time. For a test of non-uniformity, the Null-Hypothesis is that the plot is uniform and a ${\chi }^2$ test is implemented on a 10 by 10 grid. The level of significance is chosen to be 0.001, which is tolerant of outliers. The test statistics provide 149 as the threshold value: for values lower than 149, the Null-Hypothesis is not rejected and for larger values, a systematic error at the given level of significance is established. For the applicability of the ${\chi }^2$ test, at least 5 events need to be expected for each field of the 10 by 10 grid, which makes $N_{obs}\ge 500$ a requirement for the data set to be analyzed.

A more detailed description of the background and some simple applications are found in [1]. Although formally only $N_{obs}\ge 500$ is required, it turns out in practical applications that the ${\chi }^2$ test is the more sensitive, the larger the number of reflections in the data set. The BayCoN plots could therefore also play an important role in the diagnostics and visualization of systematic errors in macromolecular applications. Another limitation of the BayCoN plots needs to be addressed: When the $\sigma \left(I_{obs}\right)$ are on average too large, this leads may result in deceptively uniform BayCoN plots (see again examples in [1]).

BayCoN plots $(X,Y)$ are uniform in the absence of systematic errors for $X\in \{\zeta ,{\zeta }^2\}$, and $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda \}$. BayCoN plots $({\zeta }^2,Y)$ tend to be more uniform (lower ${\chi }^2$ values) compared to BayCoN plots $(\zeta ,Y)$, because non-uniformities in $\zeta$ are often “smeared out” for ${\zeta }^2$ and maybe also because the minimization of the sum of the squared residuals is the target function in the least squares refinement. The focus is on BayCoN plots $(\zeta ,Y)$ in the present work.

3.3. Complementing Metrics

The residuals are distributed symmetrically with respect to the origin in a data set without systematic errors: positive weighted residuals $\zeta =(I_{obs}-I_{calc})/\sigma \left(I_{obs}\right)>0$ appear (within the limits given by stochastic fluctuations) with the same frequency and amplitude as negative residuals $\zeta <0$. Violations of this basic required symmetry with respect to positive and negative residuals can be quantified by the mean value of the weighted residuals:

$$〈\zeta 〉={\displaystyle \frac{1}{N_{obs}}}\sum _{i=1}^{N_{obs}}{\zeta }_i$$

(2)

with the weighted residual ${\zeta }_i=\frac{I_{obs,i}-I_{calc,i}}{\sigma \left(I_{obs,i}\right)}$ and the weight $\sigma \left(I_{obs,i}\right)$ of the i-th weighted residual where i corresponds to the Miller indices $\left(hkl\right)$ of each unique reflection. The value of $〈\zeta 〉$ is close to zero, when no systematic errors apply. The significance of a deviation from zero is given by

$${\displaystyle \frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}}={\displaystyle \frac{\frac{1}{N_{obs}}{\displaystyle \sum _{i=1}^{Nobs}}{\zeta }_i}{\sqrt{var\left(\zeta \right)/N_{obs}}}}$$

(3)

$$var\left(\zeta \right)={\displaystyle \frac{1}{N_{obs}-1}}\sum _{i=1}^{N_{obs}}{\left({\zeta }_i-〈\zeta 〉\right)}^2$$

(4)

which is obtained by dividing the mean value of the residuals $〈\zeta 〉$ by the square root of the population variance $var\left(\zeta \right)$ of the residuals over $N_{obs}$ [6,7]. If a significant deviation from zero is found, it is helpful to evaluate the fraction of positive excess residuals and its significance in order to discriminate between cases where the significant deviation of $〈\zeta 〉$ is driven by individual large outliers or by frequency—many small deviations. The latter points to a shift of a substantial part of the residual distribution.

Two more metrics for the quantification of systematic errors independent from the BayCoN plots are used in this study. These are the agreement factor ratio [8]:

$$g={\displaystyle \frac{wR\left(F^2\right)}{wR{\left(F^2\right)}_{su}^{pred}}}$$

(5)

with the post refinement weighted agreement factor $wR\left(F^2\right)$ and the predicted agreement factor based on statistical weights $wR{\left(F^2\right)}_{su}^{pred}$. The latter is an idealized value calculated from the reflection input file and the number of parameters used in the model under the assumption of absence of systematic errors. This assumption obviously includes that the standard uncertainties of the observed intensities, the $s.u.\left(I_{obs}\right)$ in Equation (1), adequately describe the noise in the data. The ratio g then gives the factor by which the post refinement agreement factor is increased due to the presence of systematic errors—a metric for the quantification of systematic errors. The difference between g and one is the agreement factor gap. For more details about this concept and for applications, see the open-access publication [8]. The other additional metric used here is the fraction of systematic errors in the variance of the observed intensities:

$${\displaystyle \frac{〈X^2〉}{〈{\sigma }^2\left(I_{obs}\right)〉}}=1-{\displaystyle \frac{〈s.u{.}^2\left(I_{obs}\right)〉}{〈{\sigma }^2\left(I_{obs}\right)〉}}$$

(6)

that specifies how much the variance of the observed intensities ${\sigma }^2\left(I_{obs}\right)$ is inflated due to the presence of systematic errors. The two metrics are connected as follows: the larger $\frac{〈X^2〉}{〈{\sigma }^2\left(I_{obs}\right)〉}$, the larger g.

To summarize: $〈\zeta 〉$ is the signed (positive or negative) mean value of the residuals; $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}$ gives the significance of a deviation of this mean value from zero, g describes the increase in systematic errors in terms of the weighted agreement factor (“costs”) provided that $s.u.\left(I_{obs}\right)$ are adequate, and $\frac{〈X^2〉}{〈{\sigma }^2\left(I_{obs}\right)〉}$ descibes the “costs” of systematic errors in terms of an inflated variance. As a convention, it is adopted that $\left|\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}\right|>3$ indicates a significant deviation from zero.

4. Test Data

A full list of all references for the sample of 314 data sets follows here: [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322].

Before discussing individual data sets as examples for frequently occurring patterns of systematic errors, the sample data is characterized from which the examples are drawn. All data sets published with IUCrData between 2020 and 2022 were examined. Data sets that needed editing or were incomplete were excluded: some publications were just corrigenda without experimental data [17,323,324], in one publication, an unusual format of the embedded diffraction data was used, [325] in one publication, Chebychev polynomials were used for the weighting scheme, [326], and in some data sets, the calculated intensities were not given [327,328,329,330,331,332,333,334]. After discarding the above mentioned data sets, 314 data sets remained in the sample. The same sample was used previously to evaluate other novel data quality metrics [8]. Three from these 314 published data sets contain less then 500 observed reflections. These data sets are excluded when discussing the BayCoN plots associated ${\chi }^2$ values but are otherwise included.

4.1. Diffractometers

The majority of 58.06% of data sets was taken on different Bruker diffractometers, followed by Rigaku (15.16%), Stoe and Cie (11.29%), Oxford Diffraction (3.55%), and others (11.94%).

4.2. Weighting Scheme Parameters

The weighting scheme parameter a ranges between zero and 0.2 [33,253] with a mean value of 0.051 and a median value of 0.048. The weighting scheme parameter b ranges between zero and 226.79 [283] with a mean value of 4.22 and a median of 0.59. The large difference between median and mean is due to many outliers for large values in b like $b=11.24$ [17], $b=32.17$ [54], $b=19.15$ [56], $b=68.62$ [65], $b=74.27$ [77] $b=60.45$ [84], $b=16.23$ [93], $b=24.35$ [183], $b=11.80$ [202] $b=72.24$ [212] $b=35.93$ [233], $b=17.45$ [238], $b=14.30$ [242], $b=21.85$ [257], $b=10.12$ [265], $b=10.84$ [268], $b=173.29$ [277], $b=226.79$ [283], $b=21.85$ [288], and $b=20.50$ [313] to list only those with $b>10$.

Only two data sets (from the same author: [236,278]) employed statistical weights. That is a straightforward yet important observation, as application of a weighting scheme is not needed in the absence of systematic errors, and if the applied weighting scheme is not able to adequately correct the error it is intended to correct for, like in the case of under estimated $s.u.\left(I_{obs}\right)$ (in particular of weak intensities) this can even increase bias [335]. As noted earlier, the application of a weighting scheme is already proof of a systematic error (either in the $s.u.\left(I_{obs}\right)$ or in the model or in both), so it may not be surprising to state that at least 312 out of 314 data sets are affected by systematic errors as in 312 out of 314 data sets a weighting scheme was applied.

The widespread use of weighting schemes in published data suggest that the standard uncertainties of the observed intensities $s.u.\left(I_{obs}\right)$ are routinely underestimated and thus require correction by default. To be more precise, it is likely that, in particular, the $s.u.\left(I_{obs}\right)$ of the strong intensities are systematically underestimated, as a weighting scheme parameter a is used in virtually all cases.

The weighting scheme, however, also corrects for other errors in the model such that these two different cases—error in the $s.u.\left(I_{obs}\right)$ and error in the model—are not separately discussed. Moreover, application of a weighting scheme may introduce new systematic errors. For example, when refining using the least squares method, the weights should not depend on the intensity, but of course they do, as can be quantified by a correlation coefficient (hereafter abbreviated as “cc”) between the observed intensity and the $s.u.\left(I_{obs}\right)$. Across 314 data sets, the average correlation is $cc(I_{obs},s.u.\left(I_{obs}\right)=0.7967$. Application of a weighting scheme increases this already large correlation coefficient to an even higher value $cc(I_{obs},\sigma \left(I_{obs}\right))=0.9590$, i.e., the basic assumption of the residuals being random variables is already violated at this fundamental level. It was pointed out earlier that these correlations may lead to artificially reduced $wR\left(F^2\right)$ and $GoF$ values via induction of negative correlation coefficients $cc({\zeta }^2,{\sigma }^2\left(I_{obs}\right))<0$ (see for example chapter 5.1 in [6] and in particular chapter 5.1.8, where it is stated that “This simulation shows that the agreement factors decrease with increasing underestimation of the $\sigma \left(I_{obs}\right)$ values. This intuitively unexpected decrease is possible due to the induced correlations, which are the stronger the more severely the $\sigma \left(I_{obs}\right)$s are underestimated. From a user perspective this means that underestimation of the $\sigma \left(I_{obs}\right)$s is rewarded”. The failure to discuss this important matter leads to lack of motivation how to do better, and how to arrive at reliable standard uncertainties after data integration. The benefits of reliable standard uncertainties are that these facilitate a learning process about other systematic errors, such that the many intrinsic models invoked in the whole process of a diffraction experiment and the standard procedures could be constantly and quickly improved over time. Data of such a high accuracy may not be needed in order to determine just standard structures, but it may help in neighboring fields like in charge density studies, neutron and electron diffraction experiments, and it helps to identify and remedy methodological weaknesses and errors.

4.3. Distribution of Different Metrics for the Whole Sample

4.3.1. BayCoN Plot-Associated ${\chi }^2$ Values

Figure 1 shows the distribution of BayCoN plot-associated ${\chi }^2$ values as box-plots for the different categories $(\zeta ,Y)$ (a) and $({\zeta }^2,Y)$ (b) on a logarithmic scale to allow for displaying the whole range of values from ${\chi }^2(\zeta ,\sin \theta /\lambda =59.47)$ up to ${\chi }^2(\zeta ,I_c/\sigma \left(I_{obs}\right)=51,654.14)$. The blue horizontal line in the bottom corresponds to the threshold value 149. The median values are displayed as orange lines. The area between the 25% percentile (first quartile, Q1) and the 75% percentile (third quartile, Q3) is filled with blue color. The distance between the 75% percentile and the 25% percentile is called “inter-quartile range” (IQR). The whiskers are placed at Q1 − 1.5 IQR and Q3 + 1.5 IQR. The circles are data points outside the indicated range (flier). Box-plots are used for the compact characterization of distributions. Above each box-plot, information is provided as follows: minimum ${\chi }^2(\zeta ,Y)$ value, maximum ${\chi }^2(\zeta ,Y)$ value, and mean and median. The number of data sets with ${\chi }^2(\zeta ,Y)<149$, i.e., those with a uniform distribution, is given additionally as an absolute number and as a percentage.

All four distributions have the respective median value distinctly above the threshold value. The majority of data sets show non-uniform BayCoN plots in each category Y. The lowest median value is with 203.82 obtained for $Y=\sigma \left(I_{obs}\right)$. There are 96 data sets with ${\chi }^2$ values of the BayCoN plots $(\zeta ,\sigma \left(I_{obs}\right))$ being smaller than 149, which corresponds to a fraction of 30.87% of all data sets with more than 500 reflections ($N=311$). The largest median value is with 278.62 obtained for $(\zeta ,\sin \theta /\lambda )$. This category also shows with only 40 data sets (12.86%), the lowest number of uniform plots.

To complete the picture, Figure 1b gives the analogue information for the squared residuals. These distributions are all much more uniform as can be seen from the median values that are, in three out of four cases, now below the threshold value. The largest median values appear again for $Y=I_{calc}/\sigma \left(I_{obs}\right)$ and $Y=\sin \theta /\lambda$, but they are now with 148.32 and 204.95, and considerably lower.

The maximum ${\chi }^2$ values are, for all BayCoN plots, $({\zeta }^2,Y)$ distinctly lower compared to the corresponding values from $(\zeta ,Y)$ plots with one exception, namely for the resolution $Y=\sin \theta /\lambda$ for which the maximum value drops only slightly from ${\chi }^2(\zeta ,\sin \theta /\lambda )=5897.78$ to ${\chi }^2({\zeta }^2,\sin \theta /\lambda )=5194.57$.

From a total of 311 data sets with $N_{obs}\ge 500$, only a small minority of 11 (3.5%) show uniform BayCoN plots in both $(\zeta ,Y)$ and $({\zeta }^2,Y)$, $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda \}$ as will be discussed in Section 5.7.

4.3.2. Agreement Factor Ratio g and Other Values

The median of g was reported to be 3.31 for the same sample of 314 data sets from IUCrData discussed in the present work [8]. This is a surprisingly large number suggesting that for half or more of the data sets the agreement factor could be greatly reduced, provided that the underlying pre-assumptions are met.

The median of the fraction of systematic errors in the variance of the observed intensities was correspondingly reported to be 0.83, indicating that in half of the data sets, the $s.u{.}^2\left(I_{obs}\right)$ in Equation (1) account for only 17% or less in ${\sigma }^2\left(I_{obs}\right)$. This could be equivalently rephrased into the statement: the $s.u{.}^2\left(I_{obs}\right)$ are inflated in half of the data sets 5.88-fold or more with the help of the weighting scheme.

These large numbers should raise questions. The metrics of Equations (5) and (6) were indeed intentionally constructed to raise questions. The obvious question is that if even in the supposedly easy case of small-molecule standard structures the weighted agreement factors are so much larger than expected from the predicted agreement factor on a regular base and if the weighting scheme is used to inflate the $s.u{.}^2\left(I_{obs}\right)$ to such a large extent on a regular base, is it not likely that the presuppositions are not met? One important presupposition was correctness of $s.u.\left(I_{obs}\right)$; however, it seems plausible that these are on a regular base underestimated for the strong intensities.

5. Diagnosis of Systematic Errors—Application

Table 1. Overview over discussed and mentioned data sets: internal reference code of data set, agreement factor ratio g according to Equation (5), fraction of systematic error in mean variance of observed intensities according to Equation (6), weighting scheme parameters a and b; ${\chi }^2$ values for BayCoN plots $(\zeta ,Y)$ with $Y=I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda$ (standard order), bin scale factor K for the 10% reflections with lowest values of $I_{calc},$$K_1=\frac{〈I_{obs}〉}{〈I_{calc}〉}$ reference. Significance of mean value of weighted residuals according to Equation (3), absorption coefficient $\mu$ and reference.

Data Set	g	$\frac{〈{\mathit{X}}^2〉}{〈{\mathit{\sigma }}^2\left({\mathit{I}}_{\mathbf{obs}}\right)〉}$	a	b	${\mathit{\chi }}^2$	${\mathit{\chi }}^2$	${\mathit{\chi }}^2$	${\mathit{\chi }}^2$	${\mathit{K}}_1$	$\frac{〈\mathit{\zeta }〉}{\mathit{\sigma }\left(〈\mathit{\zeta }〉\right)}$	$\mathit{\mu }$	Reference
	(Equation (5))	(Equation (6))			$(\mathit{\zeta },{\mathit{I}}_{\mathit{calc}})$	$(\mathit{\zeta },\mathit{\sigma }\left({\mathit{I}}_{\mathit{obs}}\right))$	$(\mathit{\zeta },{\displaystyle \frac{{\mathit{I}}_{\mathit{calc}}}{\mathit{\sigma }\left({\mathit{I}}_{\mathit{obs}}\right)}})$	$(\mathit{\zeta },{\displaystyle \frac{\sin \mathit{\theta }}{\mathit{\lambda }}})$		Equation (3)	[mm−1]
Flawed $s.u.\left(I_{obs}\right)$ (Section 5.1):
2021_088	1.4539	0.1163	0.0150	0.0500	1497.03	308.92	1859.74	1083.76	2.02	4.86	0.77	[226]
2020_031	6.3721	0.9506	0.0551	0.0000	283.91	82.66	294.14	182.52	1.66	2.09	0.08	[37]
2020_064	4.7502	0.8280	0.1189	68.6175	968.90	742.32	1263.96	812.27	14.92	18.88	2.89	[65]
2022_045	1.9121	0.8852	0.0523	0.0000	2302.77	1386.49	2450.53	2255.98	1.28	3.15	0.82	[276]
Significance cut-off (Section 5.2):
2020_018	11.3657	0.9967	0.1238	3.8143	2654.92	1065.06	2600.10	762.14	−7.01	−12.28	0.08	[25]
2020_054	5.2586	0.9794	0.0445	0.6409	313.61	185.62	303.19	297.39	3.99	1.23	0.29	[58]
2020_073	7.9499	0.8915	0.0577	2.4014	271.59	236.57	264.65	246.80	4.01	0.16	0.09	[74]
2020_080	3.1707	0.5385	0.0600	0.0000	653.51	259.82	712.50	399.29	18.36	3.29	0.09	[80]
2021_012	3.7749	0.8062	0.0500	2.0000	1629.58	183.40	1694.25	249.57	5.70	0.86	0.10	[151]
2021_018	2.5073	0.0762	0.0253	3.3741	894.57	748.71	685.71	268.24	2.45	−13.25	1.63	[157]
2021_044	8.3321	0.9321	0.0494	24.3530	1744.65	1047.59	1778.95	730.88	4.74	4.15	4.12	[183]
2022_010	12.4101	0.9831	0.0953	14.3038	1812.42	735.26	1998.22	571.72	1.69	−2.29	0.07	[242]
Intensity cut-off (Section 5.3):
2020_029	7.7116	0.9861	0.1087	0.0972	305.27	171.15	390.66	155.30	2.54	1.14	0.08	[35]
2020_046	2.8304	0.8679	0.0517	6.1123	529.00	110.50	602.61	321.59	8.35	9.14	0.15	[51]
2020_074	5.5571	0.9617	0.1532	0.0304	866.30	455.34	2109.38	214.12	2.42	2.97	0.27	[75]
2020_092	1.6338	0.0171	0.0506	0.0000	1236.17	312.76	1689.98	617.73	6.65	19.82	0.09	[91]
2022_046	3.9830	0.8582	0.0359	173.2937	2016.80	1635.85	1811.10	283.84	0.33	−16.22	2.72	[277]
2022_054	2.3415	0.7791	0.0622	0.1317	358.89	457.24	321.17	166.91	0.67	−1.88	1.51	[284]
Weak $I_{obs}$ too strong (Section 5.5):
2022_016	5.6240	0.8892	0.0479	1.2289	1406.00	1187.24	1480.05	390.85	4.40	16.01	2.38	[248]
2020_011	3.0415	0.7686	0.0178	1.2762	1080.84	1012.90	1126.10	1039.08	1.25	11.46	17.34	[19]
2020_039	3.3080	0.7887	0.0717	0.7287	246.52	180.38	379.19	163.09	2.21	5.33	11.87	[45]
2020_119	5.2589	0.9600	0.0600	2.9000	1060.86	637.64	1263.86	374.88	3.19	9.85	1.15	[118]
2020_122	1.8527	0.5512	0.0240	0.0000	998.50	1385.48	1332.03	377.17	1.72	16.28	14.45	[121]
2021_061	5.0337	0.6790	0.0050	1.1433	532.78	523.53	556.01	235.15	1.87	6.89	6.08	[200]
2021_081	4.2205	0.9551	0.0529	0.6752	1369.33	992.03	1323.52	226.74	2.32	4.20	1.59	[219]
2022_052	3.3012	0.8497	0.0431	226.7933	2915.09	397.42	2973.92	423.35	11.15	41.42	23.54	[283]
2022_083	7.3501	0.9543	0.0100	25.0000	999.02	658.24	1248.98	171.44	3.61	14.64	4.46	[312]
Weak $I_{obs}$ systematically weaker than weak $I_{calc}$ (Section 5.6):
2020_003	4.8220	0.9015	0.0525	2.9966	1247.86	1007.46	1073.29	423.93	−2.93	−15.78	0.29	[11]
2020_008	3.5354	0.5087	0.0461	1.5828	783.62	728.95	701.61	175.21	−0.12	−10.91	0.19	[16]
2020_018	11.3657	0.9967	0.1238	3.8143	2654.92	1065.06	2600.10	762.14	−7.01	−12.28	0.08	[25]
2020_024	1.1999	0.3543	0.0349	0.0000	494.50	273.13	494.40	258.45	−0.87	−12.25	2.16	[30]
2021_009	2.4729	0.7696	0.0375	5.4623	1060.86	988.79	943.32	191.58	−3.00	−14.69	1.60	[148]
2021_018	2.5073	0.0762	0.0253	3.3741	894.57	748.71	685.71	268.24	2.45	−13.25	1.63	[157]
2021_054	2.8780	0.8050	0.0239	0.6589	989.51	887.61	987.29	618.01	−1.04	−13.13	1.89	[193]
2021_058	5.0823	0.9649	0.0804	1.0693	1223.38	1231.21	871.45	254.50	−0.19	−10.54	0.66	[197]
2021_085	6.1444	0.9114	0.0578	3.7997	566.19	643.44	493.76	339.27	−1.00	−11.11	0.24	[223]
2022_012	3.6222	0.9330	0.0524	1.0724	1273.89	1358.52	1028.27	290.15	−0.50	−16.81	1.37	[244]
2022_046	3.9830	0.8582	0.0359	173.2937	2016.80	1635.85	1811.10	283.84	0.33	−16.22	2.72	[277]
2022_050	2.6287	0.7725	0.0234	23.3944	1188.70	1210.66	891.84	357.53	0.70	−15.28	3.98	[281]
2022_058	2.5887	0.5885	0.0194	21.8494	1611.50	1576.42	1641.98	1109.17	0.56	−13.89	9.27	[288]
Uniform BayCoN plots (Section 5.7):
2021_042	3.2288	0.8791	0.0512	0.4385	93.14	99.99	132.37	124.08	1.19	2.29	1.20	[181]
2020_012	1.9280	0.3890	0.0065	0.5686	144.90	109.33	142.21	127.26	1.37	−1.96	16.75	[20]
2020_081	1.4835	0.3693	0.0154	0.3678	110.07	73.43	109.68	98.50	1.14	−0.84	10.45	[81]
2020_117	4.7997	0.8482	0.0443	0.2172	99.32	77.02	109.98	133.08	1.58	1.16	0.12	[116]
2020_128	1.6161	0.4453	0.0260	0.8115	83.63	76.72	103.75	125.90	1.32	0.94	0.55	[127]
2021_076	3.9178	0.2439	0.0411	0.0767	97.07	85.33	98.50	105.97	1.24	−0.52	0.49	[214]
2022_024	2.8324	0.8038	0.0321	0.6811	119.96	130.82	143.32	128.18	0.60	−1.79	1.06	[255]
2022_043	1.3355	0.5733	0.0409	0.0000	111.41	140.57	146.17	115.38	0.79	−1.93	9.62	[274]
2022_055	2.0678	0.7640	0.0317	0.0410	124.84	124.23	111.70	108.80	0.88	−0.62	0.21	[285]

displays an overview over all data sets that are discussed or mentioned together with the corresponding data quality metrics and references sorted according to the section of discussion. The individual trends are discussed in the respective section.

5.1. Errors in the $s.u.\left(I_{obs}\right)$

When using the Goodness of Fit as a quality indicator, several assumptions enter under which the $GoF$ is applicable. For example, the residuals are assumed to be random variables that do not correlate with $\sigma \left(I_{obs}\right)$ and are distributed independently and identically. An identical distribution implies that appropriately chosen subsets of the residuals tend to have the same mean value and variance. For example, the mean value and variance of the residuals in this case tend not to change when the residuals are sorted in order of (calculated!) intensity, $\sigma \left(I_{obs}\right)$, resolution, or significance (the significance needs to be based on the calculated intensity as mentioned earlier. In the following text, this will not be stressed any further. When residuals are evaluated with respect to significance, the significance based on the calculated intensity is implied). When, for example, the variance of the residuals changes distinctly, this is a sign for not identically distributed residuals. When the variance of the residuals changes smoothly and continuously with increasing reflection significance, this may indicate a systematic error. Specifically, the error may lie in the standard uncertainties of the observed intensties $s.u.\left(I_{obs}\right)$ (there may be other, yet unknown systematic errors that lead to a similar effect). The effect of flawed $\sigma \left(I_{obs}\right)$ was previously demonstrated (see Figure 5 in [1]) with the help of BayCoN plots by employing artificial data. In these earlier publications, the importance of $Y=I_{calc}/\sigma \left(I_{obs}\right)$ was not yet known, so the corresponding BayCoN plots were not included. Here, we are going to briefly discuss some examples from published data sets and include the BayCoN plots $(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))$. For this discussion, it needs to be understood that almost every data set is contaminated with several different systematic errors. Identifying all errors in each data set is out of the scope of this paper. Instead, experimental data sets are chosen to illustrate the manifestation of specific systematic errors.

Errors in the estimated standard uncertainties $s.u.\left(I_{obs}\right)$ distort the residual distribution and can be recognized through BayCoN plots. Data set 2021_088 [226], a $\mathrm{Pd}\left(\mathrm{II}\right)$ complex (orthorombic space group $P{2}_1{2}_1{2}_1$, $\mathrm{Mo}$ $K_{\alpha }$ radiation, temperature 98 K, XtaLAB AFC12 diffractometer equipped with a mirror monochromator, $a=0.0150$ and $b=0.0500$) illustrates this behavior.

A consequence of the chosen weighting scheme is that parameters of the average variance of the observed intensities $〈{\sigma }^2\left(I_{obs}\right)〉$ are 88.37% of stochastic nature (fraction of statistical weights). The fraction in $〈{\sigma }^2\left(I_{obs}\right)〉$ describing systematic errors is consequently only 11.63% (see Table 1). A weighted agreement factor solely based on statistical weights would result in the absence of systematic errors in a value close to 0.031. The agreement factor based on $\sigma \left(I_{obs}\right)$ is with 0.045 approximately 1.45 times larger. This factor reflects the “costs” of the chosen weighting scheme parameters with respect to the weighted agreement factor.

Figure 2 shows the individual BayCoN $(\zeta ,Y)$ plots with $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right)$, $\sin \theta /\lambda \}$ (standard order) for data set 2021_088 in the given order. The low-intensity reflections (located in the bottom horizontal stripe in Figure 2a) show only moderate residuals, as the points tend to accumulate in the middle of the plot where the moderate residuals reside. The extreme residuals are exclusively connected to the strong intensities (located in the top horizontal stripe in Figure 2a). The largest ${\chi }^2=1859.74$ value is obtained for the dependence of the residuals on the significance (Figure 2c), where the low significant data show only moderate residuals and the high significant data show extreme residuals. This dichotomy violates the assumption of identically distributed residuals.

As a consequence of the dichotomy, the normal probability plot [336] shows deviations for strong positive and strong negative residuals (Figure 2e). The dependence of the residuals from the significance is illustrated in Figure 2f by plotting the weighted residuals against the rank of significance. The amplitude of the residuals $\zeta$ tends to increase with significance (trumpet shape). Figure 2g shows the moving average of the squared weighted residuals, ${\zeta }^2$, which were again sorted beforehand according to the rank of significance, like in Figure 2f. The dashed black line gives the mean value of all squared residuals $\langle {\zeta }^2\rangle$, the average of the entity that is minimized in the least squares procedure and that also enters $GoF$ and $wR\left(F^2\right)$. The black line corresponds to a reference value $\alpha =(N_{obs}-N_{par})/N_{obs}<1$, that is attained in the case of absence of systematic errors. In this case of absence of systematic errors, the moving averages would fluctuate around $\alpha$, i.e., they would not change systematically with the significance (or any other appropriately chosen entity Y) [6]. The difference between the black line and the black dashed line is small compared to the large and systematic changes in the moving averages. The black dashed line describes the net average effect of the large total changes. The moving average is calculated for different windows (50, 100, and 500 consecutive points) and shows consistently under-average values for low-significance and over-average values for high significant data with increasing fluctuations for increasing significance. The missing strong fluctuations for low significant data in the left-hand side of Figure 2g and the presence of strong fluctuations for high significant data in the right-hand side of Figure 2g are interpreted as an additional confirmation for a significance-dependent systematic error that is likely in $\sigma \left(I_{obs}\right)$. This is because systematic overestimation of $\sigma \left(I_{obs}\right)$ for low-significance data would not only lead to low values of the squared residuals, but additionally also to low fluctuations; and underestimation of the $\sigma \left(I_{obs}\right)$ for high significant data would not only lead to large values of ${\zeta }^2$ but also to large fluctuations for high significant data. The plot does show all of these details.

These features indicate underestimated $s.u.\left(I_{obs}\right)$ for strong reflections. Such bias can artificially lower $wR\left(F^2\right)$ and $GoF$, creating a “rewarding” error that conceals real inaccuracies. Recognising this pattern is essential for developing reliable uncertainty estimates and for improving data quality evaluation protocols.

Some consequences of such an significance-dependent error are (i) any metric employing “rare events”, i.e., weighted residuals being larger than 3—like in the detection of contamination with low energy photons or with higher harmonics—involuntarily restrict the search to an artificial limited range in significance and resolution (see Figure 2f). In other words: metrics based on “rare events” are distorted or even compromised.

This analysis of data set 2021_088 reveals a surprising possibility: systematic errors may be embedded within the very procedures used for data quality evaluation! A flawed data evaluation process is not tolerable as the potential for improving data precision and accuracy are veiled, too, and true progress (like establishing the procedures and validation processes to obtain reliable $s.u.\left(I_{obs}\right)$ from the experiment) in the experiment and its methodological aspects and procedures is hampered or even inhibited. The indicated rewarding error (and maybe others) in standardized and accepted data evaluation processes may well be the main obstacle to quick progress in the field.

Moreover, how flawed $s.u.\left(I_{obs}\right)$ affect the model parameter values and model parameter errors in data set 2021_088 and other data sets affected by the same error is not known.

Other data sets with similar characteristics are 2020_031 [37], a pyrazine derivative crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, measured on a STOE IPDS 2T diffractometer with $\mathrm{Mo}$ $K_{\alpha }$ radiation at 193 K; 2020_064 [65] a triiodoborane benzene hemisolvate, again crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, measured on a STOE IPDS II two-circle diffractometer with $\mathrm{Mo}$ $K_{\alpha }$ radiation at 173 K despite application of large weighting scheme parameters $a=0.1189$, $b=68.6175$; and 2022_045 [276], a compound containing a decavanadate(V) anion and crystallizing in the triclinc space group $P\overline{1}$, measured on a STOE Stadivari diffractometer with $\mathrm{Ag}$ $K_{\alpha }$ radiation at 263 K. For more details see Table 1, where the agreement factor ratio is given together with the fraction of systematic error in ${\sigma }^2\left(I_{obs}\right)$, weighting scheme parameters a and b, and the ${\chi }^2$ values for the BayCoN $(\zeta ,Y)$ plots. The corresponding BayCoN plots are found in the Supplementary Materials.

5.2. Significance Cut-Off

Applying a significance or intensity cut-off is a cosmetic manipulation of the data. It artificially reduces the sum of squared residuals. This leads to a lower agreement factor and a lower $GoF$. Simultaneously, it increases the mean squared significance of the observed intensity. It introduces bias as it allows selectively observed intensities with positive fluctuations into the data set and excludes those with negative fluctuations. Also, it may help to disguise the error of underestimated $s.u.\left(I_{obs}\right)$ (in particular, when the $s.u.\left(I_{obs}\right)$ of weak reflections are underestimated), if present [335].

There is no problem with negative intensity observations when refining against $F^2$. A significance or intensity cut-off is not only not necessary, it is even of disadvantage as it hampers the analysis of weak data and of weak significant data. Errors affecting only or mostly weak or weak significant data may be disguised. Moreover, application of an intensity or significance cut-off changes locally the distribution of the weighted residuals, which is a violation of the requirement that the residuals need to be identically distributed random variables. The BayCoN plots in Figure 3 and Figure 4 illustrate this very well. Application of an intensity cut-off increases additionally artificially the mean value and median of the lowest intensity bin and breaks the requirement that positive and negative residuals be symmetric distributed. For weak significant data sets, a significance cut-off may impact a large fraction of the residuals (see Figure 3f).

When a significance or intensity cut-off is applied nevertheless, it is the duty of the author who wishes to apply the cut-off, to prove that this does not affect the model parameter values nor the model parameter value errors instead of tacitly assuming this, without giving evidence, which is the accepted common practise today. If crystallographic journals and crystallographic data banks would ask for this proof every time a cut-off is applied, it would most likely be applied less frequently in the future and simultaneously it would be made sure that the model is not negatively affected in each individual case. For example, the interesting and also disputed [337] case of a Metal Organic Framework structure that specifically targets dioxin molecules [338] could not be analyzed in detail due to application of an intensity cut-off that was not declared in the publication. Attempts to repeat the refinement without cut-off failed as the negative intensity observations were already omitted from the input reflection file. A request to the corresponding data bank to complete the experimental information and another request to the authors to provide the full input reflection file was unsuccessful.

In the popular crystallographic software SHELXL, a default significance cut-off is implemented that replaces the observed intensity with its negative standard uncertainty by default, when an “OMIT” command is not included manually. Many SHELXL users may not even be aware of this default setting. In order to overwrite the default, it is necessary to add the command “OMIT-100” or similar in the shelx input file to make sure that all negative intensity observations are included in the refinement. The command “OMIT-100”, for example, would accept all reflections $F_{obs}^2>-50\sigma \left(F_{obs}^2\right)$. If highly significant negative intensity observations occur, this could be a very valuable hint that the standard uncertainties of the weak intensities are systematically underestimated as discussed in [335].

As an example for how the significance cut-off appears in the BayCoN plots, data set 2020_018 [25] is briefly discussed with the corresponding plots depicted in Figure 3. The chemical structure is a Naphtalene derivative (${\mathrm{C}}_{24}{\mathrm{H}}_{20}{\mathrm{O}}_2$) measured with $\mathrm{Mo}$ K$\alpha$ radiation at 296 K on a Bruker Kappa Apex II CCD diffractometer equipped with a Graphite monochromator. It crystallizes in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{n}$. The weighting scheme parameters are both quite large with $a=0.1238$ and $b=3.8143$.

The fraction assigned to systematic errors in $〈{\sigma }^2\left(I_{obs}\right)〉$ is as large as 99.67% (Table 1) due to the chosen large weighting scheme parameters. This might appear to be an extreme case, however, as many as 25% of all data sets examined show a fraction in $〈{\sigma }^2\left(I_{obs}\right)〉$ assigned to systematic errors of 93.9% or larger. A weighted agreement factor based on $s.u.\left(I_{obs}\right)$ would result in the absence of systematic errors in a value close to 0.025. The agreement factor based on $\sigma \left(I_{obs}\right)$ is with 0.2814; however, $g=11.37$ times larger. The mean significance of the observed intensities with respect to statistical weights is $〈I_{obs}/s.u.\left(I_{obs}\right)〉=14.24$; however, after application of the weighting scheme, it drops to $〈I_{obs}/\sigma \left(I_{obs}\right))〉=2.27$. It is on average not even significant according to the usual 3$\sigma \left(I_{obs}\right)$ criterion. The mean squared significance drops from $〈I_{obs}^2/s.u{.}^2\left(I_{obs}\right)〉=1534.05$ to $〈I_{obs}^2/{\sigma }^2\left(I_{obs}\right)〉=12.33$, to a level of 0.8%. This reflects the combined “costs” of the chosen weighting scheme parameters and remaining systematic errors in the data set with respect to the weighted agreement factor.

The corresponding normal probability plot in Figure 3e shows a little discontinuity in the slope at $\zeta =-1$ caused by the SHELXL default setting (“OMIT-2”) for the significance cut-off. Figure 3f shows that the range of residuals is severely restricted from below for low significant data due to the significance cut-off. The sum of squared residuals, that enters $GoF$ and $wR\left(F^2\right)$, is artificially reduced by application of a cut-off. The residuals tend to spread out less for high significant reflections similar to the trumpet pattern visible in Figure 2f but in the opposite direction and truncated from below due to the applied cut-off. Unfortunately, it was not possible to reproduce the refinement without cut-off to see the full pattern as the reflection input file was not embedded. An unknown systematic error additionally leads to increasing negative median values for decreasing significance (red line). This makes the cut-off even more “effective” for the purpose of reducing the sum of squared residuals. Despite the cut-off, the induced reduced range of residuals for low significant data is the moving average for the squared residuals of the lowest significant reflections still over average as can be seen from Figure 3g. This is a hint towards other systematic errors in this data set as one would expect to have a reduced strength of residuals for low significant reflections treated with a significance cut-off.

A very large fraction of data sets published with IUCrData does show these features related to a significance cut-off, but often not as distinct as in this case. The effect becomes particular strong for data sets that show a low total significance of the observed intensities.

Examples for other data sets displaying significance cut-off effects in the BayCoN plots are, for example:

As seen in 2020_054 [58], a pyrazolinyl thiazole with sum formula ${\mathrm{C}}_{25}{\mathrm{H}}_{18}{\mathrm{ClF}}_2{\mathrm{N}}_5\mathrm{S}$ (with a disordered fluorophenyl group) that crystallizes in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$ and was measured on a Rigaku Oxford Diffraction SuperNova diffractometer with $\mathrm{Mo}$ K_α radiation at 293 K; 2020_073 [74], ${\mathrm{C}}_{19}{\mathrm{H}}_{24}{\mathrm{O}}_5$, that crystallizes in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$ and was measured on a Stoe IPDS 2T with $\mathrm{Mo}$ K_α radiation at 120 K; 2020_080 [80], ${\mathrm{C}}_{29}{\mathrm{H}}_{25}{\mathrm{FN}}_2{\mathrm{O}}_4$, crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{n}$, measured on a Enraf-Nonius CAD-4 at 293 K; 2021_012, a bis-(indolyl)-methane compound with sum formula ${\mathrm{C}}_{27}{\mathrm{H}}_{21}{\mathrm{N}}_3{\mathrm{O}}_6·0.5{\mathrm{C}}_2{\mathrm{H}}_5\mathrm{OH}$, crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, measured again on a Enraf-Nonius with $\mathrm{Mo}$ K_α radiation CAD-4 at 293 K with overlapping author’s list; 2021_018 [157], a solvated bimetallic complex with sum formula $\left[{\mathrm{Cd}}_2{\mathrm{Cl}}_4{\left({\mathrm{C}}_{13}{\mathrm{H}}_{17}{\mathrm{N}}_3\right)}_2\right]·{\mathrm{C}}_2{\mathrm{H}}_5\mathrm{OH}$ crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, measured on a Rigaku Oxford Diffraction SuperNova diffractometer with $\mathrm{Mo}$ K_α radiation at 296 K; 2021_044 [183], ${\mathrm{C}}_{12}{\mathrm{H}}_{17}{\mathrm{I}}_2{\mathrm{O}}_3\mathrm{P}$, crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on a Stoe IPDS 2T diffractometer with $\mathrm{Mo}$ K_α radiation at 120 K with a large weighting scheme parameter $b=24.3530$, and 2022_010 [242], a benzoannulated fulvene with sum formula ${\mathrm{C}}_{42}{\mathrm{H}}_{24}$ crystallizing in the monoclinic space group $\mathrm{C}2/\mathrm{c}$, measured on a Stoe IPDS 2T diffractometer with $\mathrm{Mo}$ K_α radiation at 120 K with a large weighting scheme parameter $b=14.3038$.

The agreement factor ratio ranges in this group between 2.5073 and 12.4101 and the fraction of systematic errors in the variance of the observed intensities between 0.0762 and 0.9967 (see Table 1). For a visualization of the BayCoN plots of the just-mentioned data sets, see the Supplementary Materials.

5.3. Intensity Cut-Off

Figure 4 shows the individual BayCoN plots for data set 2020_046 [51], a coumarin ring system bearing a a trimethylsilyl triazole substituent, that crystallizes in the monoclinic space group $\mathrm{C}2/\mathrm{c}$. The experiment was conducted on a Bruker Smaprt Apex CCD diffractometer with fine-focus sealed tube and Graphite monochromator at 100 K. The absorption coefficient is $\mu =0.15$ mm⁻¹. The weighting scheme parameters are $a=0.0517$ and $b=6.1123$.

The fraction of statistical fluctuations of the total variance of the observed intensities is 0.1321 and the agreement factor ratio is with $wR\left(F^2\right)=0.1249$ and $wR{\left(F^2\right)}_{su}^{pred}=0.0441$ just 2.83.

${\chi }^2(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))=602.61$ (Figure 4c) and ${\chi }^2(\zeta ,I_{calc})=529.00$ (Figure 4a) are the significance- and intensity-related BayCoN plots in accordance with the visual impression that is most non-uniform. An empty lower left corner indicates that negative residuals are missing for low-intensity and low-significance reflections. In contrast to, e.g., Figure 3c, is the bottom horizontal stripe area depopulated starting from the middle of the BayCoN plot. The vertical axis indicates that a surprisingly large fraction of 20–30% of the reflections are affected by the applied intensity cut-off.

${\chi }^2(\zeta ,\sin \theta /\lambda =321.59)$ is distinctly smaller but still greater than the threshold value 149. It would be interesting to see how strong the intensity cut-off affects the resolution-dependent BayCoN plot $(\zeta ,\sin \theta /\lambda )$. For this, it would just be necessary to refine the model against all intensities including negative intensity observations without any cut-off. This was unfortunately not possible as the list of observed intensities was not published with IUCrData and could also not be found under the corresponding CCDC number.

Applying an intensity cut-off hampers the investigation of weak intensities. It violates the assumption of identically distributed residuals. As as result, BayCoN plots become non-uniform. The mean intensity and the mean significance of the data set are both artificially increased by the cut-off and the sum of squared residuals is artificially decreased. Both lead by construction to artificially lowered values in the $GoF$ and $wR\left(F^2\right)$. Even if these artificially induced numerical changes in $GoF$ and $wR\left(F^2\right)$ are small (which needs to be shown for each individual case), the methodology is flawed and should be reconsidered or avoided in the future.

The severe effect of the cut-off on the residuals is not obvious from the normal probability plot (Figure 4e). Obvious effects of an intensity cut-off are that the mean and median of the weakest intensities are shifted to positive values (see the grey and red horizontal lines in Figure 4f), the number of negative residuals is artificially reduced, and the average strength of, in particular, the negative residuals, is artificially reduced. This can be seen from Figure 4g, the “Balance sheet”: the blue and orange bar to the left depict the fraction of positive and negative residuals that fluctuates around 0.5 in the case of a true random process. The 3$\sigma$ error bar attached to the blue bar corresponds to the scaled (because fractions are used rather than absolute numbers) error of a random walk process with equal probability for positive and negative residuals. The next pair of blue and yellow bars provide the absolute mean values of positive and negative residuals and the reference value $\alpha \sqrt{2/\pi }$ (with $0<\alpha =\frac{N_{obs}-N_{par}}{N_{obs}}<1$), which belongs to the idealized case of a Gaussian distribution [6]. The attached 3$\sigma$ error bars show that the positive residuals are significantly stronger than the reference value and that the absolute negative residuals are significantly weaker than the reference value and that positive and absolute negative residuals cannot be considered approximately equally strong. Equal strength of positive and of absolute negative residuals within the limits given by statistical fluctuations are expected for data sets with a symmetric distribution of residuals (not necessarily of Gaussian type). Finally, the last pair of blue and orange bars display the average values of the squared positive and of the squared negative residuals together with their respective error bars and the reference value ${\alpha }^2$. The reference value again conforms to the idealized case of a Gaussian distribution of residuals [6]. The positive squared residuals are again significantly larger compared to the squared negative residuals. In short, the Balance sheet shows that there are less negative residuals than positive and that the absolute negative residuals tend to be significantly smaller compared to the positive ones. For more information about the “Balance sheet” and its applications, see [7], where it is introduced and discussed in greater detail including the reference values and error bars. To which degree the detected imbalances in data set 2020_046 are caused by application of an intensity cut-off cannot be evaluated without access to the input reflection file.

Examples for other data sets displaying intensity cut-off effects in the BayCoN plots are:

As shown in 2020_029 [35], a pyrimidine derivative ${\mathrm{C}}_{10}{\mathrm{H}}_{13}{\mathrm{N}}_5$ crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, measured on a Bruker Smart Apex CCD diffractometer with $\mathrm{Mo}$ K_α radiation at 446 K; 2020_074 [75], a imidazole derivative ${\mathrm{C}}_{31}{\mathrm{H}}_{26}{\mathrm{Cl}}_2{\mathrm{N}}_2\mathrm{O}$ crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on a Bruker Smart Apex CCD diffractometer with $\mathrm{Mo}$ K_α radiation again at 446 K; 2020_092 [91], a kresoxim-methyl derivative ${\mathrm{C}}_{25}{\mathrm{H}}_{25}{\mathrm{N}}_3{\mathrm{O}}_4$ crystallizing in the orthorhombic space group $\mathrm{P}{2}_1{2}_1{2}_1$, measured on a Bruker Smart Apex CCD diffractometer with $\mathrm{Mo}$ K_α radiation at 100 K; 2022_046 [277], a complex containing a hexanuclear {${\mathrm{Nb}}_6$} cluster $\left[{\mathrm{Nb}}_6{\mathrm{Cl}}_{12}{\mathrm{I}}_2{\left({\mathrm{H}}_2\mathrm{O}\right)}_4\right]·8\left({\mathrm{C}}_4{\mathrm{H}}_8\mathrm{O}\right)$ crystallizing in the orthorhombic space group $\mathrm{Pbca}$, measured on a Bruker Smart Apex CCD diffractometer with $\mathrm{Mo}$ K_α radiation at 123 K and with a very large weighting scheme parameter $b=173.2937$ and a disordered tetrahydrofuran mojety; and 2022_054 [284], a thiazole orange derivative bearing an alkene substituent ${\mathrm{C}}_{24}{\mathrm{H}}_{25}{\mathrm{N}}_2{\mathrm{S}}^{+}{\mathrm{I}}^{-}·{\mathrm{H}}_2\mathrm{O}$ crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on a XtaLAB Mini diffractometer with $\mathrm{Mo}$ K_α radiation at 170 K.

The weighted agreement factor ratio g ranges in this group between 1.6338 (2020_092) and 7.7116 (2020_029) and the fraction assigned to systematic errors in the variance of the observed intensities ranges between 0.0171 (2020_092) and 0.9861 (2020_029) (see Table 1). The corresponding BayCoN plots are depicted in the Supplementary Materials. Not for all of these data sets were the observed intensities available.

5.4. Incomplete Absorption Correction Procedures

Incomplete absorption correction causes one-sided positive residuals $〈\zeta 〉>0$. These residuals originate from weak intensities. They contribute dominantly to the ${\chi }^2$ sum [339]. The one-sidedly positive residuals result from weak observed intensities being systematically larger than weak calculated intensities and may even lead to a significant positive shift of the weighted residuals $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}>3$. These criteria together constitute a fingerprint of this specific error. Incomplete absorption correction is a subcategory of errors leading to a trend of $I_{obs}>I_{calc}$ for weak reflections. These are discussed in Section 5.5 below. In the present paragraph, the traces of insufficient absorption correction are discussed with respect to the BayCoN plots. This important topic was not covered in the cited literature.

The mentioned fingerprint traces were observed for absorption coefficients $\mu >5$ mm⁻¹. They were developed with the help of published data sets from [340], where six crystals of different sizes and with different absorption coefficients were measured on a Bruker Apex II diffractometer with microfocus X-ray source and mirror optics at 100 K with $\mathrm{Mo}$ K_α and $\mathrm{Ag}$ K_α radiation. For more details about the crystals and measurements, the reader may consult the cited literature.

The BayCoN plots from a strongly absorbing crystal (scandium platinate, $\mu =121.02$ mm⁻¹, Crystal 1) illustrate the polarization of weak intensities towards positive residuals for incomplete absorption correction clearly, as can be seen from the top row of Figure 5, whereas the bottom row shows the BayCoN plots for a weakly absorbing crystal (inorganic cobalt complex, $\mu =2.87$ mm⁻¹, Crystal 6). The corresponding ${\chi }^2$ values for a test of the respective BayCoN plot against uniformity are given for each of these plots in

Table 2. Crystals from study [340], listed in decreasing order of absorption coefficient for $\mathrm{Mo}$ K_α radiation. Crystal 5 is too large for the diameter of the $\mathrm{Mo}$ microsource of approximately 100 $\mathsf{\mu }$m. Agreement factor ratio g (Equation (5), fraction of systematic error in mean variance of observed intensities (Equation (6)), weighting scheme parameters a and b, ${\chi }^2$ values for BayCoN plots $(\zeta ,Y)$ with $Y=I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right),\sin \theta /\lambda$, and significance of deviation of the mean value of the weighted residuals from zero.

Data Set		g	$\frac{〈{\mathit{X}}^2〉}{〈{\mathit{\sigma }}^2\left({\mathit{I}}_{\mathit{obs}}\right)〉}$	a	b	${\mathit{\chi }}^2$	${\mathit{\chi }}^2$	${\mathit{\chi }}^2$	${\mathit{\chi }}^2$	$\frac{〈\mathit{\zeta }〉}{\mathit{\sigma }\left(〈\mathit{\zeta }〉\right)}$
		Equation (5)	Equation (6)			$(\mathit{\zeta },{\mathit{I}}_{\mathit{calc}})$	$(\mathit{\zeta },\mathit{\sigma }\left({\mathit{I}}_{\mathit{obs}}\right))$	$(\mathit{\zeta },\frac{{\mathit{I}}_{\mathit{calc}}}{\mathit{\sigma }\left({\mathit{I}}_{\mathit{obs}}\right)})$	$(\mathit{\zeta },\frac{\sin \mathit{\theta }}{\mathit{\lambda }})$	Equation (3)
Crystal 1	Scandium platinate [341]	3.1987	0.6444	0.0265	6.5030	1514.93	1407.27	1686.90	386.57	17.75
Crystal 3	Sodium tungstate [342]	2.0825	0.4185	0.0103	1.1388	1745.17	1653.66	1775.41	769.16	25.13
Crystal 4	Scandium cobalt carbide [343]	5.0227	0.8754	0.0174	0.0528	236.06	212.19	252.53	292.02	4.31
Crystal 5	Dibromoacridine derivative—	4.6856	0.9371	0.0260	1.6275	168.63	149.10	177.20	292.21	4.78
Crystal 6	Inorganic cobalt complex [344]	4.0642	0.9104	0.0270	2.4392	148.27	153.40	153.68	749.06	1.43

. All ${\chi }^2$ values are far larger than 149 for scandium platinate, which proves non-uniformity at a significance level of 0.001. The plots for $Y=I_{calc}$, $Y=\sigma \left(I_{obs}\right)$, and $Y=I_{calc}/\sigma \left(I_{obs}\right)$, are—very much in line with the visual impression—similar and with correspondingly large ${\chi }^2$ values of about 1500 whereas the resolution dependence is with ${\chi }^2(\zeta ,\sin \theta /\lambda )=386.57$ and still very large, but considerably smaller compared to the other values that tend to cluster. The situation is reversed for the weak absorbing inorganic cobalt complex, with values close to 150 and thus quite uniform for the categories $Y=I_{calc}$, $Y=\sigma \left(I_{obs}\right)$, and $Y=I_{calc}/\sigma \left(I_{obs}\right)$, but with a distinctly larger value for the resolution dependence, ${\chi }^2(\zeta ,\sin \theta /\lambda )=749.06$.

These observations for the strongly and weakly absorbing crystal form the extremes between which the values for the other crystals fall: going from top to bottom through Table 2, which is also the order of decreasing absorption coefficient, and ignoring Crystal 2, which was an outlier, and is omitted, a pattern is observed as follows: crystals with absorption coefficient $\mu >10$ mm⁻¹ show large values ${\chi }^2(\zeta ,Y\ne \sin \theta /\lambda )>1000$ and are still large, but considerably smaller values ${\chi }^2(\zeta ,Y=\sin \theta /\lambda )<1000$. From $\mu \approx 10$ mm⁻¹ (Crystal 4), the pattern starts to invert, as ${\chi }^2(\zeta ,Y=\sin \theta /\lambda )$ becomes the largest value in the series and is with ${\chi }^2(\zeta ,\sin \theta /\lambda )=749.06$, the by far largest value for the weakly absorbing ($\mu =2.87$ mm⁻¹) Crystal 6 ($d_{max}/d_{min}=4.0$).

Additionally to the characteristics of incomplete absorption correction, the data sets in Table 2 have $b>0$ in common, and a characteristic resolution dependence with largest positive average residuals from low and from high resolution areas. This “spoon”-like resolution dependence was described earlier in the literature [7] as it appears to be very widespread and up to now of unknown origin. The data sets discussed in [7] showing the characteristic resolution dependence all have small absorption coefficients $\mu \le 1.318$ mm⁻¹, and from the Supplementary Materials of the cited publication, it can be seen that the ${\chi }^2(\zeta ,Y)$ values follow the pattern “small, small, small, much larger”, for the BayCoN plots in standard order, similar to Crystal 6 from Table 2. These data sets were also taken from the literature [345] and have the following in common with the data sets from Table 2: (i) they were measured on diffractometers equipped with microfocus and mirror technology at 100 K; (ii) they show $b>0$, and (iii) all have crystal dimensions ranging between $0.14$ mm $<d_{max}<0.32$ mm, i.e., with $d_{max}$ being larger than the beam diameter. This is interpreted as a hint that the characteristic “spoon-like” resolution dependence of the weighted residuals may be connected to anisotropies, that are modelled during absorption correction procedures and may become the dominant source of error in absorption correction procedures for low absorption coefficients, leading to $b>0$ for crystals with virtually no absorption. Yet this is again only a side observation; the focus of the present work is not to identify sources of errors, but merely to demonstrate the usefulness of the BayCoN plots in the description and categorization of systematic errors, the important and often neglected step prior to diagnosing.

Another important topic that is not in the main focus of this work, but should be mentioned, is concerned with the interpretation of the weighting scheme parameters a and b. They may affect the variance of the observed intensities ${\sigma }^2\left(I_{obs}\right)$ quite differently for different data sets. As an example, Crystal 1 ($a=0.0265$) and Crystal 5 ($a=0.0260$) have similar values of the weighting scheme parameter a. Crystal 1 ($b=6.5030$) shows a distinctly larger value of the weighting scheme parameter b compared to Crystal 5 ($b=1.6275$). Yet the average percentage of systematic errors in the variance of the observed intensities is with 93.71% much larger for Crystal 5 that shows lower values in both a and b! How the weighting scheme parameters affect the variance of the observed intensities depends on many factors, like the distribution of intensities and the total scattering power. These factors limit a direct comparison particular of the weighting scheme parameter b between different structures. However, the effect of the weighting scheme parameters on the average percentage of systematic errors in the variance of the observed intensities is still possible and presumably more meaningful.

Crystal 5, with $d_{max}=0.2$ mm the largest Crystal in the set, is the one with largest systematic error in the variance of the observed intensities. g ranges in this set between $g=2.0825$ (Crystal 3) and $g=5.0227$ (Crystal 4), which means that provided the $s.u.\left(I_{obs}\right)$ describe the noise in the $I_{obs}$ adequately, there are remaining systematic errors in Crystal 3 and in Crystal 4, which, when removed, would lead to a weighted agreement factor of $wR\left(F^2\right)=0.017$ for Crystal 3 (instead of 0.033) and of $wR\left(F^2\right)=0.012$ for Crystal 4 (instead of 0.054), a huge potential for improvements—or exaggerated small $s.u.\left(I_{obs}\right)$, or both. Only the weakly absorbing Crystal 6 does not show a significant positive deviation of the mean value of the weighted residuals from zero (see last column in Table 2).

5.5. Weak Observed Intensities Stronger than Weak Calculated Intensities

Insufficient absorption correction leads to characteristic patterns in the BayCoN plots as seen in Figure 5 (top). The data sets in Table 3 show the just-discussed pattern of weak observed intensities being larger than the corresponding weak calculated intensities, and bear similarities in the BayCoN plots with the just discussed pattern for insufficient absorption correction. They were chosen by visual inspection. Table 1 displays the corresponding ${\chi }^2$ values. Most of the data sets in Table 3 follow the pattern “large, large, large, smaller” in the ${\chi }^2$ values of the corresponding BayCoN plots in standard order, like 2022_016, 2020_119, 2020_122, 2021_061, 2021_081, and 2022_083. From these, the data sets 2020_119 and 2021_081, show comparatively small absorption coefficients of only $\mu =1.15$ mm⁻¹ and $\mu =1.59$ mm⁻¹. These two data sets simultaneously show a high anisotropy of the crystal morphology as given by the ratio of the largest and smallest crystal dimension, $\frac{d_{max}}{d_{min}}=5.71$ and $\frac{d_{max}}{d_{min}}=4.70$, respectively. This may be interpreted as a hint that a high anisotropy of a crystal’s morphology may contribute to the pattern of incomplete absorption correction. In that case, it is expected that the pattern of incomplete absorption correction may appear also with small absorption coefficients when the anisotropy of the crystal is very high. Column 12 in Table 3 gives the product between these two entities. It is seen that the low absorbing crystals from data sets 2020_119 and 2021_081 show a similar large value $\frac{d_{max}}{d_{min}}$ $\mu =6.58$ and 7.48, respectively, like data set 2021_061 with larger absorption coefficient $\mu =6.08$ mm⁻¹ and lower anisotropy $\frac{d_{max}}{d_{min}}=1.25$. More research with a larger number of data sets is needed in order to substantiate this assertion or to falsify it.

The data sets not conforming to the pattern “large, large, large, smaller” in the ${\chi }^2$ values of the corresponding BayCoN plots in standard order are 2020_011, that shows an unexpectedly large ${\chi }^2(\zeta ,\sin \theta /\lambda )=1039.08$, i.e., an unexpected strong dependence of the residuals from the resolution. The corresponding BayCoN $(\zeta ,\sin \theta /\lambda )$ plot shows strong and changing polarizations of the residuals with resolution that may be related to problems in scaling or merging of different batches, but this is clearly speculative. Comments from the authors or other participating crystallographers who are confronted with a similar error are highly welcome to clarify this error; 2020_039, with ${\chi }^2(\zeta ,I_{calc}/\sigma \left(I_{obs}\right)=379.19$ being the largest value in the series which frequently happens in disordered structures. Disorder was, however, not mentioned in the manuscript; and 2022_052, where the comparatively low value ${\chi }^2(\zeta ,\sigma \left(I_{obs}\right)=397.42$ breaks the pattern. This low value is accompanied by a large weighting scheme parameter $b=226.7933$ that may serve as an explanation already as the large b value intentionally increases $\sigma \left(I_{obs}\right)$. This data set was refined as an inversion twin (other data sets refined as an inversion twin with a similar pattern of showing low or the lowest ${\chi }^2(\zeta ,Y)$ for $Y=\sigma \left(I_{obs}\right)$ compared to $Y\in \{I_{calc},I_{calc}/\sigma \left(I_{obs}\right)\}$ are 2020_025 (${\chi }^2(\zeta ,I_{calc})=725.88$, ${\chi }^2(\zeta ,\sigma \left(I_{obs}\right))=442.88$, ${\chi }^2(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))=1070.70$, ${\chi }^2(\zeta ,\sin \theta /\lambda )=414.36$, $\mu =0.32$ mm⁻¹), 2020_051 (177.12, 119.83, 207.33, 146.92, $\mu =8.05$ mm⁻¹), 2020_052 (173.20, 106.48, 189.84, 222.21, $\mu =1.60$ mm⁻¹), 2022_020 (359.13, 169.07, 359.92, 574.61, $\mu =1.71$ mm⁻¹).

Some general observations from Table 3 are that only four out of nine data sets are from microsources or with microfocus devices. All structures contain elements at least as heavy as $\mathrm{S}$ or heavier, and all but one contain either $\mathrm{S}$ or $\mathrm{Cl}$ or both. Five out of nine are described as either needle (3) or plate (2), and most are distinctly anisotropic in shape as given by the ratio $d_{max}/d_{min}$ (total sample median: $2.47$). Absorption coefficients tend to be large compared to the whole sample (total sample median: $0.738$ mm⁻¹). The 3rd quartile of absorption coefficients starts at $\mu =2.735$ mm⁻¹, i.e., all absorption coefficients in Table 3 are above median and most are larger than 75% of all absorption coefficients. The four data sets with lowest absorption coefficients $\mu <5.0$ mm⁻¹ are all from the same diffractometer type. For the total sample ($N=314$) $median{(\frac{d_{max}}{d_{min}}·\mu )}_{total}=2.15$. All values in Table 3 are well above the median. This is interpreted as evidence that the product $\frac{d_{max}}{d_{min}}·\mu$ is more important than the individual factors.

A little further down, the data set 2022_016 is discussed in greater detail. The remaining data sets in order of Table 3 are

Table 3. Data sets with weak $I_{obs}$ being systematically larger compared to weak $I_{calc}$: data set, indication of microsource or microfocus techniques, elements in the unit cell, habitus, monochromating device, temperature in the experiment, largest dimension of crystal in mm as given in the respective publication, ratio of largest and smallest crystal dimensions, and diffractometer type. The data sets tend to be connected to larger absorption coefficients and to anisotropic habitus. For references and additional information, see Table 1.

		Radiation	Elements	Habitus	Mono	T	${\mathit{d}}_{\mathit{max}}$	${\mathit{d}}_{\mathit{max}}·\mathit{\mu }$	$\frac{{\mathit{d}}_{\mathit{max}}}{{\mathit{d}}_{\mathit{min}}}$	$\mathit{\mu }$	$\frac{{\mathit{d}}_{\mathit{max}}}{{\mathit{d}}_{\mathit{min}}}·\mathit{\mu }$	Diffractometer
					Chromator	[K]	[mm]			[mm−1]	[mm−1]
2022_016	ms	Cu K$\alpha$	S, O, N, C, H	needle	mirror	100	0.04	0.10	4.00	2.38	9.54	XtaLAB Synergy
2020_011	—	Cu K$\alpha$	Bi, S, N, C, H	needle	mirror	100	0.12	2.11	4.51	17.34	78.14	Agilent SuperNova
2020_039	—	Mo K$\alpha$	Cu, I, S, N, C, H	plate	graphite	293	0.50	5.93	3.85	11.87	45.63	Stoe IPDS 1
2020_119	—	Mo K$\alpha$	Cu, S, F, O, N, C, H	plate	mirror	100	0.40	0.46	5.71	1.15	6.58	XtaLAB AFC12
2020_122	ms	Mo K$\alpha$	Pb, O, C, H	prism	mirror	120	0.25	3.60	1.54	14.45	22.21	Bruker D8
2021_061	ms	Mo K$\alpha$	Nb, Cl, O, H	block	mirror	123	0.20	1.22	1.25	6.08	7.60	Bruker Apex II CCD
2021_081	—	Mo K$\alpha$	Cu, Cl, S, F, O, N, C, H	block †	mirror ‡	98	0.47	0.75	4.70	1.59	7.48	XtaLAB AFC12
2022_052	—	Mo K$\alpha$	Cu, S	fragment	?	296	0.11	2.47	1.62	23.54	38.02	Bruker D8
2022_083	ms	Cu K$\alpha$	Zn, Cl, S, F, O, C, H	needle	mirror	100	0.22	0.97	5.59	4.46	24.93	XtalAB Synergy Dualflex

^† Actual dimensions are 0.47, 0.17, and 0.10 mm. ^‡ The monochromating device was not given in the cif file but in another publication [226] with the same corresponding author and diffractometer type, the monochromating device is specified as mirror. The question mark indicates that this information was not specified.

Table 3. Data sets with weak $I_{obs}$ being systematically larger compared to weak $I_{calc}$: data set, indication of microsource or microfocus techniques, elements in the unit cell, habitus, monochromating device, temperature in the experiment, largest dimension of crystal in mm as given in the respective publication, ratio of largest and smallest crystal dimensions, and diffractometer type. The data sets tend to be connected to larger absorption coefficients and to anisotropic habitus. For references and additional information, see Table 1.

		Radiation	Elements	Habitus	Mono	T	${\mathit{d}}_{\mathit{max}}$	${\mathit{d}}_{\mathit{max}}·\mathit{\mu }$	$\frac{{\mathit{d}}_{\mathit{max}}}{{\mathit{d}}_{\mathit{min}}}$	$\mathit{\mu }$	$\frac{{\mathit{d}}_{\mathit{max}}}{{\mathit{d}}_{\mathit{min}}}·\mathit{\mu }$	Diffractometer
					Chromator	[K]	[mm]			[mm−1]	[mm−1]
2022_016	ms	Cu K$\alpha$	S, O, N, C, H	needle	mirror	100	0.04	0.10	4.00	2.38	9.54	XtaLAB Synergy
2020_011	—	Cu K$\alpha$	Bi, S, N, C, H	needle	mirror	100	0.12	2.11	4.51	17.34	78.14	Agilent SuperNova
2020_039	—	Mo K$\alpha$	Cu, I, S, N, C, H	plate	graphite	293	0.50	5.93	3.85	11.87	45.63	Stoe IPDS 1
2020_119	—	Mo K$\alpha$	Cu, S, F, O, N, C, H	plate	mirror	100	0.40	0.46	5.71	1.15	6.58	XtaLAB AFC12
2020_122	ms	Mo K$\alpha$	Pb, O, C, H	prism	mirror	120	0.25	3.60	1.54	14.45	22.21	Bruker D8
2021_061	ms	Mo K$\alpha$	Nb, Cl, O, H	block	mirror	123	0.20	1.22	1.25	6.08	7.60	Bruker Apex II CCD
2021_081	—	Mo K$\alpha$	Cu, Cl, S, F, O, N, C, H	block †	mirror ‡	98	0.47	0.75	4.70	1.59	7.48	XtaLAB AFC12
2022_052	—	Mo K$\alpha$	Cu, S	fragment	?	296	0.11	2.47	1.62	23.54	38.02	Bruker D8
2022_083	ms	Cu K$\alpha$	Zn, Cl, S, F, O, C, H	needle	mirror	100	0.22	0.97	5.59	4.46	24.93	XtalAB Synergy Dualflex

^† Actual dimensions are 0.47, 0.17, and 0.10 mm. ^‡ The monochromating device was not given in the cif file but in another publication [226] with the same corresponding author and diffractometer type, the monochromating device is specified as mirror. The question mark indicates that this information was not specified.

In 2020_011 [19], a bismuth compound $\mathrm{Bi}{\left({\mathrm{C}}_9{\mathrm{H}}_6{\mathrm{NS}}_2\right)}_3$ with absorption coefficient $\mu =17.34$ mm⁻¹ crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on an Agilent SuperNova Dual diffractometer with $\mathrm{Cu}$ K_α radiation and mirror monochromator at 100 K. The crystal dimensions are given with 0.12, 0.06, and 0.03 mm and the crystal is described as a needle;

2020_039 [45], a three-dimensional copper polymer with ${\left[C{u}_X{I}_x\right]}_n$ staircase motif ($x=4$) and sum formula $[{\mathrm{Cu}}_4{\mathrm{I}}_4{\left({\mathrm{C}}_8{\mathrm{H}}_8{\mathrm{N}}_2{\mathrm{S}}_2\right]}_{\mathrm{n}}$ crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on a Stoe IPDS-1 diffractometer equipped with a plane graphite monochromator. $\mathrm{Mo}$ K_α radiation was used at 293 K; the absorption coeffcient is given with $\mu =11.87$ mm⁻¹ and the crystal is described as a plate with dimensions 0.50, 0.35, and 0.13 mm;

2020_119 [118], a copper salt ${\left\{\left[\mathrm{Cu}\left({\mathrm{CF}}_3{\mathrm{SO}}_3\right){\left({\mathrm{CH}}_3\mathrm{CN}\right)}_2\left({\mathrm{C}}_{12}{\mathrm{H}}_{12}{\mathrm{N}}_2{\mathrm{O}}_2\right)\right]\left({\mathrm{CF}}_3{\mathrm{SO}}_3\right)\right\}}_{\mathrm{n}}$ with cations forming polymeric chains crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on an XtaLAB AFC12 diffractometer equipped with mirror monochromator and rotating anode tube with $\mathrm{Mo}$ K_α radiation at 100 K. The absorption coefficient is $\mu =1.15$ mm⁻¹, the crystal is described as a plate with dimensions 0.40, 0.10, and 0.07 mm;

2020_122 [121], a redetermination of $\{{\left[{\mathrm{Pb}}_3{\left({\mathrm{C}}_4{\mathrm{H}}_7{\mathrm{O}}_3\right)}_6{\left({\mathrm{H}}_2\mathrm{O}\right)}_2\right]}_{\mathrm{n}}$ (space group $\mathrm{P}{2}_1/\mathrm{n}$, T = 120 K, $\mu =14.45$ mm⁻¹) measured on a Bruker D8 Venture diffractometer with multilayer optics and $\mathrm{Mo}$ K_α radiation. The crystal is described as a prism with dimensions 0.25, 0.19, and 0.16 mm;

2021_061 [200], a niobium cluster cation with two non-coordinating charge-balancing iodide ions $\left[{\mathrm{Nb}}_6{\mathrm{Cl}}_{12}{\left({\mathrm{H}}_2\mathrm{O}\right)}_6\right]{\mathrm{I}}_2$, $\mu =6.08$ mm⁻¹, crystallizing in the trigonal space group $\mathrm{P}\overline{3}1\mathrm{m}$, measured on a Bruker Apex II CCD diffractometer equipped with microfocus sealed tube with $\mathrm{Mo}$ K_α radiation at 123 K. The crystal is described as a block with dimensions 0.20, 0.20, and 0.16 mm;

2021_081 [219] reported as the first example of a binuclear cation of the type $\left[\mathrm{Cu}{\left(\mathrm{terpy}\right)}_2\right){\mathrm{Cl}}_2{]}^{2+}$ with trifluoromethanesulfonate counter ions, $\mu =1.59$ mm⁻¹, crystallizing in the triclinic space group $\mathrm{P}\overline{1}$, measured on a XtaLAB AFC12 diffractometer with $\mathrm{Mo}$ K_α radiation at 98 K. The crystal is described as a block with dimensions 0.47, 0.17, and 0.10 mm.

2022_052 [283], a djurleite crystal ${\mathrm{Cu}}_{61.39}{\mathrm{S}}_{32}$ crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{n}$, $\mu =23.54$ mm⁻¹, measured on an Bruker D8 Venture Photon 100 CMOS diffractometer with $\mathrm{Mo}$ K_α radiation at 296 K and with extreme weighting scheme factor $b=226.7933$. The crystal is described as a fragment with dimensions 0.11, 0.07, and 0.07 mm and refined as a two-component inversion twin;

2022_083 [312] a ${\mathrm{Zn}}^{\mathrm{II}}$ complex $\left[\mathrm{Zn}{\left({\mathrm{C}}_{15}{\mathrm{H}}_{10}{\mathrm{ClN}}_3\right)}_2\right]{\left({\mathrm{CF}}_3{\mathrm{SO}}_3\right)}_2$ with $\mu =4.46$ mm⁻¹, crystallizing in the monolinic space group $\mathrm{P}{2}_1/\mathrm{c}$ measured on a microfocus XtaLAB Synergy Dualflex HyPix diffractometer equipped with mirror monochromator, measured with $\mathrm{Cu}$ K_α radiation at 100 K (weighting scheme factor $b=25.00$). The crystal is described as a needle with dimensions 0.22, 0.07, and 0.04 mm.

The weighted agreement factor ratio g ranges in this group between 1.8527 (2020_122) and 7.3501 (2022_083) and the fraction of the systematic error in the variance of the observed intensities ranges between 0.5512 (2020_122) and 0.9600 (2020_119, see Table 1).

A particular interesting example is data set 2022_016 [248], a coumarin derivative with sum formula ${\mathrm{C}}_{16}{\mathrm{H}}_9{\mathrm{NO}}_2\mathrm{S}$ that crystallizes in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$. The experiment was conducted on a XtaLAB Synergy diffractometer with a microfocus $\mathrm{Cu}$ K_α source at 100 K. The absorption coefficient is given with $\mu =2.38$ mm⁻¹. The crystal is described as a needle with small dimensions 0.04, 0.01, and 0.01 mm which fit in the focused X-ray beam. The weighting scheme parameters are $a=0.0479$, $b=1.2289$. This choice of weighting scheme parameters leads to a fraction of 11.08% of statistical weights on average and to an agreement factor ratio of 5.62, i.e., the resulting agreement factor is 5.62 times as large as expected in the absence of systematic errors.

The corresponding BayCoN plots in Figure 6 show a striking similarity with those from the strong absorbing Crystal 1 from above as diplayed in Figure 5. The ${\chi }^2$ values tend to be larger than 1000 with the exception of $Y=\sin \theta /\lambda$, which shows the lowest value ${\chi }^2=390.85$.

The corresponding normal probability plot in Figure 6e shows one-sidedly positive outliers in the residuals as a consequence of too strong weak observed intensities. The median of the residuals from the weakest calculated intensities is shifted to positive values (Figure 6f). Positive and negative residual values are “out of balance”: There are significantly more positive residuals than negative ones (Figure 6g, left), the positive residuals are significantly larger than the negative ones (middle) and the squared positive residuals are, on average, approximately twice as large (exact value: 2.09) as the negative squared residuals (right). A consequence of the disbalance is that the residuals are significantly shifted to positive values: $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}=+16.01$. All data sets of this error type show this significant positive shift (see Table 1) that affects the weak intensities and weak significant intensities most (Figure 6h).

Why does the relative weak (weak absolute value of 2.38 mm⁻¹, however, above the overall median value of the whole sample) absorbing coumarine ring system from [248] show a similar error profile to the strongly absorbing scandium platinate (Crystal 1)? In both cases, tiny crystals in combination with microsource techniques were used. Both crystals are anisotropic in shape. A plausible explanation could be a not well-centered crystal in [248].

This hypotheses could probably be answered by the authors of the study, who are kindly invited to collaborate on this question. It would also explain the unexpected existence of uniform BayCoN plots in another data set (2020_081 [81], see also Section 5.7 and Table 1 and Table 5) from an octahedral platinum(IV) complex measured with microfocus technique on an XtaLAB diffractometer with a tiny, needle-shaped crystal (0.05, 0.02, and 0.01 mm) with absorption coefficient $\mu =10.45$ mm⁻¹ at 100 K and with $\mathrm{Mo}$ K_α radiation. Like in the above cases, the crystal in [81] is very small, the absorption coefficient is comparable to Crystal 4 from Table 2, and yet the BayCoN plots are all uniform and a significant shift of the mean value of the residuals with $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}=-0.84$ also does not exist.

When the centering of the crystal is an essential source of this specific error, it does make sense that this is seen in the data sets with microfocus techniques, as it is the most sensitive technique with respect to centering due to the small beam diameter and the strongly inhomogeneous beam without a top-hat profile. It also makes sense that this is seen for crystals that are small enough to fit the beam size like in data set 2022_016 and for crystals with relative weak absorption coefficients and with anisotropic crystal morphology, that easily leads to only parts of the crystal being temporarily out of the focus of the beam. Both absorption and anisotropy of the crystal are contributing factors. Furthermore, the error may be expected for conventional X-ray sources as well if the crystal is severely not centered. Conventional sources are included in Table 3, indeed. Yet, if the crystal is well centered, there is nothing to amplify and this would lead to data sets without the characteristic polarization of the residuals with respect to $I_{calc}$ like in the above mentioned data set 2020_081 [81].

If the centering of the crystal is of such great importance, corresponding appropriate metrics describing the centering should be included in the cif files. The authors of [81,248,340] are kindly invited and encouraged to participate in this development by providing metrics for the centering of the crystals in their respective studies to validate or to falsify the findings. In summary, it can be said that the appearance of a pattern of insufficient absorption correction for comparatively weak absorbing crystals is unexpected and may stress the need for further improvements in absorption correction procedures.

5.6. Weak $I_{obs}$ Systematically Weaker than Weak $I_{calc}$

If weak observed intensities ($I_{obs}$) are systematically lower than calculated ones ($I_{calc}$), a polarization of residuals occurs. This polarization is most evident when the residuals are plotted against the order of $I_{calc}$ (see Figure 7f). Often, additionally also, polarization with respect to $Y\in \{\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right)\}$ is observed. Consequently, the ${\chi }^2$ values for the BayCoN plots $(\zeta ,Y)$, $Y\in \{I_{calc},\sigma \left(I_{obs}\right),I_{calc}/\sigma \left(I_{obs}\right)\}$ are often distinctly larger compared to $Y=\sin \theta /\lambda$ again, like in the case just discussed in Section 5.4 and Section 5.5. This again leads to a recognizable pattern in the ${\chi }^2$ values of the corresponding BayCoN plots. This pattern is followed even more strictly as can be seen from Table 1. However, application of an intensity or a significance cut-off may counteract, to some extent, and may even hamper observation of this phenomenon in BayCoN plots involving $Y=\sigma \left(I_{obs}\right)$ and $Y=I_{calc}/\sigma \left(I_{obs}\right)$. When in the SHELXL instruction file a significance cut-off is not specified, the software silently applies the command “OMIT-2” by default. As a consequence, reflections $I_{obs}<-\sigma \left(I_{obs}\right)$ are replaced by $-\sigma \left(I_{obs}\right)$, leading to $I_{obs}/\sigma \left(I_{obs}\right)\ge -1$ for negative intensity observations and to an artificial high number of weighted residuals $\zeta \approx -1$ which all stem from those negative intensity observations, where simultaneously $I_{calc}\approx 0$. This unjustified procedure has severe effects: negative intensity observations are truncated, leading to an artificial increase in the mean value and the significance of negative intensity observations and truncating the lower limit of residuals from negative intensity observations at −1. As all of this happens silently in the background, many crystallographers are not even aware of this procedure. All data sets with too low weak observed intensities listed in Table 1 also show a significant negative shift of the mean value of the residuals with respect to zero (see Table 1). The bin scale factor $K=〈I_{obs}〉/〈I_{calc}〉$ for the bin of lowest 10% of $I_{calc}$ and $I_{calc}/\sigma \left(I_{obs}\right)$ is often distinctly smaller than one or even negative (see Figure 7h and Table 1)—despite the counteracting effect of the application of significance cut-off!

Common features in

Table 4. Overview over data sets with weak $I_{obs}$ weaker than weak $I_{calc}$.

		Radiation	Elements	Habitus	Monochromator	T	${\mathit{d}}_{\mathit{max}}$	${\mathit{d}}_{\mathit{max}}·\mathit{\mu }$	$\frac{{\mathit{d}}_{\mathit{max}}}{{\mathit{d}}_{\mathit{min}}}$	$\mathit{\mu }$	$\frac{{\mathit{d}}_{\mathit{max}}}{{\mathit{d}}_{\mathit{min}}}·\mathit{\mu }$	Diffractometer	Reference
						[K]	[mm]			[mm−1]	[mm−1]
2020_003	—	Mo K$\alpha$	S, O, N, C, H	block	?	293	0.56	0.16	4.85	0.29	1.39	Bruker D8	[11]
2020_008	—	Mo K$\alpha$	Cl, O, N, C, H	block	?	297	0.45	0.08	1.29	0.19	0.24	Bruker Smart Apex	[16]
2020_018	—	Mo K$\alpha$	O, C, H	block	Graphite	296	0.20	0.02	1.33	0.08	0.11	Bruker Kappa Apex	[25]
2020_024	—	Mo K$\alpha$	Br, S, O, N, C, H	block	Graphite	293	0.30	0.65	1.50	2.16	3.25	Bruker Kappa Apex	[30]
2021_009	—	Mo K$\alpha$	Ba, Co, S, O, N, C, H	block	?	296	0.20	0.32	1.33	1.60	2.13	Bruker Apex	[148]
2021_018	—	Mo K$\alpha$	Cd, Cl, N, O, C, H	block	?	296	0.16	0.27	1.66	1.63	2.70	Rig. Ox. Diff. SuperNova	[157]
2021_054	—	Mo K$\alpha$	Cd, Ni, N, C, H	block	Graphite	123	0.15	0.29	2.34	1.89	4.41	Rig. Ox. Diff. Excal.	[193]
2021_058	—	Mo K$\alpha$	Ni, O, N, C, B, H	block	Graphite	150	0.59	0.39	1.85	0.66	1.22	Bruker Apex	[197]
2021_085	—	Mo K$\alpha$	Cl, O, C, N, H	block	?	293	0.22	0.05	1.38	0.24	0.32	Bruker Smart Apex	[223]
2022_012	—	Mo K$\alpha$	Ni, Cl, S, C, H	column	Graphite	170	0.27	0.37	2.85	1.37	3.92	Bruker D8	[244]
2022_046	mf	Mo K$\alpha$	Nb, Cl, I, O, C, H	block	?	123	0.23	0.63	1.64	2.72	4.47	Bruker Apex	[277]
2022_050	—	Mo K$\alpha$	Ir, Cl, P, F, C, N, H	plate	?	100	0.27	1.07	3.86	3.98	15.35	Bruker Apex	[281]
2022_058	ms	Cu K$\alpha$	Ir, P, F, O, N, C, H	column	?	100	0.21	1.86	3.24	9.27	30.06	Bruker D8	[288]

are diffractometers with microfocus or microsource technology are only represented two times. With the exception of data set 2020_018, all data sets show elements at least as heavy as $\mathrm{S}$. Block-shaped crystals appear frequently, resulting in a majority of the crystals having $d_{max}/d_{min}<2$ in contrast to Table 3. Only two absorption coefficients are in the 3rd quartile. The product $\frac{d_{max}}{d_{min}}·\mu <5$ mm⁻¹ for 11 from 13 data sets in Table 4. The experimental temperatures also tend to be larger compared to Table 3. The bin scale factor $K_1={\displaystyle \frac{〈I_{obs}〉}{〈I_{calc}〉}}$ for the 10% of the weakest reflections in $I_{calc}$ range between −7.01 for data set 2020_018 and 2.45 for data set 2021_018 with 9 from 13 values $K_1<0$ and 12 from 13 $K_1<1$ as can be seen from Table 1. All data sets in this category show highly significant negative deviations $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}<-3$.

As an example, data set 2020_003 [11], a salt composed of a benzene-1,2-diaminium dication and a pair of sulfonate anions is briefly discussed. The structure was taken on a Bruker diffactometer at 293 K with $\mathrm{Mo}$ K_α radiation. It crystallizes in the monoclinic space group $\mathrm{C}2/\mathrm{c}$. The crystal was described as block with dimensions 0.56 × 0.14 × 0.12 mm. The weighting scheme parameters were set to $a=0.525$, $b=2.9966$. Figure 7a–d show the corresponding $(\zeta ,Y)$ BayCoN plots and corresponding ${\chi }^2$ values.

${\chi }^2(\zeta ,\sin \theta /\lambda )=423.93$ is by far the lowest ${\chi }^2$ value, but still larger than the threshold value 149. The disruptive effect of an applied significance cut-off is clearly visible in Figure 7a,c, as it distorts the distribution of the residuals for weak $I_{calc}$.

The polarization of the residuals due to weak $I_{obs}$ being smaller than the weak $I_{calc}$ is therefore best visible in the BayCoN $(\zeta ,\sigma \left(I_{obs}\right))$ plot (Figure 7b), where it appears as increased density of points in the lower left and upper right corners and diffuse depopulated areas in the top left and bottom right.

The normal probability plot (Figure 7e) indicates presence of systematic errors. The weighted residuals are polarized with respect to $I_{calc}$: the residuals of the weak $I_{calc}$ tend to negative values, those of the stronger $I_{calc}$ to positive values (Figure 7f). In total, there are significantly more negative than positive residuals (Figure 7g, left) and the negative residuals tend to be absolute stronger (Figure 7g, middle, right) compared to the positive residuals. The ratio $K=〈I_{obs}〉/〈I_{calc}〉$ shows for the bin of the 10% lowest calculated intensities a distinct negative value $K\approx -3$ and similar for the 10% lowest significant reflections (Figure 7h) despite application of a significance cut-off that artificially shifts this ratio in favour of positive values.

Figure 7e,g together show that the significant shift of the residuals towards negative values (as given in Table 1) is not driven by a few exceptional very large negative outliers—which would appear in the left part of Figure 7e, but by a shift of the total distribution of residuals to negative values resulting in a significant larger frequency of negative residuals compared to positive ones, Figure 7g (left). This shift of the mean value of the residuals to negative values also leads to slightly stronger absolute negative residuals compared to the positive ones, Figure 7g (middle), but not to exceptional large contributions from very strong negative residuals, as this would drive the mean value of the negative squared weighted residuals in Figure 7g (right) to much larger values. The negative shift of the residuals is best thought of as a shift of the median and it is a common pattern in all data sets of this category listed in Table 1.

To discuss the other data sets:

In some data sets, an extinction coefficient was refined. For example, in data set 2020_008, $x=0.0135$ for x in $F_{calc}^{\ast }=k{F}_{calc}{\left[1+0.001x{F}_{calc}^2{\lambda }^3/\sin \left(2\theta \right)\right]}^{-1/4}$ with the overall scale factor k and the corrected intensity $F_{calc}^{\ast }$. The corresponding scatter plot of observed versus calculated intensities is depicted in Figure 8a. It shows two outliers $I_{obs}>I_{calc}$ for the strongest reflections. When an extinction model is not applied, the scatter plot in Figure 8b results. While Figure 8b indeed seems to indicate that extinction should be modelled, Figure 8a seems to indicate that extinction was over-corrected, as it almost shows an inverted extinction pattern with $I_{obs}>I_{calc}$, or, that an extinction correction was applied erroneously to compensate for what was in fact another error. Similar for data set 2021_085 with extinction coefficient $x=0.0169$: the scatter plot shows many cases $I_{obs}>I_{calc}$ for the strong reflections (Figure 8c). If an extinction coefficient is not refined, $I_{obs}<I_{calc}$ results (Figure 8d). This phenomenon, where an extinction coefficient $x>0$ results in $I_{obs}$ being increasingly larger than the corresponding $I_{calc}$, and which results in $I_{obs}<I_{calc}$ for strong reflections, when extinction is not modelled ($x=0$), may be called the “inverse extinction” phenomenon. Its occurrence in combination with data sets showing $I_{obs}<I_{calc}$ for the weak intensities may be accidental or not. This is a separate topic and out of the scope of the present work. Another interesting case is data set 2020_024, where extinction was not part of the model, resulting in $a=0.0349$, $wR\left(F^2\right)=0.0833$, a highly significant negative mean value of the residuals $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}=-12.25$, and the plot shown in Figure 8e. Modelling extinction ($x=0.002711$, Figure 8f) resulted in a smaller weighting scheme parameter $a=0.0217$, a lower agreement factor $wR\left(F^2\right)=0.0761$, but changes the significance of the mean value of the residuals only very little, $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}=-11.94$. It remains highly significant and negative. The scatter plot of data set 2021_054 also shows systematically $I_{obs}<I_{calc}$.

The authors of the data sets listed in Table 1 are invited to help identifying individual sources of systematic errors that all lead to a significant shift of residuals to negative values driven by frequency rather than strength. In doing so, they would greatly support the process to find specific sources of errors and to make this knowledge publicly available to the whole crystallographic community to the benefit of data submitting authors, journals, crystallographic data banks, and users of crystallographic data banks.

In summary, it is seen from the present and from the preceding paragraph, that the analysis of the weighted residuals for weak $I_{calc}$ leads to patterns of over- and underestimation of $I_{obs}$ as compared to $I_{calc}$ (note that a comparison of $I_{obs}$ and $I_{calc}$ leaves the question open, where the error emerges from—a model ($I_{calc}$), or experimental data ($I_{obs}$)—which is indeed not known for most cases discussed here with the exception of insufficient absorption correction, where it is known from the literature that the weak $I_{obs}$ tend to be overestimated). This leads to the polarization of the residuals with respect to $I_{calc}$ and to significant (positive and negative) shifts of the whole distribution of the weighted residuals as quantified by $〈\zeta 〉/\sigma \left(〈\zeta 〉\right)$. The polarization of the residuals can be further quantified for example by correlation and rank correlation coefficients $cc(\zeta ,I_{calc})$, $cc(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))$, $rcc(\zeta ,I_{calc})$, $rcc(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))$, where “$cc$” abbreviates Pearson’s correlation coefficient and “$rcc$” abbreviates Spearman’s rank correlation coefficient. As a consequence, the residuals are not random variables any more. Metrics like the Goodness of Fit and the weighted agreement factor are based on the assumption that the residuals are uncorrelated random variables (the $GoF$–and indeed the whole least squares process—relies on the ${\chi }^2$ distribution, which relies on the distribution of identical and independent Gaussian random numbers). The metrics based on this requirement are biased themselves when this requirement is not met. As a consequence, there is a systematic error in the fit evaluation process itself, which often lets the fit appear to be of a higher quality than it actually is—the systematic error is rewarding.

5.7. Uniform BayCoN Plots

There are 11 data sets with uniform BayCoN plots in the sample. None of those 11 data sets employ statistical weights. Two are excluded from the analysis due to a very small number of $N_{obs}=33$ (2022_051, [282]) and 250 (2021_074, [213]). Some details of the remaining data set are given in Table 5.

From the remaining nine data sets are all but one in the lowest quartile of $N_{obs}$ (first quartile: 2860; median: 3981; third quartile 6315) and none in the first quartile of mean significance $〈I_{obs}/\sigma \left(I_{obs}\right)〉$ of the data set (which are surprisingly low: first quartile: 5.68; median: 7.56; third quartile 10.74). The relative high significance $〈I_{obs}/\sigma \left(I_{obs}\right)〉$ of the data sets may be owed to the large fraction of experimental microsource or microfocus setups.

The ${\chi }^2$ test for the BayCoN plots is more sensitive with a larger number of reflections. With a low number of reflections, even visible systematic errors may fall below the detection limit. An example for this is data set 2021_076: the BayCoN plots $(\zeta ,Y)$, $Y\in \{I_{calc},I_{calc}/\sigma \left(I_{obs}\right)\}$ clearly indicate the application of an intensity cut-off; however at the given level of significance and with $N_{obs}=562$ only, this clearly visible feature remains insignificant. A feature not being significant does not imply absence of systematic errors, it just implies that a possibly existing systematic error falls below the sensitivity threshold of this method at the chosen level of significance. It may be detectable by other metrics.

Surprisingly, there are three data sets with comparatively large absorption coefficients $\mu =9.62$ mm⁻¹ (2022_043), $\mu =10.45$ mm⁻¹ (2020_081), and even $\mu =16.75$ mm⁻¹ (2020_012) in this set.

As an example data set, 2020_012, with the highest absorption coefficient in this subset, ($\mu =16.75$ mm⁻¹) does show a spoon-like resolution dependence, despite uniform BayCoN plots ($N_{obs}=669$).

Observations from Table 1 and Table 5 for the subset of uniform BayCoN plots are that all data sets show insignificant deviations from zero for the mean value of the weighted residuals, $-3<〈\zeta 〉/\sigma \left(〈\zeta 〉\right)<3$. This is taken as evidence for the necessary and plausible, but not sufficient requirement that the mean weighted residuals need to be zero within the limits of statistical fluctuations for obtaining uniform BayCoN plots. Without giving further proof of the following as it appears to be also self-evident, it should be mentioned that data sets with insignificant shifts of the mean value of the residuals tend also to show more balanced residuals in all three aspects: the number of positive and negative residuals tend to be equal (within the limits of statistical fluctuations) as well as the mean value of positive and absolute negative residuals and the mean values of the squared positive and negative residuals.

The crystal dimensions tend to be too large for a microfocus beam, which is obviously not an obstacle for obtaining uniform BayCoN plots. As discussed further above, accurate centering of the crystal may be an important factor in this matter. A characteristic number for describing the centering of the crystal should be included in the cif file if this assumption is substantiated.

Table 5. Data sets with uniform BayCoN plots: data set, indication of microsource or microfocus techniques, elements in the unit cell, habitus, monochromating device, temperature in the experiment, largest dimension of crystal in mm as given in the respective publication, absorption coefficient in mm⁻¹, and diffractometer type. For references and additional data, see Table 1.

		Radiation	Elements	Habitus	Monochromator	T	${\mathit{d}}_{\mathit{max}}$	${\mathit{d}}_{\mathit{max}}/{\mathit{d}}_{\mathit{min}}$	${\mathit{N}}_{\mathit{obs}}$	Diffractometer
						[K]	[mm]
2021_042	mf	Mo K$\alpha$	Co, F, N, C, H	prism	mirror	183	0.18	4.50	3329	Bruker D8
2020_012	mf	Mo K$\alpha$	Cs, Br, F	block	—	100	0.11	1.83	669	Bruker D8
2020_081	mf	Mo K$\alpha$	Pt, S, N, C, H	plate	mirror	100	0.05	5.00	1556	XtaLAB Synergy
2020_117	ff	Mo K$\alpha$	O, N, C, H	block	graphite	170	0.35	1.75	1238	XtaLAB mini
2020_128	mf	Mo K$\alpha$	Cl, S, O, N, C, H	plate	mirror	100	0.32	5.33	1968	Bruker Apex II
2021_076	—	Mo K$\alpha$	S, N, C, H	plate	—	150	0.18	4.50	562	XtaLAB Synergy
2022_024	mf	Ag K$\alpha$	P, O, N, C, H	prism	—	298	0.12	1.33	608	Bruker D8
2022_043	ms	Mo K$\alpha$	Br, N, C, H	neelde	graphite	200	0.12	3.00	1715	Bruker Apex II
2022_055	ms	Cu K$\alpha$	F, O, N, C, H	column	—	100	0.28	5.60	1309	Bruker D8

mf: microfocus; ff: fine-focus; ms: microsource.

Table 5. Data sets with uniform BayCoN plots: data set, indication of microsource or microfocus techniques, elements in the unit cell, habitus, monochromating device, temperature in the experiment, largest dimension of crystal in mm as given in the respective publication, absorption coefficient in mm⁻¹, and diffractometer type. For references and additional data, see Table 1.

		Radiation	Elements	Habitus	Monochromator	T	${\mathit{d}}_{\mathit{max}}$	${\mathit{d}}_{\mathit{max}}/{\mathit{d}}_{\mathit{min}}$	${\mathit{N}}_{\mathit{obs}}$	Diffractometer
						[K]	[mm]
2021_042	mf	Mo K$\alpha$	Co, F, N, C, H	prism	mirror	183	0.18	4.50	3329	Bruker D8
2020_012	mf	Mo K$\alpha$	Cs, Br, F	block	—	100	0.11	1.83	669	Bruker D8
2020_081	mf	Mo K$\alpha$	Pt, S, N, C, H	plate	mirror	100	0.05	5.00	1556	XtaLAB Synergy
2020_117	ff	Mo K$\alpha$	O, N, C, H	block	graphite	170	0.35	1.75	1238	XtaLAB mini
2020_128	mf	Mo K$\alpha$	Cl, S, O, N, C, H	plate	mirror	100	0.32	5.33	1968	Bruker Apex II
2021_076	—	Mo K$\alpha$	S, N, C, H	plate	—	150	0.18	4.50	562	XtaLAB Synergy
2022_024	mf	Ag K$\alpha$	P, O, N, C, H	prism	—	298	0.12	1.33	608	Bruker D8
2022_043	ms	Mo K$\alpha$	Br, N, C, H	neelde	graphite	200	0.12	3.00	1715	Bruker Apex II
2022_055	ms	Cu K$\alpha$	F, O, N, C, H	column	—	100	0.28	5.60	1309	Bruker D8

mf: microfocus; ff: fine-focus; ms: microsource.

As an example, data set 2021_042 [181] is discussed. It is the data set with largest $N_{obs}=3329$ in this subset. The experiment was conducted with $\mathrm{Mo}$ K_α radiation on a Bruker D8 Quest diffractometer with multilayer optics monochromator at 183 K. The absorption coefficient is given with $1.20$ mm⁻¹. Weighting scheme parameters $a=0.0512$, $b=0.4385$ are reported. A total of 87.91% of the variance of the observed intensities is due to systematic errors despite the seemingly small weighting scheme parameters. All systematic errors together increase $wR\left(F^2\right)$ by a factor of 3.23.

Figure 9 shows the BayCoN plots $(\zeta ,Y)$ and the corresponding ${\chi }^2$ values, that are all below 149. Signs of application of a significance cut-off are visible in the BayCoN plot $(\zeta ,I_{calc})$ (Figure 9a) and, more pronounced, in $(\zeta ,I_{calc}/\sigma (,I_{obs}))$ (Figure 9c) by the slightly depopulated areas in the lower left corner. The command “OMIT-1” was found in the corresponding SHELXL input file, i.e., a slightly harder significance cut-off was chosen in comparison to the default setting that corresponds to “OMIT-2”. In Figure 9c, the top right corner is also slightly depopulated. This is a consequence of $a=0.0512$ and appears, for example, in conjunction with not (fully) modelled disorder, and twinning, which will be shown in a follow-up publication. However, both effects are small enough such that they do not establish a systematic connection between the residuals and the significance of the reflections at the chosen level of significance of 0.001.

A refinement with statistical weights leads to an expected and considerable reduction in the weighted agreement factor from 0.093 to 0.064, that is, however, not as distinct as indicated by $g=3.2288$ and to an increase in the $GoF$ from 1.05 to 2.20 compared to a refinement with weights ($a=0.0512$, $b=0.4385$). The new agreement factor ratio is $g=2.2007$. The output list file issues a warning about two $\mathrm{F}$ atoms in the hexafluoridphosphate anion ${\mathrm{PF}}_6^{-}$ that may be split. Although chemically not of any extra value, the model could still be improved by taking the disorder into account and removing the significance cut-off.

The applied weights obviously reduce $GoF$, which correctly indicates the presence of other errors when using statistical weights. The $GoF$ ceases to be an independent measure of fit quality when extra parameters are introduced to fabricate lower values—a point that is often overseen. Figure 9e–h show the BayCoN plots for a refinement with statistical weights and corresponding ${\chi }^2$ values, that all increased considerably with the exception of $(\zeta ,\sigma \left(I_{obs}\right))$. They additionally show the typical signs for underestimated $\sigma \left(I_{obs}\right)$ particularly in Figure 9g (compare with Figure 2c).

The other data sets with uniform BayCoN plots are 2020_012 [20], a redetermination of the crystal structure of caesium tetrafluoridobromate(III), ${\mathrm{CsBrF}}_4$ with $\mu =16.75$ mm⁻¹, crystallizing in the orthorhombic space group $\mathrm{Immm}$, measured on a Bruker D8 Quest diffractometer with microfocus technique with $\mathrm{Mo}$ K_α radiation at 100 K. The crystal is described as a block with dimensions 0.11, 0.09, and 0.06 mm;

2020_081 [81], a platinum (IV) complex, $\mu =10.45$ mm⁻¹, crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, measured on a Rigaku XtaLAB Sybergy diffractometer equipped with microfocus technique and hybid pixel detector with $\mathrm{Mo}$ K_α radiation at 100 K. The crystal is described as a plate with dimensions 0.05, 0.02, and 0.01 mm;

2020_117 [116], pyridine-4-carboxamidoxime N-oxide, ${\mathrm{C}}_6{\mathrm{H}}_7{\mathrm{N}}_3{\mathrm{O}}_2$, crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{c}$, data taken on a Rigaku XtaLAB mini diffractometer with fine focus sealed tube at 170 K with $\mathrm{Mo}$ K_α radiation, $\mu =0.12$ mm⁻¹. The crystal is described as a block with dimensions 0.35, 0.20, and 0.20 mm. An extinction model was applied (extinction coefficient $x=0.007$);

2020_128 [127], a weak absorbing ($\mu =0.55$ mm⁻¹) hydrated molecular salt ${\mathrm{C}}_{10}{\mathrm{H}}_8{\mathrm{NS}}^{+}·{\mathrm{Cl}}^{-}·{\mathrm{H}}_2\mathrm{O}$ crystallizing in the monoclinic space group $\mathrm{P}{2}_1/\mathrm{n}$, measured on a Bruker diffractometer with Incoatec microfocus sealed tube with $\mathrm{Mo}$ K_α radiation at 100 K. Disorder was refined for the cation with occupation factors 0.853 (3) and 0.147 (3). The dimensions of the crystal are given with 0.32, 0.25, and 0.06 mm;

2021_076 [214], Pyrazine-2(1H)-thione (${\mathrm{C}}_4{\mathrm{H}}_4{\mathrm{N}}_2\mathrm{S}$, $\mu =0.49$ mm⁻¹, monoclinic spacegroup $\mathrm{P}{2}_1/\mathrm{m}$, XtalLAB Synergy Dualflex diffractometer with HyPix detector, T = 150 K, dimensions of crystal 0.18, 0.06, and 0.04 mm);

2022_024 [255] a phosphonate coordination polymer with divalent ${\mathrm{Zn}}^{2+}$ cation ($\left({\mathrm{H}}_3\mathrm{O}\right)\left[\mathrm{Zn}{\left({\mathrm{CH}}_4{\mathrm{N}}_2\mathrm{PO}\right)}_3\right]$, $\mu =1.06$ mm⁻¹, $\mathrm{Ag}$ K_α radiation, hexagonal spacegroup $\mathrm{P}{6}_3/\mathrm{m}$, Bruker D8 diffractometer with microfocus sealed tube, T = 298 K, and dimensions of crystal 0.12, 0.11, and 0.09 mm);

2022_043 [274], a dibromo methylaniline with hydrogen bonds generating $C\left(2\right)$ chains along $\left[100\right]$ direction (${\mathrm{C}}_7,{\mathrm{H}}_7{\mathrm{Br}}_2\mathrm{N}$, $\mu =9.62$ mm⁻¹, $\mathrm{Mo}$ K_α radiation, orthorhombic spacegroup $\mathrm{P}{2}_1{2}_1{2}_1$, Bruker Apex II Quazar CCD diffractometer, T = 200 K, and dimensions of crystal 0.12, 0.05, and 0.04 mm);

2022_055 [285], the first example of a perfluorinated pyridine ring with a formamide functional group. Chains are formed along the direction of the $b-$axis (${\mathrm{C}}_6,{\mathrm{H}}_2{\mathrm{F}}_4{\mathrm{N}}_2\mathrm{O}$, $\mu =0.21$ mm⁻¹, $\mathrm{Mo}$ K_α radiation, orthorhombic spacegroup $\mathrm{P}{2}_1{2}_1{2}_1$, Bruker D8 diffractometer with Incoatec source, T = 100 K, dimensions of crystal 0.28, 0.06, and 0.05 mm);

The agreement factor ratio ranges in this group between 1.3355 and 4.7997, the fraction of systematic error in the variance of the observed intensities ranges between 0.2439 and 0.8791.

5.8. Other Patterns

There are a number of other interesting patterns in the BayCoN plots that have not been addressed yet. Among these are individual cases with very unusual and highly non-uniform BayCoN plots that might go back to data processing errors, a number of cases that do show highly non-uniform BayCoN plots $(\zeta ,I_{calc}/\sigma \left(I_{obs}\right))$ which appear to be connected to disorder and twinning problems, and others, that need to be addressed in a future publication.

6. Discussion

Section 5.1 showed that there is a severe, and with the help of BayCoN plots, easy detectable problem with the $s.u.\left(I_{obs}\right)$ in some data sets; however, this may be only the tip of the iceberg, as there is plenty of evidence that the correct determination of $s.u.\left(I_{obs}\right)$ is a central problem in general: (i) all data sets but two employ a weighting scheme which implies that all of them are affected by a systematic error (in the $s.u.\left(I_{obs}\right)$, in the model or in both). It is reasonable to assume that the $s.u.\left(I_{obs}\right)$ are in the overwhelming majority of all analyzed data sets affected by a systematic error as (ii) even those 11 data sets with uniform BayCoN plots still employ weighting scheme parameters. That underestimation of the $s.u.\left(I_{obs}\right)$ is a common phenomenon also emphasized earlier [346]. Use of statistical weights for one of the uniform data sets (2021_042) in Section 5.7 led to the conclusion that the weighting scheme was used in order to force the Goodness of Fit to an acceptable value, which otherwise would correctly indicate the presence of systematic errors. This is only the generally used procedure, and by no means particular to the authors of this publication. This commonly established procedure is in need of revision.

Flawed $s.u.\left(I_{obs}\right)$ may not present a problem for standard structure determination as the independent atom model is rather rigid, and systematic errors tend to accumulated in the anisotropic displacement parameters that are regarded not as essential as for example bonding distances. Yet, they constitute a methodological deficit that prevents improvements and may strike when it comes to cases where correct $s.u.\left(I_{obs}\right)$ are more important, for example in charge density studies, in studies with Metal Organic Frameworks, or with thermal diffuse scattering, or when the methodological developments from single-crystal experiments are transferred to neighboring fields like neutron and electron diffraction experiments, in dynamic structure crystallography or in cases with superstructures. Moreover, every single experiment gives the opportunity to learn how to improve the methodology and the intrinsic models and this opportunity is just missed by the rather indiscriminate application of intensity and significance cut-off and weighting scheme. What if systematic errors are the key for progress in the field, if taken more seriously?

A particular sensitive topic is the recurring theme of self-reinforcing errors. Example (i): Overestimation of weak intensities (initial error), leads to artificially reduced agreement factors, which are interpreted to indicate high data precision or high data accuracy or high fit quality (confirmation bias, interpretation error) [339]. Example (ii): Underestimation of the $s.u.\left(I_{obs}\right)$ of strong intensities (initial error) artificially increases the mean squared significance of the observed intensities (confirmation bias, interpretation error). The weighted agreement factor is, as a consequence, artificially low. A high significance of the observed intensities and low agreement factors are interpreted as a sign of high fit quality (confirmation bias, consequential error). Example (iii): The default application of a significance cut-off (initial error) may conceal issues with weak intensities or their standard uncertainties. This artificially lowers the Chi square sum, which directly affects $wR\left(F^2\right)$ and $GoF$. Consequently, these reduced values may be misinterpreted as a indicators of high fit quality—an instance of confirmation bias. Example (iv): Truncation of large residuals (initial error): large positive residuals are truncated by application of a weighting scheme; large negative residuals are truncated by a significance cut-off. Both truncation processes lead to artificial lowered values of $wR\left(F^2\right)$ and $GoF$ which are again likely interpreted as a sign of high fit quality (confirmation bias, interpretation error). The simple example discussed in [335], where the $s.u.\left(I_{obs}\right)$ are all too small by the exact same factor, belongs to this category. Example (v): incomplete absorption correction (initial error) results via scaling in overestimated weak intensities. These lead to artificial low anisotropic displacement parameter values and artificial reduced values of agreement factors that are interpreted as a sign of high quality (confirmation bias) [339]. The list is not complete. Such errors impair the data quality assessment because they only lead to seemingly good results by simultaneously blocking real progress. This recurring theme of shielded errors may be a sign of flawed thinking when assessing (fit) data quality. The heretical question that must be asked concerns the appropriate use and limitations of the weighting scheme. There is no question that a flat variance is desirable; however, the means to achieve this goal are in question: by describing, identifying, and removing systematic errors that lead to a non-flat variance or by forcing the variance to be as flat as possible despite remaining systematic errors by applying a weighting scheme.

The answer to this question is not purely technical—it is political. It reflects the values of the crystallographic community. Should the main goal be to produce as many publishable results as possible? Or should it be to increase data quality by learning from errors? In the personal view of the author, the methods to describe systematic errors in diffraction experiments are currently underdeveloped and the whole topic “systematic errors” is neglected given the potential it has for improvements.

Decades of crystallography passed by with very limited computer power, computer memory, and data storage. During this period, it was very important to have tools like the weighting scheme available in order to achieve any results at all. However, today computing power and data storage are available at low costs. The focus must be shifted to obtaining the most accurate information possible from each experiment. This may feel a bit uncomfortable and like a shift of paradigm, but this step is necessary in order to learn from each experiment and to turn crystallography in a high-precision, high-accuracy technology again over the next decades. The ability to detect and quantify systematic errors in diffraction experiments has not kept pace with the rapid development of instrumentation. As a consequence, this ability has not sufficiently developed and flawed data quality assessment protocols as well as misconceptions have not been adequately corrected. The proper development of methods for the quantification and visualization of systematic errors may result in a higher accuracy of all instruments; in fact, it may be the only way to use the full potential of instrumentation. From the author’s point of view, the next decade of crystallography may well be defined by a renewed focus on data quality.

For progress in the field, it would be important to discriminate at least between the cases where weighting scheme parameters are applied due to flawed $s.u.\left(I_{obs}\right)$ and cases where they are applied due to model deficiencies. Authors should be encouraged to assess these cases and to add a short section about systematic errors that comprises measures of the extent of systematic errors and likely sources as well as the impact on model parameters and model parameter errors. The agreement factor ratio g and the fraction of systematic errors in the variance of the observed intensities ${\sigma }^2\left(I_{obs}\right)$ could be used as metrics for this assessment. Even if sources of errors are not known or cannot be quantified it would be helpful to state this openly in order to stimulate progress. Also, weighting scheme parameters $a>0$ and $b>0$ should not be used uncommented. Why are these needed? Why do they often render the $s.u.\left(I_{obs}\right)$ unimportant, by inflating the variance of the observed intensities to a degree that lets random fluctuations as indicated by the $s.u.\left(I_{obs}\right)$ appear small? Rather than inflating the variance of the observed intensity to clearly non-physical limits involving weighting scheme parameters like those mentioned in Section 4.2 with $b>10$ and even $b>100$, these values should be at least explained or commented on in order to learn how to avoid these in future.

There is yet another problem with the application of the weighting scheme that has not yet been mentioned, as follows: it increases the correlation coefficient between the observed intensities $I_{obs}$ and $\sigma \left(I_{obs}\right)$. This may pose a problem to the validity of the least squares procedure by inducing autocorrelation among the residuals that are supposed to be statistically independent. The mean correlation coefficient for these 314 data sets with $cc(I_{obs},s.u.\left(I_{obs}\right))=0.7967$ is already quite strong; however, $cc(I_{obs},\sigma \left(I_{obs}\right))=0.959$!

Significance and intensity cut-offs are unfortunately still applied in the overwhelming majority of all published data sets examined here. In the perspective of the author, these should be banned completely as they prevent the analysis of the residuals for the weak and the weak significant data and are of a purely cosmetic nature. A strict scientific procedure would lay the burden on authors to provide evidence that application of a cut-off does not affect the model parameter and model parameter errors instead of tacitly assuming this without giving evidence for this optimistic assumption. The application of a significance or an intensity cut-off may affect a large fraction of the data when the data set is, in total, only weak significant. For example, Figure 3a,c show a disruptive effect of a significance cut-off on the distribution of the weighted residuals in the BayCoN plots and Figure 3f demonstrates that approximately 50% of the reflections are affected by the significance cut-off—a surprisingly large fraction! The fraction is so large because the mean significance of data set 2020_018 is with $〈I_{obs}/\sigma \left(I_{obs}\right)〉=2.27$, and thus very low. It is made so low by the weighting scheme parameters in the first place. Similar effects are visible in Figure 4c,f for the intensity cut-off. Also, the detection of data sets with too weak observed intensities for the weak intensities is hampered in this way.

The interested authors who published with IUCrData in 2020–2022 will find the BayCoN plots and corresponding ${\chi }^2$ values as well as some of the characteristic numbers as listed in Table 1 for their data sets in the Supplementary Materials.

Some procedures and data quality evaluation processes are based on misconceptions or use insufficiently precise criteria and need revision. For example, the official purpose of the weighting scheme is to reduce the influence of outliers (mainly from high-intensity and low-resolution reflections) on model parameter values. In reality, (i) a substantial fraction of the data needs to be affected by a systematic error for the weighting scheme to get invoked in the first place, not only a few outliers. Moreover, (ii) weighting scheme parameters $b>0$ point to problems with weak reflections rather than to problems with strong reflections. Also, (iii) when $s.u.\left(I_{obs}\right)$ are all substantially too small, a SHELXL-like weighting scheme is not able to handle this [335]. (iv) Incomplete absorption correction leads—quite counter intuitively—mainly to problems with weak reflections [339]. These problems are just covered up instead of tackled by flattening the variance with the help of the weighting scheme. (Note the large weighting scheme parameter $b=6.5030$ for Crystal 1 in Table 2, that keeps falling for Crystal 3 ($b=1.1388$) and Crystal 4 ($b=0.0528$) with lower absorption coefficients. The by far largest Crystal 5 with $d_{max}=0.20$ mm again shows an increased value $b=1.6275$ despite a lower absorption coefficient. Finally, Crystal 6, which is affected by contamination with low energy photons, also shows a non-zero value $b=2.4392$). The list is not complete. The abundant and indiscriminate use of the weighting scheme bears therefore the danger of not only disguising individual systematic errors in individual data sets (like undetected or not modelled disorder or insufficient absorption correction), but also, and even more concerning, additionally, there is a risk that conceptual errors and data processing errors are disguised on a regular base. Such a data processing error could, for example, be the improper application of an error model like

$${\sigma }^2{\left(I\right)}_{\mathrm{corrected}}={\left[K\sigma {\left(I\right)}_{\mathrm{raw}}\right]}^2+{\left(gI\right)}^2$$

(7)

with parameters K and g. This type of error model is used in SADABS [340], XDS [347], AIMLESS [348], and HKL-2000 [349]. Typically, for each scan, one value of K is refined and g is refined for the whole data set such that the weighted mean square deviation:

$${\chi }^2={\displaystyle \frac{\frac{1}{N-1}{\sum }_N{(I-〈I〉)}^2}{{\sum }_{N}s.u{.}^2\left(I\right)}}$$

(8)

between N equivalent reflections is as close as possible to unity over the full range of intensity. Plots of ${\chi }^2$ against the resolution are taken to evaluate the success of this procedure. It is found that these plots are often close to unity “except sometimes for a small rise at very low resolution that is clearly indicative of a residual systematic error.” (p. 4, [340]). What is the “residual systematic error” at very low resolution? A typical answer to this is that it is well known that the strong low-resolution reflections are most affected by systematic errors and therefore the small rise at low resolution is but expected. In this view, the residual systematic error is virtually unavoidable and can just be ignored. There is not much to counter this statement,; however, it can be added that there is another interpretation available: a slight but maybe quite systematic underestimation of $s.u.\left(I\right)$ specifically for the strong reflections residing at low resolution would also lead to a small rise at very low resolution, and it would lead to weighting scheme parameters $a>0$ on a regular base, i.e., even for uncomplicated small standard structures.

Some of the misconceptions in crystallography are (i) that negative intensity observations could or should be replaced by a predefined value, when they are “too” negative; (ii) that the weighting scheme can and should be applied on a regular base without considering its applicability; (iii) that large weighting scheme parameters need not be commented and need not lead to action; and (iv) that the Goodness of Fit has meaning after invoking a weighting scheme—it ceases to be an independent measure of fit quality. Measures that need to be taken are consequently (i) manually including “OMIT-100” or similar in SHELXL instruction files to make sure that all reflections are used in the least squares refinement; (ii) Application of a weighting scheme needs justification why it is needed and how it can help to obtain better results. Cases where the weighting scheme is not helpful or not appropriate need to be specified and excluded. (iii) Resulting weighting scheme parameters need discussion and should lead to changes in the model or data processing steps or at least to suggestions for such changes. (iv) The use of the Goodness of Fit needs to be reconsidered.

Systematic errors can affect even seemingly uncomplicated small molecules structures. To fully understand their impact, refinements must be performed using statistical weights. It is therefore not surprising that authors hesitate to do this frequently. For progress in the field, however, it will be necessary to encourage authors to do this and to openly discuss all systematic errors that are observed. In this way, systematic errors are embraced and used like a compass that paves the way to steadily increasing accuracy in diffraction experiments. In this view, systematic errors are invaluable hints to improvements and hidden problems—a treasure chest for data quality questions waiting to be recovered.

7. Summary

The BayCoN pots are a simple and versatile tool for the detection and visualization of systematic errors. They also help to categorize systematic errors. Detection and characterization of systematic errors is a necessary prerequisite for further differentiation, investigations about origins and remedies. The step to detect and visualize systematic errors is therefore important and valuable in itself and not only when it immediately leads to the identification and removal of the underlying systematic error. This is an important point that is often neglected. Diagnosis is prior to treatment and cure and a valuable and important step in itself even when a cure is not (yet) available—at least it is then known that a problem exists and the problem is characterized.

Application of the BayCoN plots to a sample of standard structure data sets ($N=314$) led to patterns in the corresponding BayCoN plots associated ${\chi }^2$ values and was linked to other metrics like the bin scale factor K and $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}$ and to crystal properties like absorption coefficients and crystal dimensions. The Ansatz to connect residual metrics to crystal properties could be deepened in following studies.

Another important result is that systematic errors tend to organize themselves into new categories, when the BayCoN plots are used as a tool for visualization: For example, the plots Figure 5a–d, Figure 6a–d and Figure 7a–d, clearly emphasize the importance of the weak- and weakly significant intensities and connects errors in the weak intensities to K, $\frac{〈\zeta 〉}{\sigma \left(〈\zeta 〉\right)}$ and other metrics. This leads to a residual-centered description of systematic errors in single-crystal diffraction experiments which provides a larger and neutral framework compared to an error-centered description as it allows the quantification of systematic errors on a continuous spectrum. Quantification of systematic errors on a continuous spectrum, in turn, allows for recategorizing data sets into different quality levels from a residual-centered perspective. This may be very helpful to improve overall data quality, when crystallographic data banks, crystallographic journals, and diffractometer producers support the applications of these categories.