Sensitivity of Optimal Estimation Satellite Retrievals to Misspecification of the Prior Mean and Covariance, with Application to OCO-2 Retrievals
Round 1
Reviewer 1 Report
The manuscript by Hai Nguyen et al. studies the performance of Optimal Estimation (OE) retrievals when the prior state vector estimate and / or its error covariance are
misspecified, as it always happens in real cases. The performance of the OE retrieval is evaluated in terms of the bias of the solution and of the accuracy of the error estimate provided by the covariance matrix of the solution. The authors also study an important synthetic test case, based on the current OCO-2 retrievals of carbon dioxide total column.
The methods used are appropriate and scientifically sound, I did not spot any errors in the equations. I enjoyed very much reading the manuscript, to my opinion it is very well written and plenty of comprehensive and clear explanations.
My main concern is the following one. The contents of the manuscript are illustrative of the functioning of the OE method. I would be very happy to find the provided explanations and examples in a new dedicated chapter, or in an appendix of a book, like the Rodgers (2000) book, but I really do not know if it will be useful or acceptable as a scientific paper. In other words, atmospheric measurements are made just because we want to know the true value of the atmospheric state xT and its covariance ST, which are not known a-priori. Therefore, rigorously speaking, the methods of the manuscript are applicable only to synthetic examples, not to real measurements where the real atmospheric state and its variability are unknown. Of course, the proposed approach can still be used 'to get an idea' of the performance of the OE solution also in real situations, but the results will not be fully reliable, being based just on 'estimates' of xT and ST, just as 'estimates' are xw and Sw. This is the reason why a common practice consists in providing as diagnostic data accompanying the OE solution, the covariance matrix describing the mapping of only the measurement noise onto the solution and, separately, the averaging kernel matrix (AK), describing (in the linear approximation) the spatial response function of the measurement and retrieval chain. If, in some fortunate circumstances, the atmospheric variability ST is known, then the AK can be used to evaluate the
actual smoothing error component as: (I-AK) ST (I-AK)t.
For this reason I leave to the Editor the decision on whether the content of the
manuscript is appropriate for the Remote Sensing journal. If it is appropriate, then
I have only the following minor comments.
MINOR COMMENTS
In the equations the transpose of the Jacobian K is indicated as K'. I suggest using
the standard notation Kt.
Equation after Eq.8 (not numbered): of course the bias is zero if computed assuming
the true value of x exactly equal to xw. I do not understand the meaning of this equation.
Table 1: this table is not necessary in my view.
Line 267: please adjust the sentence.
Line 268: usually the term 'accuracy' refers to the bias, while 'precision' refers
to the random error. In this case I think you get larger random error and a smaller bias.
Line 280: I liked very much the univariate example as it illustrates very clearly
the actual functioning of the OE. However, maybe for a paper it is not so essential
and Sec. 2.4, probably, could be removed to shorten the text.
Lines 330-341: this is a very long explanation for a quite simple concept. I would
condense everything in fewer sentences.
Line 444: remove one 'prior'.
Line 458: please check the sentence.
Line 568: please note that matrix (Kt S-1ε K) may be singular, especially in the inversions where OE is really needed (when the least squares approach is ill-posed), in this case your proposed strategy is not applicable.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This is a well-written paper that was interesting to read. It provides a nice overview of retrieval theory with a specific application to retrievals from the OCO-2 instrument. This paper deserves publication, after considering one major item, and a couple of minor issues.
Major issue
The paper convincingly makes the case that the choice of prior and its uncertainty is important for the final retrieved product (e.g. xCO2) and the associated uncertainty. However, in real practice the xCO2 is hardly used directly to compare to a model, because all users are aware that the modeled columns need to be convolved (using the averaging kernel) with the working prior that is given in the product. A potential bias in the working prior is accounted for by comparing with model results convolved with the prior using the averaging kernel. So, in that sense the mis-specified mean would not be a problem in itself.
However, the results in section 3 show that the problem is more complex. Here the CO2 working prior profile is actually within roughly 5 ppm from the “true” prior. Nevertheless, the resulting XCO2 is biased by as much as 22 ppm. According to the authors, this is caused by the mis-specification of other parameters in the state, like the albedo and surface pressure. My question here is if this situation would also be possible in true situations. Obviously not for OCO-2, which works with inflated priors. But for practical use of OCO-2 it would be good if the authors could (1) outline more clearly how a bias would propagate in a real inversion, where the averaging kerning is used in the analysis (2) extend their OCO-2 analysis, with less extreme errors on non-CO2 parts of the state vector.
Minor comments
Line 89: Tiknohov à Tikhonov
Line 160-161: this has been said before. Remove here, or before.
Figure 1: I notice that the x-axes have different cut-offs. I would advise to use the same (e.g. 50).
Line 458: I do not know if this is a fair assessment of the choices made by the OCO-2 team: your true prior is constructed from "retrieved" states, so, it might well be that the true knowledge about these albedos was truly limited at the time algorithms were developed.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The authors present an interesting study on the effects of "wrong" prior means and covariances of OE-based retrievals. The manuscript is very well written, and the "retrieval audience" will most likely feel at home since the authors use common terminology and notation. First, general concepts of prior misspecification are explored, and then expressions are derived for retrieval uncertainties and biases. Finally, the results are discussed in the context of a CO2 retrieval from the OCO-2 instrument. The summary is particularly well-written with some concise suggestions for retrieval designs for near-linear forward models.
Even though the authors mention it, I would have liked to see the non-linearity problem highlighted a bit more prominently. I would argue that not only "many", but "most" atmospheric retrieval problems feature a non-linear forward model - in the case of OCO-2, the inclusion of aerosol parameters make those matters worse. Intuitively one would expect the bias to be dominated by the retrieval settling into a local minimum. However, the authors do acknowledge that fact in the final summary.
Nevertheless, the work remains relevant, even for non-trace gas retrievals, such as fluorescence (which could be mentioned in the manuscript), which is almost linear.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 4 Report
The authors investigate the consequences of a suboptimal prior on the retrieval of atmospheric variables from satellite radiances. They apply their framework on a surrogate model of NASA’s OCO-2 retrieval scheme. The paper is very well written and pedagogical. However, some aspects make it disconnected from the practical context.
A beauty of the OE linear framework is that retrieval users replace or remove the retrieval prior when comparing the retrieval with other model or OE data even without knowing it, when using the retrieval averaging kernels (Connor et al., 2014). The linear assumption may not hold but the authors’ experiment with a linearized surrogate model does not explore the consequences of such invalidity. The averaging kernel power also partly vanishes away when using the column-average averaging kernel rather than the full profile averaging kernel (Chevallier, 2015), but this is not even mentioned here. Taking the specificities of OE priors into account will likely lead to a very different discussion than the present one.
A revised version should also correct the following issues:
- There is no “true prior”. The text defines this expression with respect to the natural variability of the state (e.g., l. 81), but a given prior is “an opinion”: it comes from some source of information that is independent from the observations; it is subjective. Given that many sources of information can be used in practice, many priors exist that are all valid. Equation (4) implies two requirements: the statistics of the prior errors shall be Gaussian and unbiased. We are therefore talking about the variability of the errors of some state variable estimate rather than about the variability of the state.
- The prior is a starting point: it is the experimentalist’s knowledge of the state before s/he uses the observations. A remaining bias at the end of the assimilation process may not be dramatic as long as there is an improvement with respect to the starting point. Talking about biases, this means that we expect that the retrieval bias is less than the prior bias. This also makes the issues related to the prior very different from the issues related to the observations.
- global observation offsets are rather harmless for atmospheric inversions that assimilate retrievals alone in contrast to what is said in l. 204., because they affect the estimate of the initial atmospheric state, not that of the fluxes. More generally, the OCO-2 bias-correction formulation (O’Dell et al. 2018) does not fit into the present bias framework well (it is not a simple offset). How is it impacted by the prior misspecification discussed here, if at all?
- The detail given about NASA’s retrieval prior is outdated.
- Line 235 introduces a distinction between OE and maximum likelihood that is inconsistent with the use of expression OE above in the text. It is not even clear afterwards whether NASA’s retrievals belong to one or another, in particular when talking about the column (remember that more than one degree of freedom for signal is expected, see Table 2 in Wunch et al 2017).
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 4 Report
The authors have enriched and refined their text, but there are two important points that still deserve attention before publication:
The authors have only addressed the retrieval side of my first point (“retrieval users replace or remove the retrieval prior when comparing the retrieval with other model or OE data”), leaving the model one unanswered. With the averaging kernel and within the theoretical framework used, the mode of the working prior disappears from the model-minus-retrieval differences. For atmospheric inverse modelling (a major goal for the OCO-2 mission), the whole working retrieval prior gets replaced, except when assimilating the total column rather than the profile. From this consideration, if I understand well, Chevallier (2015, https://doi.org/10.5194/acp-15-11133-2015, cited in Kaminski and Mathieu, 2017, https://doi.org/10.5194/bg-14-2343-2017) came, righty or not, to a very different recommendation from the authors for the working retrieval prior: aligning prior error statistics with atmospheric inverse modelling typical hypotheses. Migliorini (2012, https://doi.org/10.1175/MWR-D-10-05047.1, also cited in Kaminski and Mathieu) suggested an alternative to averaging kernels for data assimilation in general to ensure optimality. This deserves some discussion.
Item (d) in point #4 and all related expressions in the paper (possibly including the authors' viewpoint about the true prior) are incorrect. As the authors define it, efficiency will be better achieved by some prior that has smaller uncertainty than the natural variability. This is how I understand the sentence from Kulawik et al. (2008, https://doi.org/10.5194/acp-8-3081-2008) reproduced by the authors.
Author Response
Please see attachment
Author Response File: Author Response.pdf
