Uncertainty Analysis for RadCalNet Instrumented Test Sites Using the Baotou Sites BTCN and BSCN as Examples

Abstract

Vicarious calibration and validation techniques are important tools to ensure the long-term stability and inter-sensor consistency of satellite sensors making observations in the solar-reflective spectral domain. Automated test sites, which have continuous in situ monitoring of both ground reflectance and atmospheric conditions, can greatly increase the match-up possibilities for a wide range of space agency and commercial sensors. The Baotou calibration and validation test site in China provides operational high-accuracy and high-stability vicarious calibration and validation for high spatial resolution solar-reflective remote-sensing sensors. Two sites, given the abbreviations BTCN (an artificial site) and BSCN (a natural sandy site), have been selected as reference sites for the Committee on Earth Observation Satellites radiometric calibration network (RadCalNet). RadCalNet requires sites to provide data in a consistent format but does not specify the required operational conditions for a RadCalNet site. The two Baotou sites are the only sites to date that make spectral measurements for their continuous operation. One of the core principles of RadCalNet is that each site should have a metrologically rigorous uncertainty budget which also describes the site’s traceability to the international system of units, the SI. This paper shows a formalized metrological approach to determining and documenting the uncertainty budget and traceability of a RadCalNet site. This approach follows the Guide to the Expression of Uncertainty in Measurement. The paper describes the uncertainty analysis for bottom-of-atmosphere and top-of-atmosphere reflectance in the spectral region from 400 to 1000 nm for the Baotou sites and gives preliminary results for the uncertainty propagating this to top-of-atmosphere reflectance.

1. Introduction

Vicarious calibration methods—where satellite sensor observations are calibrated or monitored using ground observations of surface and atmospheric properties—have been used for radiometric satellite sensors operating in the solar-reflective spectral region (400–2500 nm) since the 1980s [1]. With the rapid growth in the number and variety of passive radiometric sensors operated by national and international space agencies and commercial operators, and with the growing range of applications, many of which require the combination of data from different satellite sensors or the establishment of long time series of data, such vicarious calibration methods have also become important to determine inter-sensor differences and to evaluate long-term stabilities.

Vicarious calibration sites are chosen to be radiometrically simple. Natural sand-based desert or pebbly areas are commonly used as these are spatially homogeneous, have dry, stable climates, rare cloud cover, and have a relatively flat spectral reflectance that can be easily interpolated between observations. Early vicarious calibration methods using ground observations were based on field campaigns where scientists measured such a site with handheld instruments at the same time as a satellite observation. Since the turn of the century, a small number of sites have operated automatic instruments to provide continuous data coverage (ROSAS [2] was established in 1997 at La Crau, France, and automated instruments were established in Railroad Valley Playa, USA, in 2002 [3]).

In recent years, the Working Group on Calibration and Validation of the Committee on Earth Observation Satellites (CEOS-WGCV) has worked to bring such sites into a single radiometric calibration network (RadCalNet) [4]. The aim of RadCalNet is to provide consistent top-of-atmosphere (TOA) reflectance products over several sites across the World to users through a common online portal [5] At the time of writing the portal has over 400 registered users across all continents. RadCalNet achieves consistency not by requiring a common measurement method (indeed there is a wide range of instrumentation and techniques) but through careful peer review of methodologies and analysis, through regular comparisons and quality assurance of site data, and by a common processor used to propagate from ground observations to TOA reflectance.

RadCalNet provides “fiducial reference measurements”. In this it follows the principles of the Quality Assurance Framework for Earth Observation (QA4EO [6]) that was ratified by CEOS in 2008. QA4EO is based around a core principle that “all EO data and derived products has associated with it a documented and fully traceable quality indicator” which provides sufficient information for users to judge the fitness-for-purpose of that data set for their application. It is therefore an essential component of RadCalNet that sites provide rigorous, peer-reviewed uncertainty budgets that are established from a systematic review of the full SI-traceability chain and following principles of “Earth Observation Metrology”.

In this paper we present a rigorous uncertainty analysis for the Baotou RadCalNet sites as an exemplar for how such uncertainty budgets should be considered. The paper is a collaboration between the operators of the Baotou sites (Aerospace Information Research Institute (AIR) in Chinese Academy of Sciences, formerly known as Academy of Opto-Electronics (AOE)) and two national metrology institutes (National Physical Laboratory (NPL) in the United Kingdom and National Institute of Metrology (NIM) in China). The uncertainty analysis here follows the Guide to the Expression of Uncertainty in Measurement (the GUM, [7]), a document that was formalized by the Joint Committee for Guides in Metrology, a consortium including the International Bureau of Weights and Measures (BIPM) and the international standards organizations (e.g., ISO).

Section 2 describes the Baotou sites. Section 3 explains the principles behind the uncertainty analysis and the traceability route for the Baotou sites and Section 4 describes the results of the uncertainty analysis.

2. The Baotou RadCalNet Sites

The Baotou Cal&Val site [8] (“Baotou site” in brief) is located at the Ming’an township, Urad Front Banner, Bayannur prefecture, Inner Mongolia Autonomous Region, China, about 50 km from Baotou city. There are two RadCalNet sites within the region, with BTCN at [40.8514°N, 109.6292°E] and BSCN at [40.8658°N, 109.6155°E]. The overall Baotou site extends over 35 km east-west and 16 km north-south, with total area 292.5 km². It is about 1270 m above sea level. The surface of the surrounding area is mainly desert, bare land, grassland, and farmland. Baotou features a cold semi-arid climate with average annual precipitation of 300 mm.

The Baotou site is a comprehensive calibration test site with many different types of target for radiometric and geometric calibration and validation of both spectrally-reflective sensors and synthetic aperture radar (SAR) sensors. On the Baotou site there are four targets set up with permanent instrumentation to allow for continuous measurement of ground spectral reflectance (along with atmospheric parameters). Three of these targets are artificial targets, separately made of black, grey, and white gravel, and the fourth target is a natural target within a desert environment. The grey permanent artificial target was chosen to be one of the initial four RadCalNet sites in 2014 and the natural sandy site went through a formal process during 2019 and was accepted as a new RadCalNet site in 2020.

2.1. Grey Permanent Artificial Target and Sandy Site

To meet the requirements of wide dynamic range and stability of the ground targets in sensor radiometric calibration, multi-greyscale permanent artificial targets were built in the Baotou site. Figure 1 shows the image (from Google Earth) of the Baotou site area. In the permanent target region, the grey-scale artificial target is composed of two white, one grey, and one black uniform gravel squares, each of which covers an area of 48 × 48 m. The grey target has been incorporated into the RadCalNet as BTCN (marked by red squared box in Figure 1a) [8]. A sandy site (300 × 300 m), marked by blue squared box in Figure 1a) is 1.8 km away from the permanent target region to the north-west and was flattened in October 2015. The sandy site is a new RadCalNet site (BSCN). The typical spectral reflectances of the two sites are shown in Figure 1b, with reflectance approximately 20% (BTCN) and 30% (BSCN).

2.2. Site Instrumentation for RadCalNet

At the Baotou site, the surface-reflected radiance is measured by several automatic observation systems of ground-reflected radiance (one on the grey target and two on the sandy site) developed based on commercial Colorimetry Research (CR) series spectrometers produced by Colorimetry Research, Inc. The systems cover the spectral region from 380 to 1080 nm, with a spectral resolution (i.e., full width at half maximum, FWHM) of 2 nm, have nominally a 3° field of view and are mounted at a height of 2.5 m (BSCN) or 2.0 m (BTCN) (see Figure 2). They observe the ground at nadir every 2 min.

Aerosol and water vapor content atmospheric parameters are obtained from the AERONET [9] sun photometer that is near the BTCN grey target and 1800 m away from the BSCN sandy site. The sun photometer is a Cimel CE318 instrument operated by the Baotou Site operator and atmospheric data are available from the AERONET website [10]. An all-sky imager has been deployed at the Baotou site to acquire cloud amount, cloud picture, and cloud height.

3. Methods

3.1. A Metrological Approach

It is core to the RadCalNet philosophy that the sites are SI traceable and have uncertainty budgets determined in a metrologically-rigorous manner. The metrological approach is needed to ensure inter-site interoperability, to provide long-term stability, and to enable the sites to be compared with the satellite observations. Metrological traceability relies on three core principles: (1) an unbroken chain of calibrations back to the reference SI units, (2) the propagation of uncertainties through that chain, and (3) comparisons performed to validate uncertainty statements.

Here, we follow metrological best-practice with our analysis. The uncertainty analysis is performed according to the GUM [7] and its supplements, with the Law of Propagation of Uncertainties used to propagate uncertainties for the laboratory calibration and field operation of the instrument, and Monte Carlo Methods (GUM Supplement 1; [7]) providing uncertainties associated with the atmospheric radiative transfer.

As these sites operate continuously, we also need to consider the error correlation between different observations. The error is the unknown difference between the measured value and the conceptual true value. While this error is unknown, we can evaluate the associated uncertainty (the dispersion, around the measured value, of values that could reasonably be attributed to the measurand), and we can evaluate the error correlation, i.e., to what extent the unknown errors are common or independent between two observations. It is important to understand this to appropriately combine multiple observations. For RadCalNet data we need to consider error correlations between the two instruments on the BSCN site and between those and the instrument on the BTCN site, and also to understand the error correlation between multiple measurements by the same instrument. Finally, we need to consider error-correlation between the measured values in different spectrometer spectral channels.

Here we adapt a framework developed for documenting the uncertainty associated with satellite observations [11]. That approach, developed from metrological principles, involves the following steps: (1) We define the traceability chain and the measurement function for the observed measurand; (2) we present an “uncertainty tree diagram” that shows the effects (sources of uncertainty) that influence our measured value and how these propagate to the measured value; (3) for each effect we create an ‘effects table’ that documents the uncertainty, the error correlation structures on different dimensions, and the sensitivity coefficient (that translates the uncertainty associated with that effect into an uncertainty associated with the measurand), and (4) we provide a combined standard uncertainty associated with all effects, with independent effects (no error correlation), with common effects (full error correlation), and with structured effects (partial error correlation).

With the satellite sensors considered in [11], the error correlation structures were provided for within scanlines, between scanlines, and between orbits. Here we consider error correlation structures between observations of a single spectrometer, between spectrometers, and between wavelengths.

3.2. Traceability for Ground and Top-of-Atmosphere Reflectance

3.2.1. Principles of Radiative Transfer

To convert the observed radiance into reflectance, and to propagate reflectance to TOA, atmospheric radiative transfer must be considered. Figure 3 gives a simplified (ignoring multiple scattering) visualization of the relevant light paths. The red lines represent the illumination of the Baotou site. Some light illuminates the Baotou site directly from the sun. A proportion of this is lost through atmospheric scattering on the way to the site. The site is also illuminated by the sky through light that is scattered in the atmosphere onto the site (sky diffuse) and light which has reflected off nearby locations and scattered onto the site (sky scatter). We use the radiative transfer model MODTRAN-5 [12] to estimate both the direct solar irradiance (direct incoming beam minus the lost irradiance) and the sky irradiance (the combination of sky diffuse and sky scatter). When propagating to top of atmosphere we must consider light that reaches the satellite directly, the loss of radiation through scattering on the upward path and light that has scattered into the beam from the sun (without reaching the ground) or from other parts of the ground background. Again, MODTRAN-5 is used for this evaluation. Note that as a default MODTRAN-5 assumes that the surrounding ground has the same reflectance as the area of interest. Where this is not the case (e.g., for the grey site BTCN, which is an artificial target), there is an “adjacency effect” that must be separately considered (see Section 4.6).

3.2.2. Overview of Traceability

The observational spectrometers at the Baotou RadCalNet sites measure upwelling radiance from the surface. This upwelling radiance is converted to ground reflectance using the measurement equation:

$${\rho }_{\mathrm{gnd}}\left(\theta ,t;{\lambda }_i\right)=\frac{\pi L_{\mathrm{gnd}}\left({\lambda }_i,t\right)}{E_{\mathrm{sun}}\left(\theta ,d;{\lambda }_i\right)+E_{\mathrm{sky}}\left({\lambda }_i,t\right)}+0$$

(1)

Here ${\rho }_{\mathrm{gnd}}\left(\theta ,t;{\lambda }_i\right)$ is the observed ground reflectance (which is actually a hemispherical-conical reflectance factor [13]) at time $t$ for solar zenith angle $\theta$ and at wavelength ${\lambda }_i$. This is calculated from the measured ground radiance $L_{\mathrm{gnd}}\left({\lambda }_i,t\right)$ and the calculated solar, $E_{\mathrm{sun}}\left(\theta ,t,d;{\lambda }_i\right),$ and sky, $E_{\mathrm{sky}}\left({\lambda }_i,t\right),$ irradiances. The “plus zero” term follows the approach described in [11], as representing the extent to which this measurement model approximates reality. Here it represents the assumption that for a spectrometer we can ignore the spectral bandwidth of observation and assume the radiance is measured “at a wavelength” rather than “integrated within a spectral band”. It also represents the assumption that downwelling irradiance can be simply calculated as a sum of solar and sky irradiances calculated from the MODTRAN-5 model.

This ground reflectance and its associated uncertainty is the main product provided to RadCalNet. RadCalNet performs its own propagation to top-of-atmosphere reflectance and provides that to users through the forum. Separately, AIR provides a top-of-atmosphere reflectance product matched to the spectral bands of desired satellite sensors to commercial and state customers. In this paper we concentrate on the evaluation of uncertainty for the ground reflectance, however we give some indication of how this is propagated to top-of-atmosphere in Section 4.6.

Traceability to SI comes from the calibration of the spectrometers used to measure ground radiance. These were calibrated at the Chinese national metrology institute, NIM, against primary standards. As a national metrology institute, NIM participates in the “Mutual Recognition Arrangement” [14]. The Mutual Recognition Arrangement is an agreement between the world’s metrology institutes to ensure global consistency of the SI. Institutes participate in regular international comparisons that are operated under strict procedures (e.g., results are submitted to a pilot who is the only institute who sees all results before publication) and are either formally accredited by standards agencies or peer reviewed by equivalent international institutes through formal audits of measurement procedures, analysis protocols, and uncertainty budgets.

In this case, NIM calibrated the spectrometers against a radiance source that had been created from an FEL lamp (FEL is a designation for the type of lamp and not an abbreviation) illuminating a diffuser panel. The spectral irradiance of the lamp $E_{\mathrm{FEL}}\left(\lambda \right)$ had been calibrated by direct comparison with a high temperature blackbody. The spectral radiance factor of the diffuser $\beta \left(\lambda \right)$ for a 0°/45° geometry (that is when the lamp illuminates it at 0° incidence angle and the spectrometer views it at 45° observation angle) was calibrated on NIM’s primary reflectance facility. At each wavelength, the gain of the spectrometer was calculated as:

$$G\left({\lambda }_i\right)=\frac{L_{\mathrm{source}}\left({\lambda }_i\right)}{C_{\mathrm{cal}}\left({\lambda }_i\right) }=\frac{E_{\mathrm{FEL}}\left({\lambda }_i\right)\beta \left({\lambda }_i\right)}{\pi C_{\mathrm{cal}}\left({\lambda }_i\right) }+0$$

(2)

where $G\left({\lambda }_i\right)$ is the gain of the spectrometer, $L_{\mathrm{source}}\left({\lambda }_i\right)$ is the radiance of the source made from the lamp and diffuser, and $C_{\mathrm{cal}}\left({\lambda }_i\right)$ is the measured count signal during calibration. The plus-zero term again represents the approximations inherent in this model, here including the assumption that the spectrometer is linear and the assumption that the spectrometer’s spectral bands are sufficiently narrow to be able to be treated as though the measurement was at a single wavelength.

When this instrument is used in the field, the measured field radiance is given by:

$$L_{\mathrm{gnd}}\left({\lambda }_i\right)=G\left({\lambda }_i\right)C_{\mathrm{gnd}}\left({\lambda }_i\right)+0$$

(3)

where $G\left({\lambda }_i\right)$ is the gain obtained in Equation (2) and $C_{\mathrm{gnd}}\left({\lambda }_i\right)$ is the signal when observing the ground. Here the plus zero again represents the linearity and monochromatic assumptions. It also assumes that the gain in the field is the same as the gain during calibration. This means that there is an assumption that the gain has not changed due to instrument ageing, transportation vibrations, or operational temperature differences. In practice, a difference has been observed due to temperature, which means that the error does not have an expected value of zero and therefore this equation has been modified to:

$$L_{\mathrm{gnd}}\left({\lambda }_i\right)=G\left({\lambda }_i\right)f\left({\lambda }_i,T\right)C_{\mathrm{gnd}}\left({\lambda }_i\right)+0$$

(4)

where,

$$f\left(\lambda ,T\right)=1+\left(m_0+m_1\lambda +m_2{\lambda }^2\right)\left(T-T_{\mathrm{ref}}\right)+0$$

(5)

for a temperature T. The coefficients $m_k \left(k=1,2,3\right)$ have been established through an empirical fit to experimental data and $T_{\mathrm{ref}}$ is the reference temperature, i.e., the temperature maintained during calibration, here 25 °C. More details on this fit process are given in Section 4.3.1.

The sun and sky downwelling irradiance terms in Equation (1) are calculated using MODTRAN-5 from the observation time and location, which determines the solar zenith angle, θ, and from the atmospheric parameters measured by the AERONET station, in particular the aerosol optical thickness and the water vapor column. MODTRAN-5 calculates the solar irradiance as:

$$E_{\mathrm{sun}}\left(\theta ,d;{\lambda }_i\right)={{\displaystyle \int }}_{{\lambda }_i-\delta \lambda }^{{\lambda }_i-\delta \lambda }\frac{E_0\left(\lambda \right)\cos \left(\theta \right)}{d^2}\tau \left(\lambda ,\theta \right){\xi }_{\mathrm{MDTN}}\left(\lambda \right)\mathrm{d}\lambda$$

(6)

where $E_0\left(\lambda \right)$ is the solar spectral irradiance at 1 astronomical unit based on the Thuillier solar irradiance model [15], $d$ is the sun–earth distance at the time of observation, $\theta$ is the solar zenith angle, $\tau \left(\lambda \right)$ is the calculated atmospheric transmittance based on the atmospheric parameters, and ${\xi }_{\mathrm{MDTN}}\left(\lambda \right)$ is the assumed (normalized to unit area) spectral bandpass function within MODTRAN-5 (here a triangle with a base of 20 cm⁻¹).

The sky irradiance is similarly calculated in MODTRAN-5 from the same solar irradiance model, the same atmospheric parameters, and the same spectral integral.

3.2.3. Uncertainty Tree Diagram

The uncertainty tree diagram is a conceptual diagram introduced in [11] which shows the origin of each term in the primary measurement equation (here Equation (1)). The diagram shows the sources of uncertainty that affect each term and gives the sensitivity coefficients—that is, the conversion factors that convert an uncertainty associated with an input quantity into the uncertainty associated with the measurand. Where an input quantity is itself calculated from its own input quantities, this is shown through additional sensitivity coefficients and equations. The sensitivity coefficients in any “branch” of the uncertainty tree can be multiplied together (chain rule) to provide the sensitivity of the primary measurand to the input quantity twig. Each source of uncertainty identified as a twig on this diagram should be evaluated both for magnitude and to understand the error correlation forms.

The uncertainty tree diagram for the Baotou measurements of ground spectral reflectance is given in Figure 4, which is reproduced at larger scale in Figure A1 (see Appendix A).

4. Results: Uncertainty Budget

4.1. Uncertainty Associated with Laboratory Calibration of the Field Spectrometer: Spectral Calibration

The spectrometers used at BTCN and BSCN were calibrated for wavelength accuracy and bandwidth using a mercury line source. The spectral radiance of the Hg source was determined, and from this for “clean” lines, with sufficient separation from other lines, a Gaussian distribution was fitted to the observed radiance. This provided an estimate of peak wavelength, which was compared to the Hg line wavelengths in air, and an (approximate) estimate of the instrument bandwidth. Note that future work is planned to characterize the spectrometer’s performance using a tunable laser system (see also Appendix B). This will provide improved information about the bandwidth and wavelength accuracy, which are limited here to the very small number of available cleanly separated spectral lines from the Hg source.

Across the spectrometer, the bandwidth (the standard deviation of the Gaussian fitted to the measured values) was consistently between 2.5 and 3 nm. This is wider than the MODTRAN-5 bandwidth assumption used in the solar and sky irradiance calculations. MODTRAN-5 uses a triangular bandpass function with a half base width of 10 cm⁻¹, corresponding to 0.2 nm at 400 nm and 1 nm at 1000 nm. Note that the official RadCalNet product integrates over a 10 nm bandwidth.

The wavelength error of the spectrometer was approximately 1.7 nm (the spectrometer measured a wavelength that was 1.7 nm shorter than the true wavelength). No correction has been made until now for this wavelength error. To account for the uncertainty associated with the measured reflectance due to wavelength error, we need to consider both the impact on the calibration of the spectrometer gain using the calibration source and the impact on the field measurement. In both cases, the error can be approximated by:

$${\delta \left(L_{\mathrm{meas}}\right)|}_{\delta \lambda }=\frac{\partial L_{\mathrm{meas}}}{\partial \lambda }\delta \lambda$$

(7)

where the first derivative can be estimated numerically. Using a balanced numerical derivative:

$$\frac{\partial L_{\mathrm{meas}}}{\partial \lambda }\approx \frac{1}{2\delta \lambda }\left(L_{k+1}-L_{k-1}\right)$$

(8)

we could estimate the error on each measured spectrum. We then calculated the ratio of the field measurement to the laboratory measurement with and without correcting this error. The results are shown in Figure 5.

To account for the time when this was not corrected, an uncertainty was applied to the measured signal that is 1.5% from 500 to 700 nm and from 780 to 900 nm, and ranges from 2% to 5% elsewhere. This is used rather than an error correction because the exact shape of the curve in Figure 5 will depend on the specific spectrum measured for each observation.

4.2. Uncertainty Associated with Laboratory Calibration of the Field Spectrometer: Radiometric Calibration

The laboratory calibration of the field spectrometer is described by Equation (3). It is calibrated against a known-radiance source made from a lamp-illuminated diffuser panel. There are uncertainties associated with each term in the measurement equation, and additional uncertainties associated with assumptions implicit in the measurement equation. These assumptions include assumptions about source uniformity and instrument linearity.

4.2.1. Lamp-Diffuser Panel Radiance

The lamp was calibrated for spectral irradiance on NIM’s primary facility through a direct comparison with a blackbody source. The uncertainties associated with this calibration have been documented previously and validated through comparisons as part of the Mutual Recognition Arrangement of national metrology institutes [14]. As is common for national metrology institute calibrations, the calibration certificate for the lamp provides uncertainties at the 95% confidence level. The standard uncertainty is obtained by dividing the certificate uncertainty by the certificate-provided coverage factor, here k = 2 for each wavelength value in turn. The lamp calibration was performed in 50 nm spectral steps and was interpolated to intermediate wavelengths using a cubic spline; the uncertainty was similarly interpolated.

There are additional uncertainties associated with the lamp spectral irradiance at the time of the spectrometer calibration due to lamp ageing since calibration and lamp current setting. The uncertainty associated with these quantities was estimated to be between 0.17% and 0.32% for all wavelengths through the annual stability of standard lamp and current setting of the power supply.

The diffuser was calibrated for spectral radiance factor on NIM’s reflectance facility as an absolute calibration; again the 95% confidence level uncertainty provided for each wavelength was converted to a standard uncertainty. The spectral radiance factor is defined as the “ratio of the radiance due to reflection of the medium in the given direction to the radiance of a perfect reflecting diffuser identically irradiated” in the International Lighting Vocabulary [16]

There are additional uncertainties associated with the distance setting and angular alignment of the lamp-diffuser pair. The lamp was set at 500 mm from the diffuser, which was the calibration distance for the lamp irradiance. However, because the diffuser is a bulk diffuser, the exact distance is difficult to set, and a residual distance uncertainty of 0.5 mm is assumed (from the inverse square law, a 0.5 mm uncertainty in 500 mm corresponds to a 0.20% uncertainty in irradiance). Similarly, the diffuser was calibrated for radiance factor for a 0°/45° geometry, and a spectrally-flat uncertainty of 0.20% was included to account for angular differences from this condition.

4.2.2. Source Non-Uniformity for Spectrometer Calibration

A source produced by a lamp-diffuser combination shows significant non-uniformity. Centrally with the lamp’s filament, the diffuser is at its brightest and this drops away from that central position (towards the outside and corners of the diffuser). This bright area may not align with the defined optical axis. The radiance of the panel is calculated from the irradiance of the lamp—which has in turn been measured over a small area around the optical axis—and the reflectance of the panel. The spectrometer used at Baotou has a field of view of 3°, and was set at 250 mm from the panel giving an observational area with diameter 13 mm.

The non-uniformity of the lamp-illuminated diffuser panel was measured at NIM. A CS-2000 spectroradiometer was used to measure the non-uniformity of diffuser, and the full field of view (FOV) of the spectroradiometer is 0.2°. The spectroradiometer at the same distance was used to scan the spectral radiance of the diffuser at 1 mm intervals over its surface. The measurement distance between the spectrometer and the diffuser is 250 mm, so the diameter of the field of view on the diffuser is 0.9 mm. The non-uniformity was calculated as the difference between an average radiance over the 13 mm area observed by the Baotou spectrometer and the point observation for brightest radiance. This was 0.29%.

4.2.3. Combined Uncertainty Associated with Source Radiance

The uncertainty associated with the source radiance comes from a combination of all the factors described above. These uncertainty effects are independent of each other and therefore inter-term error correlation does not need to be considered. However, we do still need to consider other error correlations. A single spectrometer is calibrated once a year against the lamp-diffuser panel radiance source. All uncertainties associated with that calibration will create an unknown calibration error that will be common for all measurements by the spectrometer in the field until the next calibration.

Because the same lamp and diffuser panel were used to calibrate all three spectrometers in position at BTCN and BSCN, many uncertainty effects will also lead to a common error between those spectrometers. An error in the lamp irradiance or diffuser reflectance will be common to all measurements by all spectrometers. However, an error due to the alignment of the lamp-diffuser panel combination will be different for each spectrometer as the system was entirely realigned between spectrometers.

We also need to consider error correlation scales from wavelength to wavelength. Uncertainties associated with alignment, distance, and stray light (all geometrical) create errors that are fully correlated between wavelengths. Any error in the lamp current setting will also be fully correlated from wavelength to wavelength, although as shorter wavelengths have a larger sensitivity to current setting, the uncertainty will be larger at shorter wavelengths (see Section 3.3.3 in [11] for a similar example). The lamp and diffuser calibrations at NIM create partially correlated errors from one wavelength to the next as some aspects of the primary calibration (e.g., alignments, reference blackbody temperature) are fully correlated and others (e.g., noise during calibration) are not correlated. As NIM repeated the calibration multiple times and had taken care to reduce the uncertainty associated with random effects as far as possible, the error correlation for these are “mostly fully correlated”.

The uncertainties associated with reference source radiance are listed in

Table 1. Uncertainties associated with reference source radiance.

	Effect	Lamp Irradiance Calibration	Lamp Ageing and Current Setting	Diffuser Calibration	Lamp-Diffuser Distance	Lamp-Diffuser Alignment	Source Non-Uniformity
	Term affected	$E_{\mathrm{FEL}}\left({\lambda }_i\right)$ in Equation (2)	$E_{\mathrm{FEL}}\left({\lambda }_i\right)$ in Equation (2)	$\beta \left({\lambda }_i\right)$ in Equation (2)	$+0$ in Equation (2)	$+0$ in Equation (2)	$+0$ in Equation (2)
Error correlation	All measurements of a single spectrometer	Fully correlated	Fully correlated	Fully correlated	Fully correlated	Fully correlated	Fully correlated
	Between spectrometers	Fully correlated	Partially correlated 1	Fully correlated	Independent	Independent	Fully correlated 2
	Between wavelengths	Partially correlated 1	Fully correlated	Partially correlated 3	Fully correlated	Fully correlated	Fully correlated
Uncertainty magnitude	400 nm	0.42%	0.29%	0.28%	0.20%	0.50%	0.29%
	500 nm	0.35%	0.23%	0.28%	0.20%	0.50%	0.29%
	600 nm	0.31%	0.20%	0.28%	0.20%	0.50%	0.29%
	700 nm	0.29%	0.17%	0.29%	0.20%	0.50%	0.29%
	800 nm	0.29%	0.17%	0.31%	0.20%	0.50%	0.29%
	900 nm	0.30%	0.29%	0.32%	0.20%	0.50%	0.29%
	1000 nm	0.31%	0.32%	0.32%	0.20%	0.50%	0.29%

¹ Lamp ageing is likely to have an error that is fully correlated between spectrometers as long as the lamp is not operated for a long time between the two spectrometer calibrations. Current setting is independent between spectrometers as it is dominated by short term effects. ² Source non-uniformity is dominated by effects that are common for all calibrations as the spatial distribution stays basically constant for all alignments and the spectrometers have near identical fields of view. ³ The lamp irradiance calibration and the diffuser calibration were performed on NIM’s primary facilities. The uncertainty budget for that has components that are fully correlated between wavelengths and components that are independent between wavelengths.

, which acts as an “effects table” (to use the term introduced in [11]).

4.2.4. Spectrometer Noise during its Calibration

The uncertainty associated with this noise in the calibration signal was estimated from the relative standard deviation of 20 successive measurements of the stable calibration source. The uncertainty associated with spectrometer noise is very close to zero at most of wavelengths, which was estimated to be less than 0.1%.

4.2.5. Spectrometer Nonlinearity

Equations (2) and (3) assume that the spectrometer is linear. We determine a single radiance calibration factor (radiometric gain) from a calibration against a single source at one radiance level. However, spectrometers often suffer from non-linearity [17], usually caused by processes in the read-out electronics. There are two types of nonlinearity that must be considered: radiance level nonlinearity (gain is not constant with changing radiance) and integration-time nonlinearity (the measured counts are not proportional to the integration time). To evaluate these nonlinearities, the spectrometer was tested using a special integrating sphere system designed by NIM. The spectrometer remained on its “auto ranging” setting, so the integrating time varied dynamically.

The integrating sphere is illuminated by two lamps, each with an adjustable aperture in front of it. In this way the overall radiance of the sphere could be varied and each lamp in turn could be closed off. The sphere’s spectral radiance was varied so that at 800 nm it ranged from 0.005 to 0.25 W m⁻² sr⁻¹. The normal calibration radiance at 800 nm is 0.071 W m⁻² sr⁻¹ and the grey target radiance at 800 nm ranges between 0.005 W m⁻² sr⁻¹ (lowest typical values) and 0.1 W m⁻² sr⁻¹ (highest typical values).

The nonlinearity factor σ of the spectrometer is calculated as:

$$\sigma =\frac{I\left(\mathrm{A}+\mathrm{B}\right)}{I\left(\mathrm{A}\right)+I\left(\mathrm{B}\right)}-1$$

(9)

where $I\left(X\right)$ is the response of the spectrometer for a radiance level $X,$ where $X$ is the source with only lamp A, with only lamp B, or with lamps A and B simultaneously. The results are shown in Figure 6 and show an increase in nonlinearity for the shortest and longest wavelengths. Because nonlinearity is not corrected in the measurements during calibration or in the field, we treat the observed non-linearity as an uncertainty component. At the shortest wavelengths it is difficult to separate nonlinearity from noise, but taking into account that nonlinearity is unlikely to be significantly different from pixel to pixel, an uncertainty curve has been approximated using a local standard deviation. At longer wavelengths there is a noticeable trend, and uncertainty bounds have been drawn to include this trend. From this we assume that the uncertainty associated with nonlinearity is 0.28% from 540 nm to 850 nm, increasing to 0.6% linearly for longer and shorter wavelengths.

4.2.6. Stray Light during Calibration and Field Operation

Equation (3) assumes that the measured count signal comes from the source radiance, both in the laboratory and in the field environment. In practice there are several effects that alter this assumption.

First, there may be an electronic bias, a “dark count”, that is present when there is no illumination. This dark count is likely to be temperature sensitive and thus will change over time. In the laboratory the dark count can be measured by closing the input optics. In the field, such measurements are not possible routinely, but some tests can be performed manually. Beyond 500 nm the ratio of dark signal to light signal is less than 0.1%.

Second, there may be external stray light that reaches the instrument sensors from outside the main field of view. In the laboratory such stray light will come from any reflections of the source into the beam. In the field, it will come from the bright surrounding area. This is discussed in Appendix B.1. The uncertainty associated with this stray light has been estimated to be 0.1% for all wavelengths.

Third, there is internal stray light. Internal stray light comes from light that is scattered onto the “wrong” pixel from the wrong wavelength. Appendix B discusses how the stray light was evaluated using cut on filters. The error associated with not correcting for stray light is 25% at 400 nm, dropping to 10% at 420 nm, 5% at 440 nm, and 3% at 460 nm. While Appendix B provides a method that could be used to correct for stray light, in practice this has not been applied to data collected at BTCN and BSCN to date. The RadCalNet WG has agreed to operate with a “collection” process, where data that are on the portal can only be updated retrospectively at defined times, known as “collections” and the next collection is likely to be during 2021. Therefore, at present there is no stray light correction and a correction cannot be performed until 2021. Thus, the uncertainty is at its full magnitude. By the next collection we hope to have calibrated the instruments using tunable lasers so that the full correction algorithm discussed in Appendix B.3 can be applied.

4.2.7. Repeatability of the Calibration/Transportation Stability

The repeatability of the calibration and the sensitivity of the spectrometer to transportation was tested using a stable laboratory source at NIM. The spectrometer was calibrated on three occasions: twice while realigning but not transporting the instrument, and a third occasion where the instrument was transported then realigned. The relative difference of the second and third calibrations compared to the first calibration is shown in Figure 7. This suggests that the dominant effect is realignment. For the purposes of this analysis the uncertainty associated with transportation is less than 0.5%.

4.2.8. Combined Uncertainty Associated with the Laboratory Calibration of the Spectrometer

The combined uncertainty associated with the laboratory calibration of the spectrometer is a combination of the effects given in this section.

Table 2. Uncertainties associated with laboratory calibration of the spectrometer.

	Effect	Source Radiance	Wavelength Accuracy	Spectrometer Noise
	Term affected	$L_{\mathrm{source}}\left({\lambda }_i\right)$ in Equation (2)	${\lambda }_i$ in Equation (2)	$C_{\mathrm{cal}}\left({\lambda }_i\right)$ in Equation (2)
Error correlation	All measurements of a single spectrometer	Mixed. See Table 1	Fully correlated	Independent
	Between spectrometers	Mixed. See Table 1	Independent	Independent
	Between wavelengths	Mixed. See Table 1	Partially correlated	Independent
Uncertainty magnitude	400 nm	0.84%	5.00%	0.10%
	500 nm	0.79%	1.50%	0.1 0%
	600 nm	0.77%	1.50%	0.10%
	700 nm	0.76%	1.50%	0.10%
	800 nm	0.76%	1.50%	0.10%
	900 nm	0.81%	1.50%	0.10%
	1000 nm	0.82%	5.0%	0.10%
	Spectrometer Nonlinearity	Internal Stray Light	External Stray Light	Calibration Repeatability
	$+0$ in Equation (2)	$+0$ in Equation (2)	$+0$ in Equation (2)	$+0$ in Equation (2)
Error correlation	Fully correlated	Fully correlated	Fully correlated	Fully correlated
	Independent	Independent	Fully correlated	Independent
	Fully correlated	Fully correlated	Fully correlated	Fully correlated
Uncertainty magnitude	0.60%	25%	0.10%	0.12%
	0.37%	1.30%	0.10%	0.15%
	0.28%	0.20%	0.10%	0.24%
	0.28%	0.10%	0.10%	0.34%
	0.28%	0.10%	0.10%	0.46%
	0.38%	0.10%	0.10%	0.58%
	0.60%	0.10%	0.10%	0.79%

lists the different effects, the magnitude of their uncertainty, and the error correlation structures. The uncertainty associated with source radiance is obtained by combining the different components in Table 1. The wavelength accuracy is a setting of a spectrometer. In this way the associated error is fully correlated for all observations with that spectrometer and it is independent between spectrometers. Because the wavelength error changes across the spectrum, we consider its error partially correlated from wavelength to wavelength, however, the error is dominated by a common component. Noise is naturally entirely independent from observation to observation and because external stray light is a property of the calibration set up, this is considered to have a fully correlated error. Nonlinearity and internal stray light are properties of the spectrometer and have a stable spectral feature. Finally, calibration repeatability, which includes the stability of the instrument on transfer to the field, has an error considered to be fully correlated for all measurements by that spectrometer but independent between spectrometers.

4.3. Uncertainty Associated with the Field Measurement of Radiance

In this section we consider uncertainties associated with the measured field radiance obtained from Equation (4). Uncertainties associated with the calibration of the instrument, described above, affect the gain term, noise affects the measured in-field signal, and the temperature correction has uncertainties related to the uncertainty in the instrument in-field temperature, and the uncertainty associated with our knowledge of the coefficients and form of the correction Equation (5). Uncertainties associated with the stability of the instrument gain have already been considered in the “calibration repeatability” term in the section above and are not included here to avoid double-counting.

4.3.1. Spectrometer Temperature Stability

Equation (5) gives an empirically determined model for the temperature correction. This was obtained by calibrating the instrument against a stable source while its temperature was varied from 10 to 40 °C and the results compared to the reference calibration at 25 °C. The fitted curves as a function of wavelength are shown in Figure 8.

To work out the uncertainty associated with this correction, there are two components. First the uncertainty associated with the gain correction due to an uncertainty in instrument temperature of 2 °C can be estimated by applying the law of propagation of uncertainty to Equation (10). Here the notation ${u\left(G_T\right)|}_{u\left(T\right)}$ describes the uncertainty associated with the gain at a temperature T due to the uncertainty associated with that temperature $u\left(T\right)$. This is used because there are other uncertainties associated with the gain that would need also to be considered in a full uncertainty analysis of the gain.

$$\frac{{u\left(G_T\right)|}_{u\left(T\right)}}{G_{\mathrm{cal}}\left(\lambda \right)}=\frac{1}{G_{\mathrm{cal}}\left(\lambda \right)}\frac{\partial G_T}{\partial T}u\left(T\right)=\frac{\left(m_0+m_1\lambda +m_2{\lambda }^2\right)}{G_{\mathrm{cal}}\left(\lambda \right)}u\left(T\right).$$

(10)

Second, there is an uncertainty associated with the plus-zero term in Equation (5). This is the uncertainty associated with the suitability of the model to represent the true temperature sensitivity of the instrument. This uncertainty was estimated from the residual of the original measurement points to the model (Figure 9). The uncertainty associated with temperature correction includes both the propagated uncertainty (due to a 2 °C uncertainty associated with the temperature of the instrument in the field) and the model uncertainty given here.

4.3.2. Noise in the Field Measurements

During field measurements a single reading is taken every two minutes with an integration time determined automatically. The uncertainty associated with noise on this measurement cannot therefore be estimated from operational data.

However, to complement the laboratory measurements of repeatability described above in Section 4.2.3, 20 measurements were taken in the field within 30 s. The standard deviation of those 20 measurements was slightly higher than the 0.1% seen in the laboratory and ranged from 0.4% at 400 nm to 0.1% at 600 nm.

4.3.3. Combined Uncertainty Associated with the Field Measurement of Radiance

The combined uncertainty associated with field radiance measured by the spectrometer is obtained by combining the different components in

Table 3. The uncertainty associated with field radiance measured by the spectrometer.

	Effect	Laboratory Calibration	Noise During Field Measurement	Spectrometer Temperature Sensitivity
	Term affected	$G\left({\lambda }_i\right)$ in Equation (4)	$C_{\mathrm{cal}}\left({\lambda }_i\right)$ in Equation (4)	$f\left({\lambda }_i,T\right)$ in Equation (4)
Error correlation	All measurements of a single spectrometer	Mixed. See Table 2	Independent	Fully correlated
	Between spectrometers	Mixed. See Table 2	Independent	Independent
	Between wavelengths	Mixed. See Table 2	Independent	Partially correlated 1
Uncertainty magnitude	400 nm	25.43%	0.41%	3.38%
	500 nm	2.16%	0.19%	0.25%
	600 nm	1.74%	0.12%	0.23%
	700 nm	1.75%	0.11%	0.30%
	800 nm	1.77%	0.19%	0.54%
	900 nm	1.85%	0.2%	0.96%
	1000 nm	5.16%	0.28%	1.97%

¹ The error due to temperature sensitivity is partly correlated from wavelength to wavelength. The component that relates to uncertainty in the model is independent, while the component that is from the propagation of temperature uncertainty to radiance uncertainty is fully correlated.

. We have not performed that combination at this stage because we want to combine uncertainties with different error correlation structures separately. The measurement noise during a single field measurement leads to an error that is entirely independent from one measurement to the next, from one spectrometer to the next, and from one wavelength to the next. For the uncertainty associated with the temperature sensitivity coefficient, this can be considered fully correlated for all measurements by a single spectrometer (the effect is stable) and independent between spectrometers, because there is a separate thermometer in each spectrometer and they are mounted in separate housings, and because the effect was separately calibrated for each spectrometer. The wavelength-to-wavelength error correlation structure is mixed. Figure 9 above shows that the residual has almost no spectral pattern (except perhaps for central wavelengths where it is small) and therefore it is reasonable to assume that the error is uncorrelated. On the other hand, the propagation from an uncertainty associated with temperature to that associated with scene radiance is fully correlated between wavelengths as is related through an algebraic expression.

4.4. Uncertainty Associated with the Ground Reflectance

4.4.1. Atmospheric Measurements from the AERONET Sun Photometer

The ground-measured radiance is converted into ground reflectance using Equation (1). This equation requires an estimate of the solar irradiance as transmitted through the atmosphere and the sky irradiance as scattered by the atmosphere. Both quantities are evaluated using MODTRAN-5, which requires atmospheric parameters as inputs.

These atmospheric parameters are measured using the Cimel CE318 sun photometer which is at the Baotou site. The sun photometer forms part of AERONET and has been calibrated according to the AERONET procedures [9], originally at NASA and more recently at the Beijing XMWK Technology Co. Ltd., China. The measurement data are transmitted to the AERONET data processing center, automatically processed and made available at the AERONET website. Then, the aerosol optical depth (AOD) at 550 nm is calculated via logarithmic interpolation from the AOT in the 440, 670, 870, and 1020 nm channels. The measurements in the 936 nm channel of the solar radiometer are used to calculate the water vapor column (WVC), using a modified version of the Langley algorithm [18]. We have taken the uncertainties associated with AOD and WVC from the literature, in particular [19]. These are estimated as less than 0.01 (absolute uncertainty) and less than 12% (relative uncertainty), respectively.

4.4.2. Sensitivity Analysis of MODTRAN-5 to Atmospheric Conditions

Monte Carlo (MC) analysis techniques were used to determine the sensitivity of the MODTRAN-5 sky irradiance and solar irradiance calculations to uncertainties in the input parameters. Errors drawn from a Gaussian distribution with a standard deviation of 12% (relative) and 0.01 (absolute) were added randomly to WVC and AOD respectively, for a range of realistic conditions for the Baotou site. Each combination of parameters was separately tested, with for each test, either the AOD or the WVC varied and the other component kept constant. The corresponding total downwelling irradiance, i.e., the relative uncertainty associated with $E_{\mathrm{sun}}\left(\theta ,d;{\lambda }_i\right)+E_{\mathrm{sky}}\left({\lambda }_i,t\right)$ in Equation (1), was simulated 1000 times by MODTRAN-5. The standard deviation of each 1000 sets of simulated irradiances was the uncertainty associated with downwelling irradiance due to AOD or WVC uncertainties. A further test was performed where these two parameters were varied together (in a way that was consistent with the fact they were derived together, so including any error correlations) and this gave the same uncertainty as the combined uncertainty from the two separate components.

Figure 10 shows the relative uncertainty associated with the total downwelling irradiance due to WVC uncertainties of 12%. Similarly, Figure 11 shows the relative uncertainty associated with the total downwelling irradiance due to AOD uncertainties of 0.01. These graphs are for two of the representative conditions studied; similar results were obtained for other conditions. In the MC simulations, the mean value of AOD is set as 0.2, the mean value of WVC is set as 0.5 g/cm², and solar zenith angle (SZA) and view zenith angle (VZA) are set as 30° and 0°, respectively.

4.4.3. Other Uncertainties Associated with the MODTRAN-5 Processing

In addition to the uncertainties associated with the input parameters to MODTRAN-5 (the AOD and WVC), the downwelling irradiance calculated by MODTRAN-5 has uncertainties associated with other assumptions in the MODTRAN-5 model. MODTRAN-5 makes assumptions on the heights of different atmospheric layers and defines a set of “atmospheric models” to describe these conditions. It also makes assumptions on the type and size of aerosols and defines a set of defined “aerosol models” to describe different assumptions. For Baotou two atmospheric models are used, i.e., the one defined as “mid-latitude summer” is used from April to September and the “mid-latitude winter” model is used from October to March.

The aerosol model used is the “rural” type. This was chosen because the site is 70 km from the nearest city, Baotou, China. It is possible that the “desert” aerosol type may be more appropriate for some periods of the year. To understand the difference between “desert” and “rural” aerosol models for the site, data from one measurement were processed using both aerosol models. The relative difference between the models is given in Figure 12. The uncertainty associated with aerosol type is taken to be a rectangular distribution within this range, thus the standard uncertainty is $D/2\sqrt{3}$, where D is the full difference (Figure 12).

The overall accuracy of the MODTRAN-5 evaluation of downwelling irradiance also depends on the uncertainty associated with the radiative transfer model within MODTRAN-5 and the uncertainty associated with the solar spectral irradiance model used. Uncertainties associated with the solar zenith angle and solar distance (θ, d, respectively, in Equation (6)) are considered negligible. In the future, this uncertainty estimate could be evaluated using in-field measurements, but that has not yet been done.

The uncertainty associated with the MODTRAN-5 radiative transfer model is described in the literature [20] as being 1%–2% for radiance predictions. The solar spectral irradiance model used is the Thuillier model [15,21] and this has a standard uncertainty of 1.5% at 450 nm, 0.9% at 650 nm, 1.1% at 850 nm, and 0.8% at 1550 nm. As the spectral uncertainty is not known, these values at 450, 650, 850, and 1550 nm were linearly interpolated to provide intermediate wavelength uncertainties.

4.4.4. Combined Uncertainty Associated with Ground Spectral Reflectance

Overall, ground reflectance is calculated using Equation (1). As we have considered in the previous subsections the total downwelling irradiance $E_{\mathrm{sun}}\left(\theta ,d;{\lambda }_i\right)+E_{\mathrm{sky}}\left({\lambda }_i,t\right)$ as a single quantity, we also do not separate them in our combined uncertainty table. The combined uncertainty associated with ground reflectance are obtained by combining the different components in

Table 4. The uncertainty associated with ground reflectance measured by the spectrometer.

	Effect	Field Radiance	Solar Irradiance Model	MODTRAN	AOT	WVC	Aerosol Model
	Term affected	$L_{\mathrm{gnd}}\left({\lambda }_i,t\right)$ in Equation (1)	$E_{\mathrm{sun}}\left(\theta ,d;{\lambda }_i\right)+E_{\mathrm{sky}}\left({\lambda }_i,t\right)$ in Equation (1)
Error correlation	All measurements of a single spectrometer	Mixed. See Table 3	Partially correlated	Partially correlated	Partially correlated	Partially correlated	Partially correlated
	Between spectrometers	Mixed. See Table 3	Fully correlated	Fully correlated	Fully correlated	Fully correlated	Fully correlated
	Between wavelengths	Mixed. See Table 3	Fully correlated	Fully correlated	Fully correlated	Fully correlated	Fully correlated
Uncertainty magnitude	400 nm	25.66%	1.50%	2.00%	0.28%	0.10%	0.19%
	500 nm	2.18%	1.35%	2.00%	0.23%	0.10%	0.08%
	600 nm	1.76%	1.05%	2.00%	0.20%	0.03%	0.22%
	700 nm	1.78%	0.95%	2.00%	0.17%	0.18%	0.43%
	800 nm	1.86%	1.05%	2.00%	0.16%	0.12%	0.47%
	900 nm	2.09%	1.05%	2.00%	0.15%	0.84%	0.39%
	1000 nm	5.53%	0.95%	2.00%	0.13%	0.03%	0.40%

. Figure 13 shows the uncertainty associated with the bottom-of-atmosphere (BOA) reflectance from 400 nm to 1000 nm.

The error correlation structures due to the solar and sky irradiances are considered partially correlated from one observation to the next with a single spectrometer; the assumption is that these change on longer time scales (hours to days), but are constant for shorter timescales. This assumption could be refined by assuming a triangular or bell-shaped correlation structure but that has not yet been analyzed. Between spectrometers the error correlation is considered fully correlated—the three spectrometers are all on the same super-site and the same AERONET CIMEL instrument is used to obtain corrections for all three instruments. To evaluate the error correlation structure between wavelengths, we analyzed the Pearson correlation coefficient for the errors obtained in the Monte Carlo simulation. This showed a very high error correlation (above 0.95) for almost all wavelengths. We therefore treat this as fully correlated from wavelength to wavelength. The same analysis has not been done for the solar irradiance model, but we assume that it is also fully correlated for the purposes of a precautionary analysis (fully correlated components are not reduced by averaging).

4.5. Uncertainty Associated with the RadCalNet BOA Reflectance Product

The BOA reflectance measurement uncertainty established in the previous section is for a measurement with a single spectrometer at a single time at the native wavelengths of the spectrometer. These measurements are every 2 min on site. For the official RadCalNet BOA reflectance product we (a) increase the wavelength step to 10 nm, (b) provide a product for the full site area (45 × 45 m for BTCN, 300 × 300 m² for BSCN), and (c) average readings to create a value every 30 min.

4.5.1. Wavelength Sampling

To obtain data at 10 nm intervals there are two options. One option is to provide the narrow bandwidth information at 10 nm intervals. The other option is to perform a spectral integral to combine data over a 10 nm bandwidth. At present, the Baotou 10 nm data are obtained by picking the data at the 10 nm intervals. Therefore, this does not alter the uncertainty analysis.

4.5.2. Representativeness (Site Homogeneity)

The measurement of the site reflectance by the spectrometers is at one point (for BTCN) or two points (BSCN). However, the satellite observations will be averaged over the larger area of the defined site. To understand the representativeness of a single/two measurement(s) for the site as a whole, tests were performed where a portable spectrometer was used to make measurements at multiple points across each target. The spectrometers have a field of view of 3° and therefore see an area with a diameter of ~104 mm for BTCN (2 m above ground), and ~131 mm for BSCN (2.5 m height). For BSCN, the site is a natural sand with a ~1 mm grain size. For BTCN, the site has pebbles which are typically ~15 mm across. The portable spectrometer used for this analysis imaged a similar area to the permanent instruments.

The uncertainty associated with the representativeness of the point measurement was estimated based on the uniformity of the whole target, which is defined using the following equation:

$$u\left(\lambda \right)=100\%\times \frac{\sigma \left(\lambda \right)}{\overline{\rho }\left(\lambda \right)}$$

(11)

4.5.3. 30 Minute Temporal Averages

To obtain BOA reflectance every 30 min, the reflectance averages of 15 min before and after a given time (i.e., 9:00, 9:30, 10:00, etc.) are used as the final BOA reflectance. Since the measurement interval of spectrometer is 2 min, 15 BOA reflectance values are averaged. The variation in those readings comes from two origins: first, we expect variability due to all the effects listed in the tables above that have “independent” as their reading-to-reading error correlation structure and second, we expect variability due to the change in solar zenith angle over the 30 min averaging period. To validate the uncertainty associated above with “independent” effects we compared the standard deviation of the 15 readings with the combined uncertainty and obtained close agreement at noon. At 9 am and 3pm, the standard deviation was higher than the combined uncertainty of the “independent” effects, because of the change in the solar zenith angle. However, at present no uncertainty has been introduced to account for solar zenith angle changes as the time period of averaging is symmetrical about the declared time and the difference between the measured mean and a mean corrected for solar zenith angle is negligible.

4.5.4. Uncertainty Associated with the BOA Reflectance

The combined uncertainty associated with the BOA reflectance is a combination of the effects given in Section 4.1, Section 4.2, Section 4.3, Section 4.4 and Section 4.5. For BTCN and BSCN for the initial period when only one spectrometer was in operation, the official RadCalNet product is the uncertainty associated with the temporal mean of 15 measurements with a single spectrometer. To establish the uncertainty associated with this mean, we combined uncertainties that were associated with errors that were “partially” or “fully” correlated with time (any partial correlation is likely to be extremely high over a 30 min window) and separately combined uncertainties associated with errors that were “independent” in their error correlation structure. The uncertainty associated with independent effects could be reduced by the square root of the number of independent observations, here $\sqrt{15}$, and this was then added in quadrature (square root of the sum of the squares) with the uncertainties associated with correlated effects.

For the calculation of the uncertainty associated with the mean of the two spectrometers at BSCN, the uncertainty components were separated into (a) fully correlated for both time and between spectrometers, (b) fully independent for both time and between spectrometers, (c) correlated between spectrometers but independent from measurement to measurement, and (d) correlated between measurements but independent between spectrometers. Then, we separately calculated the uncertainty associated with each of these. The uncertainty associated with the mean of two spectrometers based on 15 measurements each was calculated by dividing (a) by 1, (b) by $\sqrt{30}$, (c) by $\sqrt{15}$, and (d) by $\sqrt{2}$. Finally, the combined uncertainty was calculated as the square root of the sum of the squares of these four parts. For this combination, we have assumed that the atmospheric effects are fully correlated over the 30 min of the averaging.

Figure 14 shows the uncertainty associated with the BOA reflectance. A summary for the uncertainty budget of BTCN and BSCN BOA reflectance is shown in

Table A1. Uncertainty budget for BTCN and BSCN BOA reflectance. The abbreviations FC, PC, and I mean fully correlated, partially correlated, and independent, respectively.

Effect	Error Correlation			Uncertainty Magnitude (%)
	All Measurements of a Single Spectrometer	Between Spectrometers	Between Wavelengths	400 nm	500 nm	600 nm	700 nm	800 nm	900 nm	1000 nm
Lamp Irradiance Calibration	FC	FC	PC	0.42	0.35	0.31	0.29	0.29	0.30	0.31
Lamp Ageing and Current Setting	FC	PC	FC	0.29	0.23	0.20	0.17	0.17	0.29	0.32
Diffuser Calibration	FC	FC	PC	0.28	0.28	0.28	0.29	0.31	0.32	0.32
Lamp-Diffuser Distance	FC	I	FC	0.20	0.20	0.20	0.20	0.20	0.20	0.20
Lamp-Diffuser Alignment	FC	I	FC	0.50	0.50	0.50	0.50	0.50	0.50	0.50
Source Non-Uniformity	FC	FC	FC	0.29	0.29	0.29	0.29	0.29	0.29	0.29
Wavelength Accuracy	FC	I	PC	5.00	1.5	1.50	1.50	1.50	1.50	5.00
Spectrometer Noise	I	I	I	0.10	0.10	0.10	0.10	0.10	0.10	0.10
Spectrometer Nonlinearity	FC	I	FC	0.60	0.37	0.28	0.28	0.28	0.38	0.60
Internal Stray Light	FC	I	FC	25	1.30	0.20	0.10	0.10	0.10	0.10
External Stray Light	FC	FC	FC	0.10	0.10	0.10	0.10	0.10	0.10	0.10
Calibration Repeatability	FC	I	FC	0.12	0.15	0.24	0.34	0.46	0.58	0.79
Noise during Field Measurement	I	I	I	0.41	0.19	0.12	0.11	0.19	0.20	0.28
Spectrometer Temperature Sensitivity	FC	I	PC	3.38	0.25	0.23	0.30	0.54	0.96	1.97
Solar Irradiance Model	I	FC	FC	1.50	1.35	1.05	0.95	1.05	1.05	0.95
MODTRAN	PC	FC	FC	2.00	2.00	2.00	2.00	2.00	2.00	2.00
AOT	PC	FC	FC	0.28	0.23	0.20	0.17	0.16	0.15	0.13
WVC	PC	FC	FC	0.10	0.10	0.03	0.18	0.12	0.84	0.03
Aerosol Model	PC	FC	FC	0.19	0.08	0.22	0.43	0.47	0.39	0.40
Representativeness of the Point Measurement (BTCN)	FC	I	FC	5.42	5.54	5.48	5.60	5.65	5.71	5.52
Representativeness of the Point Measurement (BSCN)	FC	I	FC	4.09	3.76	3.52	3.45	3.42	3.50	3.67

(see Appendix C).

4.6. Uncertainty Associated with Propagation to TOA

Comparing a satellite sensor observation with the Baotou calibration site requires TOA reflectance. As discussed in Section 3.1, propagation to TOA requires the determination of direct light from the site, light lost through scattering between the site and the satellite, and light gained from scattering in the atmosphere, both directly and having reflected off nearby ground locations. The influence on the measured radiance of light that has reflected off nearby ground locations (background radiation) is described by the adjacency effect [22].

AIR has performed a detailed analysis on the adjacency effect when calculating the TOA reflectance from the BOA reflectance. A method considering the adjacency effect has been proposed and detailed in another paper [23]. In this method, a local atmospheric point spread function (PSF) for the Baotou site was constructed, and this was used to calculate an effective background reflectance that was used in MODTRAN-5 simulation as the reflectance of the surrounding area. The TOA reflectance calculated using this method with the consideration of adjacency effect was compared with several satellites’ observations. The uncertainty associated with the effective background reflectance was obtained by considering several uncertainty components, such as the constructed atmospheric PSF, the error of AOD, and the seasonal change of the surrounding area. Then, the uncertainty associated with the TOA reflectance simulated using the effective background reflectance was also estimated.

However, for the official RadCalNet product, the propagation to TOA is performed through the RadCalNet processing. This does not have the option to include the correction of the adjacency effect. Thus, we determined the uncertainty introduced by not correcting for the adjacency effect, based on the constructed effective background reflectance of Baotou site in the previous study. This was evaluated by running MODTRAN-5 for the different targets assuming first the background reflectance discussed above and then a background reflectance that is the same as the target reflectance. The difference in the TOA reflectance between correcting and not correcting the adjacency effect in MODTRAN-5 simulations for BTCN and BSCN targets are shown in Figure 15.

In addition to the uncertainties associated with the BOA reflectance measurements, uncertainties associated with the radiative transfer modelling to TOA performed by MODTRAN using the input atmospheric conditions, such as AOD, WVC, aerosol type, and MODTRAN model, are also considered. To determine the uncertainty associated with propagation to TOA reflectance, a similar approach to that used for the determination of the uncertainty associated with $E_{\mathrm{sun}}\left(\theta ,d;{\lambda }_i\right)+E_{\mathrm{sky}}\left({\lambda }_i,t\right)$ was used (Section 4.4). The TOA reflectance is calculated using MODTRAN-5 from solar spectral irradiance, sun–earth distance factor and the solar zenith angle. MC analysis techniques were also used to determine the uncertainty associated with the TOA radiance due to WVC and AOD uncertainties. The uncertainties associated with the assumption of aerosol type, solar irradiance model, and MODTRAN-5 radiative transfer model were also considered.

According to the analysis results, the total uncertainty associated with the TOA reflectance propagated from the BOA reflectance is given in Figure 16.

5. Discussion

The uncertainty of BOA reflectance (strictly “hemispherical-conical reflectance factor”) measured by a single spectrometer for BTCN (about 6%) is greater than that for BSCN (about 4.8%), which is to a large extent caused by the surface uniformity. Because there are two spectrometers deployed in BSCN, the uncertainty for BSCN is further decreased to about 3% across most of the wavelength range (see Figure 14). Thus, the combined uncertainty of BSCN associated with propagation to top-of-atmosphere reflectance is almost always better than 5% within the spectral range 450–1000 nm, and in about half of the spectral range the uncertainty is less than 4%. This result indicates that the high uncertainty due to the poor surface uniformity for the BTCN site could be reduced by using multiple observation devices. At the shortest wavelengths (below 450 nm) the uncertainty is dominated by internal stray light effects within the spectrometer and reaches ~20% at 400 nm.

This paper focuses on the uncertainty analysis method applied to the official RadCalNet reflectance products using the standard RadCalNet processing, without adjacency effect correction and at 10 nm intervals. Reduced uncertainties are available from a processing to TOA that includes adjacency effect correction.

Performing a thorough uncertainty analysis always identifies areas of potential improvement. Here we have identified that the wavelength error could be reduced and that stray light is the dominant source of uncertainty for the shortest wavelengths and therefore needs a more thorough investigation and the establishment of a matrix-based correction algorithm. We anticipate that these investigations will lead to an improved uncertainty by the time of the next RadCalNet Collection (2021) and some of the corrections will be able to be applied retrospectively (particularly for stray light). The uncertainty associated with homogeneity is also a significant uncertainty for BTCN and could, perhaps, be improved using an instrument with a wider field of view, however, such an instrument would be more difficult to calibrate. The uncertainty budget allows us to consider such trade-offs in future improvements.

6. Conclusions

This paper showed an example of a metrological uncertainty analysis for a RadCalNet site. For the first time, we considered the error correlation structures for each uncertainty component. These error correlation structures enabled us to perform a robust propagation to a product that averages two spectrometers and 15 measurements by each spectrometer. In the future, we will use the wavelength-to-wavelength error correlation structure to obtain a reliable estimate of the uncertainty associated with a spectrally integrated product (e.g., to match the spectral response function of a satellite sensor). The temporal and spectrometer-to-spectrometer error correlation structures will also be needed to determine a robust uncertainty associated with the comparison to a satellite that makes multiple overpasses of the sites during a comparison period. To perform that analysis completely, we would also need to account for the error correlation structures for the satellite product. The paper that we used as the basis of the method described here [11] has shown how this was done for the Advanced Very High Resolution Radiometer (AVHRR) sensors. A simplified version of this has also been done for Sentinel-2 [24].

As an example of the metrological uncertainty analysis, this paper described the determination of ground reflectance and TOA reflectance for the Baotou BTCN and BSCN sites. We analyzed the sources of uncertainty from the laboratory calibration of the spectrometer, through to field observations of radiance and the calculation of ground reflectance, and to the propagation to TOA reflectance. The uncertainty tree diagram for the Baotou measurements of ground spectral reflectance was drawn, and every uncertainty component was considered and analyzed based on tests in the laboratory and field, and Monte Carlo modelling of the atmospheric corrections.

The preliminary results for the uncertainty propagating to the TOA reflectance were analyzed. The uncertainty associated with the TOA reflectance propagated from BOA reflectance measured by a single observation is approximately estimated as 6% for BTCN and 4.8% for BSCN if the adjacency effect can be corrected for. Since the RadCalNet BOA product on BSCN is generated based on the observations from two spectrometers, its uncertainty is reduced to about 3%. The uncertainties of the official RadCalNet TOA product (which does not consider the adjacency effect) for BTCN and BSCN are estimated to be approximately 7% and 4%–5%, respectively.