The basis clear sky LUT consists of spectrally resolved RTM results for aerosols with different values of aerosol optical depth (aod), single scattering albedo (ssa) and asymmetry parameter (gg). The basis LUT is dedicated to consider the effect of aerosol scattering and absorption. The absorption by water vapour and ozone as well as the surface reflection are predominantly independent from the aerosol state and fixed values have been used for the calculation of the basis LUT, i.e., a water vapor column of 15 mm, an ozone content of 345 DU, and a surface albedo of 0.2. The atmospheric values conform to the US standard atmosphere. The effect of the solar zenith angle on the transmission, hence the solar irradiance, is considered by the use of the Modified Lambert–Beer (MLB) function developed by Mueller et al.[22], which is given in Equation (2). (2) I Λ = I 0 , enh , Λ * exp ( − τ 0 Λ cos a Λ ( θ z ) ) * cos ( θ z )where τ_0Λ is the optical depth of the vertical column, I_Λ is the solar radiation at ground for a solar zenith angle (SZA) of θ_z and I_0,enh,Λ is based on the extraterrestrial irradiance according to Equation (3) at wavelength (wavelengths band) Λ. The cosine of the solar zenith angle accounts for the decrease of the surface flux density due to the increase of the solar zenith angle (the same amounts of photons are distributed over a larger area for increasing solar zenith angles). This relation is identical to the Lambert–Beer function (or MLB function) with exception of an additional “empirical” correction exponent a, hence a correction of the parameter τ cos ( θ z ) [22]. The correction parameters a_Λ are calculated based on two RTM runs, one at θ_z =0 and the other at θ_z =60°, hence the correction parameters a_Λ can be calculated without the need for a numerical fit. I_0,enh is based on the extraterrestrial irradiance at the top of atmosphere and estimated using Equation (3)[22]. (3) I 0 , enh , Λ = ( 1 + I 0 Λ ⋅ D Λ B Λ ⋅ I Λ ) ⋅ I 0 ΛHere D_Λ and B_Λ are the diffuse and direct (beam) component of the solar surface irradiance I_Λ = B_Λ + D_Λ at a SZA of zero. The application of this equation is needed in order to preserve a good match of the MLB relation with RTM results for high optical depths, where the use of the extraterrestrial irradiance I_0Λ fails. It is important to notice that the Modified Lambert–Beer relation (Equation (2)) is also used for the calculation of the direct (beam) irradiance B_Λ, with the exception that instead of using I_0,enh,Λ the original extraterrestrial irradiance I_0Λ must be used. The fitting parameters a_Λ have different values for direct irradiance and (total) solar surface irradiance. Using the MLB function, the calculated direct irradiance, diffuse irradiance as well as the solar surface irradiance can be reproduced very well (see Figure 2 as an example). The MLB function is discussed in detail by Mueller et al.[22], including proof and verification. Further validations are given by Ineichen et al.[23].

The MLB parameters a_Λ, I_0,enh,Λ and τ_0Λ are saved in the basis LUT for each wavelength band and different values of aerosol optical depth (aod), single scattering albedo (ssa) and asymmetry parameter (gg). The resulting basis LUT is three dimensional and contains the MLB-parameters to calculate I_basis and B_basis with Equation (2). The effect of aod is significantly larger than the effect of both ssa and gg, so the basis LUT contains MLB parameters for 23 aod values times three ssa values times two gg values (ssa ∈ {0.7, 0.85, 1.0} and gg ∈ {0.6, 0.78}). The spectral effect of aerosols is considered by application of a standard aerosol model [24,25].

The calculation of the solar irradiance is done as follows: for any given aerosol state the nearest neighbors are selected from the LUT and the solar irradiance is calculated for these nearest neighbors at any given solar zenith angle by application of the MLB relation. Subsequent interpolation between the nearest neighbors provides the solar irradiance for a given aerosol state, but only for fixed water vapor, surface albedo and ozone values. In order to correct for variations from the fixed values occurring in nature, the parameterizations described in Sections 3.2 and 3.3 are applied as consecutive steps.

The sensitivity of surface irradiance on atmospheric variables has been evaluated for all steps in order to reduce the mesh points of the needed RTM calculations as much as possible. Decision criteria for the mesh width has been the effect of the parameters on the atmospheric transmission in combination with its degree of non-linearity, e.g., for a parameter with large effect on the atmospheric transmission but a predominantly linear behavior the interpolation grid can be wide-meshed, while with increasing non-linearity the distance between the grid-points has to be decreased.

The results of radiative transfer modeling reveals that the effect of variations in water vapor and ozone is predominantly independent of the atmospheric state (in particular the aerosol state) [13]. Taking benefit of this feature the basis LUT has been calculated for a fixed value of water vapor (15 mm) and deviations relative to this value are corrected subsequently by application of Equation (4), which is based on the broadband correction applied in [13]. RTM runs for the clear-sky case have been performed, in order to derive the atmospheric transmissivity correction (Equation (4)) for water vapor variations relative to the fixed standard values used in the basis LUT. (4) I basis , Λ H 2 O = I basis , Λ + Δ I H 2 O , Λ ⋅ cos b Λ θ z .

ΔI_H2O,Λ is the difference between the irradiance I basis , Λ H 2 O for the real amount of water vapor and I_basis,Λ resulting from interpolation within the basis “aerosol” LUT. ΔI_H2O,Λ is calculated for the following set of input parameters (θ_z =0, a rural aerosol type with aod = 0.2, ssa = 0.94, gg = 0.75, 345 DU ozone, and SAL_Λ = 0.2). b_Λ is a “fitting” parameter applied to match the solar zenith angle dependency of water vapour absorption.

ΔI_H2O,Λ is pre-calculated for 18 water vapor columns, seven of them in the range of [0...15] mm and eleven within [20...75] mm. Respective values of ΔI_H2O,Λ and b_Λ are saved in an LUT for each wavelength band. The use of this LUT enables the calculation of the irradiance for the real water vapor values given for the specific pixel and time. Only 19 of the 32 Kato bands are influenced by water vapor absorption, consequently only for these the LUT contains non-zero values for ΔI_H2O,Λ and b_Λ. The parameter b_Λ is limited by b_Λ ⩽ 1.

The differences between explicit RTM runs and results derived using Equation (4) are in general quite small as visualized in Figure 3 for a sun zenith angle of 20° as an example.

For solar surface irradiance the deviations between radiative transfer results and the parameterization are negligible for the majority of cases. Only for extreme atmospheric conditions (high SZA, very high or low amount of water vapor), considerable deviations between explicit RTM runs and parameterization occur. The wavelength bands Λ ∈ {724 nm, 932 nm, 1,355 nm} are most sensitive to changes in water vapor content. For those, deviations in the irradiance are given in Table 1 for the worst case of 80° SZA and 70 mm water vapor content.

In a further independent step the same approach as in Equation (4) is used for ozone. (5) I basis , Λ H 2 O , O 3 = I basis , Λ H 2 O + Δ I O 3 , Λ ⋅ cos c Λ θ z .

ΔI_O3,Λ is the difference between the irradiance I basis , Λ H 2 O , O 3 for the real amount of ozone and I basis , Λ H 2 O derived for the fixed value of ozone. c_Λ is a “fitting” parameter applied to match the solar zenith angle dependency of ozone absorption.

The effect of ozone on the broadband solar surface irradiance is quite small, therefore three pre-calculated ΔI_O3 values are sufficient. In contrast, for spectrally resolved irradiance eight pre-calculated ΔI_O3 values are needed (210 to 525 DU, in steps of 45 DU). The LUT for ΔI_O3 and c_Λ contains nonzero values for 13 of 32 Kato bands. For broadband irradiance the parameter c_Λ < 1. This relation also holds for direct irradiance for most wavelength bands, but fails to generate reliable results for the ultraviolet range. Here, appropriate modifications of Equation (5) might resolve the problem.

Again small differences between explicit RTM runs and results derived using Equation (5) are observed, illustrated for a sun zenith angle of 20° for global irradiance in Figure 4. Nevertheless, for high SZA considerable deviations between explicit RTM runs and parameterization occur. For the wavelength bands Λ ∈ {317 nm, 586 nm, 646 nm} that are highly sensitive to changes in ozone content, deviations in the irradiance are given in Table 2 for the worst case of 80° SZA and 525 DU ozone content.

RTM calculations for a variety of atmospheric states have shown that the effect of the surface albedo on the solar surface irradiance is predominantly independent of the atmospheric clear sky state.

The following formula was derived for broadband solar surface irradiance [13]: (6) I Λ = I basis , Λ H 2 O , O 3 * ( 0.98 + 0.1 ⋅ S A L Λ )

I basis , Λ H 2 O , O 3 is the solar surface irradiance derived from the basis LUT for a surface albedo of 0.2 for all wavelength after Equations (4) and (5) have been applied. SAL_Λ is the variable surface albedo and I_Λ is the solar irradiance after the surface albedo correction has been applied.

This formula was derived for broadband surface irradiance and albedo [13]. Nevertheless, within the spectral model, we assume that this formula can also be applied to spectral reflectance SAL_Λ. However, the applied equation simplifies the spectral effect of the surface albedo. Increased surface albedo will increase particularly short wavelengths because of the strongly wavelength-dependent backscattering by the atmosphere as discussed, e.g., by Lenoble et al.[26]. This effect is not accounted for by the applied equation.

As a database for spectral reflectance we use the spectral albedo functions assigned to each of 20 land-use types, originating from the NASA CERES/SARB Surface Properties Project ([27,28]). The land-use types are fixed throughout the year, hence variation of the surface albedo over the year are not considered. The spectral resolution of the derived surface albedo equals the grid of the correlated-k Fu/Liou parameterization in the range of 800–4,000 nm. Within the 290–800 nm range the surface-type dependent albedo files provided with libRadtran can be used, which have a significantly higher spectral resolution. These are not defined for the same land-use types, but for a smaller set of classes like lawn, grass, water and rye. To solve this problem we use a feasible linear combination of libRadtran spectral reflectances to estimate the reflectivity of the NASA CERES/SARB classes. The diurnal variation of the spectral surface albedo is determined as a function of solar zenith angle, as given by [29]. The scene dependent solar zenith adjustment factors needed for this function are also taken from the NASA CERES/SARB Surface Properties Project.

The atmospheric background profile provides the vertical information on the total number density needed for the calculation of the Rayleigh scattering. In addition, it provides the vertical profile of water vapor and ozone, which are scaled according to the column amounts. Yet, using different background profiles (e.g., mid-latitude summer, polar winter, ...) has predominantly no effect on the solar surface irradiance. Consequently, in the new approach all RTM calculations are only performed for the US standard atmosphere rather than for 5 atmospheric background profiles as before in the CM-SAF prototype.

The basis LUT is three dimensional and contains the MLB-parameters to calculate I_basis and B_basis with Equation (2). The irradiances are interpolated from this basis LUT for the respective aerosol state, expressed by specific values of aod, ssa and gg. The spectral effect of aerosols is considered by application of a standard aerosol model [24,25]. The effect of deviations in water vapor, ozone and surface albedo relative to the fixed values used in the calculation of the basis LUT are corrected by application of the parameterizations described in detail in Sections 3.2 and 3.3.

Figure 5 illustrates the new clear-sky scheme and the context and order of the parameterizations.

Due to optimizations of the interpolation grid and the application of the water and ozone correction as well as the surface albedo parameterization the amount of needed RTM calculations is enormously reduced, e.g., by a factor of 10,000 without losing accuracy. This is illustrated in Table 3.

The broadband cloud transmission is calculated from the effective cloud albedo derived with the Heliosat method ([14,15]). The effective cloud albedo is defined as the normalized difference between the all sky and clear sky reflection in the visible observed by the satellite. The deviation of the all sky reflection of the pixel from the clear sky reflection is a measure of the cloud effect on the Earth’s solar radiation budget. In order to derive the effective cloud albedo, different luminance conditions arising from variations in the observed reflections due the Sun-Earth distance and the solar zenith angle have to be corrected. Furthermore, the dark offset of the instrument has to be subtracted from the satellite image counts. The observed reflections are therefore normalized by application of Equation (7): (7) ρ = D − D 0 f ⋅ cos ( θ )Here, D is the observed digital count including the dark offset of the satellite instrument. D₀ is the dark offset, θ is the solar zenith angle and f corrects the variations in the Sun-Earth distance. The effective cloud albedo (CAL) is then derived from the observed normalized reflections by Equation (8): (8) CAL = ρ − ρ cs ρ max − ρ csHere, ρ is the observed reflection for each pixel and time. ρ_cs is the clear sky reflection, which is a monthly value derived for every pixel and time slot separately. This is essentially done by using the reflection of the pixel in a cloud free case. This is usually the relative minimum of the reflections during a certain time span (e.g., a month) derived for each pixel of the satellite image. Further details on the method to derive ρ_cs are given in [15]. ρ_max is the “maximum” reflection. It is determined by the 95 percentile of all reflection values at local noon in a target region, characterized by high frequency of cloud occurrence for each month. Changes in the sensitivity of the satellite instrument would lead to a respective change in the 95 percentile. In this manner changes in the satellite brightness sensitivity are accounted for.

The wavelength dependency of the effective cloud albedo and hence the cloud transmission is accounted for in a second step (Section 4.2).

The broad band cloud transmission is equivalent to the clear sky index k_bb defined as the ratio of solar surface irradiance I to clear sky irradiance I_cs: (9) k b b = I / I cs .

This value is related to the effective cloud albedo given by the Heliosat relation (e.g., [14]or[15]) via: (10) k b b = { 1.2 for   CAL   ≤ − 0.2 1 − CAL for   − 0.2 <   CAL   ≤ 0.8 1.1661 − 1.781   CAL + 0.73   CAL 2 for   0.8   <   CAL   ≤   1.05 0.09 for   CAL   >   1.05.

The occurrence of clouds results in a shift of clear sky spectrum from the red towards the blue range. Hence, a conversion from the broadband cloud transmission described by the clear sky index k_bb to a wavelength dependent transmission k_Λ is required. The clear sky state does not predominantly affect the spectral correction discussed hereafter. Hence the standard atmosphere of the basis LUT has been used for the derivation of the spectral correction.

For this, libRadtran RTM runs have been performed for the different wavelength bands and for the broadband with SAL and SZA set to zero and for cloud layers with cloud optical depth values of COD = 0, 10, 20, 40, 80 and 160. The corresponding wavelength dependent clear sky index values are gained through division of the resulting solar surface irradiance by the irradiance for COD = 0. Hence clear sky indices for all wavelength bands and the broadband are available after this step. The broadband clear sky index values for the different CODs are k_bb = 1, 0.525, 0.244, 0.197, 0.102 and 0.050, respectively.

Conversion factors are calculated for each wavelength band with a radiative transfer model by Equation (11) in order to consider the spectral effect of clouds on the broadband transmission. (11) f Λ = k Λ RTM k b b RTMHere, f_Λ is the conversion factor, k Λ RTM and k b b RTM are the clear sky indices for the wavelength bands (denoted by Λ) and the broadband (denoted by bb) calculated with radiative transfer model (denoted by RT M)

The resulting conversion factors are shown in Figure 6.

The conversion factors are saved in a LUT and applied to correct the wavelength effect of clouds on the clear sky transmission for every satellite pixel and time by application of Equation (12): (12) k Λ = f Λ ⋅ k bb sat .and subsequently by application of Equation (13): (13) I Λ = k Λ ⋅ I cs , Λhere, k_Λ is the derived all sky spectrally resolved clear sky index and k bb sat is the broadband clear sky index observed by satellite. I_Λ is the all sky and I_cs,Λ the clear sky spectrally resolved irradiance, respectively. I_cs,Λ is derived with the clear sky method described in Section 3.

The surface reflection is already accounted for within the retrieval of the effective cloud albedo which motivates the setting of the surface albedo to zero. However, the effective cloud albedo is retrieved with a broadband visible channel, hence, the spectral dependency of multiple scattering between clouds and the surface is not accounted for within the applied spectral correction. Furthermore, the approach discussed in this manuscript simplifies the spectral dependency of clouds on the solar zenith angle. Radiation with shorter wavelengths in the visible-UV penetrate clouds more effectively than at longer wavelengths, and the difference increases with increasing solar zenith angle [30]. This effect is not accounted for by the discussed approach.

The information on the effective cloud albedo is derived with the Heliosat method, described in Section 4. This methods accounts also for the effect of surface reflection in cloudy situations by consideration of the observed clear sky reflection (see Section 4.1). The effective cloud albedo is retrieved in pixel resolution from the broadband visible channel of the Meteosat satellites. This means that CAL is available in 30 min resolution for Meteosat First Generation and in 15 minute resolution for Meteosat Second Generation (MSG). For this study CAL derived from MSG’s high resolution visible broadband channel has been used, hence a CAL value has been available every 15 min at a given location. Further information on MSG channels are given in [1]. The spectral effect of aerosols is considered by application of a standard aerosol model [24,25]. The irradiances are interpolated from this basis LUT for the respective aerosol state. For information about the aerosol state an aerosol climatology based on AEROCOM data [31] merged with ground based measurements from the AErosol RObotic NETwork (AERONET) [32] are used. It comes as a climatology of monthly means in 1 × 1 degree resolution. Alternatively aerosol information can be taken from the Global Aerosol Date Set (GADS/OPAC) [33] using relative humidity from the National Centers for Environmental Prediction (NCEP) [34] in order to consider the effect of relative humidity on aerosol optical depth, single scattering albedo and asymmetry parameter.

The water vapor profile results from the reanalysis of the European Centre for Medium-Range Weather Forecasts at a 0.25 × 0.25 degree grid [35,36]. Monthly values have been used. For ozone, climatologies based on the Total Ozone Mapping Spectrometer (TOMS) and the Ozone Measuring Instrument (OMI) are used [37].

The treatment of the surface reflection in clear sky situation has been discussed in Section 3.3. The surface albedo does not vary throughout the year but is based on long term annual means. Details are given in Section 3.3. This means that seasonal and year to year variations in snow cover are not considered. This induces uncertainties of about 2%–4% in regions with frequent variations in snow cover. Occasionally higher uncertainties might occur.

Two data sets of spectral measurements are used for the validation of the method. They cover one year and are specified in Table 4. Both data sets were quality checked by the institutes that carried out the measurements. Please note, that these measurements are performed on a tilted plane, because they are intended to be used within photovoltaic performance studies.

To make the datasets comparable they were averaged to hourly values. Moreover, they have been integrated to the spectral bands used in our approach. As the instruments are limited within the interval of 300 to 1,700 nm, we can perform this integration for the 20 Kato bands defined within the interval of 317–1,613 nm. Both data sets have been used in a preceding study in a comparison with SOLIS (SOLar Irradiance Scheme) [22].

Different aspects were used for validation of the data retrieved with SPECMAGIC.

First, the retrieved annual averages of I_Λ have been compared with measurements for each wavelength band for cloud free skies. Cloud free cases are classified by the ratio between diffuse and global irradiance, for a ratio below 0.3 clear sky cases are assumed. The applied error measures, Root Mean Square Deviation (RMSE) and Bias, are based on comparison of hourly values and are defined in the Appendix. Please note that the comparison on hourly basis leads to high values of the root mean square deviations. Therefore, the absolute and relative bias values are the main quantities applied for the evaluation of the ability of the method to provide accurate spectral resolved irradiance. The resulting spectra are given for both sites together with the bias in Figure 7.

In general a good agreement between the retrieved and the measured spectrum is apparent. The retrieved annual spectrum is in better agreement with Stuttgart measurements. This is also depicted in the bias. The bias is closer to zero at Stuttgart than at Loughborough. The good match in the UV at the Stuttgart site in contrast to the mismatch at the Loughborough site might be a result of an outdated calibration of the instrument in the UV where aging and blurring of optical devices happens much faster. The bias shows no striking spectral feature or trend between 400 and 1400 nm. Secondly, SPECMAGIC has been compared with SOLIS) [22] for cloud-free skies. SOLIS follows a concept of integrated radiative transfer runs, but takes benefit of the modified Lambert–Beer relation in order to improve the computing performance. The use of identical atmospheric input for SOLIS and SPECMAGIC avoids the occurrence of differences in the comparison that are induced by uncertainties in the atmospheric input. As the MLB approach is applied in both schemes also uncertainties induced by this approach are avoided. Hence, the comparison results provide information about the accuracy of the applied hybrid eigenvector approach and parameterizations for water vapor, ozone and surface albedo corrections relative to explicit radiative transfer modeling for typical atmospheric conditions.

Figure 8 shows the comparison results. It is observed that the SPECMAGIC annual spectrum is in good agreement with the SOLIS annual spectrum. However, the bias indicates a slight tendency of SPECMAGIC to underestimate the SOLIS results for wavelength bands with Λ < 700 nm and to overestimate the SOLIS results for wavelength bands with Λ > 700 nm.

The relative bias and root mean square deviation (RMSE) resulting from the comparison of SPECMAGIC with the ground measurements are illustrated for both sites in Figure 9.

In order to investigate differences in the spectral features between cloudy and cloud free situations, the respective statistical quantities are diagrammed separately in addition. It can be seen that the relative RMSE is dominated by the cloudy situations, and it is significantly reduced for clear sky situations. A striking feature is the large all sky relative bias above 1,300 nm. For clear sky irradiance an significant increase in the relative bias is also apparent, which indicates that the clear sky model might be a main driver. However, the results indicate that also the spectral cloud correction contributes significantly to the bias. Further analysis is needed before final conclusions can be drawn, especially as systematic measurement errors can not be excluded a priori, but have to be seriously taken into account. Below 1,200 nm the retrieval shows a good performance. This indicates that the applied spectral correction of clouds works well here.

The final analysis concentrates on four wavelength bands. We use an ozone absorption band centered around 345 nm (328–363 nm). A second band is chosen in the visible range (center 485 nm, 452–517 nm). The third band is a water vapor absorption band (center 931 nm, 889–975 nm). The following band is a near infrared band that is not affected by water vapor absorption (center 1,010 nm, 975–1,046 nm). For these four bands the yearly course is given for both sites in Figures 10 and 11. The irradiance values at the specified bands are normalized with the broadband irradiances to concentrate the analysis on purely spectral effects. The monthly averages as well as the 10th and 90th percentile of normalized irradiance are given in these figures.

A first inspection of the SPECMAGIC retrievals shows that the annual cycle of the four chosen bands are similar for both sites. In summer the UV and VIS bands have a higher share in the broadband irradiance than in wintertime. The opposite is apparent for the water vapour and NIR bands. However, a remarkable misfit between SPECMAGIC and ground measurements occurs at Loughborough for the UV band. Here, the ground measurements shows only a slight annual cycle, which seems to be unlikely. The more pronounced annual cycle given by SPECMAGIC seems to be more realistic. The good match of the annual cycle at Stuttgart supports this hypothesis. This is another hint for an outdated or misaligned calibration in the UV at Loughborough, which has been already indicated by the comparison of the annual averages (see Figure 7). The difference of the WV absorption band at Loughborough needs further investigation. The observations at Stuttgart show an overall good agreement with SPECMAGIC.