A Methodology for Verifying Cloud Forecasts with VIIRS Imagery and Derived Cloud Products—A WRF Case Study

Abstract

A methodology is presented to evaluate the accuracy of cloud cover fraction (CCf) forecasts generated by numerical weather prediction (NWP) and climate models. It is demonstrated with a case study consisting of simulations from the Weather Research and Forecasting (WRF) model. In this study, since the WRF CCf forecasts were initialized with reanalysis fields from the North American Mesoscale (NAM) Forecast System, the characteristics of the NAM CCf products were also evaluated. The procedures relied extensively upon manually-generated, binary cloud masks created from VIIRS (Visible Infrared Imager Radiometry Suite) imagery, which were subsequently converted into CCf truth at the resolution of the NAM and WRF gridded data. The initial results from the case study revealed biases toward under-clouding in the NAM CCf analyses and biases toward over-clouding in the WRF CCf products. These biases were evident in images created from the gridded NWP products when compared to VIIRS imagery and CCf truth data. Thus, additional simulations were completed to help assess the internal procedures used in the WRF model to translate moisture forecast fields into layered CCf products. Two additional sets of WRF CCf 24 h forecasts were generated for the region of interest using WRF restart files. One restart file was updated with CCf truth data and another was not changed. Over-clouded areas in the updated WRF restart file that were reduced with an update of the CCf truth data became over-clouded again in the WRF 24 h forecast, and were nearly identical to those from the unchanged restart file. It was concluded that the conversion of WRF forecast fields into layers of CCf products deserves closer examination in a future study.

1. Introduction

The accuracies of cloud model predictions play critical roles in many real-time meteorological applications including air quality [1] and solar energy management [2] as well as a host of military and civilian aerodrome operations [3]. However, the verification of cloud model forecasts can be challenging [4], which has led the WMO (World Meteorological Organization) to recommend methods for evaluating clouds and related parameters [5].

In an earlier publication, new procedures that conform to these WMO recommendations were presented. They exploit remotely-sensed satellite imagery and derived cloud data products in the quantitative assessment of clouds in datasets commonly used in numerical weather prediction (NWP) and climate modeling [6]. The procedures were applied to quantitatively assess cloud model performance of lower-level water clouds. That study focused on these clouds because they play a critical role in cloud feedbacks, which have been identified as the leading source of spread in estimates of climate sensitivity [7,8]. The procedures relied extensively upon manually-generated cloud/no cloud (MGCNC) masks created from satellite imagery as the basis for establishing cloud cover fraction (CCf) truth or CCf_truth data. In the application that was demonstrated, CCf_truth was derived from imagery collected by the VIIRS (Visible Infrared Imager Radiometry Suite) sensor flying on the NASA/NOAA (National Aeronautics and Space Administration/National Oceanic and Atmospheric Administration) Suomi NPP (National Polar-orbiting Partnership) mission. The procedures were applied to reanalysis fields created from the North American Mesoscale (NAM) Forecast System and simulations generated with them using the Weather Research and Forecasting (WRF) model. In essence, VIIRS imagery and cloud data products derived from them were temporally and spatially collocated within NAM gridded fields to identify grid cells for comparison against the CCf_truth data. VIIRS cloud phase products, quality controlled using color-composites of VIIRS imagery, were used to ensure only lower-level water clouds were considered in the match-up data sets. Comparisons between NAM CCf or CCf_NAM and the CCf_truth data, created from the VIIRS imagery for the case study presented, revealed a bias toward under-clouding in the NAM re-analysis data [6]. An in-depth analysis of the WRF cloud forecast products was not completed at that time.

Therefore, the focus of this study is a more detailed analysis of the WRF cloud cover fraction (CCf_WRF) forecast characteristics of lower-level water clouds, for the same case study of 18–19 September 2014. The initial CCf_WRF results presented below, suggest a tendency for WRF to over-specify clouds compared to CCf_truth. Since WRF cloud forecasts are predicted at each vertical layer in the model and then composited into a total cloud cover for a given grid point while CCf_NAM is a single-valued parameter at each grid point in the dataset, additional tools were needed to investigate the internal WRF model logic that translates forecast moisture parameters in the layered CCf_WRF fields. To support this analysis, it also became necessary to evaluate the cloud spin-up characteristics of the WRF model to ensure the reliability of CCf_WRF forecasts, which is highlighted in Section 2 along with an overview of the study domain and data used in the WRF simulations. In Section 3, procedures to match-up VIIRS cloud products with CCf_NAM and CCf_WRF data are presented along with initial results from comparisons between CCf_truth, CCf_NAM, and CCf_WRF data. In Section 4, the techniques used in this analysis are presented to characterize and update CCf_WRF data products in a WRF restart (WRFrst) file. Comparisons between CCf_truth and CCf_WRF data created from simulations based upon the modified and unmodified WRFrst files are also shown. A summary is provided in Section 5 along with implications for future research.

2. Data Sources

Since characteristics of the datasets and procedures used in this research were reported earlier [6], they are only briefly highlighted here. However, methods of exploiting these datasets are covered in greater detail to describe new procedures developed for this study.

2.1. NAM Analysis Fields

A number of datasets have been developed by NCAR (the National Center for Atmospheric Research) for use as input in the WRF Preprocessing System (WPS) including the NAM datasets, which can be downloaded from the NCAR Research Data Archive (RDA) website: http://www2.mmm.ucar.edu/wrf/users/download/free_data.html. Different cycles of NAM (ds609.0) datasets were used to initialize WRF in this study as discussed in Section 2.3. NAM data archived at the NCAR RDA cover the continental United States at a 12 km spatial resolution every six hours, continuously from 0000 UTC on January 1, 2012. Variables contained in the ds609.0 datasets are shown at the NCAR RDA website: http://rda.ucar.edu/datasets/ds609.0/. These NAM data include a total cloud cover fraction (CCf_NAM) variable that can be compared to collocated CCf_truth data generated from VIIRS data, as discussed in Section 3.2. A graphic of the NAM region of interest is shown in Figure 1 along with the general area where VIIRS satellite data are temporally and spatially coincident with these data.

2.2. VIIRS Satellite Data

The VIIRS imagery and cloud data products used in this study were collected by the NASA NPP spacecraft that was launched on 28 October 2011. The satellite is in a sun-synchronous, near-circular polar orbit at an altitude of 824 km with an inclination of 98.74 deg, which creates a period of 101 min. The satellite ascends, heading north, across the equator at about 13:30. http://database.eohandbook.com/database/missionsummary.aspx?missionID=413. The VIIRS sensor collects data in 22 spectral bands between about 0.4–12.0 µm. At nadir, the cross-track scanning sensor captures imagery (375 m or high resolution) data and radiometric (750 m or moderate resolution) data. The resolution of both types of data increases by a factor of 2 as the VIIRS sensor scans from nadir to the edge of its 3000 km data swath. Comprehensive details about the VIIRS instrument design have been published [9]. An overview can be found at http://database.eohandbook.com/database/instrumentsummary.aspx?instrumentID=412.

VIIRS clouds products are created at the NASA NPP ground processing segment, including the VIIRS cloud mask (VCM) product that contains both cloud mask and cloud phase analyses [10,11], which were used extensively in this research. These VIIRS products are available via the NOAA (National Oceanic and Atmospheric Administration) CLASS (Comprehensive Large Array-data Storage System) server: https://www.bou.class.noaa.gov/saa/products/catSearch. The performance characteristics of the VCM algorithms, which create cloud confidence and cloud phase products at a horizontal cell size (HCS) of the VIIRS moderate resolution data [9] are well documented [12,13,14]. VIIRS images are used to create manually-generated cloud/no cloud (MGCNC) masks, which becomes the basis for the CCf_truth data as demonstrated in Section 3.2, using procedures that have been well documented [9,13,15].

Figure 1 shows the approximate location of the VIIRS granules used in this case study. Two sets of VIIRS imagery were used. The first consists of VIIRS data collected at approximately 1800 UTC on 18 Nov 2014 while the second set was collected approximately 24 h later. The VIIRS observation time interval for 18 Nov 2014 data ranged from 1813 to 1820 UTC over the WRF region of interest. These data were used to create the CCf_truth data needed to assess and update cloud fields in a WRFrst file valid at 1800 UTC on this date. The observation times for the second set of VIIRS granules, which were used to quantitatively assess the CCf_NAM data and to verify the WRF cloud terminus forecasts valid at 1800 UTC on 19 Nov 2014, ranged from 1757 to 1803 UTC over the region of interest. These data were also used to evaluate the cloud spin-up characteristics of the WRF model as discussed in the next section. Imagery from the two sets of VIIRS data are further discussed in Section 3.2.

2.3. WRF Simulations

Five relatively short-range (6–24 h) forecasts were generated using WRF ver. 3.7.1 from different cycles of the NAM data. The first three sets of simulations were used to assess the cloud spin-up characteristics of the WRF model while the remaining simulations were used first to evaluate clouds in the WRFrst file and then to assess the utility of assimilating VIIRS cloud products into the final WRF forecast, sometimes called the WRFout file. Key parameters used in these WRF simulations are summarized in

Table 1. Weather Research and Forecasting (WRF) ver. 3.7.1 simulation configuration for verification of cloud cover fraction.

Simulation Characteristics	Parameter Settings
Period of simulation	Variable (e.g., 6, 12, and 24 h case 1800 UTC 18–19 November 2014)
Meteorological data	NAM reanalysis (ds609.0)
Horizontal spatial resolution	12 km
Time step	60 s
Number of vertical levels	42
Top pressure in profiles	10 hPa
Shortwave radiation	Dudhia scheme
Longwave radiation	RRTM scheme
Surface boundary layer	Monin-Obukhov similarity
Land surface layer	USGS
Planetary boundary layer	YSU
Cloud microphysics	Milbrandt 2-mom
Cumulus physics	Grell-Devenyi
Demonstrated model spin-up	6 h

. No nesting was used in the simulations. The WRF model was configured to create restart files every six hours. The WRFrst file contains all the information needed to restart a simulation while the final WRFout file does not. Thus, WRFrst was considered as an intermediate forecast while the WRFout files were considered for the terminus forecast files.

In the WRF system, QCLOUD (i.e., liquid water mixing ratio (LWMR)) is converted into cloud fractional coverage at each WRF output (eta or pressure) level, as described by Equation (4) in Xu and Randall [16]. Total cloud fractional coverage can then be derived from the cloud fractional cover layers using different methods as described by the Unified Post Processing (UPP) User’s Guide, [17,18,19,20]. Since this study focuses on single-layered water clouds in the truth data, ideally the values for layered cloud cover fraction and total cloud cover fraction in WRF should be the same. However, that was not always found to be the case.

In the first phase of this study, three cycles of NAM data were used to generate WRF forecasts of variable length with each terminating at 1800 UTC on 19 Nov 2014. These simulations were used to examine the cloud spin-up characteristics of the WRF model. The WRF forecast length and NAM datasets used for these simulations were: a 24 h forecast from NAM data at 1800 UTC on 18 Nov 2014, a 12 h forecast from NAM data at 0600 UTC on 19 Nov 2014, and a 6 h forecast from NAM data at 1200 UTC on 19 Nov 2014. Comparisons between CCf_WRF and CCf_truth data for each simulation provide insight into the cloud spin-up forecast characteristics of the WRF model. (Note that the VIIRS data are within ± 3 min of the WRF forecast valid time, i.e., 1800 UTC on 19 Nov 2018.) Comparisons between these WRF forecasts and the manually-generated cloud truth from VIIRS imagery, revealed (results not shown) that all key cloud features present in the 24 h forecast were captured in the 6 h forecast, including the lower-level water clouds. Therefore, it was determined sufficient that the second set of WRF forecasts, valid at 1800 UTC on 19 Nov 2014, be generated from NAM data at 1200 UTC on the same date in order to ensure cloud and moisture levels were not deficient in the WRFrst file.

Thus, for the second phase of the study, a WRFrst file valid at 1800 UTC on 18 Nov 2014 was created from the 1200 UTC NAM data. This WRFrst file was analyzed extensively, as noted in Section 4.3, using results from the CCf_truth data on 18 Nov 2014. A copy of this WRFrst file was then modified using procedures outlined in Section 4.2. Subsequently, two additional WRF simulations, of 24 h forecast length, were generated based upon the modified and unmodified WRFrst files using the settings listed in Table 1. The results from these simulations were then compared to the corresponding set of VIIRS CCf_truth data, valid between 1757–1803 UTC on 19 Nov 2014, to further evaluate the cloud products generated by the WRF model.

3. Methods and Procedures

3.1. Generating VIIRS and NAM(WRF) Match-Up Data

VIIRS moderate resolution data were temporally and spatially collocated with the NAM (WRF) gridded data to identify regions that contain either cloud-free or single-layered water clouds. These procedures were applied first with the NAM profile datasets to characterize CCf_NAM and again with the CCf_WRF forecasts generated by WRF from the NAM data to characterize model cloud forecast performance. First, an extended bow-tie trim was applied to the VIIRS data to reduce any overlapping pixels that occur toward the edge of the VIIRS scan. The NAM (WRF) data were mapped to a Lambert Conformal projection where the data were essentially equally spaced. An N × M grid was established with the NAM (WRF) data at the center of each grid. VIIRS data were matched to the NAM (WRF) grid by mapping the VIIRS data into the same projection and calculating the NAM (WRF) grid index into which it falls. VIIRS data outside the bounds of the NAM (WRF) grid and NAM (WRF) grid cells not containing VIIRS data were eliminated for further analyses. Next, temporal and spatial restrictions were applied to the VIIRS data. In general, scan times of the VIIRS data were checked to include only those that fall within ± 30 min of NAM analysis or WRF forecast times. However, as mentioned earlier, the temporal differences between match-up datasets used in this study were much smaller. Distances from the grid center were calculated using the Vincenty formula to identify every VIIRS cloud product within 6.5 km of each 12 km NAM (WRF) grid center. The VIIRS cloud products were then used again to identify occurrences where only cloud-free and/or single-layered water clouds exist in the match-up data, since this phase of the research was focused on these types of clouds as previously noted. Cloudy and cloud-free pixels were identified in a binary, manually-generated cloud/no cloud (MGCNC) mask product [14,15], which is derived from the moderate resolution VIIRS imagery. The process of generating this mask is discussed in the next section. The cloud phase product, created by the VCM algorithm and quality controlled with VIIRS imagery, was used to establish the cloud top phase of the pixels in the cloud truth product. NAM (WRF) grid locations found to contain any ice clouds or mixed phase clouds were excluded from further analysis with only pixels containing clear pixels, low-level water clouds or their combination remaining. Finally, a 12 km resolution CCf_truth was determined by calculating the mean of the MGCNC data matched to each cell. This CCf_truth was used to quantify CCf_NAM and CCf_WRF performance in the match-up data.

3.2. Generating VIIRS Mean Cloud Cover Fraction Truth Data

VIIRS mean cloud cover fraction was derived from the MGCNC data product. This (offline) process of generating the MGCNC data product is demonstrated with the VIIRS imagery used to construct the VIIRS-WRF match-up data at 1800 UTC on 19 Nov 2014. First, color composites of the VIIRS data for the set of granules collected between 1757–1803 UTC on 19 Nov 2014 are shown in Figure 2a. The corners of this set of images have the following latitude/longitude pairs starting with the upper-left corner and proceeding clockwise: 35° 36′ N, 91° 18′ W; 40° 48′ N, 56° 43′ W; 16° 06′ N, 53° 48′ W; 11° 54′ N, 81° 48′ W. Spaces are inserted between the granules to emphasize granule boundaries. The color composites are created by placing the 0.65 µm VIIRS (M5) channel in the red gun of the display, the 1.6 µm (M10) channel in the green gun and the 1.38 µm (M9) band in the blue gun [21]. In this display, water clouds appear yellow, thin ice clouds blue and thick ice clouds purple. See Chapters 6 and 7 in Hutchison and Cracknell for more examples of cloud and background signatures in false color composites [9]. Land appears green, snow (if present) is red, and the ocean black. The south-eastern coast of the USA is clearly visible in the color composite images as are the islands of Cuba, Haiti, the Dominican Republic and Puerto Rico.

The manually-generated cloud/no cloud (MGCNC) masks are based upon binary analyses of the moderate resolution VIIRS granules [9,15]. Figure 2b contains the binary MGCNC for the 19 Nov 2014 imagery shown in Figure 2a, where white is cloud and black is cloud-free. A similar truth analysis for one of the VIIRS granules collected on 18 Nov 2014 is shown in Figure 2 of Hutchison et al. [6]. This MGCNC mask was then converted into a mean cloud cover fraction truth (CCf_truth) dataset by aggregating it into the 12 km NAM (WRF) grid projected as shown in Figure 2c. (Note that in this image, only the intersection of the NAM and VIIRS region is shown.) The cloud-filled grids are white while the cloud-free grids are black. Grids with CCf values in the 10–90% range are assigned the middle shade of gray and can be seen to contain water clouds through comparisons with the clouds found in the false color image shown in Figure 2a. In this study, a cloud phase restriction was enforced and the MGCCf truth data became the CCf_truth for low-level water clouds after the MG-CCf_truth data were further merged with the VCM cloud phase analysis, similar to the one shown in Figure 2d. This particular cloud phase analysis revealed that the scene contains relatively few water clouds (green) compared to ice clouds (red, brown, and orange) and mixed phase clouds (yellow). Partly cloudy (light blue) pixels indicate sub-pixel water clouds, since this condition is determined from a spatial test of VIIRS 0.65 µm imagery resolution (375 m) data that is applied only over oceans backgrounds [10]. This same test does not detect sub-pixel (optically thin) ice clouds, which are (1) highly transparent at this wavelength and (2) readily detected and identified using the M9 band.

Quality control of the MGCNC data occurs at several stages in the process. First, color composites are essential to differentiate between cloudy and cloud-free surfaces and to understand the phase of cloud features in the scene. For the case shown in Figure 2a, scene interpretation, i.e., understanding which pixels are cloud-contaminated and which are not, is easy with VIIRS data under most global, daytime conditions. However, the task can become daunting under some conditions, e.g., during polar night-time conditions when thermal contrasts between clouds and backgrounds can be very small and no reflectance data are available [22]. Thus, a thorough understanding of the scene content is the starting point for making the MGCNC analysis. Secondly, several color composites are useful to discriminate between water-clouds and those containing ice particles. In Figure 2a, lower-level water clouds appear yellow since they are highly reflective in the M5 band (red gun) and M10 band (green gun) but reflect much lower in the M9 band (blue gun). Similarly, thin ice (cirrus) clouds are blue since they reflect highest in the M9 (blue gun) compared to the other channels in the composite, while thicker ice clouds appear purple since they can be highly reflective in the M9 band (blue gun) and M5 band (red gun) but have a low reflectance in the M10 (green gun). Middle level water clouds in this display may appear a shade of gray because they are reflective in all three of the display bands. Replacing the M9 band in Figure 2a with the VIIRS M15 band (12.0 µm), inverted to show colder objects as lighter than warmer objects, provides information on cloud top temperatures, which assists in separating lower-level water clouds from higher-level water clouds. In the color combination using the M15 band (not shown) water clouds with colder temperatures have an orange hue compared to the warmer low-level water clouds, which would appear yellow to gray. If additional insight is needed, a gray-scale image of the VIIRS M9 band is useful to accentuate thin cirrus clouds along with higher level water clouds since the 15 nm wide M9 band (see Table 4.4; Hutchison and Cracknell [9]) is centered on a strong water vapor absorption line (see Figure 4.9; Hutchison and Cracknell [9]) that suppresses the signatures of lower-level clouds under most global atmospheric moisture conditions [15,23,24].

3.3. Clouds in the NAM Dataset

Figure 3 shows qualitative comparisons between the CCf_NAM values for all clouds, i.e., all cloud phases contained in the ds609.0 datasets on 18 and 19 Nov 2014 at 1800 UTC and the CCf_truth data created from the MGCNC images of the collocated VIIRS data collected on the same dates. These data were projected into the WGS 84 (World Geodetic System 1984) reference frame and show only those regions of the NAM/VIIRS grid intersections. The 18 Nov 2014 NAM data were used to generate the initial WRF simulations, discussed in the next section, using settings listed in Table 1. In all images shown in Figure 3, coastlines are black followed by mostly cloud-free conditions (<10%) which are darker gray (as shown in most regions between areas A and B), partly cloudy conditions (10–90%) which are lighter gray (as best seen in the lower-right corner of Figure 3a) with cloudy grids (>90%) in white, as most evident along the prominent frontal boundary extending from the upper right toward the lower-left in each figure. As noted previously, partly cloudy pixels, in general, may be considered lower-level water clouds, which can be confirmed by comparing them with the water cloud pixels found in the color image shown in Figure 2a. Comparisons, between Figure 3a (based upon the MGCNC cloud mask mapped to the NAM grid) and Figure 3b (CCf_NAM for all clouds in the intersection of the NAM and VIIRS dataset) showed NAM under-clouds the lower-level water clouds (i.e., middle gray shade) in areas A, B, and especially in area C. Similar results are found in Figure 3c,d for the NAM and CCf_truth images, respectively, on 19 Nov 2014 at 1800 UTC.

Table 2. Comparisons between cloud cover fraction in NAM data (CCf_NAM) and truth (CCf_truth) for 19 Nov 2014 at 1800 UTC from manual masks of VIIRS imagery between 1757–1803 UTC on 19 Nov 2014.

Bin Number	CCftruth Interval (%)	Performance Metric	CCfNAM	CCftruth
1	0 ≤ CCftruth < 10	count	12,689
		mean (%)	8.0	1.6
		standard deviation (%)	24.6	2.7
2	10 ≤ CCftruth < 20	count	2365
		mean (%)	8.9	14.5
		standard deviation (%)	25.8	2.8
3	20 ≤ CCftruth < 30	count	1342
		mean (%)	9.8	24.6
		standard deviation (%)	26.5	2.8
4	30 ≤ CCftruth < 40	count	731
		mean (%)	10.1	34.5
		standard deviation (%)	25.6	2.9
5	40 ≤ CCftruth < 50	count	388
		mean (%)	16.7	44.4
		standard deviation. (%)	33.2	2.8
6	50 ≤ CCftruth < 60	count	238
		mean (%)	18.9	54.6
		standard deviation (%)	34.4	2.8
7	60 ≤ CCftruth < 70	count	165
		mean (%)	26.8	64.5
		standard deviation (%).	39.7	2.9
8	70 ≤ CCftruth < 80	count	90
		mean (%)	28.4	74.0
		standard deviation (%)	38.6	2.7
9	80 ≤ CCftruth < 90	count	63
		mean (%)	30.5	84.9
		standard deviation (%)	42.0	3.0
10	90 ≤ CCftruth < 100	count	71
		mean (%)	38.0	95.5
		standard deviation (%)	46.7	2.8
11	CCftruth = 100	count	71
		mean (%)	63.1	100.0
		standard deviation (%)	43.5	0.0

shows quantitative results for matchups of NAM grids at 1800 UTC on 19 Nov 2014 that contain only lower-level water clouds in the VIIRS data collected between 1757–1803 UTC. These data are presented in a format similar to those shown earlier for the 18 Nov 2014 timeframe [6], i.e., they are binned in 10% mean CCf_truth intervals (column 2). With an additional bin for 100% CCf_truth, there are 11 CCf_truth bins (column 1). The performance metrics, shown in column 3, include the total number of VIIRS-NAM matchups for each bin (counts) contained in the CCf_truth data, along with the mean and standard deviation. Column 4 shows the performance statistics for the CCf_NAM data while column 5 shows similar results for the CCf_truth data. The results revealed similar biases toward under-clouding in the NAM data for single-layered, lower-level water clouds, as was reported in the earlier study [6]. For the most cloud-free bin (0 ≤ CCf_truth < 10), the mean CCf_NAM is within the 0 to 10% CCf_truth bin range at 8.0% but higher than the mean CCf_truth, which is 1.6%. For the remaining CCf_truth bins, the CCf_NAM mean values appear to be about 60% less than the CCf_truth mean values in all the bins except for bins 2 and 11 where mean CCf_NAM values are about 40% of the mean CCf_truth values. The standard deviations for CCf_NAM results are also substantial, which suggests the NAM data are composed largely of completely cloudy and cloud-free CCf values, i.e., the data tend not to capture the smaller-scale clouds that are present in the VIIRS imagery. When the CCf_truth equals 100%, the CCf_NAM is 63.1% and the standard deviation is 43.5%.

For the two datasets used in this case study, NAM data appear to lack skill in analyzing lower-level water clouds across the full range of cloud cover fraction. Thus, it is reasonable to assume that WRF would under-predict these clouds in simulations based upon NAM these datasets. However, just the opposite was found to occur and this is discussed in the next section.

3.4. Cloud Cover Fraction in the Initial WRF Simulations

A baseline 24 h WRF simulation was generated from the NAM dataset valid at 1800 UTC on 18 Nov 2014; thus, making the WRF forecast terminus 1800 UTC on 19 Nov 2014. The results of that simulation are shown qualitatively in Figure 4, which contains the mean CCf_truth data in Figure 4a,b shows the CLDFRAmax (i.e., WRF maximum cloud cover fraction at any single eta level in the WRF profile for a given grid point). The WRF CLDFRAmax represents a minimum of the total CCf_WRF value at each WRF grid point since stacking of multiple cloud layers is not taken into consideration. In this case, the mean CCf_truth data was created by mapping the MGCNC cloud mask from VIIRS imagery, as shown in Figure 2b, into the WRF grid. The VIIRS imagery has an observation time between 1757–1803 UTC on 19 Nov 2014. Figure descriptions are identical to those used in Figure 3. Again, all partly cloudy pixels are lower-level water clouds as confirmed by examining the yellow pixels in the color image shown in Figure 2a. The qualitative results in these data clearly show that WRF generates substantially more fully cloudy grids but fewer partly cloudy grids than were found in the CCf_truth data. In addition, WRF generates extensive areas of cloudy conditions in Areas A, B, and C in grids that are less cloudy in the CCf_truth data. The reason for this over-prediction of cloudy grids by the WRF model in this case study, in light of the fact that NAM data was found to underspecify clouds in the image of the CCf_NAM data, warrants additional analyses; this is highlighted in the next section.

4. Evaluating the WRF Moisture to Cloud Conversion Process

4.1. Background

A simple experiment was undertaken to input lower-level water clouds from the CCf_truth data into a WRFrst file, in order to help better evaluate the moisture to cloud conversion process used in the WRF model. It was postulated that improved WRF CLDFRAmax data fields might be achieved in the terminus forecast by initializing the WRF model with more accurate cloud cover fraction data from the truth dataset. The approach follows operational cloud forecasting techniques employed for decades at the United States Air Force (USAF) Weather Central and involves a concept known as “bogusing”. The bogusing concept relies upon the use of satellite imagery to quality control the clouds at some intermediate time, by adding or removing clouds from the forecast fields based upon those observed in the satellite imagery before restarting the forecast model. This bogus process has been described in the literature [25,26].

The bogusing experiment requires a thorough examination of the atmospheric profiles in the WRF eta level coordinate system, i.e., to avoid interpolations of moisture from eta into pressure coordinates or vice versa, and to ensure clouds are bogused at the correct eta level. It also requires some knowledge of cloud spin-up time in the WRF model to ensure any changes to a WRFrst file are allocated time to reach equilibrium in the next forecast cycle, since CCf_truth was used to update a WRF simulation based upon an earlier NAM data set. Thus, in the bogusing approach used herein, a WRFrst file was created from the WRF simulation that was initialized from a NAM data set at 1200 UTC on 18 Nov 2014. The simulation generated two WRFrst files valid at 1800 UTC on the same date, which were evaluated using the CCf_truth data set. Cloud fields in one WRFrst file were then bogused while those in the other WRFrst file were unchanged. Results based upon the unchanged WRFrst file were used to establish the baseline. The WRF simulations were restarted to reach a terminus point at 1800 UTC on 19 Nov 2014 where another CCf_truth data set was used to quantify the forecast performance of both WRF simulations, i.e., using the baseline (unmodified) and bogused WRFrst files.

4.2. WRF Forecast Parameters to Cloud Conversion Procedures

As noted above, in the WRF system, QCLOUD (i.e., LWMR) is converted into cloud fractional coverage at each WRF eta level using Equation (4) of Xu and Randall [16]. Therefore, the procedures developed to bogus VIIRS clouds into the WRFrst file employ an inversion of the Xu and Randall equation using the WRF relative humidity data along with CCf_truth as the parameter to bogus the QCLOUD variable in the WRFrst file. Thus, in this bogus process, the Xu and Randall relationship is inverted to solve for QCLOUD given CCf_truth and relative humidity from the WRF profiles, as shown in Equation (1):

QCLOUD = (b/α₀) ln [1/(1 − CCf_truth/rh^k)]

(1)

where rh (relative humidity) < 1.0 and b, α₀, and k are empirically determined constants. The inversion uses the following procedures:

• First, the eta-level of the maximum cloud cover fraction (CLDFRAmax) in the WRF restart file cloud (CCf_WRFrst) is located for a given WRF grid. Then a decision is made to bogus the QCLOUD field for each grid that contains only a single-layered water cloud. (Note: all analyses are conducted on the WRFrst file eta-levels to avoid interpolations of moisture fields onto standard pressure levels.)
• Next, the temperature, pressure and relative humidity at the eta-level of the clouds are retrieved from the WRF data.
• Finally, CCf_WRFrst, calculated from the Xu and Randall Equation (4), is replaced by CCf_truth and the updated QCLOUD (QCLOUD_new) is calculated for the eta-level using Equation (1). The WRFrst file is updated for that grid by replacing CCf_WRFrst with CCf_truth while QCLOUD is replaced with QCLOUD_new at the eta-level.

4.3. Identifying Grids for Bogusing in the WRF Restart File

Figure 5 shows comparisons between CCf_truth and CCf_WRFrst (i.e., CLDFRAmax) results with cloud legends identical to those in Figure 3. Figure 5a contains the mean CCf_truth for all clouds in the VIIRS data on 18 Nov 2014 after mapping the MGCNC image to the WRF grid. Figure 5b displays the CLDFRAmax value in all eta levels for each grid location in the baseline WRFrst file which is valid for 1800 UTC 18 Nov 2014 from a WRF simulation initialized with a NAM dataset at 1200 UTC 18 Nov 2014.

The images in Figure 5a,b show two key deficiencies in the WRF simulations. First, the CLDFRAmax values tend to be binary, as expected based upon the results shown in Section 3.4. Far fewer WRFrst grids contain clouds in the lighter gray shade in Figure 5b, i.e., within the 10–90% CCf range, than are found in the CCf_truth image (Figure 5a).

Table 3. Frequency of cloud cover fraction (CCf) for all clouds contained in the mean CCf_truth data and the maximum value found at all eta levels for each collocate grid in the WRFrst file.

CCf Interval (%)	CCfWRF Counts	CCftruth Counts
0 ≤ CCf ≤ 10	9496	11,011
10 < CCf ≤ 20	67	1646
20 < CCf ≤ 30	23	1059
30 < CCf ≤ 40	16	781
40 < CCf ≤ 50	20	709
50 < CCf ≤ 60	6	655
60 < CCf ≤ 70	12	649
70 < CCf ≤ 80	11	774
80 < CCf ≤ 90	4	961
90 < CCf ≤ 100	21,672	13,082
Total	31,327	31,327

quantifies the difference between the CCf_truth values and those in the CLDFRAmax image created from the WRFrst file. Binary cloud fields account for 99.5% of the forecasts in the CCf_WRFrst file while they account for only 76.9% of the grids in the CCf_truth image. Secondly, the WRFrst file contains too many clouds, i.e., the model over-clouds the low as well as high level clouds in most regions of the scene. For example, over-clouding in the WRFrst file is seen in the stratocumulus fields (Area A) over the Gulf of Mexico, in most areas over the southeastern-eastern US landmass (Area B), and over the open Atlantic Ocean area northeast of Cuba (Area C). Thus, matchups that contain lower-level water clouds, which appear as the middle gray shade in Figure 3a and Figure 5a, become key candidates for update (bogusing).

4.4. WRF Simulation Results with Baseline and Updated WRF Restart Files

Therefore, an update was made to the WRFrst file using the procedures described above. After this update, another image was generated of all cloud fields in the WRFrst file using the CLDFRAmax from each WRF profile. That image is shown in Figure 5c. It is evident that clouds in the updated WRFrst file are in better agreement with those in Figure 5a than was found in the unmodified WRFrst file in Figure 5b, although Figure 5c still contains far too many clouds compared to the truth image.

Next, an additional WRF simulation was generated to confirm whether or not the improved cloud fields bogused into the WRFrst file would persist through a 24 h forecast cycle. These WRF simulations were generated from the modified and unmodified WRFrst files valid at 1800 UTC on 18 Nov 2014. Both the baseline WRF run, shown in Figure 4b, and the updated forecasts from the bogused WRFrst file, shown in Figure 5c, used the original simulation settings listed in Table 2. Each has a forecast length of 24 h, and both are valid at 1800 UTC on 19 Nov 2014. Images of these results are shown in Figure 6b,c, respectively along with the mean CCf_truth image in Figure 6a, which was created from VIIRS imagery collected within minutes of the forecast termination time. Images in Figure 6 are displayed in the same manner described in Figure 3, Figure 4 and Figure 5.

The results in Figure 6 revealed that WRF simulations based upon the modified WRFrst file show few differences from simulations based upon the unmodified WRFrst file. Additionally, the reduction in over-clouding obtained through the bogusing of the WRFrst file does not persist across a 24 h forecast cycle. CCf_WRF forecasts continue to be highly binary with little to no cloud fields found in the 10–90% cloud cover fraction range. This tendency to create binary forecasts is most evident in area C, but is also obvious in area A and B. The over-clouding is qualitatively evident in all areas, where large regions of clouds are seen in the WRF results that are not contained in the CCf_truth imagery. Thus, it is concluded that the over-clouding of WRF forecasts coupled with the binary nature of them suggests a lack of skill in the internal procedures used to translate WRF forecast variables into cloud cover fractional coverage within the WRF model.

5. Conclusions

In this case study, techniques originally developed to quantitatively assess the accuracy of cloud cover fraction (CCf) in the 12 km NAM ds609.0 databases were extended to analyze CCf forecasts from the WRF model. While the procedures focused on quantitatively evaluating single-layered water clouds are considered responsible for most of the variations in the simulation sensitivities found in global climate models, they also support qualitatively assessing all clouds created in the WRF simulations.

The procedures require the registration of VIIRS imagery and cloud products, including manually-generated cloud analyses generated from VIIRS imagery, with NAM initialization and WRF forecast datasets. The manually-generated cloud masks created from VIIRS imagery serve as truth measurements of mean cloud cover fraction (CCf_truth) once mapped to the WRF and NAM grids along with the cloud phase analysis from the VIIRS cloud mask.

The results from this case study suggest that NAM data are biased toward under-clouding of lower-level water clouds as reported previously [6]. However, the bias shifts strongly toward over-clouding in the WRF forecasts based upon these NAM data. In addition, the WRF cloud forecast fields are highly binary, i.e., predictions are mostly all clear or completely cloudy, even across large geographic areas shown to contain large areas of stratocumulus as evident in temporally and spatially coincident satellite imagery. Furthermore, WRF was found in this case study to over-predict cloud cover fraction for lower-level water as well as higher-level clouds and frequently forecasts multiple cloud layers (high and low) when only a single layer is present. While it is possible to assimilate, through the bogus process, more accurate cloud analyses into the WRF simulations, e.g., using VIIRS imagery and cloud products to update a WRFrst file, it was found that corrections to improve these cloud fields do not persist through a 24 h forecast cycle. Based upon the results of this case study, it is postulated that the current methodology to translate the WRF forecast parameters into cloud cover fraction at each of the WRF eta levels is shown to be inadequate in the current WRF implementation. It is concluded that additional studies with larger datasets are needed to more fully evaluate this conversion process and possibly improve the parameterization of the WRF cloud cover fraction forecast methodology.