Image Collection Simulation Using High-Resolution Atmospheric Modeling

: A new method is described for simulating the passive remote sensing image collection of ground targets that includes e ﬀ ects from atmospheric physics and dynamics at ﬁne spatial and temporal scales. The innovation in this research is the process of combining a high-resolution weather model with image collection simulation to attempt to account for heterogeneous and high-resolution atmospheric e ﬀ ects on image products. The atmosphere was modeled on a 3D voxel grid by a Large-Eddy Simulation (LES) driven by forcing data constrained by local ground-based and air-based observations. The spatial scale of the atmospheric model (10–100 m) came closer than conventional weather forecast scales (10–100 km) to approaching the scale of typical commercial multispectral imagery (2 m). This approach was demonstrated through a ground truth experiment conducted at the Department of Energy Atmospheric Radiation Measurement Southern Great Plains site. In this experiment, calibrated targets (colored spectral tarps) were placed on the ground, and the scene was imaged with WorldView-3 multispectral imagery at a resolution enabling the tarps to be visible in at least 9–12 image pixels. The image collection was simulated with Digital Imaging and Remote Sensing Image Generation (DIRSIG) software, using the 3D atmosphere from the LES model to generate a high-resolution cloud mask. The high-resolution atmospheric model-predicted cloud coverage was usually within 23% of the measured cloud cover. The simulated image products were comparable to the WorldView-3 satellite imagery in terms of the variations of cloud distributions and spectral properties of the ground targets in clear-sky regions, suggesting the potential utility of the proposed modeling framework in improving simulation capabilities, as well as testing and improving the operation of image collection processes. and interactions of cloud hydrometeors in the S.E.; writing—review and M.M., A.C., S.Y., and visualization, S.E., and A.K.; M.M.; project M.M., A.K., S.E., and S.Y.; M.M., A.K., S.E., A.C.,


Introduction
The simulation of image collection for airborne and satellite sensors has many applications. Simulated video and image products make it possible to measure the value of potential imagery collections before they are carried out. Accurate simulations can help mission planners use collection platforms more efficiently, safely, and effectively, and predict the detection and visibility of targets and other scene features under anticipated collection conditions [1,2]. Simulation is useful for prototyping sensors [3] and for testing algorithms in the imaging chain [4]. Simulation can be used to predict, analyze, and reverse engineer artifacts and system issues that degrade image and video quality.
Simulations can generate large quantities of annotated training data for computer vision machine learning systems.
The research described in this article is concerned with passive remote sensing in visible and near-infrared spectral bands. Some researchers have generated synthetic scenes using emulations of physical effects modeled on input images [5]. However, the main goal of this project was to create end-to-end 3D simulations of image collection that included all the scene features, such as the target, atmosphere, and sensor characteristics. The simulations must also include time-stepping to account for target motion, changes in ambient illumination, sensor platform motion, and atmospheric dynamics.
Several image collection simulation tools have been developed for this purpose, such as the Digital Imaging and Remote Sensing Image Generation (DIRSIG) model [1][2][3][4]6] and Monte Carlo Scene (MCScene) [7][8][9][10]. DIRSIG and MCScene account for scene features and sensor specifications well in many cases but do not account for dynamical, heterogeneous atmospheric effects at high spatial resolution. Both DIRSIG and MCScene accept, as input, collection platform and trajectory parameters, atmospheric parameters, and target parameters and generate an output of simulated images by simulating photon scattering from the target and through the atmosphere. It is possible to specify the spectral properties of ground targets and target scenes, and to include sensor design features in the models, but the atmosphere is generally treated as an unchanging feature with no spatial or temporal variations. Furthermore, there is no model that ingests existing weather data in real time, conducts high-resolution processing, and simulates potential imagery resulting from impending weather conditions. These are significant shortcomings, because atmosphere and weather are often the primary and most unpredictable factors that contribute to image quality degradation.
Weather conditions applied to image processing, correction, or simulation are generally homogeneous parameters, because atmospheric data have not been available that have the spatial scale of most imagery. When ambient weather conditions are used as parameters for correcting or calibrating imagery, single parameters for temperature, etc. may be applied to an entire scene of tens of square kilometers. This spatial scale is much coarser than the resolution scale of image data, which can be 1-4 m for a standard commercial satellite multispectral image.
Advanced weather modeling and increasing computing resources have improved the resolution of weather simulation. Large-eddy simulation (LES) explicitly resolves clouds and energetic turbulent eddies at fine scale (10-100 m) by calculating atmospheric dynamics and physics. LES has been increasingly applied to simulations of more realistic atmospheric conditions by constraining model inputs (which include aerosol size distribution parameters, hygroscopicity parameters, large-scale forcing and surface forcing) with locally measured boundary conditions and global-or regional-scale, low-resolution weather data [11].
It is worth considering the feasibility of applying high-resolution weather information to image collection simulations. The possible advantages include a more accurate spectral characterization of ground targets, more accurate measurement of temperature in thermal data, and better prediction of atmospheric impediments to imaging, such as clouds, haze, hail, sleet, and aerosols, that might influence satellite or airborne tasking decisions.
The concept of operation for this idea is that the weather model is represented as a 3D voxel grid placed over the scene. At each time step, LES calculations produce a new voxel grid for each parameter of interest, such as the cloud water content, droplet radius, humidity, temperature, and pressure. The image collection simulation accounts for the high-resolution weather data by masking or otherwise demarcating cloudy regions in the simulated imagery output. This process makes it possible to account for the high-spatial-resolution variability of effects from clouds and other atmospheric phenomena across a scene.
Section 2 of this article explains the method of image collection simulation and describes a ground truth experiment in which real imagery was collected. Section 2 also describes the process of developing a computer model of the 3D scene and calibrating the computer model to the physical properties of the scene. Section 3 explains the high-resolution atmospheric simulation. Section 4 describes the results of the image collection simulation, and Section 5 provides conclusions and possible paths for future work.

Image Collection Simulation Using DIRSIG
The purpose of image and video collection simulation is to account for all possible factors in the end-to-end image chain that can degrade video and image quality, and to be able to reproduce and explain artifacts or target signatures. The main factors to simulate include platform characteristics such as the camera specs, wavelength bands, sensor response functions, spectral linewidth, timing, jitter, altitude, and elevation and azimuth angles. The target factors include the reflectance, bidirectional reflectance distribution function (BRDF), and material composition of the facets. The atmosphere and environmental factors include the sun direction, temperature, moisture, haze, clouds, precipitation, and downwelling and upwelling radiation. The product of the image collection simulation is an image expressed in units of radiance. Video can be simulated as a concatenation of image frames collected at typical video rates.
The software package used in this paper was DIRSIG, which is a radiation transfer simulation program designed by the Carlsen Lab in the Rochester Institute of Technology [1][2][3][4]. DIRSIG simulates image collection by ingesting a computer-aided design (CAD) scene for which all the objects and environmental features have been assigned spectral characteristics. DIRSIG can be set up to use the Air Force Research Lab program MODTRAN to calculate atmospheric radiative transfer to supplement the target photon scattering and sensor response functions simulated by DIRSIG [12]; MODTRAN accepts an input formatted description of atmospheric characteristics, and outputs line-of-sight spectral transmittances and radiances calculated by solving the radiative transfer equation to account for absorption, emission, and scattering. DIRSIG models sensor characteristics in great detail, such as the sensor response at specific wavelengths and camera specifications. DIRSIG also models platform motion and trajectory, and moving objects on the ground [1,2]. DIRSIG and MODTRAN have been used for a long time to measure radiance transfer through the atmosphere and simulate the effects on imagery. However, existing image collection simulations in the open scientific literature cannot accommodate atmospheric dynamics with high spatial or temporal resolution [13]. Therefore, we developed an image collection simulation that includes the dynamical effects of the atmosphere on radiation transfer.

Ground Truth Experiment Using WorldView-3 Satellite Imagery
To test the validity of the image collection simulation, WorldView-3 commercial satellite images of the Atmospheric Radiation Measurement Southern Great Plains (ARM/SGP) site in Lamont, OK, were obtained. The images studied in this experiment were collected on 21, 22, 26, 27, and 28 May ( Table 1). The satellite image data set includes visible spectral bands ranging from blue to red, infrared bands ranging from near infrared (NIR) to shortwave infrared (SWIR), and atmospheric sounding data from the Cloud, Aerosol, Water Vapor, Ice, Snow (CAVIS) instrument. Each image data set consisted of 15-17 files including multispectral, atmospheric, panchromatic, NIR, and SWIR bands. (See Supplementary Materials). The cloud cover during the period ranged from nearly clear to overcast conditions. In several cases, the images had slight opacity from haze or thin cloud layers, which was ideal for testing the image collection simulation with atmospheric dynamics.
Calibration objects consisting of colored tarps were placed on the ground (Figures 1 and 2). Two sites in the SGP central facility were used for the placing objects: a site at the north end of the compound and another to the south. The reflectance of the calibration tarps was later measured in a laboratory at the National Ground Intelligence Center (NGIC), as described in Appendix A. The calibration process helped to align the spectral characteristics of the DIRSIG scene model with the real-world imagery.

CAD Model
A CAD model of the ARM/SGP site ( Figure 3) was built with Blender [14], an open source CAD software package. Existing CAD building models developed for the ARM/SGP site in FBX, DWG, and SQLITE formats were imported as input into Blender, and other objects in the scene were manually generated. The scene was draped onto a Digital Elevation Model (DEM), which also had a layer corresponding to terrain type, e.g., soil, grass, rock, etc. Vegetation and other simple scene objects were manually generated in Blender and added to the CAD model. The output of Blender is a 3D CAD model in a proprietary format that can be read by DIRSIG. An annotated map of the vegetation and terrain types was derived from Google Maps imagery and a United States Department of Agriculture (USDA) model, CropScape, which provided ground truth values for the terrain classifications (soil, grass, etc.) [15]. A segmentation image product of the terrain classification was developed for the Google Maps imagery using a K-means unsupervised training model in the Harris Corporation ENVI software, so the image pixels in the Google Maps imagery could be assigned material values that could be mapped onto the CAD model. This process made it possible to combine the existing CAD model, which only included manmade buildings, equipment, and features, with an annotated background, thus providing material characteristics for the entire CAD model, which could be assigned spectra. The spectra of scene vegetation and other background objects were assigned from the Nonconventional Exploitation Factors Data System (NEFDS) database [16]. The spectra of target objects were derived from laboratory measurements of the BRDF. This terrain classification map of the scene was used by the DIRSIG model to simulate the reflectivity properties of the objects in the scene. The objects in the Blender model were exported into OBJ and GLIST formats; the latter is used by DIRSIG to assemble groups of objects into a scene.

WorldView-3 Simulation
The image collection was simulated in DIRSIG by modeling the trajectory of the WorldView-3 satellite. The following satellite parameters were extracted from the WorldView-3 XML metadata: the altitude, image dimensions, mean azimuth angle, and mean elevation angle. The ground sampling distance (GSD), or image resolution, was taken from the WorldView-3 metadata; the WorldView-3 GSD values ranged from 1.3 to 2.6 m, and the simulated GSD values were set to match the GSD values in the metadata. Other features of the camera model such as the pixel pitch, focal length, and aperture were unavailable but were able to be approximated from first-principle optics, based on knowledge of the image resolution and other parameters in the metadata.
The imaging bands used by WorldView-3, which ranged from coastal blue to SWIR, were used in the DIRSIG sensor model. The DIRSIG wavelengths used in the cloud fraction comparisons below were only the visible wavelengths, which were set as 450, 550, and 650 nm for blue, green, and red, respectively. This can be compared to the sensor bands WorldView-3 uses for red (630-690 nm), green (510-580 nm), and blue (450-510 nm). The laboratory spectra that were measured for the calibration tarps (Appendix A) were ingested into the DIRSIG model of the scene.

Atmospheric Model
The atmospheric model used in the image collection simulation is the Weather Research and Forecasting (WRF) model version 3.8.1 [17], which adopts a fully compressible equation system as a dynamical solver as well as advanced physics packages for subgrid-scale processes. For a flexible LES configuration, we utilized the version enhanced by implementing initialization, forcing, and statistics modules [11]. The LES model produces a time-stepping 3D distribution of atmospheric state variables, such as the pressure, temperature, water vapor, and cloud variables, as well as a variety of other statistical variables, which can be utilized as an input atmospheric model (i.e., 3D atmospheric data) for the image collection simulation.
The The simulation uses the following physics packages: Thompson cloud microphysics [18], Rapid Radiative Transfer Model for General Circulation Models (RRTMG) radiative transfer [19], and 1.5-order Turbulence Kinetic Energy (TKE) subgrid-scale turbulence [20]. The Thompson scheme is a bulk cloud microphysics scheme that calculates evolutions and interactions of cloud hydrometeors in the categories of cloud liquid, cloud ice, rain, snow, and graupel; the RRTMG scheme calculates radiative transfer and heating due to longwave and shortwave fluxes based on the correlated k-distribution method [21][22][23]; the 1.5-order TKE scheme represents diffusion due to subgrid-scale turbulence based on the turbulent kinetic energy predicted based on wind shear, buoyancy, or external forcing [24][25][26].
The simulations were performed for each of five days on which the WorldView-3 images were collected. The model was initialized with a sounding profile at 12 UTC (6 local time) and was driven for 15 h. The simulation applies large-scale and surface forcing from the ARM VARANAL product [27], which is based on a variational analysis using rawinsondes, ground measurements at ARM/SGP, and National Oceanic and Atmospheric Administration (NOAA) Rapid Refresh (RAP) analysis data as input. To prevent the drift of simulated fields from VARANAL fields, a relaxation of the thermodynamic profile is applied using a 12 h time scale.

Comparison of Cloud Measurements and LES Atmospheric Simulation
First, we tested the quality of the time-stepping atmospheric model data. The simulated cloud fields were compared to observed cloud occurrence from the Active Remote Sensing of Clouds (ARSCL) product [28], which is based on vertically pointing radar and lidar at the ARM/SGP site. Figure 4 shows cloud occurrences from the ARSCL measurements and LES atmospheric simulation for the period of 21, 22, and 26-28 May. The cloud occurrence was hourly averaged to reduce the scale mismatch between the 1D measurement and the 3D simulation. The simulations generally captured the observed peaks of cloud layers at the lower and upper levels. The observations and simulations generally show that the clouds with more coverage also generally occupied more height and had a longer duration during the course of the day. For specific times and heights, the simulated clouds could be overpredicted (e.g., upper-level clouds on 21 May) or underpredicted (e.g., upper-level clouds on 22 May). The larger difference at upper levels suggest that the quality of the simulation could be improved by better large-scale forcing and ice microphysics, which are responsible for the cloud generation at the upper levels.
We performed several tests to examine the simulation's sensitivities to the choice of microphysics scheme and relaxation time scale (not shown). By switching from Thompson microphysics to Morrison microphysics [29], upper-level cloud occurrence and duration increased in most cases.
Lower-level clouds were almost identical in the two simulations using different microphysics schemes. With a reduced relaxation time scale (i.e., stronger relaxation), the simulated clouds were more persistent at lower levels but were diminished at upper levels. The sensitivity tests produced mixed results depending on the different times, heights, and case days. Since parameter tuning for each case is not preferable for operational purposes and for prototyping the framework, we used simulations with a combination of Thompson microphysics and a 12 h relaxation time scale for the following analysis, and planned a separate study for more consistent improvements.

Comparison of Image Collection Simulation and Satellite Ground Truth
Next, the image collection simulation was performed with DIRSIG using the LES high-resolution atmosphere and modeled 3D scene at ground level, and it was evaluated with a ground truth experiment using WorldView-3 satellite images. This procedure was performed according to the following steps:

1.
Run the LES models using the forcing data taking into account the SGP site observations during the period of the ground truth experiment.

2.
For the five days and times when imagery was collected at the SGP (Tables 1 and 2), capture the 3D atmosphere, particularly the cloud water.

3.
Align the 3D atmosphere with the scene as projected onto a 2D ground plane.

4.
Integrate the cloud water along parallel lines of sight extending from each ground pixel to the satellite's position for each day.

5.
These integrated lines of sight form a cloud mask (Figure 5), and the cloud coverage percentage is the percent of pixels for which clouds obscure the line of sight.   Table 2 compares the cloud coverage from the image collection simulation to the cloud coverage reported in the metadata of the WorldView-3 images. Unlike the previously shown comparison with ARSCL (Figure 4), the cloud coverage here is derived from image collection simulations that consider the target-to-sensor angle and field of view of the WorldView-3 satellite measurements. The results for the simulated cloud coverage agree with the WorldView-3 cloud coverage within 23% for all days except 26 May. The WorldView-3 image on 26 May had 99% cloud cover, and the atmospheric model missed the clouds completely. Since the upper-level stratiform clouds are dependent on the input large-scale forcing rather than model dynamics, the discrepancy is likely due to the insufficient quality of the input forcing data. A possible mitigation for the forcing uncertainty would be to use regional-scale simulations that incorporate more observational data to constrain the simulations. Using multiple sets of LES forcing data and driving multiple LES simulations would also be useful for quantifying the potential uncertainty of the forcing data.
Examples of DIRSIG simulations that include the high-resolution atmospheric modeling are shown in Figure 5. The currently used modeling framework simulates small-scale clouds and turbulence structures for given large-scale meteorological conditions. Thus, the simulations are not designed to capture the physical location, size, or structure of individual clouds but to capture the cloud statistics such as the fractional coverage and the effects on visibility. The colors in the simulations represent classes of objects (grass, dirt, etc.). The simulated images on the left in these figures were not formed to match the red-green-blue colors as the WorldView-3 images on the right, but the spectral properties of the cloud-free areas in the DIRSIG products were calibrated to match the spectra of the WorldView-3 image products across as many bands as possible in the wavelength range from coastal blue to short-wave infrared. A description of this process is provided in Appendix A.
We generated spectral plots over the range of the WorldView-3 imagery wavelengths (420-830 nm). Figure 6 shows the DIRSIG simulation radiance values, which include the spectra of the NGIC lab measurements of tarp reflectance, compared to the radiance values measured on the tarp in the WorldView-3 image from 22 May. The radiance values from the WorldView-3 image represent average values from 3-4 samples obtained with the ENVI image viewer [13]. The agreement of measurements is good for green and red but worse for the blue band.
Though DIRSIG is unable to calculate cloud radiometry, it is possible to use the atmospheric simulation to calculate the lower bounds of visibility, by integrating the 3D cloud water. The extinction coefficient σ ext determines the range of visibility S m , which can be calculated, assuming a constant extinction coefficient across the visible light spectrum, as [30] S m = 3.91/σ ext. (1) The extinction coefficient can be calculated from the cloud liquid water content C w , density of water ρ water , and effective radius of water droplets a ef .
Therefore, the visibility can be calculated from Equations (1) and (2) as It is possible to calculate the liquid water content from the cloud water mixing ratio q c and the density of air ρ air as C w = ρ air · q c (4) Along a slant range line of sight h, C w would be calculated as an integral, All the quantities mentioned here can either be treated as constants or calculated by the LES simulation. The cloud masks in Figure 5 were set to the lowest possible threshold, so the masks indicated either the presence or absence of clouds along the line of sight.

Discussion
The high-resolution atmospheric model provided cloud predictions for which the predicted cloud coverage was within 23% of the measured cloud coverage, except for one case in which clouds were not successfully predicted, likely due to the uncertainty of the input large-scale forcing dataset. The simulated imagery of the ARM/SGP site compared favorably with the WorldView-3 satellite imagery in terms of the variations of cloud distributions and spectral properties of ground target in clear-sky regions, suggesting a potential capability of the proposed modeling framework in testing and improving the operation of image collection processes.
In this study, the cloudy regions were flagged by a cloud mask since DIRSIG is unable to model the radiometric scattering and absorption properties of the clouds. It would be possible, in future work, to modify DIRSIG or other image collection tools to include the effects of attenuation, opacity, reflection, shadows, and other cloud features. The MCScene image simulation code includes a capability for incorporating cloud distributions produced by an empirical model, which increases the resolution of regional-scale weather simulations [9]; high-resolution weather calculations based on real atmospheric dynamics and cloud physics, such as the ones in this study, could be applied and tested in such simulations.

Conclusions
This study tested a new process of combining image collection simulations with high-resolution atmospheric model simulations. The simulated results were evaluated against ground-based remote sensor and WorldView-3 satellite imagery. While the simulation accuracy for producing the right cloud type and the timing of the cloud formation still needs improvement, the simulated cloud fields have the detailed structure that makes them feasible for the image collection simulation process.
It is desirable to have good quality in the input large-scale forcing datasets to constrain the regional-scale conditions for the high-resolution atmospheric simulations. Future research could consider running the atmospheric model at a regional scale by incorporating available observations to generate the forcing or boundary conditions for the high-resolution LES model. It would also be valuable to include interactions with vegetation and terrain in future atmospheric simulations. The sensitivity of the atmospheric modeling to the various input measurements can be tested to see if a smaller suite of input measurements could be used. If this turns out to be the case, this image simulation technique may be able to be applied in other locations, using a smaller set of instruments that could make it practical. In some environments, unmanned aerial vehicles could serve as a source of local atmospheric parameters [31].

Conflicts of Interest:
This article was approved by the NGA for public release . However, the NGA and the authors' affiliated institutions had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Appendix A
The reflectance of the tarps for the wavelengths used by WorldView-3 was measured in a laboratory by the National Ground Intelligence Center (NGIC). The procedure was as follows.
Reflectance measurements at wavelengths from 350 to 2500 nm were collected with an Analytical Spectral Devices ASD FieldSpec3 spectroradiometer, and they were calibrated via the Spectralon polymer panel with illumination provided by a tungsten halogen lamp. Three spectra with twenty co-added spectral measurements each were averaged together to yield a single signature. All signatures underwent a Spectralon correction routine to remove the signature contribution of the Spectralon polymer, which is not 100% reflective across the 350 to 2500 nm region. The resulting corrected signatures are expected to be within 4% of the absolute reflectance.