2.1. Remote Sensing Imaging Systems under Low-Light Conditions
On dark nights, the sky is still full of light, called “nocturnal radiation”. Nocturnal radiation comes from natural sources of radiation, such as the Sun, the Earth, the Moon, other planets, clouds, and the atmosphere. However, the luminosity is too weak for visual perception by human eyes (below the eye’s visual threshold). Enhancing the weak radiation to the level required for normal vision is a key task of low-light remote sensing technology.
Low-light remote sensing technology aims to acquire, convert, enhance, record, and display target image information at night and under other low-light conditions. Its achievements are concentrated on the effective expansion of remote sensing vision in the time, space, and frequency domains.
In terms of the time domain, low-light remote sensing technology overcomes the “night blindness” barrier, doubling the range of qualifying periods. In terms of the space domain, it enables remote sensing vision to achieve normal observations in low-light spaces. In terms of the frequency domain, it extends the visual frequency band to the shortwave region.
In the military field, low-light remote sensing technology has been used in night reconnaissance and other battlefield operations, and it can be combined with infrared, laser, radar, and other technologies to form a complete photoelectric reconnaissance, measurement, and alarm system. Low-light remote sensing imaging has become essential in the weapons and equipment available to troops. Furthermore, it has been applied in aerospace reconnaissance [
2].
With the continuous development of low-light remote sensing technology, the design and implementation of low-light imaging systems face many challenges. First, the system design is difficult to optimize, and in the case of high-performance requirements, the analysis and matching design of the system opto-electro-mechanical indicators are challenging. Additionally, the information acquisition and processing capacity of the low-light imaging system is expected to become a technical bottleneck.
Assuming that the satellite orbital height is
H, the surface pixel spatial resolution is
Dg, the camera focal length is
f, and the detector cell size is
d, then
.
Figure 1 shows the imaging principle of the spaceborne remote sensing camera.
According to the above equation, when given the system orbital height
H, if we want to increase the system spatial resolution
Dg (that is, make
Dg smaller), we must increase the focal length
f of the camera or use a detector with a smaller cell size
d. In a spaceborne ground-to-ground push-broom imaging system, the speed at which a satellite platform moves relative to the ground is
where
μ is the Earth’s gravitational constant,
R is the Earth’s radius, and
H is the satellite’s orbital altitude relative to the surface. Thus, the operating frequency of the probe
ffrm and the working period
Tfrm are
Therefore, in an imaging system with a given orbital height, the higher the spatial resolution, the higher the system operating frame rate (line frequency), the shorter the system operating cycle, and consequently, the shorter the cell integration time.
Assuming that the target characteristics satisfy Lambert’s law, the low-light imaging system receives a faint light signal reflected by the target, and the number of photogenerated electrons generated on the cell is
Ns:
where
Tint is the integration time of the imaging system;
L(λ) is the brightness of the spectrum on the target by night sky or sunlight;
F is the F-number of the optical system;
Apix is the single-cell area of the detector;
τo(λ) is the spectral transmittance of the optical system;
τa(λ) is the atmospheric spectral transmittance,
η(λ) is the quantum efficiency of the detector;
ρ(λ) is the target reflectivity;
h and
c are Planck’s constant and vacuum speed of light, respectively; and
λ1 and
λ2 are the cutoff wavelengths of the system response. The equivalent aperture of the optical system is
D. The F-number of the optical system satisfies the equation
. The three-dimensional instantaneous field of view of the imaging system,
Ω, is related to the cell size, system focal length, ground resolution, and orbital height:
In Equation (4), Dg is the resolution size of a target on the ground, H is the orbital height of the camera, d is the cell size of the imaging system, and f is the system focal length.
Assuming that the total number of electrons in the system noise is ntotal, the system signal-to-noise ratio RSN can be obtained as .
In low-light imaging, when the imaging system works in a low-illumination region, the target receives a spectral radiation brightness L(λ), which is 108–109 orders lower than the visible light panchromatic remote sensing imaging system working in the insolation region. To obtain a sufficient system signal-to-noise ratio, a large cell size detector is needed to improve the detector integration time and reduce the optical system’s F-number. However, to maintain the spatial resolution of the imaging system, increasing the cell size increases the focal length of the optical system. To improve the system signal-to-noise ratio, the F-number in the optical system needs to be maintained or reduced, and the optical system aperture increases.
Therefore, solving the contradiction between a high resolution and system signal-to-noise ratio by only adjusting the detector cell size and the F-number of the optical system leads to a very large scale of the imaging optical system. This brings challenges to optical design, processing, and assembly, and the excessive volume and weight of the optical system cannot adapt to the spaceborne platform.
Therefore, a more effective method is needed to improve the system performance through the information processing system, and super-resolution reconstruction can be a good choice [
3,
4].
2.2. Super-Resolution in Remote Sensing Images
In remote sensing image processing, single- and multi-image super-resolution methods have been proposed in the past few decades [
5]. A brief overview of different methods is shown in
Figure 2.
In 1955, Toraldo di Francia defined the concept of super-resolution in the field of optical imaging, using optical knowledge to recover data information beyond the diffraction limit. Around 1964, Harris and Goodman applied image super-resolution to the process of synthesizing a single-frame image with richer detail information by extrapolating the spectrum. In 1984, based on predecessors, Tsai, Huang, and others designed a method of reconstructing high-resolution images using multi-frame low-resolution images, and the super-resolution reconstruction technology began to receive widespread attention and research interest from academia and industry [
6,
7].
Figure 2.
An overview of the super-resolution on remote sensing images [
8,
9].
Figure 2.
An overview of the super-resolution on remote sensing images [
8,
9].
For the super-resolution process on single-frame remote sensing images, the statistical sparsity prior of ordinary images [
10,
11] is most utilized. In 2013, inspired by the theory of compressive sensing and self-structural similarity, Pan [
12] increased the resolution of a remote sensing image merely based on raw pixels. Discrete wavelet transform and sparse representation also proved useful [
13] for super-resolution reconstruction when combined. For hyperspectral images, sparse properties in the spatial and spectral domains have been well researched, playing a vital role in super-resolution reconstruction.
Although the above approaches have had an important role in the field of remote sensing image super-resolution, their defects remain obvious. First, these methods were proposed on the basis of features on lower levels [
14,
15]. The performance of machine learning algorithms is determined based on how the features of the images are represented [
16]. Low-level features, such as raw pixels [
11], are far from satisfactory. So far, the most popular scheme has been to extract high-level features via deep convolutional neural networks (CNNs). The results of CNNs used in image classification and object detection also demonstrate their possibility for more complex applications, such as image classification [
17] and object detection [
18]. The spatial distribution of remote sensing images is more complicated than that of natural images, making higher-level features vital to obtain and yield a better representation of the data. CNN-based [
19,
20,
21] methods are useful for learning end-to-end mapping directly between low- and high-resolution images, achieving state-of-the-art performance on natural image super-resolution reconstruction.
Second, unlike in natural images, objects on the ground in remote sensing images usually share a wider range of scales; the object (e.g., airplane) and its surrounding environment (e.g., airport) are mutually coupled in the joint distribution of their image patterns. However, most of the methods mentioned above learn data priors on a limited single-object scale, which means the information of the surroundings is usually neglected. Moreover, the process of super-resolution requires the inference of missing pixels, resulting in a more challenging problem.