On-Board Image Enhancement on Remote Sensing Payload

Abstract

CubeSats are designed to optimize applications within the strict constraints of space and power. This paper presents an On-Board Image Enhancement technique for remote sensing payloads, focusing on achieving Auto White Balance (AWB) with limited resources and enhancing the capabilities of small/microsatellites. The study introduces hardware-based techniques, including histogram adjustment, De-Bayer processing, and AWB, all tailored to minimize hardware resource consumption on CubeSats. The integrated 1U CubeSat system comprises a sensor board, an Image Data Processor (IDP) unit, and onboard computing, with a total power consumption estimated at 2.2 W. This system facilitates image capture at a resolution of 1920 × 1200 and utilizes the proposed algorithm for image enhancement on remote sensing payloads to improve the quality of images captured in low-light environments, thereby demonstrating significant advancements in satellite image processing and object-detection capabilities.

1. Introduction

The field of small-satellite technology has witnessed rapid growth in recent years. These satellites offer high cost-effectiveness, shorter development cycles, and extensive commercial application potential in areas such as communication [1], Earth observation [2], and scientific research [3]. With continuous technological advancements, small satellites are poised to drive further innovation and applications in the space industry, opening up new markets and application domains.

Designing remote sensing payloads for small satellites presents multiple challenges. First, image resolution is of paramount importance. Due to the limited space on CubeSats [4], achieving sufficient image resolution within this confined space is crucial to ensure the accuracy and reliability of remote sensing data. Second, the use of high-dimensional lenses is restricted by the size limitations of CubeSats, which may impact the performance and functionality of the remote sensing payload. Additionally, the hardware systems on CubeSats face strict power consumption limitations, potentially restricting the performance of sensors and image-processing units. Thus, optimizing power consumption is essential to ensure a system operates effectively within a limited energy supply. Finally, it is necessary to limit the amount of incoming light to protect the sensor, which may affect the brightness and contrast of the images, consequently impacting the quality of satellite imagery. In summary, the design of remote sensing payloads is a complex task that requires multiple constraints to be overcome to ensure the performance and effectiveness of the payload.

For efficient image processing and capturing on satellites, the implementation of algorithms on embedded processors is required. Field-Programmable Gate Arrays (FPGAs) [5] are highly suitable for this purpose due to their balance between parallel processing capabilities and low power consumption [6].

This study utilizes FPGA logic for onboard object detection missions. The design of an effective imaging payload necessitates the careful consideration of power consumption, resource allocation, and image quality. The adoption of FPGA technology facilitates the design of a compact development board, enabling parallel data processing while ensuring low power consumption within the limited space of CubeSats. This research introduces FPGA firmware implementation and its technical specifications, focusing particularly on FPGA logic design. These applications have found wide usage in CubeSat. To achieve parallel-processing and low-power image capture architecture, the development of system architecture for an imaging payload on a 1U CubeSat (10 cm × 10 cm) is presented. This system efficiently coordinates the available volume and power consumption of CubeSats and is designed based on FPGA technology. The image-enhancing De-Bayer [7] and AWB [8] algorithms play a critical role in enhancing the clarity of satellite imagery. The AWB algorithm automatically corrects color deviations, ensuring the naturalness and accuracy of colors under different lighting conditions. Simultaneously, the De-Bayer algorithm reconstructs full-color images from raw image data, improving color accuracy and image detail clarity. As both algorithms utilize histograms, this study stores the parameters computed from histograms in DDR to enhance computational efficiency and leverages the parallel-processing capabilities of FPGA to accelerate the processing speed. This capability enables the leveraging of image recognition technology for the identification og objects on the Earth’s surface or in maritime areas, such as detecting vehicles on roads or ships at sea.

2. Proposed Algorithm for Image Enhancement on Remote Sensing Payload

The spatial constraints of CubeSats preclude the accommodation of high-resolution lenses, leading to reduced variations in captured light [9]. This limitation results in images with a lack of lighting variations and subtle details, reducing their quality and usability. Additionally, the necessity to protect the sensor results in less incoming light being permitted, leading to darker images with diminished details. Such exposure limitations affect the dynamic range of the images, thus further diminishing their quality. Consequently, preprocessing images captured on satellites enhances their quality. It compensates for the limitations of light variations and exposure time, making the images clearer and more suitable for subsequent analysis and applications. This study proposes an algorithm for image enhancement on remote sensing payloads, including FPGA-based De-Bayer and Auto White Balance (AWB) processes to address the issues arising from light variations and exposure-time limitations. AWB automatically adjusts the color saturation of images, rendering them more true-to-life and natural, while the De-Bayer process addresses color distortion issues in images, further enhancing their quality.

2.1. FPGA-Based De-Bayer Algorithm

This study employs an innovative approach, starting with the histogram processing of RGB channel data. Subsequently, the calculated maximum and minimum values of the RGB channel parameters are stored in DDR via DMA. During the normalization process, parameter values are retrieved from DDR through DMA for normalization. Then, the image undergoes iteration over each pixel using the De-Bayer filter. Following that, color interpolation is performed on each pixel, converting the result back to RGB values to obtain the output image. Figure 1 illustrates the FPGA logic design of the De-Bayer algorithm, where RGB channel data are collected nine at a time into the De-Bayer matrix generator. Subsequently, the data are divided into groups of three using an MUX. Further, it undergoes multiplication with a kernel mask, followed by outputting channel data. Finally, the data are sent to DMA via FIFO and then written into the specified DDR location.

The De-Bayer algorithm is a digital image-processing technique used to reconstruct a full-color image from incomplete color samples output by an image sensor overlaid with a Bayer filter. It assumes the reflectance of all the image channels as expressed by Equation (2), with the RGB image and De-Bayer filter iterating through the process.

$$N_c\left(x,y\right)=C\left(x,y\right)-\min \left(C\left(x,y\right)\right)/\max \left(C\left(x,y\right)\right)-\min \left(C\left(x,y\right)\right)$$

(1)

where $N_c\left(x,y\right)$ represents the normalized pixel value by Equation (1), $C(x,y)$ represents the original channel, the pixel value at the $(x,y)$ position, and $\min (C\left(x,y)\right)$ and $\max \left(C\left(x,y\right)\right)$ represent the minimum and maximum values of the channel data, respectively.

$$N_c\left(x,y\right)\times D{B}_C\left(x,y\right)=R_c(x,y)$$

(2)

where $D{B}_C\left(x,y\right)$ represents the De-Bayer filter and $c$ is the channel, $N_c\left(x,y\right)$ is one channel of the image R, G, and B, and $R_c(x,y)$ is the channel result after applying the De-Bayer process.

2.2. FPGA-Based Auto White Balance (AWB) Algorithm

This FPGA-based AWB algorithm aims to preprocess and enhance low-light satellite images. First, it uses histogram preprocessing to calculate the maximum, minimum, and total values for each channel. Subsequently, these values are stored in DDR via DMA. Next, the algorithm extracts the maximum, minimum, and total parameters through DMA for image compensation. Finally, the output image is obtained. The FPGA logic design diagram, as shown in Figure 2, illustrates the process where RGB channel values enter a buffer, while addresses are calculated to read black patterns and compute black averages from ROM. Additionally, differential gain calculations are performed from ROM, followed by a right shift of 16 bits to obtain constrained minimum and maximum values. Subsequently, the results are sent to DMA via FIFO and stored in the specified location of DDR.

This assumption is articulated by Equation (4), detailing the FPGA-based AWB function.

$$H_c\left(v\right)={\sum }_{x=1}^N{\sum }_{y=1}^M{I}_c(x,y)$$

(3)

$$I_c(x,y)=H_c\left(v\right)\times W_c$$

(4)

For each color channel (red, green, blue) of the RGB image, its histogram is calculated separately by Equation (3). A histogram is a statistical chart that displays the frequency of each brightness value, where $H_c\left(v\right)$ is the histogram of color channel $c ϵ \{R,G,B\}$ with the value $v=N\times M$, where N is the image height and M is the image width. $I_c(x,y)$ is mean the channel $c$ place $(x,y)$. $W_c$ is calculated based on the assumption of the white point for channel $c$. The $W_c$ parameters of white point assumption are adjusted based on the results obtained from the white reference objects in the experiment. Each color channel is adjusted using the calculated gains to achieve color balance. $I_c(x,y)$ represents the brightness value of channel $c$ at position $(x,y)$ after adjustment.

3. Hardware Logical On-Board Image Enhancement Design

The collection of satellite imagery faces challenges such as terrain and shadow effects, particularly in mountainous areas. Atmospheric factors like clouds and fog also impact image clarity. The proposed methodology involves histogram adjustments for contrast and brightness, De-Bayer processing for enhancing image quality, and Auto White Balance (AWB) to address lighting variations and atmospheric effects, ensuring accurate color representation. AWB adjusts white balance based on light source colors, providing realistic surface feature depiction despite changing conditions.

In the design of remote sensing payloads, the computation resource and power consumption play an important role. This paper proposes the FPGA-based algorithm and the hardware design of image enhancement with fewer computing resources and lower power consumption, which means the image enhancement can be processed in real time. The improved De-Bayer algorithm and AWB FPGA logic design is described in Figure 3. In this design, preprocessing of the histogram is performed first upon image input. These processed histograms are stored in memory through the APB Bus [10] and APB arbiter controller, along with DMA. Subsequently, the max, min, and sum parameters of the histogram are extracted via DMA and provided to the De-Bayer and AWB modules. Upon receiving the parameters, the De-Bayer module performs normalization processing, followed by De-Bayer processing, and finally saves the result to DDR via DMA. Following this, AWB adjusts white balance and color balance upon receiving parameters, eventually saving the result to DDR via DMA.

To implement an imaging payload system on a 1U CubeSat, this study designs the image capture and image enhancement functionalities for 1U CubeSats, as illustrated in Figure 4. Figure 5 showcases the lens (Schneider Xenoplan 1.9/35 (manufacturer Schneider Kreuznach, city of Bad Kreuznach, Germany)), CMOS Sensor (Python 2000 Global Shutter (manufacturer onsemi, city of Phoenix, AZ, USA)), and the Image Data Processor (IDP) utilized. The primary task of the IDP involves receiving and preprocessing image data. The main FPGA processing unit on the Image Acquisition Board is the Microsemi SmartFusion2 M2S060 FPGA System (Aliso Viejo, CA, USA) on Chip (SoC). The FPGA facilitates basic processing of the image data.

As depicted in Figure 4, the sensor board captures the image through the lens and transmits the image’s raw data through a twisted pair cable connected to the IDP [11]. The IDP board is responsible for image-processing functions, including histogram adjustment, De-Bayer, and AWB. The processed image result is sent to the onboard computer via a board-to-board connection.

The FPGA Logic Design of the On-Board Image Enhancement on Remote Sensing Payload

The actual 1U CubeSat architecture is illustrated in Figure 5, which depicts the design of the imaging payload system on the 1U CubeSat. The module structure of the imaging payload system is shown in part (a), and the interior of the imaging payload system is presented in part (b). This system includes the On-Semiconductor PYTHON 2000 sensor board and the Image Data Processor (IDP). Detailed specifications of the modules are provided in

Table 1. 1U CubeSat hardware specifications.

Sensor Board	On-Semi PYTHON 2000
Sensor types	Color, monochrome, and NIR-enhanced monochrome
Optical format	2/3 inch
Active pixels	1920 × 1200
Pixel size	4.8 × 4.8 µm
Power consumption	1 W
Image Data Processing (IDP)
SoC	Microsemi Smartfusion2 M2S060
MCU	Microsemi Smartfusion2 ARM Cortex-M3 MCU
OS	FreeRTOS
On-board memory	256 MB ECC LPDDR
On-board flash	2 GB NAND + 16 MB NOR
Power consumption	1.2 W

. Table 1 lists the sensor types as color, monochrome, and NIR-enhanced monochrome; the optical format as 2/3 inch; active pixels as 1920 × 1200; pixel size as 4.8 × 4.8 µm; and the fully functional power consumption of the sensor board [11] as 1W. Additionally, it details the SoC as Microsemi SmartFusion2 M2S060 [12], the MCU as Microsemi SmartFusion2 ARM Cortex-M3, the operating system as FreeRTOS, the on-board memory as 256 MB ECC LPDDR, and the on-board flash as 2 GB NAND and 16 MB NOR, with the fully functional power consumption of the IDP as 1.2 W. The total power consumption of the modules in this study, including the sensor board and IDP, is 2.2 W. This indicates that the design is well suited for FPGA utilization in CubeSats, highlighting an efficient power consumption profile for the IDP.

Table 2. Resource usage of the De-Bayer FPGA logic design comparison.

Type	Proposed	Smart Stereo Camera [13]	Heterogeneous Systems-on-Chip [14]
Feature map	3 × 3	2560 × 720	2048 × 2048
CLB LUTs	173	32,665	5184
CLB Registers	194	None	None
LUT as memory	10	6910	48
FPGA	Microsemi Smartfusion2 M2S060	zynq xc7z030 soc	Xilinx Virtex-7

and

Table 3. Resource usage of the AWB FPGA logic design comparison.

Type	Proposed	Camera Systems [15]	Multi-Camera GigE Vision IP Core [16]
Feature map	3 × 3	1920 × 1080	2048 × 2048
CLB LUTs	972	5802	813
CLB Registers	2703	None	610
LUT as memory	4	None	8
FPGA	Microsemi Smartfusion2 M2S060	Zynq-7020 SoC	virtex-4 fx100-11

, respectively, compare the FPGA logic design and resource utilization of the proposed de-Bayer and AWB methods with other methods.

4. Experiment Results

Using satellite images captured in low-light environments, this study simulated the Auto White Balance (AWB) function. The input images and histogram, as shown in Figure 6, underwent AWB processing. The output images and histogram, depicted in Figure 7, demonstrate that the experimental approach effectively addresses the issue of CubeSats’ inability to accommodate high-resolution lenses in space. These satellites encounter situations where lighting conditions undergo substantial changes, necessitating the restriction of incident light to safeguard the sensors and thereby generating low-dynamic-range (LDR) images. Such issues could potentially affect the brightness and contrast of the images. However, the results of this experiment successfully mitigate these concerns, resulting in clearer image rendering. The histogram images in Figure 6c,d and Figure 7c,d, consist of lines rep-resenting the red, green, and blue channels, respectively.

Additionally, this paper utilizes the proposed module of the imaging payload on 1U CubeSat (Figure 5) to obtain satellite images through the sensor payload. Figure 8 displays the raw low-dynamic-range (LDR) image captured by the imaging payload. The proposed on-board image enhancement for remote sensing payload is simulated and presented in Figure 9. It is observed that the brightness of the original images significantly improves after processing by the Image Data Processor (IDP). The red rectangle highlights how the Auto White Balance (AWB) result enhances the clarity of the buildings.

Furthermore, the IDP captures satellite images over the sea to detect ships. These images serve the purpose of ship detection. The results of ship object detection performed pre-training with the YOLO v7 model [17] and conducting training using the AI-TOD DataSets [18] are shown in Figure 10. In the test images, two boats are present, and with YOLO v7 object detection, it is evident that the ships captured by the method proposed in this study, as shown in Figure 10b, are more accurately detected than those in the unprocessed image, as shown in Figure 10a, due to the presence of two ships in the test image.

5. Conclusions

This study develops an on-board image enhancement technique for remote sensing payloads aimed at enhancing image quality and facilitating object detection. Auto White Balance (AWB) is utilized for image enhancement, resulting in clearer satellite imagery. The FPGA-based design enables the use of smaller development boards and reduces power consumption, making it highly suitable for CubeSat applications. Through experiments with satellite images captured in low-light environments, this study addresses concerns related to image quality, leading to improved image rendering. Furthermore, the developed Image Data Processor (IDP) captures satellite images over the sea for ship detection. The results from YOLO v7 object detection demonstrate enhanced accuracy in identifying boats compared to traditional methods, underscoring the effectiveness of the proposed approach in satellite-based object detection.