Lightweight Multi-Scales Feature Diffusion for Image Inpainting Towards Underwater Fish Monitoring

Abstract

In the process of gradually upgrading aquaculture to the information and intelligence industries, it is usually necessary to collect images of underwater fish. In practical work, the quality of underwater images is often affected by water clarity and light refraction, resulting in most fish images not fully displaying the entire fish body. Image inpainting helps infer the occluded fish image information based on known images, thereby better identifying and analyzing fish populations. When using existing image inpainting methods for underwater fish images, limited by the small datasets available for training, the results were not satisfactory. Lightweight Multi-scales Feature Diffusion (LMF-Diffusion) is proposed to achieve results closer to real images when dealing with image inpainting tasks from small datasets. LMF-Diffusion is based on guided diffusion and flexibly extracts features from images at different scales, effectively capturing remote dependencies among pixels, and it is more lightweight, making it more suitable for practical deployment. Experimental results show that our architecture uses only $48.7\%$ of the parameter of the guided diffusion model and produces inpainting results closer to real images in our dataset. Experimental results show that LMF-Diffusion enables the Repaint method to exhibit better performance in underwater fish image inpainting. Underwater fish image inpainting results obtained using our LMF-Diffusion model outperform those produced by current popular image inpainting methods.

1. Introduction

Aquaculture is one of the important sources of global marine and freshwater food supply. By farming various aquatic organisms, such as fish, shellfish, and crustaceans, it can meet people’s demand for protein and nutrient-rich foods, ensuring food security and stability of supply. In the process of gradually upgrading aquaculture to the information and intelligence industries, it is usually necessary to collect images of underwater fish. Researchers use these images to display underwater fish and analyze the density and distribution of fish [1,2]. Considering factors such as cost, portability, and underwater corrosion protection, in practical production applications, the resolution of most underwater cameras is relatively low. The underwater environment is usually dimly lit, and the shooting process is easily affected by water refraction and scattering. In the real natural environment, substances like sediment and algae in the water can cause turbidity, leading to issues such as dark images, color distortion, and excessive noise in underwater photography [3,4,5]. Therefore, the image quality of underwater fish in natural environments highly depends on the conditions. When the lighting is suitable and the water is clear, the image quality of underwater fish is relatively high, making it easier to analyze fish school density and distribution. However, the water quality can vary significantly due to seasonal changes, precipitation, and surrounding human activities, leading to substantial differences in the quality of underwater images captured at different times. As shown in Figure 1, not only will different fish block each other when the fish swarm is dense, but especially when the water quality is turbid, the fish body in the underwater image cannot be fully displayed.

Image inpainting is to redraw unknown areas based on the information from neighboring regions and the overall structure of the image to recover unknown information. When using existing image inpainting methods for underwater fish images, the results were not satisfactory. The usual practice of existing image inpainting methods are typically trained on large-scale datasets of over 10,000 images [6,7,8,9], using random masks on images as input for the model training [10,11,12,13,14,15]. Some of these algorithms utilize free-form mask generation [16], and some of these are trained on datasets with irregular masks [15]. However, due to the limited availability of underwater fish image datasets for training, these methods are not appropriate for our application. If the above-mentioned methods were to be used for underwater fish image inpainting, the masks in the training set need to cover a part of the fish body. It is obvious that random masks are unlikely to meet this requirement. We try to use the recent methods Diverse Structure Inpainting (DSI) [17], Aggregated Contextual Transformations (AOT) [18], and Repaint [19] for underwater fish image inpainting. DSI and AOT are unable to inpaint the complete shape of the fish. Compared to the two methods above, Repaint yields inpainting results closer to real images. Since it is drawn using the resampling process, no masks need to be entered to pretrain, simplifying the model training process. But it still has shortcomings, such as the unnatural transition between fish bodies and underwater backgrounds, leading to instances like a fish having two heads in the inpainting image. Moreover, models of the Repaint method have a large number of parameters, which are not practical for real-world deployment. To solve the above problems, we present Lightweight Multi-scales Feature Diffusion (LMF-Diffusion). This is an architecture based on the guided diffusion model, building upon the foundation of an Attention-based UNet [20]. To enhance the learning capability of models on small datasets, we aim for the model to capture long-range dependencies among pixels and extract features from images at different scales. Therefore, the Multi-scale Cross-axis Attention (MCA) [21] module is introduced into the architecture. Depthwise separable convolutions [22] are also used in LMF-Diffusion, which reduce the parameters of the model and make it more suitable for real-world deployment. In summary, our main contributions can be concluded as follows:

• LMF-Diffusion is proposed. Employing LMF-Diffusion models for image inpainting tasks has shown promising results in underwater fish image inpainting, producing results that closely resemble real images.
• Within LMF-Diffusion, MCA is integrated to allow the model to effectively capture distant dependencies among pixels and extract features from images at various scales and orientations.
• To reduce parameters, depthwise separable convolutions are used. Models trained using LMF-Diffusion possess only $48.7\%$ of the parameters compared to the guided diffusion model.

The remainder of this paper is organized as follows. Section 2 provides an overview of the relevant research. Section 3 introduces the collection of the image dataset, followed by the proposed LMF-Diffusion. Section 4 describes the experimental setup. Section 5 is the results part, where the results of different experiments are presented. Section 6 analyzes and discusses the experimental results. Finally, the last section provides a summary of the entire paper.

2. Related Work

2.1. Image Inpainting

Image inpainting is the process of inpainting missing areas in an image based on the neighboring position information and the overall structure of the image, following certain rules to repaint the unknown areas and recover the missing information [23]. The current scope of image inpainting tasks is broad, including rectangular mask inpainting, irregular mask inpainting, object removal, watermark remova, old photo colorization, and so on [6,24].

Traditional image inpainting algorithms are mainly classified into structure-based image inpainting algorithms [25,26,27] and texture-based image inpainting algorithms [28,29,30,31]. Structure-based methods use geometric techniques to restore missing parts of an image, assuming that the damaged and known parts of the image have similar content. It finds the most matching feature blocks in the known areas of the image and then replicates information at the pixel level to fill in the missing areas. Texture-based image inpainting algorithms gradually spread pixel information around the damaged pixels in the image and synthesize new textures to fill in the missing parts, which are more suitable for restoring backgrounds with structured textures and removing small objects from images. However, the inpainting effect is related to the size of the area to be mended. Traditional image inpainting algorithms perform well in simple regular-texture structures or small-area mask complementation tasks but have poor results in images with complex textures for larger areas.

Deep learning-based image inpainting methods, such as generative networks [32], VAE, GAN, and the denoising diffusion probabilistic model (DDPM), can effectively learn and model the distribution of real data from training data. The training of the VAE-based image inpainting method [33] is usually stable, but the resulting results are likely to be fuzzy. The GAN-based image inpainting method [34,35,36,37,38,39] can inpaint higher-quality images. Increasing the width and depth of the network can better fit the data features, but too large a network depth can cause gradient explosion. Therefore, current image inpainting network models use thin and deep network structures to reduce and control the number of parameters, and employ multi-scale features or skip-connection residual structures to address gradient vanishing issues. The recent Repaint method [19] uses the DDPM prior and repeated sampling, leveraging a pre-trained unconditional DDPM as a generative prior to sampling the uncovered areas based on the given image information, altering the reverse diffusion iteration to complete the image inpainting task.

2.2. Denoising Diffusion Probabilistic Models

First and foremost, Jascha et al. put forth the concept of the diffusion probabilistic model [40], which operates as a Markov chain known for its ability to convert a known distribution into a data distribution. It was upon this foundation that, in 2020, Jascha et al. took a step further with the introduction of the DDPM [41]. They established a connection between the diffusion probabilistic model and denoising score matching with Langevin dynamics. Building upon the DDPM, Prafulla et al. increased the number of attention heads, using BigGAN residual blocks for both upsampling and downsampling the activations to propose the guided diffusion model [42]. Notably, guided DDPMs outperformed GANs in unconditional image synthesis tasks.

Diffusion models are deep generative models consisting of two processes: the forward process and the reverse process. The diffusion model has been widely applied to various artificial intelligence tasks, such as image generation [43,44,45], image super-resolution [46,47], image inpainting [19,48], target recognition [49]. The forward process entails a series of iterative steps, during which minor details of noise are introduced to each initial input image $x_0$. At each step, the intensity of the Gaussian noise $q\left(x_t\right|x_{(t-1)})$ varies, progressively altering the input data until it converges to pure Gaussian noise $x_T$. On the other hand, the reverse diffusion process symbolizes the inverse of the forward diffusion process. It follows a similar iterative pattern, steadily eliminating the reverse noise $p\left(x_{(t-1)}\right|x_t)$ to reconstruct the original image. This entire process is exemplified in Figure 2.

2.3. Attention Mechanisms

In the course of the development of deep learning, the attention mechanism has given rise to many variants. Self-attention has shown good performance, but its use leads to a sharp increase in model parameters and computational complexity. Jonathan et al. proposed axial attention [24] that applies self-attention separately in the vertical and horizontal directions of images, significantly reducing computational complexity. Ashish et al. introduced the multi-head attention mechanism, using multiple attention heads in parallel, each independently learning and outputting, then combining these outputs to enhance the model’s learning ability. Additionally, more improved attention mechanisms have been proposed. For example, Zhu et al. designed Bi-Level Routing Attention [11], a dynamic and query-aware sparse attention mechanism that filters out most irrelevant key-value pairs at a coarse regional level, retaining fine-grained details and reducing computational load.

In the task of image inpainting, attention mechanisms have been widely utilized. To meet the requirements of edge consistency and semantic integrity in image inpainting, Yu et al. [14] employed contextual attention to optimize the inpainting of blurry results. Li et al. [50] proposed GAP-Net, which leverages self-attention mechanisms to guide feature mapping in the transformation process from latent feature space to image information, resulting in more realistic image inpainting. The guided diffusion mentioned earlier [42] adjusted the use of the multi-head self-attention mechanism based on the original DDPM, significantly enhancing the model’s learning ability.

Although DDPM can achieve better sample quality than GAN in many image generation tasks, the model has high computational complexity and a large number of parameters in DDPM training. What is more, previous image inpainting approaches based on the guided diffusion model are difficult to adapt to fish image inpainting tasks, and they have low content awareness so that the boundaries of a fish body in image inpainting results are not naturally transitioned enough. The rich colors and textures of koi make image inpainting more challenging. Different from the existing image inpainting methods mentioned above, we propose LMF-Diffusion to address the aforementioned issues and conduct ablation studies of our method.

3. Materials and Methods

3.1. Koi Dataset

To evaluate the performance of image inpainting methods on underwater images in practical applications, an underwater koi image dataset was constructed. This dataset was collected from an outdoor fish pond in Guangdong, China, where the majority of the fish have diverse colors and patterns. As shown in Figure 3, a regular underwater camera was submerged to capture video footage of the pond during the daytime. Multiple video recordings were made in the same water area during the summer and autumn, with the total duration of the footage exceeding one hour. From this extensive footage, image segments displaying the complete shapes of fish were extracted, resized to $256\times 256$ pixels, and compiled to form the dataset. A regular underwater camera is one of the most popular models currently available for purchase online, which is beneficial for the practical application and promotion of this research, and is typically used to help anglers explore underwater conditions. Due to the open and flowing nature of the water where the data are collected, it is not possible to estimate how many fish are present. From the gathered images, it is obvious that the sizes of the koi in the images vary and are mixed throughout the water. The same fish occupies different proportions of the frame depending on its distance from the camera.

In building the dataset, 1500 images for the training set and around 100 images for the test set were selected. It was strictly required that the shape of the fish in each image included in the dataset must be fully visible, ensuring the representativeness and effectiveness of our dataset.

It is worth noting that our dataset encompasses a wide range of diversity, including various colors, patterns, angles, and poses of koi images. Such diversity will aid in evaluating the performance of image inpainting methods across different scenarios and provide ample challenges and opportunities for the generalization ability of the models. During the data collection process, efforts were made to ensure the richness and diversity of the data, aiming for the dataset to play a more representative and reliable role in the evaluation and application of underwater image inpainting methods.

3.2. LMF-Diffusion

3.2.1. LMF-Diffusion Model Architecture

Underwater lighting conditions are complex, such as light attenuation and color shifts, which pose challenges for image inpainting tasks. Unlike images of common objects on land, in underwater fish images, fish postures vary and fish heads at similar angles in different images may correspond to completely different fishtail postures, making it more challenging for models to capture image features. At the same time, limited by factors such as lighting, seasons, underwater camera equipment, and so on, capturing clear and complete underwater fish body images suitable for model training is extremely difficult. Specifically, out of around 70,000 underwater fish images captured, only 1600 were selected to compose the dataset. This poses higher demands on the learning capability of the model, requiring more accurate and reliable inpainting results based on a limited number of real images.

When using Repaint for underwater fish image inpainting, it is necessary to first pretrain the unconditional DDPM model using the guided diffusion architecture [38]. Utilizing the prior model obtained with the guided diffusion architecture for image inpainting results in relatively rigid fish body boundaries in the inpainting images. Issues such as significant discrepancies in koi patterns compared to real images and incorrectly inpainting the fish body area as the fish head arise, as shown in Figure 4. Clearly, using this method for underwater fish image inpainting makes it difficult to inpaint the natural form of the fish. Moreover, the model pretrained with the guided diffusion architecture has a large number of model parameters, thus occupying a significant amount of storage space, which is not convenient for practical applications. LMF-Diffusion is proposed to improve the above issues.

The structure of the LMF-Diffusion is shown in Figure 5. LMF-Diffusion is based on the UNet architecture. UNet is a fully convolutional network with an encoder–decoder structure. The structure of the encoder part and the decoder part is symmetrical, and the number of decoders is the same as the number of encoders. The encoder part performs feature extraction and compression by downsampling the input image, while the decoder part inpaints the size of the image and combines the extracted features from the encoder part with the corresponding layers of the decoder part. To enhance the model’s feature extraction capability and address the challenge of training image inpainting models on small datasets, MCA is incorporated as the attention module in UNet. MCA helps the model capture long-range dependencies among pixels effectively, handling features at different scales and orientations. In the subsequent experimental section, it is demonstrated that with the inclusion of the MCA module, the model can more accurately inpaint the details and edge information of fish shapes. To facilitate the practical deployment and application of the model, the use of depthwise separable convolutions is proposed to reduce the model’s parameter size and computational cost.

Algorithm 1 displays the complete training procedure of LMF-Diffusion. Referring to the training method of DDPM, the loss function is derived as a simplification of the one presented by Ho et al. [40]. ${\overline{\alpha }}_t$ represents the total noise variance, ${\beta }_t$ is the variance of the normal distribution, and the noise generated at step t is sampled from $\mathcal{N}(0,{\beta }_t)$. In the specific training process, ${\beta }_t$ is determined by different beta schedules. For example, when the beta schedule is linear, the variance is sampled evenly from the minimum to the maximum value at each step.

Algorithm 1 Training LMF-Diffusion model

Input:  input image $x_0$ Output:  prediction ${ϵ}_{pred}$ 1: repeat 2:     $t\sim Uniform\left(\left\{1,\dots ,T\right\}\right)$ 3:     $ϵ\sim \mathcal{N}(0,I)$ 4:     ${\overline{\alpha }}_t={\prod }_{s=1}^t(1-{\beta }_s)$ 5:     ${ϵ}_{pred}={ϵ}_{\theta }(\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,t)$ 6:     $Loss=E_{t,x_0,ϵ}\left[{∥\begin{array}{c}ϵ-{ϵ}_{pred}\end{array}∥}^2\right]$ 7:     Take gradient descent step on ${▽}_{\theta }{∥\begin{array}{c}ϵ-{ϵ}_{pred}\end{array}∥}^2$ 8:     Update the weights of the LMF-Diffusion 9: until converged return ${ϵ}_{pred}$

3.2.2. Multi-Scale Cross-Axis Attention (MCA)

Underwater images suffer from issues such as different scales of blurriness and color distortion, requiring the model to have strong modeling capabilities to more flexibly learn the structure and texture features of fish bodies in underwater images, and to more accurately inpaint lost details. Moreover, due to the limited amount of training data in underwater fish image inpainting tasks, the long-range dependencies among pixels in the model are challenging. In the original guided diffusion model, multi-head attention was used as the attention module. When using it as the pretraining model for the Repaint method, the resulting image was unsatisfactory, with an error of two fish heads in one fish and stiff transitions around the fish edge. Therefore, to enhance the image inpainting capability for underwater fish images, an attempt was made to introduce a more flexible attention module.

MCA was proposed for solving image segmentation problems in the medical field. Their MCA-Net, based on this proposal, achieved superior results to state-of-the-art methods in several typical medical image segmentation tasks. MCA not only effectively captures multi-scale information, but also improves the handling of challenges in achieving long-distance interactions in many medical image segmentation tasks with relatively small datasets. Therefore, introducing the MCA module into the architecture is considered. MCA can help the model establish remote dependency relationships among pixels, fully considering the overall structure and continuity of underwater fish images. By introducing multi-scale information and cross-axis correlations, the model can more accurately inpaint details or the edge information of fish bodies.

MCA is divided into two directions. Given the feature map $F(H\times W\times C^1)$, it uses three parallel 1D convolutions to encode F. The outputs of these convolutions are fused via summation and fed into a $1\times 1$ convolution. The formulation is as follows.

$$F_x=Con{v}_{1\times 1}\left(\sum _{i=0}^{2}Conv1D_1^x\left(Norm\left(F\right)\right)\right)$$

(1)

$$F_y=Con{v}_{1\times 1}\left(\sum _{i=0}^{2}Conv1D_1^y\left(Norm\left(F\right)\right)\right)$$

(2)

where $Conv1D_1^x(·)$ represents one-dimensional convolution along the axis dimension, and $Norm(·)$ represents layer normalization. $F_x$ and $F_y$ are the output of two directions, respectively.

Conduct the y-axis attention using $F_x$, and calculate the cross-attention between $F_x$ and $F_y$. That is, using $F_x$ as the key and value matrices and $F_y$ as the query matrix, the calculation result $F_T$ is obtained. Similarly, using $F_y$ as the key and value matrices and $F_x$ as the query matrix, the calculation result $F_B$ is obtained.

$$F_T=MHC{A}_y(F_y,F_x,F_x)$$

(3)

$$F_B=MHC{A}_x(F_x,F_y,F_y)$$

(4)

where $MHC{A}_y$ and $MHC{A}_x$ represent the multi-head cross-attention along the the y-axis and x-axis, respectively. The final output of MCA is:

$$F_o=Con{v}_{1\times 1}\left(F_T\right)+Con{v}_{1\times 1}\left(F_B\right)+F$$

(5)

In subsequent experiments, the same number of attention heads is maintained in the MCA as in the multi-head attention module of the guided diffusion architecture.

Compared to MCA, ordinary multi-head attention is relatively limited in handling multi-scale features and cross-scale information. For some more complex long-range dependency relationships, traditional multi-head attention may be difficult to fully capture all the necessary contextual information.

In practical application, MCA not only improves the inpainting effect of underwater fish images, but also reduces the number of parameters. MCA integrates multi-scale features and uses attention mechanisms to extract global context information. After integrating MCA, LMF-Diffusion shows a significant performance improvement in fish image inpainting tasks, enabling more accurate inpainting of fish colors and textures.

3.2.3. Depthwise Separable Convolution

When using deep learning methods for image inpainting tasks on underwater fish images, the model parameters pretrained through the guided diffusion architecture have a relatively large quantity, leading to high memory consumption, which poses certain difficulties in practical deployment and application. Constrained by conditions such as computational resources, the aim is to reduce the complexity of the model, decrease the number of parameters and computational costs, and improve the model’s operational speed while maintaining performance. Therefore, depthwise separable convolutions are introduced in LMF-Diffusion.

The depthwise separable convolution decomposes standard convolution into two processes, consisting of depthwise convolution and pointwise convolution. The depthwise convolution performs convolution operations on each input channel, resulting in an output feature map with the same number of channels as the input feature map. Since each channel only needs to be convolved once instead of being repeated over all output channels, this reduces a lot of redundant calculations. The number of output feature maps in depthwise convolution equals the number of input feature maps, which makes effective dimension expansion impossible. Therefore, pointwise convolution is introduced, where pointwise convolution is a normal $1\times 1$ convolution that can change the number of channels and aggregate several feature maps obtained from depthwise convolution.

Referring to Figure 6, assume the standard convolution input is $D_F\times D_F\times M$ and the kernel size is $D_K\times D_K\times M\times N$, where $D_F$ and $D_K$ represent the spatial dimensions of the input and the kernel, M is the number of input feature channels, and N is the number of output channels. Depthwise separable convolution first performs depthwise convolution on the same input, applying a kernel of size $D_K\times D_K$ to each channel of the input to generate the same number of channels, and then stacks these channels along the channel dimension. Pointwise convolution applies a $1\times 1\times M\times N$ convolution operation to the feature maps obtained from the previous step. Therefore, the parameter count for standard convolution can be succinctly represented as $D_K·D_K·M·N$, and similarly, the parameter count for depthwise separable convolution is $D_K·D_K·M+M·N$. The reduction in parameter count is as follows:

$${\displaystyle \frac{D_K·D_K·M+M·N}{D_K·D_K·M·N}}={\displaystyle \frac{D_K·D_K+N}{D_K·D_K·N}}$$

(6)

Therefore, when $D_K$ and N values satisfy certain conditions, depthwise separable convolution can significantly reduce model computation compared to standard convolution, but it may also lead to performance degradation. Not all convolutions in the training network are replaced with depthwise separable convolutions. Based on the actual effect, some of them are adjusted to a depthwise separable convolution.

In the final architecture, the parts using depthwise separable convolutions are the upsample block, the downsample block, and the input layer of residual block.

4. Experiments Setup

4.1. Experimental Platform and Parameters

All experiments in this study were conducted on the same software and hardware platform, as shown in

Table 1. Experimental hardware and software platform.

Name	Configure
CPU	Intel(R) Core (TM) i9-10850K CPU @ 3.60 GHZ (Intel Corporation, Santa Clara, CA, USA)
GPU	Nvidia GeForce RTX 3090 (Nvidia Corporation, Santa Clara, CA, USA)
Operating system	Ubuntu 20.04.6 LTS
CUDA	10.1
CUDNN	8.0.5
Python	3.7.6

. This study was implemented on Ubuntu using Conda, which allows environment migration across different operating systems. The Repaint method using pretrained guided diffusion model is chosen as the baseline. When using the Repaint method, models are pretrained with 250,000 iterations using the AdamW optimizer for both guided diffusion and LMF-Diffusion. With $T=250$ timesteps, the Repaint approach was applied using $r=10$ times resampling and a jumpy size of $j=10$.

4.2. Evaluation Metrics

Params are used to evaluate the lightweighting effect of the architecture. LPIPS and PSNR are utilized to assess the final image inpainting effect. PSNR is a metric used to evaluate the quality of an image by comparing it to the original one, with higher values indicating better quality. Proposed in [51], LPIPS learns a generator that enforces the inverse mapping from generated images to ground truth images, prioritizing the perceptual similarity between them. A smaller LPIPS value indicates a better image inpainting effect.

4.3. Mask Settings

The ablation study is evaluated under four different mask settings. To compare the effectiveness of methods in underwater image inpainting, fish masks are designed for the test set to assess whether methods can inpaint the entire fish image based on just half of the fish body. Additionally, thin, thick, and half masks are included in the test set to evaluate the performance of the model.

5. Results

In this section, the ablation study results of LMF-Diffusion on the dataset are reported, comparing them with other existing image inpainting methods and attempting to inpaint obscured fish images. It is noted that model pretraining was conducted on LMF-Diffusion, and the Repaint method was used for underwater fish image inpainting based on this model to evaluate its performance.

5.1. Hyperparameter Optimization

First, the optimal hyperparameter settings for the model are selected. A Repaint method was employed for underwater image inpainting under the fish mask setting, and the model was evaluated using the LPIPS and PSNR metrics.

Initially, comparative experiments were conducted on different learning rates. Under the same batch sizes, LMF-Diffusion was trained with varying learning rate settings, and the experimental results are presented in the

Table 2. Comparison of experimental results with different batch size settings under the same learning rate.

	Learning Rate
	1 × ${10}^{-\mathbf{3}}$	1 × ${10}^{-\mathbf{4}}$	1 × ${10}^{-\mathbf{5}}$	1 × ${10}^{-\mathbf{6}}$
PSNR	14.962	24.505	28.812	26.641
LPIPS	0.425	0.165	0.116	0.148

. The results indicate that the model achieved the best performance in underwater fish image inpainting when the learning rate was set to 1 × ${10}^{-5}$.

Furthermore, models were trained under the same learning rate but with different batch sizes. The experimental results, as shown in

Table 3. Comparison of experimental results with different learning rate settings under the same batch size.

	Batch Size
	16	32	64	128	256	512
PSNR	24.861	26.966	25.977	28.812	25.782	27.661
LPIPS	0.150	0.135	0.142	0.116	0.159	0.132

, indicate that the best performance was achieved with a batch size of 128.

Therefore, in the subsequent experiments, our architecture was configured with a batch size of 128 and a learning rate of 1 × ${10}^{-5}$.

5.2. The Results of Ablation Studies

Influence of convolution. Depthwise separable convolutions were incorporated to substitute a portion of the convolutions within the guided diffusion architecture. This adjustment has led to a notable reduction in the number of parameters, while causing only a minor impact on the performance of image inpainting. As illustrated in Table 1, after introducing depthwise separable convolutions, our architecture is only $48.7\%$ of the guided diffusion.

Influence of attention block. MCA was chosen in LMF-Diffusion. The utilization of the MCA block has demonstrated its ability to augment the efficacy of image inpainting without inflating the number of parameters. Referring to Table 1, it becomes evident that the architecture integrated with MCA showcases a reduction in LPIPS and an increase in PSNR, indicating an improvement in the perceptual similarity between inpainted images and ground truth.

Analysis. According to the results in

Table 4. Ablation studiy results of our LMF-Diffusion on our dataset.

Model	Params (M)	Mask Settings
		Fish		Thin		Thick		Half
		PSNR↑	LPIPS↓	PSNR↑	LPIPS↓	PSNR↑	LPIPS↓	PSNR↑	LPIPS↓
Guided diffusion model	932.08	28.461	0.121	36.137	0.070	33.879	0.087	24.775	0.192
+ DSC	486.86	28.617	0.119	36.138	0.071	33.832	0.089	24.571	0.199
+ MCA	899.74	29.342	0.112	36.109	0.069	33.984	0.084	24.573	0.194
Our architecture	454.53	28.812	0.116	36.154	0.068	33.965	0.084	24.761	0.192

, on the fish mask setting, the architecture improved with MCA achieved better image inpainting results, with a significant improvement in image quality but an increase in the number of parameters. The model parameter size significantly decreased with the improved architecture using depthwise separable convolutions, but there was a slight decrease in image quality, as indicated by LPIPS and PSNR values. In the other three mask settings, our architecture achieved better inpainting results in thin and thick mask settings. In the half-mask setting, where half of the image was masked for inpainting, MCA did not show significant improvement in image inpainting, but the results were comparable to the guided diffusion architecture. Therefore, considering the above cases, LMF-Diffusion combines these two improvements, reducing the number of model parameters while enhancing the image inpainting results.

In the fish mask setting, the results of our LMF-Diffusion compared with the original guided diffusion architecture for underwater fish image inpainting are shown in Figure 7. Obviously, from a human perspective, the fish boundaries in the images inpainted by our LMF-Diffusion model are smoother, and the pose and structure of the fish are more similar to the input image. When using guided diffusion, there can be cases where the inpainting image shows two fish heads in a fish, as the original guided diffusion architecture cannot accurately capture underwater fish features. Furthermore, the lines of the fish’s tail cannot be rendered naturally in guided diffusion results, leading to an illogical enlargement of the tail. Our LMF-Diffusion has addressed the above issues. MCA can better capture multi-scale features in input data, reduce information redundancy, and enhance model generalization capability. By relying on MCA to extract global context information, when using LMF-Diffusion, there are no occurrences of one fish body corresponding to two fish heads in the inpainting results. As shown in the sixth column of Figure 7, LMF-Diffusion can better match the color and patterns of the fish head and body compared with guided diffusion. As indicated in the first column of Figure 7, LMF-Diffusion also depicts the tail fin more realistically and naturally. Figure 8 shows the underwater image inpainting results of guided diffusion and LMF-Diffusion under different mask settings. Both guided diffusion and LMF-Diffusion yield visually pleasing results with thin masks and thick masks. Combining these observations with the scores in Table 4, LMF-Diffusion demonstrates slightly better inpainting performance than guided diffusion. However, with half masks, guided diffusion exhibits issues such as mismatches between the fish head and body, or half of the fish body disappearing, while LMF-Diffusion produces better results.

5.3. Comparison with Other Methods

In this section, a comparison is made between the proposed method and other image inpainting methods. The test set was covered with fish masks and inpainted using different methods.

The experimental results are presented in Figure 9 and

Table 5. Comparison with other methods.

			Repaint
	DSI [17]	AOT [18]	Guided Diffusion [42]	LMF-Diffusion
PSNR↑	21.497	30.084	28.461	28.812
LPIPS↓	0.241	0.082	0.121	0.116

. Using the DSI method [17] for underwater fish image inpainting, the unknown areas in the resulting images can be filled with colors similar to the known background. However, due to the small size of the training data set, the transition between unknown and known areas was abrupt, making the shape of the mask clearly visible, and the fish bodies are difficult to distinguish. According to the data in Table 4, AOT [18] performs the best in terms of LPIPS and PSNR values but differs in image inpainting results. The AOT method can inpaint the rough outline of the fish body with smoother color transitions, causing the color of the repaired fish tail to dissolve in the water. As a result, the LPIPS and PSNR values are closer to the original image, but the transition with the water is too blurred, making it difficult to depict the shape of the fish tail, and the resulting inpainting does not meet our expectations. The Repaint method demonstrates a higher fidelity in inpainting results that closely resemble real images. However, it is not without its drawbacks. One notable limitation is the unnatural blending between fish subjects and underwater environments, resulting in anomalies like fish appearing to have two heads in the result images. From the results, our model alleviates the above problems. Not only do the inpainting images more closely resemble real images, but the matching issue between a fish head and body has significantly improved, with no case of one fish head corresponding to two fish bodies.

5.4. Inpainting Obscured Fish Images

To simulate the practical application of image inpainting techniques in underwater fish images, instance segmentation technology was used instead of manually adding masks to demonstrate the feasibility of the method.

The YOLOv8 method was adopted for instance segmentation to inpaint the complete form of the target fish obscured by other fish. Through this method, masks specific to the occluded fish were successfully generated and then used along with the original images as inputs for our image inpainting method. The resulting images, as shown in Figure 10, present clear and complete underwater fish images. From the inpainting results, it is evident that the model reasonably infers the swimming posture of the occluded fish beneath the masks, while the transition of fish body color and patterns appears natural, and the inpainting portions seamlessly blend with the underwater background, resulting in a fairly good visual effect.

6. Discussion

Our experimental section demonstrated the effectiveness of LMF-Diffusion in underwater image inpainting tasks and its real-life applications, proving its practical feasibility. However, there are still limitations to LMF-Diffusion. As mentioned in the ablation studies section, when the input mask covers a large portion of the image, distortion persists in the inpainting result.

Furthermore, the proposed architecture only enhances underwater image inpainting results from pretrained models, without improving the inference process (i.e., the Repaint method). In the “Inpainting obscured fish images” part of the experiments, after generating masks based on instance segmentation, we use Repaint for image inpainting with LMF-Diffusion as a prior. This method requires time for mask generation and inference, making it unsuitable for video inpainting. This presents an area for future research.

7. Conclusions

In this paper, LMF-Diffusion is proposed for model pretraining in underwater fish image inpainting. In LMF-Diffusion, the MCA module helps the model establish long-range dependencies among pixels and extract features from images at different scales and orientations. Considering practical deployment, depthwise separable convolutions are utilized to reduce the model’s parameters and spatial footprint. Experimental results show that proposed LMF-Diffusion enables the Repaint method to exhibit better performance in underwater fish image inpainting. Compared with the original guiding diffusion model, it not only reduces the parameter size to about $48.7\%$ of the original, but also demonstrates smoother and more realistic results. Additionally, underwater fish image inpainting results obtained using our LMF-Diffusion model outperform those produced by current popular image inpainting methods.