ResShift-4E: Improved Diffusion Model for Super-Resolution with Microscopy Images

Abstract

Blind super-resolution algorithms based on diffusion models still face significant challenges at the current stage, including high computational cost, long inference time, and limited cross domain generalization ability. This paper aims to apply super-resolution algorithms to the field of optical microscopy imaging to reveal more microscopic structures and details. Firstly, we proposed a lightweight super-resolution model called ResShift-4E, which is an optimized model from two important aspects: reducing the diffusion steps in ResShift and strengthening the influence of the original residuals on model learning. Secondly, we constructed a dataset of Multimodal High-resolution Microscopy Images (MHMI) including a total of 1220 images, which is available on line. Moreover, we extended our model to application-oriented research on blind image super-resolution of optical microscopy imaging. The experimental results demonstrate that our ResShift-4E model outperforms other models on various microscopy images.

1. Introduction

Super-resolution reconstruction is an important computer-vision task that aims to reconstruct an input low-resolution image into a high-resolution image with more textures and details [1,2,3]. Therefore, super-resolution reconstruction techniques have an indispensable place in tasks such as microscopy imaging, medical imaging, image segmentation, image classification, pedestrian re-identification, and geographic remote sensing. However, in real-world environments, super-resolution reconstruction suffers from inconsistency between the predicted downsampling method and the real image, which leads to reconstruction bias and poor quality of the recovered image [4,5].

Most existing methods assume a predefined degradation process (e.g., bilateral downsampling) from the high-resolution image to the low-resolution image, which is difficult to apply to real images with unknown complex degradation types [6,7,8]. The blind image super-resolution algorithm based on diffusion models aims to use diffusion models for the reconstruction by learning a more realistic downsampling method. This downsampling method is predicted by human beings to increase the high-frequency feature information such as texture, morphology, and intrinsic semantics of images [9]. Nevertheless, the blur information of real low-resolution images in practical applications is often inconsistent with the model prediction, which often leads to poor quality of reconstructed high-resolution images and inadequate recovery of high-frequency information. Therefore, how to reconstruct images effectively, i.e., accurately extracting image blur information by mining potential correlations in pairs of clear and blurred images, has become a key research direction in the field of blind image super-resolution [10,11,12].

Specifically, for microscopy images, high-resolution microscopy techniques provide us with unique access to deeper insights into the inner workings of cells and a wide variety of biological processes [13]. For instance, photoactivated localization microscopy, stimulated radiation loss microscopy, and structured light illumination microscopy. However, these methods typically rely on complex optical setups, specific fluorescent markers, and sample processing methods, and require extensive computational post-processing [14]. It becomes essential to further acquire and analyze more details in microscopy images, accurate image enhancement, and super resolution. Although microscopy images have less shooting instability compared to normal images, their super-resolution effects are still restricted by the finite nature of textures and details in the images.

To address the mentioned issues, this paper aims to explore an effective solution to implement super-resolution reconstruction using diffusion models. Meanwhile, it is also expected to achieve a balance between the computational cost of the model and the quality of the output image. In this paper, we employ the fuzzy kernel prediction algorithm in combination with the diffusion model to achieve good performance in the field of single-image super-resolution. Our primary contributions are as follows:

(1) Proposing our ResShift-4E model. It is more lightweight, and is twice as computationally efficient as the original ResShift model.

(2) Constructing the Multimodal High-resolution Microscopy Images (MHMI) dataset. MHMI includes images of four different microscopy techniques, 1220 in total.

(3) Applying the modified ResShift-4E model to the field of biological image analysis. ResShift-4E is trained on the MHMI dataset and compared to other models, the results of which demonstrate the advantage of ResShift-4E over others under various types of microscopy images.

2. Related Works

2.1. Super-Resolution Image Datasets

Researchers use unified datasets to present the quantitative and visual comparison of the super-resolution models. In the field of blind super-resolution, the widely used datasets include DIV2K [15], Flickr2K [16], OutdoorSceneTraining [17], and 2020Track [18], all of which have more than 1000 high-resolution images. Limited by the acquisition difficulty, real image datasets are more suitable for super-resolution reconstruction. Cai et al. [19] firstly developed a real image dataset, RealSR, with 595 LR-HR image pairs of the same scene at different focal lengths with different scales, accomplishing accurate alignment. Wei et al. [20] developed a larger dataset called DRealSR including 840 image pairs. Similarly, traditional test sets include Set5 [21], Set14 [22], BSD100 [23], Urban100 [24], Manga109 [25], 2020Track1 [18], and 2020Track2 [18], with 5, 14, 100, 100, 100, 109, 100, and 100 images, respectively. Meanwhile, test sets such as RealSR, OST300 [26], DPED [27], and ADE20K [28] are more frequently used in the field of real image super-resolution. In summary, it is still restricted by the open datasets of microscopy images for training. Thus, it is necessary to build up a new dataset for blind super-resolution task.

2.2. Super-Resolution Methods Based on Diffusion Models

With the introduction of Denoising Diffusion Probabilistic Models (DDPM) [29] across the board, diffusion models have been progressively simplified with great success in the field of image generation. A common approach is to insert LR images into the input of the current diffusion model (e.g., DDPM) and retrain the model from scratch on the training data. Ramesh et al. [30] firstly applied diffusion models efficiently to the multimodal domain by learning a low-resolution model and multiple super-resolution diffusion models to save computation costs. Latent Diffusion Models (LDMs) [31] creatively learned feature distributions on the latent space with diffusion models instead of directly letting the diffusion models learn the images, greatly reducing the learning cost of diffusion models. Sahak et al. [32] proposed a diffusion-model-based blind super-resolution model, SR3+, which combined self-supervised training with noise modulation enhancement during training and testing.

An unconditionally pretrained diffusion model is another approach, utilizing its inverse path to generate the expected HR image [33,34,35,36,37,38]. ResShift [33] limits the diffusion model from indeterminate hundreds of steps to 15 steps. It also proposes a noise table (schedule) that can effectively control the noise intensity and transition speed during diffusion, which strengthens the inference efficiency. Gu et al. [34] took an alternative approach in high-resolution image and video synthesis by integrating low-resolution diffusion into high-resolution generation, which demonstrates a strong zero-sample generalization ability on untrained resolution. Li et al. [35] converted Gaussian noise into SR prediction through Markov chain, which could effectively solve the problems of overly smooth outputs, pattern collapse, and large model occupancy area, etc. DDIM [36] generalized DDPM through a class of non-Markovian diffusion processes, which generated implicit models with high-quality samples faster and greatly improved the iterative rate of DDPM.

In summary, a light-weighted model and guided super-resolution reconstruction are the current mainstream research directions of diffusion models. These techniques are worth investing in to achieve a good trade-off between the computations and the output image quality.

3. The Proposed Method

We propose a diffusion-based image super-resolution model, ResShift-4E, which can magnify the input low-resolution (LR) image by a factor of 4 to output an HR image. It has obvious advantages over the ResShift-15 model [33]. The proposed ResShift-4E significantly reduces the hardware requirements and runtime while maintaining a higher level of super resolution.

Figure 1 shows an overview of our ResShift-4E model. It reduces the diffusion step of the ResShift model to 4 steps and constructs a loss function $L_{contrast}$, which focuses on the original residuals by adding a residual structure between the initial LR image and the target HR image. The reason is derived from the fact that the diffusion model can better learn the objective difference between the LR image and the HR image, so as to achieve the effect of limiting the diffusion model’s overly powerful random generation ability. It is expected to improve the objective evaluation metrics, and to ensure that the subjective evaluation metrics are stable enough.

In the Figure 1, $x_T$ denotes the initial state of the model and ${\widehat{x}}_0$ denotes the inference result of the model with $N(y,k^{2}I)$ and $p_{\theta }\left(x\right|x_T,y)$ distribution, respectively, where y is an intermediate state in the Markov chain and may represent a feature map, a specific parameter, or a set of transformed image features depending on the model architecture; during the forward diffusion process from ${\widehat{x}}_0$ to $x_T$, the model learns the unknown image degradation by random Gaussian sampling with the sampling parameters ${\mu }_{\theta }(x_t,y,t)$ and $ϵ$. The parameter $ϵ$, which dominates the randomness, follows the distribution of ${\sum }_{\theta }(x_t,y,t)$.

In contrast with traditional diffusion models that generate images from scratch, the ResShift model uses a degraded image as the initial input, embedding it in the initial state. We follow the diffusion structure of ResShift and reduce the number of diffusion steps. A contrast loss function is added at the end of the forward diffusion process to strengthen the constraints on random sampling.

In the original ResShift model, the forward diffusion process is as below

$$q\left(x_t\right|x_{t-1},y_0)=N(x_t;x_{t-1}+{\alpha }_t{e}_0,k^2{\alpha }_{t}I),t=1,2,\dots ,T$$

(1)

Compared with this, the ResShift-4E Markov chain transition formula for the forward process is modified as follows:

$$q\left(x_t\right|x_{t-1},y_0)=N(x_t;x_{t-1}+{\alpha }_t{e}_0,k^2{\alpha }_{t}I,(1-{\eta }_t)e_0),t=1,2,\dots ,T$$

(2)

The newly added transition parameter $(1-{\eta }_t)e_0$ is a value that is independent of random noise and decreases with increasing diffusion steps. Thus, each diffusion step adds an offset towards the original residuals, not just the residuals from the HR image at the diffusion step, and the effect of this offset gradually decays as the diffusion progresses, ensuring that the simulated noise is not overfitting. Similarly, in order to strengthen the influence of the original residual features on the diffusion model, a contrast loss function $L_{contrast}$ is added. It is defined as follows:

$$\begin{array}{cc}\hfill & L_{contrast}(y,x_0,x_t)={\displaystyle \frac{1}{N}}\sum _{n=1}^N\beta D_W^2(x_0,x_t)+(1-\beta )max{(m-D_w(y,x_t),0)}^2\hfill \\ & D_W(x,y)={\parallel x-y\parallel }_2={(\sum _{i=1}^P{(x^i-y^i)}^2)}^{{\displaystyle \frac{1}{2}}}\hfill \end{array}$$

(3)

$D_W$ denotes the Euclidean distance between two samples; P denotes the feature dimension of the sample; $\beta$ is the weight of the comparison with HR image and LR image, respectively, under the diffusion step already taken; m is the LR comparison threshold; and N is the number of samples. Specifically, $\beta$ controls the regularization strength to prevent overfitting, while m determines the degree of smoothness in the image processing pipeline. Of course, the optimization of the model weights is still mainly determined by the form of diffusion in ResShift, which usually needs to minimize its negative variational lower bound, i.e.,:

$$\underset{\theta }{min}\sum _t{D}_{KL}[q\left(x_{t-1}\right|x_0,x_t,y_0)\parallel p_{\theta }\left(x_{t-1}\right|x_t,y_0)],$$

(4)

where $D_{KL}[·||·]$ stands for Kullback–Leibler (KL) scatter, and more mathematical details can be found in Sohl-Dickstein et al. [39] or Ho et al. [29] Combining Equations (2) and (3) yields the new loss function we defined, which is used to update the model weights.

4. Experiments

In this section, we first introduce our Multimodal High-resolution Microscopy Images (MHMI) dataset. Then, our method is compared with BSRGAN [40], Real-ESRGAN [41], and other models. The experiments on the MHMI dataset are evaluated in terms of subjective and objective metrics. Finally, the strengths, weaknesses, and practical prospects of this application are discussed.

4.1. The MHMI Dataset

In this paper, we collected and constructed a multimodal pure optical microscopy image dataset called Multimodal High-resolution Microscopy Images (MHMI). Their scale bars are 0.5 μm per pixel. Advanced optical microscopes boost modern biology, which requires necessitating specific image processing algorithms to enhance microscopy images. MHMI is constructed through multiple helpers. It consists of 1220 images, including images captured by four microscopy techniques, namely, differential interference contrast (DIC), fluorescence, phase contrast, and brightfield. It is believed that this dataset can be used to research deep-learning-based microscopy image processing. It can be accessed by the link https://github.com/JimmCao/ResShift-4E (accessed on 21 January 2025).

The MHMI dataset comes from the following three main sources: (a) 240 ground truth images selected from the Fluorescence Microscopy Denoising (FDM) [42] dataset; (b) 920 labeled images selected from the NeurIPS22-CellSeg (2022 Weakly Supervised Cell Segmentation Images) [43] dataset; and (c) 60 real-captured fluorescence microscopy images acquired from laboratories specializing in optical microscopy.

As shown in Figure 2, based on the optical microscopy classification, we visualize the four microscopy images after filtering and cropping the images to show their main features. Among them, the fluorescence images and phase contrast images are mainly 512 × 512 pixels, the differential interference contrast images are mainly 1024 × 1024 pixels, and the brightfield images are mainly 2048 × 1536 pixels. We divide the MHMI dataset into the training set, validation set, and test set. The total number for the training set, validation set, and test set is 1000, 120, and 100, respectively.

As in Figure 3, we downsampled the high-resolution images in the dataset using bicubic interpolation and add random Gaussian blur, where the parameter of downsampling is four times, and the blur parameter interval is taken as [0.45, 2], in which the two values represent the minimum and maximum values of the blur interval, respectively. To facilitate visual comparison and training, we uniformly crop the high-resolution images of the validation set and the test set to 512 × 512 pixels, and, correspondingly, the LR of the low-resolution images is 128 × 128 pixels. Metrics such as PSNR, LPIPS, and NIQE are based on the comparison of high-resolution image (HR) and Ground Truth (GT).

4.2. Experiment Settings

Experiments are conducted under the same settings. The operating system is Ubuntu 22.04, and the CPU is an Intel Xeon E5-2673. The used GPUs are four NVIDIA GeForce GTX 1080Ti GPUs, each with 11 GB of memory. For GPU acceleration, CUDA 11.7 and cuDNN 8.5 are employed. The programming language is Python 3.10, and the deep learning framework used is Pytorch 2.0.1.

For the hyperparameters, “degradation_type” is set to “real-esrgan” degradation channel, which consists of two identical degradation steps containing the same fuzzy kernel with multiple random noise range values. The “crop_pad_size” is set to 300 (pixels), which indicates that the oversized images in the training set are cropped into small images of less than 300 × 300 pixels to improve the overall learning efficiency and training speed of the model. In addition, “iterations”, “microbatch”, “epoch”, “warmup_iterations”, “milestones”, and “save_freq” are set to 1000, 2, 200, 5000, [68,000, 90,000], and 10,000, respectively. Moreover, we crop the high-resolution images of the validation and test sets into small images of 512 × 512 pixels, which is convenient for quick validation, testing, and subsequent comparison.

4.3. Effect of Different Noise and Blur on Training Results

The resolution of optical microscopy images is usually limited by diffraction, so optical super-resolution methods are used to obtain a higher resolution. These methods utilize techniques in the frequency or spatial domain to obtain magnified images by separating light of different wavelengths and performing band broadening operations on the captured light. However, such super-resolution methods may introduce problems such as the irregular noise and blurring, which can affect the output super-resolution quality. During the training process, different types of noise and blurring can have a large impact on the super-resolution results, and such ablation experiments are quite necessary. Our comparison results of ResShift-4E on MHMI are shown in

Table 1. Test results of ResShift-4E on MHMI under different noise and blur.

PSNR↑				LPIPS↓
Noise	Blur			Noise	Blur
	[0.3,1.5]	[0.2,1.5]	[0.1,1.2]		[0.3,1.5]	[0.2,1.5]	[0.1,1.2]
0.50	29.74	29.95	29.83	0.50	0.4414	0.4249	0.4300
0.45	29.70	30.00	29.79	0.45	0.4338	0.4217	0.4246
0.40	29.66	29.96	29.69	0.40	0.4338	0.4241	0.4251
NIQE↓
Noise		Blur
		[0.3,1.5]		[0.2,1.5]		[0.1,1.2]
0.50		10.2901		10.2517		10.8768
0.45		9.8009		10.0176		10.7537
0.40		9.6013		9.8927		10.2258

, Figure 4 and Figure 5.

As shown in Table 1, to quantitatively evaluate the performance of ResShift-4E, we calculate the Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS) [44], and Natural Image Quality Evaluator (NIQE) [45] under different noises and blurs. In the degradation model, the Gaussian noise is a mean value, the Gaussian blur is an interval, and the actual blur assigned to each image is randomly selected within this interval. It can be seen that the optically processed microscopy images are not sensitive to noises. A Gaussian noise difference of 0.05 only has a fluctuation of about 0.05 on the PSNR, LPIPS, and other evaluation metrics of the referenced images. The best values of both are reached when noise is 0.45 and blur is in [0.2, 1.5] at the same time.

We find that the smaller the noise and the larger the blur, the better the performance of NIQE, which reaches the best value at noise of 0.40 and blur in [0.3, 1.5]. Based on the performances in Figure 4 and Figure 5, it can be inferred that the larger the blur and the smaller the noise, the better the ResShift-4E model is able to simulate the granularity of the original GT image. It is quite important in the analysis and research of microscopy images, even though it may lead to a certain degree of over-sharpening.

Moreover, we subsequently evaluated the results by several professionals in computer and optical microscopy, and finally chose the training results with noise of 0.50 and blur in [0.2, 1.5] as our benchmarks for application.

4.4. Visual Results

Blind super-resolution models based on diffusion models often provide better visualization results. In order to further verify the application advantages of the ResShift-4E model, we choose the GAN-based blind super-resolution models such as BSRGAN [40] and Real-ESRGAN [41]. ResShift-4E is superior to the compared methods on three main types of microscopy images, and the comparison results are shown in Figure 6, Figure 7 and Figure 8. In which, the images labeled “GT” are the cropped, zoomed-in panels for the original photo, and “GT” means ground truth.

To ensure fairness, we trained the BSRGAN [40] model and the Real-ESRGAN [41] model using the MHMI dataset as well, and then compared the three models on the MHMI test set. It can be seen that the ResShift-4E model shows better results in fluorescence images, brightfield images, and phase-contrast images. Compared with BSRGAN [40] and Real-ESRGAN [41], which focus on local features, our ResShift-4E model makes good use of the diffusion model’s advantage of focusing on the whole noise, and achieves a better balance between local and whole. It demonstrates that our ResShift-4E is competitive when applied to super-resolution of microscopy images.

In most cases, our ResShift-4E model produces good super-resolution results on the MHMI dataset and has a clear advantage. However, our model has poor super-resolution results on differential interference micrographs. Subsequent optimization for diffusion-model-based super-resolution algorithms is needed.

5. Conclusions

In this paper, we propose a diffusion model called ResShift-4E for blind super-resolution. The main novelty is optimizing ResShift by reducing diffusion steps and strengthening the influence of the original residuals. We also construct a Multimodal High-resolution Microscopy Image (MHMI) dataset and apply our ResShift-4E model to MHMI, which experimentally demonstrates the advantages over other models. The proposed model can be used for automatic image enhancement, noise reduction, or resolution enhancement in biological imaging, medical diagnostics, or materials science.

In the future, we will further cooperate with laboratories in related fields to collect relevant data, and expand the scale of various types of microscopic images. In addition to expanding the data, another expectation of our work lies in optimization of the original framework of ResShift. For instance, by combining various acceleration algorithms like the Teacher–Student model, we could even reduce diffusion steps to 1 step, which will greatly improve its performance. We will also explore how ResShift-4E can be integrated into current microscopy workflows, either as a standalone tool or as part of an existing pipeline, improving processing efficiency, image quality, and overall workflow throughput.