Study on Joint Intensity in Real-Space and k-Space of SFS Super-Resolution Imaging via Multiplex Illumination Modulation

Abstract

This paper studied the general mechanism of spatial-frequency-shift (SFS) super-resolution imaging based on multiplex illumination modulation. The theory of SFS joint intensity was first proposed. Experiments on parallel slots with discrete spatial frequency (SF) distribution and V-shape slots with continuous SF distribution were carried out, and their real-space images and k-space images were obtained. The influence of single illumination with different SFS and mixed illumination with various combinations on SFS super-resolution imaging was analyzed. The phenomena of sample SF coverage were discussed. The SFS super-resolution imaging characteristics based on low-coherence illumination and highly localized light fields were discovered. The phenomenon of image magnification during SFS super-resolution imaging process was discussed. The differences and connections between the SF spectrum of objects and the k-space images obtained in SFS super-resolution imaging process were explained. This provides certain support for optimization of high-throughput SFS super-resolution imaging.

1. Introduction

Due to the diffraction limit of optical microscopy systems, the resolution of optical microscopes is generally around half of the wavelength [1]. For example, in the visible light spectrum, the resolution is restricted to approximately 200 nm. Thus, many researchers have begun to study super-resolution microscopies to break the diffraction limit, including laser scanning confocal microscopy (LSCM) [2,3], stimulated emission depletion microscopy (STED) [4,5], single-molecule localization microscopy (SMLM) [6,7] and so on. Regardless of the type of super-resolution microscopy, it is necessary to detect the high spatial frequency (SF) information of the sample beyond the diffraction limit using an optical system with a cut-off SF. A common method is to modulate the SF of the illumination field interacting with the sample, which shifts the high SF information of the sample into the cut-off SF range of the optical system. Hence, super-resolution imaging in far-field detection based on spatial-frequency-shift (SFS) effect was developed, such as structured illumination microscope (SIM) [8,9,10,11,12,13], Fourier ptychographic microscopy (FPM) [14,15,16,17,18,19,20] and on-chip SFS super-resolution imaging [21,22,23,24]. The resolution improvement of the SFS super-resolution imaging technology based on illumination modulation depends on the transverse wave vector (k_x) of the complex illumination field. The SIM and FPM depend on the illumination modulation with k_x provided by the free-space propagation wave, which is always smaller than the free-space wave vector k₀. To achieve a higher resolution, it is necessary to modulate and generate an illumination mode with a higher k_x. Therefore, on-chip SFS super-resolution imaging technologies were proposed, which used illumination modulation with a higher transverse wave vector k_x provided by the evanescent wave. However, such an illumination field has extremely high localization and weaker photon intensity. These two characteristics will lead to weaker intensities in both the k-space and real space in far-field detection, which will have a certain impact on the image quality and resolution of SFS super-resolution microscopic imaging. Based on the general SFS illumination mechanism, this work proposed the theory of joint intensity of real-space intensity and k-space intensity for SFS super-resolution imaging technology based on multiplex illumination modulation. The influence of the multiplexing of different illumination modes on super-resolution imaging was analyzed, providing certain theoretical support for the development of more effective illumination modulation methods and weak-signal demodulation techniques in SFS super-resolution microscopy.

2. Discussion and Results

2.1. Experimental Setup

We built a dual-channel optical SFS super-resolution microscopic imaging system, as shown in Figure 1 and Figure S1. The objective lens is 0.4 NA/×20. The pixel size of the camera in the real space is 1.85 μm. A programable LED array with a center wavelength of 530 nm and a full width at half maximum of 30 nm is set as the low-coherence illumination light source, which can provide vertical illumination and oblique illumination, leading to wide-field far-field imaging and SFS super-resolution imaging. All the LEDs are encoded, which means they can be lit individually or simultaneously with different combinations to generate the various modulated illumination modes. The LEDs illuminate the object through the waveguide, which can support evanescent illumination mode with high k. Then, through a beam splitter (BS), one path of the scattering field forms an image of the object, which is the real-space image of the object; the other path forms an image of the back focal plane (BFP) of the objective lens. The BFP of the objective lens possesses unique Fourier transform characteristics in optical systems. According to Fraunhofer’s diffraction theory, when the light passes through an object, the diffracted rays at different angles (corresponding to different SF) are converged by the objective lens to distinct positions on the BFP. This one-to-one mapping relationship establishes the BFP as the Fourier transform plane of the incident light: the light intensity distribution at each point on the BFP directly reflects the intensity of the corresponding SF component of the object, thereby constituting the SF power spectrum of the object. Consequently, the BFP image essentially represents the k-space image of the object.

2.2. Theory of Joint Intensity

k-space intensity (KI) is the ratio of efficient photons in the object area (SFS_eff) to all photons in the upper interface of the waveguide from the LED. As shown in Figure 2a, the divergence angle of LED is θ. When the oblique incident angle is smaller than the critical angle, the propagation wave is used to illuminate the object area, which is regarded as SFS_kp. The KI is defined as SFS_eff/SFS_kp. When the oblique incident angle is larger than the critical angle, the highly localized evanescent wave is used to illuminate the object area, which is regarded as SFS_keva. The KI is defined as SFS_eff/SFS_keva. In our experimental setup, five LEDs are used as the light source. The first LED is centrally positioned beneath the sample area, providing vertical illumination with a transverse wave vector of k₀, which is the traditional illumination method. The other four LEDs are lined up in a row beginning with the first LED, providing oblique illumination. Therefore, the four LEDs induce different incident angles in the sample area. Total internal reflection occurs on the surface of the sample area when the incident angle exceeds the critical angle. The waveguide material is K9 glass with a refractive index of 1.52 at a wavelength of 530 nm (the influence of waveguide imperfections on SFS super-resolution imaging is detailed in Supplementary Materials S2). Then, the critical angle is 41°. To uniformly increase the incident angle from 0° to over 41°, the four incident angles φ are designated as 8°, 17°, 30°, 49°, providing transverse wave vectors of k₁, k₂, k₃, k₄. Notably, depending on whether the incident angle is larger than the critical angle, k₁–k₃ are the transverse wave vectors of propagation illuminations, while k₄ is the transverse wave vector of evanescent illumination. As plotted in Figure 2b, as the SFS becomes deeper, the KI drops steeply, especially at the point from propagation wave illumination to evanescent wave illumination. The real-space intensity (RI) was detected in the object area by an optical power meter (S130C, Thorlabs, Newton, MA, United States), as plotted in Figure 2b. Notably, although the different LEDs have slightly different intensities, all data were averaged over 10 independent measurements to enhance the generalizability of the results. In the range of propagation illumination, as the SFS goes deeper, the RI decreases slightly. From propagation illumination to evanescent illumination, the RI drops extremely steeply. Once the LED is switched on and then illuminates the sample, the image sensor in the real-space path can collect the real-space image, and another image sensor in the k-space path can obtain the k-space image. The illumination from LEDs has the properties of both KI and RI. Although KI and RI are independent, the obtained image is influenced by illumination intensity, which is the combined effect of KI and RI. To analyze it, joint intensity (JI) is proposed, defined as JI = RI × KI. As shown in Figure 2c, the JI decreases as the SFS goes deeper. Specifically, the JI of the fourth SFS (the deepest SFS, provided by evanescent illumination in the x-axis) is 100-fold weaker than that of the first SFS (provided by propagation illumination).

In our experiment, a series of parallel slots were used as objects to analyze the influence of illumination with multiplex SF modulation. The schematic of parallel slot distribution is illustrated in Figure 3j, where the periods of the slots are 860 nm, 815 nm, 659 nm, 491 nm, 351 nm, 304 nm, 243 nm and 237 nm. The origin of the coordinate system is set at the midpoint between the two slots. The object function of the slots can be expressed as

$$O(x)=\mathit{rect}({\displaystyle \frac{x-p/2}{l_w}})+\mathit{rect}({\displaystyle \frac{x+p/2}{l_w}})$$

(1)

where p is the period of the slots, and l_w is the line width of the slots. In the Fourier domain, this can be expressed as (a detailed derivation process is provided in the Supplementary Material S1)

$$F(\xi )=\mathcal{F}(O(x))=C\cdot \mathrm{sinc}(l_w\xi )\cdot \cos (\pi p\xi )$$

(2)

where C is a constant. Notably, only the term of $\cos (\pi p\xi )$ includes p, indicating that only this $\cos (\pi p\xi )$ term determines the SF of the slots, further demonstrating that the expected resolution of SFS super-resolution imaging required for the slots to be distinguished can be analyzed based on the $\cos (\pi p\xi )$ term, especially the p parameter.

Based on the periodicity of the cosine function, in the frequency domain, slots with high SF can cover some slots with low SF because high SF denotes a large period of cosine function in the frequency domain and a small spatial period of the slot structure in the spatial domain. In other words, in the spatial domain, if slots with small spatial periods can be resolved under the corresponding k_x, some slots with large spatial periods may also be resolved simultaneously.

Imaging resolution refers to the smallest distinguishable detail that an optical microscope system can resolve. The LEDs are used as the light sources; thus, this is the partial coherence imaging. The resolution falls in a lager range between coherent imaging and incoherent imaging. The coherent imaging resolution is calculated by

$${\Delta }_r=\lambda /(NA+N{A}_{illu})$$

(3)

The incoherent imaging resolution is calculated by

$${\Delta }_r=\lambda /1.67(NA+N{A}_{illu})$$

(4)

λ is the center wavelength of the illumination light. NA is the numerical aperture of the objective lens. NA_illu = n cos(φ) is the expanded numerical aperture from the illumination modulation. n is the refractive index of the waveguide material, and φ is the incident angle. k₀ = 2π/λ is the transverse wave vector provided by vertical illumination. k_xi = 2π NA_illu/λ is the transverse wave vector provided by oblique illumination. The expected resolutions are illustrated in

Table 1. The expected resolution of each SFS depth.

SFS Depth	Resolution
1	869–520 nm
2	624–373 nm
3	457–274 nm
4	342–205 nm

. The LEDs can be modulated to generate single illumination and mixed illumination with various combinations. When a single LED illuminates the objects, the light field interacts with the sample; the scattering field of the illuminated object is calculated as

$$U({\overrightarrow{r}}_{x,i})=O({\overrightarrow{r}}_x)E({\overrightarrow{r}}_{x,i})=O({\overrightarrow{r}}_x)\exp (i{k}_{xi}x)$$

(5)

where the complex amplitude of the object is $O({\overrightarrow{r}}_x)$, and the electric field of the illumination field is $E({\overrightarrow{r}}_{x,i})$.

The light field, which encompasses the complex information of the object after being collected by the microscope, travels to the plane of the image sensor, which can be represented as

$$G({\overrightarrow{r}}_{x,i})=U({\overrightarrow{r}}_{x,i})\otimes H({\overrightarrow{r}}_x)$$

(6)

where the transfer function of the linear time-invariant imaging system is $H({\overrightarrow{r}}_x)$, and $\otimes$ denotes the convolution calculation.

Then, the image sensor obtains the intensity of the above light field, which is denoted as

$$I({\overrightarrow{r}}_i)=|G({\overrightarrow{r}}_{x,i}){|}^2=|U({\overrightarrow{r}}_{x,i})\otimes H({\overrightarrow{r}}_x){|}^2$$

(7)

The Fourier transform of the intensity is calculated as

$$\mathcal{F}\left\{I({\overrightarrow{r}}_i)\right\}=\mathcal{F}\left\{|O({\overrightarrow{r}}_x)\exp (i{k}_{xi}x)\otimes H({\overrightarrow{r}}_x){|}^2\right\}$$

(8)

When multiple LEDs illuminate the object (multiplex SFS mode), which is a process of incoherent imaging with mixed SF, the image sensor records the superposition of intensities, which is expressed as

$${\displaystyle \sum I({\overrightarrow{r}}_i)}={\displaystyle \sum |G({\overrightarrow{r}}_{x,i}){|}^2}={\displaystyle \sum |U({\overrightarrow{r}}_{x,i})\otimes H({\overrightarrow{r}}_x){|}^2}$$

(9)

The corresponding Fourier form is calculated as

$$\mathcal{F}\left\{{\displaystyle \sum I({\overrightarrow{r}}_i)}\right\}=\mathcal{F}\left\{{\displaystyle \sum |O({\overrightarrow{r}}_x)\exp (i{k}_{xi}x)\otimes H({\overrightarrow{r}}_x){|}^2}\right\}$$

(10)

Based on Equation (9), when mixed illumination is executed, the real-space intensity recording by the image sensor depends on the sum of intensities of different SFS modes. Based on Figure 2c, the JI of the illumination with a highly localized field is greatly weaker than that of the illumination with a propagation field. Therefore, when mixed illumination with both a highly localized field and a propagation field is employed, its real-space images mostly show photon information in the spatial domain from the illumination provided by the propagation field. Moreover, based on Equation (10), the k-space images modulated by the illumination from multiplex LEDs also depend on the sum of JI, obviously dominantly influenced by the most intense JI.

2.3. Experimental Results and Discussion

As shown in Figure 3a, parallel slots cannot be resolved under vertical illumination. As shown in Figure 3b–e, parallel slots with corresponding periods matching the set SFS modes can be resolved under single oblique illumination with SFS1, SFS2, SFS3, SFS4, respectively, which are within the expected resolution ranges. Furthermore, based on the above analysis of the object function of slots, SF information of the SFS3 mode contains partial SF information of the SFS1 mode. And SF information of the SFS4 mode contains partial SF information of the SFS2 mode. Thus, under modulation by SFS3, as shown in Figure 3d, parallel slots with periods of 304 nm, 351 nm, 860 nm can be distinguished, covering the partial expected resolution under the SFS1 mode. Under modulation by SFS4, as shown in Figure 3e, slots with periods of 237 nm, 243 nm, 659 nm can be distinguished, covering the partial expected resolution under the SFS2 mode. Moreover, Figure 3k–n plot the intensity profiles of parallel slots through dot lines. The colors of the intensity profiles correspond to the colors of the dot lines in Figure 3b–e. For the same parallel slots, the photon intensity in the image plane drops with the SFS going deeper. The intensity profiles clearly illustrate that the slots with the expected periods are all distinguishable under the corresponding SFS modes. The intensity of real-space image analysis reveals significant variations across different SFS modes: SFS1 exhibits 1.71×, 6.86× and 40× stronger intensity compared to SFS2, SFS3 and SFS4 modes, respectively, in their resolved slots.

Interestingly, super-resolution imaging with multiplex SFS modes produces different properties compared to using the single SFS mode, including image intensity and resolution. As shown in Figure 3f, only parallel slots with the 659 nm period (they are the expected resolution of the SFS2 mode, and they can also be resolved under the SFS4 mode) are resolved under the mixed illumination of SFS2 and SFS4. This is because the JI of the illumination with SFS2 is 85-fold stronger than that of SFS4. The SFS4 mode is depressed, and the information of slots with the expected resolution of the SFS2 mode and the SFS4 mode is dominant. As shown in Figure 3g, under the mixed illumination of SFS3 and SFS4, parallel slots with the 860 nm period (they can be resolved under the SFS3 mode) are vaguely distinguished, while parallel slots within the expected SFS3 and SFS4 resolution cannot be distinguished. Because the JI of the illumination with SFS3 is 60-fold stronger than that of SFS4, the SFS3 mode and the SFS4 mode are mutually inhibitory to some extent, though the SFS3 mode is slightly dominant. As shown in Figure 3h, none of the parallel slots are distinguished under the mixed illumination of SFS2, SFS3 and SFS4. The JI of the illumination with SFS2 is 1.42-fold stronger than that of SFS3; thus, the SFS4 mode is inundated by the SFS2 mode and the SFS3 mode. And the SFS2 mode and the SFS3 mode are mutually constrained. As shown in Figure 3i, only parallel slots with the 860 nm period (they are the expected resolution of the SFS1 mode, and they can also be resolved under the SFS3 mode) can be slightly distinguished under the mixed illumination of SFS1, SFS2, SFS3 and SFS4. The JI of the illumination with SFS1 is 1.16-fold stronger than that of SFS2. Therefore, the SFS1 mode, the SFS2 mode and the SFS3 mode are mutually constrained to some extent, and the SFS4 mode is inundated by the SFS1 mode, the SFS2 mode and the SFS3 mode, though the SFS1 mode is slightly dominant.

The above parallel slots sample with different definite periods has a discrete SF distribution. Therefore, the experiment of the parallel slots sample can clearly calibrate the resolution of the imaging system. However, it fails to fully reflect the imaging resolution range of each SFS mode. Moreover, it cannot intuitively demonstrate the image magnification issue in the SFS super-resolution imaging process. Thus, V-shape slots with continuously changing periods were selected as the sample for our SFS super-resolution imaging. As shown in Figure 4b, the V-shape slots in the yellow box cannot be distinguished at all under wide-field vertical illumination. The image of V-shape slots obtained via field emission scanning electron microscopy (FESEM, Zeiss Ultra 55, Oberkochen, Germany) is shown in Figure 4a, whose resolution is within the range from 200 nm to 1200 nm. The following super-resolution images of V-shape slots obtained via modulated illuminations all illustrate the yellow box parts. In SFS super-resolution imaging, the modulated scattering photons carrying high SF information of the objects are collected by a microscope and then coupled into the image sensor. Hence, the high SF is shifted into the passband of the objective lens and reflected in the digital images provided by the image sensor. Objects with high SF information have sub-diffraction-limit structures in the spatial domain, and these fine structures are distorted the real size because of the optical magnification provided by the SFS method. In other words, SFS modulation provides extra magnification to the traditional microscope. Therefore, the magnification factor of SFS super-resolution imaging can be defined as

$$M_{SFS}=(k_c+k_{xi})/k_c$$

(11)

k_c is the transverse wave vector of the cut-off SF of the objective lens, and k_xi is the transverse wave vector of modulated illumination. The relationship between the pixel number (px_num) of SFS images and the real size of the objects l_r is

$$l_r=(\mathit{px}\_\mathit{num}\times \mathit{px}\_\mathit{size}/M_{\mathit{obj}})/M_{\mathit{SFS}}$$

(12)

where px_num denotes the center-to-center pixels of a pair of slots; px_size is the pixel size of the image sensor; and M_obj is the magnification of the objective lens.

As shown in Figure 4c, under the illumination with SFS1, the continuous area of V-shape slots around the orange arrow is resolved. The normalized intensity profile pointed by the orange arrow is plotted in the inset, indicating there are 11 pixels between two peeks (the distance between the middle of a pair of slots). Based on Equation (12), the period of the slots is 841 nm, which is within the expected resolution range. As shown in Figure 4d, under the illumination with SFS2, the continuous range of V-shape slots around the yellow arrow is resolved. The normalized intensity profile pointed by the yellow arrow shows 10 pixels, demonstrating that the period of the slots is 638 nm, which is also within the expected resolution range. Notably, it can be observed that there is a significant magnification in the distinguishable areas. As shown in Figure 4e, under the illumination with SFS3, two continuous areas are distinguished, where the resolutions are within the range of the SFS1 mode and the SFS3 mode. Compared with the orange area under the SFS1 mode, the orange area under the SFS3 mode has a more obvious magnified distortion. In other words, as the SFS becomes deeper, the magnification increases, which is consistent with Equation (11). The normalized intensity profile pointed by the green arrow is plotted in the inset, showing there are nine pixels between the peaks. After calculation, the resolution at the green point is 473 nm. The reason for the calculated resolution (473 nm) being slightly distant from the resolution range of SFS3 (274–459 nm) is imprecise computations due to the larger pixel size of the image sensor. As shown in Figure 4f, under the illumination with SFS4, similar phenomena are observed as with SFS3, but the intensity is smaller. And two continuous areas around the yellow and pink arrows are resolvable. The normalized intensity profile illustrates that there are eight pixels between the peaks at the pink point, indicating that the corresponding resolution is 344 nm, which is in line with expectations.

Furthermore, the multiplex SFS modes for V-shape slots are also studied. As shown in Figure 4g, under the mixed illumination of SFS2 and SFS4, the yellow area is magnified and distinguished. As shown in Figure 4h, under the mixed illumination of SFS3 and SFS4, the orange area is magnified and distinguished. As shown in Figure 4i, under the mixed illumination of SFS2, SFS3 and SFS4, the orange area is subtly magnified and slightly distinguished. As shown in Figure 4j, under the mixed illumination of SFS1, SFS2, SFS3 and SFS4, the orange area is subtly magnified and clearly distinguished. These results validate the conclusion from the parallel slots experiment, and furthermore, they demonstrate the situations encountered in objects with continuous SF.

While obtaining the real-space images of the object, its k-space images are also obtained. The parallel slots and V-shape slots are in the same substrate of the sample. Therefore, for the parallel slots and V-shape slots, the same k-space images are studied. Figure 5a shows the SF spectrum image of the object, which is obtained via Fourier transform from the SEM image of the object. The parallel slots and V-shape slots only contain SF information in the horizontal and vertical directions, as expected. The SEM image has a high resolution, resolving slots with both small and large periods. Thus, the SF spectrum image of the object contains low SF information and high SF information. The k-space image under wide-field vertical illumination is shown in Figure 5b, containing only cut-off SF information with four mutually perpendicular directions. The k-space images under single illumination with SFS1, SFS2, SFS3 and SFS4 are shown in Figure 5c–f. As the SFS goes deeper, the higher SF information shifts into the cut-off SF passband of the objective lens to the image sensor. The SF information in each SFS step corresponds to each part of the SF spectrum of the object in Figure 5a. Additionally, as the SFS goes deeper, the intensity of the k-space drops according to the JI. Furthermore, k-space images under multiplex SFS modes are studied, as shown in Figure 5g–j. The dominant role of SFS with a high JI can be observed, along with the mutual constraints and superposition effects of different SFS modes. These results in the k-space images are consistent with findings from the real-space images.

3. Conclusions

In conclusion, our work studied the mechanism of SFS super-resolution microscopic imaging based on illumination modulation. The SFS super-resolution imaging system was built to obtain real-space and k-space super-resolution images. The JI theory based on RI and KI was proposed to study its influence on SFS super-resolution imaging. As the SFS goes deeper, the JI drops. Especially at the point where the illumination changes from the propagation wave field to the evanescent field, the JI decreases significantly. Parallel slots with discrete SF distribution and V-shape slots with continuous SF distribution were used as the imaging objects. The effects of modulated illumination (different single SFS illumination and various mixed illumination) on SFS super-resolution imaging were explored. Partial coherent imaging illuminated by LEDs was observed. The phenomenon of high SF being able to cover part of low SF was revealed. The magnifying issue during SFS super-resolution imaging process was explained. The k-space images of parallel slots and V-shape slots were also studied. The effects of JI on the k-images were discussed. This also intuitively revealed that high SF information beyond the diffraction limit can be transferred into the objective lens passband through modulated illumination, thus achieving SFS super-resolution imaging. Any sample can be regarded as a superposition of numerous gratings with various frequencies, amplitudes and phases. Parallel slots and V-shape slots are the simplest gratings. Therefore, the theory of parallel slots and V-shape slots is possibly suitable for other complex samples. This work provides a certain basis for further optimization of SFS super-resolution microscopy imaging.