In Situ Measurement Method Based on Edge Detection and Superpixel for Crystallization Imaging at High-Solid Concentrations

Abstract

To facilitate measuring crystal sizes during batch crystallization at high-solid concentrations by using an invasive imaging system, an in situ imaging measurement strategy based on edge detection and superpixel is proposed for the ambiguous boundary problem of large amounts of crystals. Firstly, an image filtering is employed to cope with image degradation caused by noise disturbance and suspension turbulence in the crystallizer. Subsequently, an image segmentation method is developed by utilizing improved edge detection and superpixel, which can be easily performed for crystal extraction. Accordingly, crystal size measurement can be developed for evaluation of the crystal size distribution. The experiment results on α-form L-glutamic acid present the effectiveness of the proposed method.

1. Introduction

Crystallization has been widely applied across the majority of solid product industries, including the pharmaceutical, food, microelectronics and materials industries [1]. For maintaining a desired quality of the crystal products, it is necessary to monitor crystal growth characterization for automation control and operation optimization [2,3,4]. The crystal size distribution (CSD) and particle shape (PS) as the key properties of crystal products have a considerable impact on final product quality as well as the downstream processes [5,6].

With the rapid progress in high-speed digital imaging sensors, there is enormous potential for using on-line imaging techniques to monitor crystal features during crystallization processes [7,8,9,10]. The on-line imaging system of crystallization can be divided into two major classes: invasive system and non-invasive system [5,11,12,13]. The invasive systems with the probe-like design can be inserted into the solution to capture high-quality images, whereas the non-invasive systems are placed outside the reactor to avoid the contamination of the camera lens. In early research, Wilkinson et al. [14] developed an on-line imaging instruction for crystallization monitoring instead of using off-line measurement instruments, which were not capable of real-time particle measurement. In the last decade, on-line, image-processing strategies have become popular for product quality with desired characteristics including CSD and PS in an in-reactor environment [11,15,16]. Researchers have attempted to develop the advanced crystal segmentation methods, which are equally important for improving the robustness of crystal measurement. Calderon De Anda et al. [15] proposed an effective multi-scale segmentation method for extracting crystal blocks from the ambiguous background of an on-line image. Cardona et al. [6] proposed an advanced image-processing framework based on filtering and edge detection to extract size and shape information from in-line images with Mettler Toledo Particle Vision and Measurement (PVM). In addition to the abovementioned edge detection, the image segmentation methods based on thresholding were also employed for crystal images [16]. Zhang et al. [17] presented a novel image-processing technique based on combining wavelet transform and fuzzy C-means clustering for crystal image segmentation, and the crystal size distribution was described by the Weibull density probability function. Gao et al. [18] utilized a deep-learning method based on Mask R-CNN to detect L-glutamic acid (LGA) crystals and to measure the sizes of crystals. However, these algorithms did not take into account high-concentration crystallization, resulting in the occurrence of overlapping particles especially in industry production. To deal with the overlap of needle-like particles, Larsen et al. [19] proposed a SHARC (segmentation for high-aspect-ratio crystals) algorithm to extract the crystals effectively for particle-size distribution information from images of high-concentration slurry. Ferreira et al. [20] developed a novel image-analysis technique that combined discriminant factorial analysis to assess the agglomeration of crystals. In our previous work [21], an image-based analysis technology was presented for segmentation of overlapping needle-like crystals. However, crystal segmentation at higher concentration can be still confronted with a number of bottlenecks [2], in particular for poor-imaging conditions and large amounts of overlapped crystals with ambiguous boundaries during crystallization. Most of the previous work focused on needle-like crystals. The proposed method can be used to segment prismatic and blocky crystals.

In this work, considering the fact that a mass of crystals are overlapping in a stirred suspension at high-solid concentrations, an image analysis methodology based on superpixel segmentation is proposed for crystal-size distribution. This systematic, image-analysis method includes primarily image filtering, image segmentation and crystal size measurement. The raw images are initially recorded with the invasive imaging instrument, which can avoid the presence of out-of-focus crystals. Furthermore, for the on-line crystal images influenced by noise disturbance and solution turbulence, image preprocessing is used with image filtering. Then, the valid crystals are extracted by using an effective, image-segmentation method based on Canny edge and superpixel for multiple touching particles. In addition, the crystal sizes can be effectively calculated after size calibration. Finally, experiments on α-form LGA demonstrate the effectiveness of the proposed image measurement method.

2. Experimental Setup

2.1. Material

L-glutamic acid (LGA), C₅H₉NO₄, was used for this work. The LGA crystadls are known to have two shapes [22]: the prismatic α and needle-like β forms. Due to the needle-like β-form being prone to breakage, α-form LGA crystals were considered to crystallize for demonstrating size measurement of prismatic crystals by the imaging analysis technique, as shown in Figure 1. It is seen that the α-form crystal is prismatic in the schematic representation. The LGA crystals with a purity of 99% were taken as the solute, and distilled water was used as the solvent in this experiment.

Figure 1. The crystal morphology of α-form LGA: (a) crystal image of α-form LGA; (b) schematic representation of α-form LGA.

2.2. Experimental Setup

Experiments were carried out with a 2 L glass jacketed crystallizer, a PTFE four-paddle agitator and a Pt100 temperature probe. To control the temperature, a Julabo-CF41 thermostatic circulator (JULABO, Seelbach, Germany) was used, as shown in Figure 2. In this study, an invasive imaging system (2D vision probe, PharmaVision Nanosonic Technology Ltd., Qingdao, China.) was able to record 60 images per second with a pixel resolution of 1600 × 1200. The camera probe was situated into the suspension to avoid reactor-wall effects on the images and to allow the imaging regions of crystals to be fixed to reduce out-of-focus visualizations. It was set to capture two images per second. In addition, a microscope device (Leica DM 2500, Leica Microsystems, Wetzlar, Germany) with LAS software (LAS v4.4, Leica Microsystems, Wetzlar, Germany) was employed to measure the crystal sizes for effectiveness verification. In the verification experiment, a batch of crystal products was firstly measured using the off-line microscope, and subsequently, the same batch was put into the stirred reactor. The on-line images were recorded and analyzed immediately in the first fifteen seconds with the in situ method.

Figure 2. Schematic drawing of the experimental setup.

The α-form LGA crystals (Sigma Chemicals, St Louis, MO, USA) were taken from the solution in a stirred reactor. According to the dissolution characteristics of LGA in water, the measurement range of solution concentration was set as 9.0–39.0 g/L, and the temperature range was 75–15 °C. In the experiment, 1.2 L LGA solution was injected into the 2 L reactor, and the agitator stirred at 200 rpm. Solution concentration was measured with ATR-FTIR spectrometer (Mettler Toledo Ltd., Zurich, Switzerland). The solution with a concentration of 33 g/L was heated to 75 °C to dissolve all LGA crystals. The solution was then linearly cooled down to the temperature 35 °C at a constant speed of 1 °C/min.

3. Crystal Image Processing

The goal of the crystal image processing methodology was to measure the crystal sizes based on the crystal images. Image segmentation is a key step in separating the crystals and the background for size measurement. The Canny edge detection method can result in an excellent performance for crystal images [15]. However, Canny edge detection method for crystal images at high-solid concentrations may face a major problem in that it is difficult to segment overlapping crystals. Therefore, considering that different crystals have different grayscale or texture, a simple linear iterative clustering (SLIC) method [23] as an efficient and simple superpixel algorithm was introduced to generate compact and nearly uniform superpixels. The steps of improved segmentation based on Canny detection and superpixel are as shown in Figure 3.

Figure 3. The illustration of the proposed segmentation algorithm.

3.1. Image Filtering

Noise is the important interference for the analysis of crystal image. For this reason, bilateral filtering [24] is popular filtering technique used to eliminate the noise in crystal images. The filtering process is as follows:

Firstly, the noise model of image is expressed by

$$g(x,y)=f(x,y)+n(x,y)$$

(1)

where $f(x,y)$ is the denoising image, $n(x,y)$ is the noise and $g(x,y)$ is the raw image.

The bilateral filter makes use of the local weighted average method to obtain the restored image pixel value $f(x,y)$ as

$$f(x,y)=\frac{{\displaystyle \sum _{(i,j)\in S_{x,y}}w(i,j)g(i,j)}}{{\displaystyle \sum _{(i,j)\in S_{x,y}}w(i,j)}}$$

(2)

where $S_{x,y}$ represents the neighborhood (size: $(2N+1)\times (2N+1)$) of the central point $(x,y)$, $(i,j)\in S_{x,y}$. For each point $g(i,j)$ within the neighborhood, $w(i,j)$ is defined by

$$w(i,j)=e^{-(\frac{|i-x{|}^2+|j-y{|}^2}{2{\sigma }_S^2}}{}^{+\frac{|g(i,j)-g(x,y){|}^2}{2{\sigma }_r^2})}$$

(3)

It is noted that $N$ denotes the neighborhood size, ${\sigma }_S$ denotes the spatial domain factor and ${\sigma }_r$ denotes the value domain factor. In this work, $N=5$, ${\sigma }_S=3$ and ${\sigma }_r=0.1$.

3.2. Improved Canny Segmentation

For an image $f(x,y)$, the smoothed image $g(x,y)$ is obtained with an appropriate Gaussian filter $h(x,y,\sigma )$ as

$$g(x,y)=h(x,y,\sigma )\ast f(x,y)$$

(4)

where $h(x,y,\sigma )=\frac{1}{2\pi {\sigma }^2}{e}^{-\frac{x^2+y^2}{2\sigma }}$ and $\sigma$ is the Gaussian filter parameter.

The gradient magnitude $M[x,y]$ and gradient direction $\theta [x,y]$ at the pixel $(x,y)$ are

$$\{\begin{array}{l}M[x,y]=\sqrt{G_x{(x,y)}^2+G_y{(x,y)}^2}\\ \theta [x,y]=\arctan (G_x(x,y)/G_y(x,y))\end{array}$$

(5)

where $G_x$ and $G_y$ are the gradient arrays defined as

$$\{\begin{array}{l}{G}_x=[f(x+1,y)-f(x,y)+f(x+1,y+1)-f(x,y+1)]/2\\ G_y=[f(x,y+1)-f(x,y)+f(x+1,y+1)-f(x+1,y)]/2\end{array}$$

(6)

If the gradient magnitude $M[x,y]$ at the pixel $(x,y)$ is less than those at its two neighbors in the same gradient direction $\theta [x,y]$, the pixel $p(x,y)$ is marked as a candidate edge point. Otherwise, the pixel $p(x,y)$ is discarded. On this basis, the final edge points $T(x,y)$ are further defined by high threshold $T_{\mathrm{H}}$ and low threshold $T_{\mathrm{L}}$, $(T_{\mathrm{L}}=0.4T_{\mathrm{H}})$. The strong edge points are defined with $T_{\mathrm{H}}$, and the weak edge points are defined with $T_{\mathrm{L}}$. Then, the contour edges are obtained for crystal projection image. Note that $T_{\mathrm{H}}$ can be calculated using Otsu threshold method [25] as

$$T_{\mathrm{H}}=\arg \underset{t}{\max }{S}_1(t)S_2(t){\left(U_1(t)-U_2(t)\right)}^2$$

(7)

where $t$ denotes the threshold, $S_1(t)$ is the number of gray values greater than the threshold $t$, $S_2(t)$ is the number of gray values less than the threshold $t$, $U_1(t)$ is the mean gray values greater than the threshold $t$ and $U_2(t)$ is the mean gray values less than the threshold $t$.

After edge detection, morphological closing is used to close the breaks of Canny edges before region filling. The rest of the edges can be easily removed by morphological opening.

3.3. Improved SLIC Superpixel Segmentation

SLIC superpixel segmentation is used to generate superpixel blocks, which will be merged with cluster algorithm to obtain extraction results of a crystal image. The main idea of SLIC superpixel segmentation is to extract the five-dimensional feature vector of CIELAB color and spatial coordinates, construct the similarity measurement standard of feature vector and then to perform local clustering of image pixels [23]. The CIELAB color space is used to measure spatial distance with color sense differences, which is convenient for color clustering. However, the crystal images are generally gray, so the grayscale is used instead of CIELAB color features. SLIC algorithm cannot segment the ideal superpixel blocks well when the target and background contain different texture information but similar color. Due to the abundant texture information in crystal images, texture feature has an important influence on the segmentation results. Therefore, texture information is considered as a SLIC segmentation information.

Suppose that the image with $N$ pixels is pre-divided into $K$ superpixels; then, the size of each superpixel is $N/K$. The distance of adjacent seed points is approximately $S=\sqrt{N/K}$. The superpixel cluster centers $C_j={[g_j,x_j,y_j,t_j]}^{\mathrm{T}}$ with $j=[1,K]$ at regular grid intervals $S$. $g$ denotes the grayscale of a pixel; $x$ and $y$ denote the row and column coordinates of a pixel, respectively; and $t$ denotes the texture value of a pixel for local image entropy [21] with the region $2S\times 2S$ as

$$t=-{\displaystyle \sum _{m=0}^{2S}{\displaystyle \sum _{n=0}^{2S}P(m,n|\mathsf{\Delta }x,\mathsf{\Delta }y)}{\log }_{2}P(m,n|\mathsf{\Delta }x,\mathsf{\Delta }y})$$

(8)

where $P(m,n|\mathsf{\Delta }x,\mathsf{\Delta }y)$ is the probability of the gray level pairs i and j in the image, $\mathsf{\Delta }x$ is the horizontal pixel offset and $\mathsf{\Delta }y$ is the vertical pixel offset for the gray level co-occurrence matrix [26].

The seed points are re-selected in the $3\times 3$ neighborhood of the seed points. The seed points move to the place with the smallest gradient in the neighborhood to avoid seed points falling on the contour boundary with large gradient. Each pixel is assigned a class label $i$ in the neighborhood around each seed point. The search region is limited to $2S\times 2S$ to accelerate the convergence of the algorithm. Next, distances are measured, including space, grayscale and texture distances. The distance between it and the seed point is calculated as follows:

$$d_c=\sqrt{{(g_j-g_i)}^2}$$

(9)

$$d_s=\sqrt{{(x_j-x_i)}^2+{(y_j-y_i)}^2}$$

(10)

$$d_t=e^{(t_j-t_i)}$$

(11)

where $d_c$ is the grayscale distance, $d_s$ is the space distance and $d_t$ is the texture distance.

The integrated distance $D$ between this pixel and the center is defined as

$$D=d_c+\frac{m{d}_s}{S}+\frac{T{d}_t}{S}$$

(12)

where $T$ is the weight value for adjusting texture distance.

The final step is to update the cluster centers iteratively until residual error meets the stop condition ($E\le 0.01$). It is noted that the residual error $E$ is defined as Manhattan distance between previous centers and new cluster centers [23]. After the iterative optimization, the neighboring merging strategy is used to merge isolated small size superpixels to ensure a good tight-fitting degree of the final result.

3.4. Image Fusion

The fusion method is proposed to fuse the binary map of Canny edge detection and the edge map of superpixel regions. In this process, bilateral filtering is employed to remove noise. The agglomeration regions that are selected by convexity feature [11] in the binary map of Canny edge detection are just fused with the superpixel edges. For morphological processing, some small particles or background spots that are not likely to provide reliable size or shape information are removed by specifying a minimum number of pixels. Moreover, the fragmentary crystal blocks that are connected to the image border are also removed to suppress border interference.

4. Size Measurement Method

Size measurement rule of crystals is still an open issue, since the shape of different crystals is varied. In this study, the shape of α-form LGA is generally prismatic, as shown in Figure 1. Therefore, the diameter of the minimum enclosing circle for crystal imaging is typically considered as the one-dimension size of prismatic crystal. It can also tolerate the problem of the incomplete particles, which are separated from overlapped crystals. Firstly, in order to objectively reflect the relationship between actual size and image size, an effective method of physical calibration [11] is utilized with a micron circle scale to compute the pixel equivalent $P_e$. This scale is located in the probe imaging location to compute the relationship between the pixel and the actual size. Based on the binary result, the pixel number of the particle size can be measured by using the minimum enclosing circle method [27], and a processing example is as shown in Figure 4. Denoted by $L_a$ the measured pixel number of the enclosing circle diameter, the actual one-dimension size of the crystal, i.e., the actual physical diameter of the minimum enclosing circle denoted by $L_{\mathrm{p}}$ is given by

$$L_{\mathrm{p}}=L_a{P}_e$$

(13)

Figure 4. An example with minimum enclosing circle method: (a) the binary image with multiple objects; (b) the result of minimum enclosing circle.

Based on the above size measurement, crystal sizes can be provided. To clarify population size information, a statistic histogram of CSD is smoothed with the probability density function of Gaussian [28], which is defined by

$$P(l)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left\{-\frac{{(l-{\mu }_l)}^2}{2{\sigma }^2}\right\}$$

(14)

where $l$ is the size variable and ${\mu }_l$ and ${\sigma }_l$ are the mean and standard deviation parameters of the Gaussian distribution, respectively.

Note that the minimum enclosing circle method is only applicable to segmented results with a mass of agglomerated particles. The method is suggested to measure blocky and circular shapes, and it facilitates further one-dimension CSD estimation. In order to better describe the size characteristics, different measurement methods should be employed according to different morphologies, which can be detected for shape features [11].

5. Experimental Results

In the experiment, the on-line crystal images of α-form LGA are recorded by the invasive imaging system. They have just double colors in a JPEG format. To simplify the calculation, the raw image can be translated into the gray scale image. It is seen that the on-line images of slurries with multiple overlapped crystals suspended in a solution contain a lot of noise, as shown in Figure 5a. It is noted that the phenomenon that a mass of crystals are overlapped in a finite space can be seen as a case of high solid density. The image preprocessing result with image filtering is shown in Figure 5b. Tenengrad $M_{\mathrm{Ten}}$ [29,30] as a quantitative index of image quality is used for comparison between Figure 5a,b. It can be seen that the preprocessed image is denoised and deblurred. Note that Tenengrad of Figure 5b is higher than that of Figure 5a higher quality and Tenengrad $M_{\mathrm{Ten}}$ is defined for image Y as

$$M_{\mathrm{Ten}}={\displaystyle \sum _i{\displaystyle \sum _j\sqrt{K_i^2(i,j)+K_j^2(i,j)}}}$$

(15)

where $i$ denotes the row, $j$ denotes the column, and $K_i$ and $K_j$ are

$$\{\begin{array}{l}{K}_i=\frac{\partial Y}{\partial i}\\ K_j=\frac{\partial Y}{\partial j}\end{array}$$

(16)

Figure 5. Segmentation results of different segmentation: (a) the in situ captured image ($M_{\mathrm{Ten}}=35.49$); (b) the preprocessed result ($M_{\mathrm{Ten}}=37.52$).

The proposed segmentation method based on superpixel and Canny detection is applied on the preprocessed image (see Figure 5b). In Figure 6a, the edge detection result is obtained by using improved the Canny edge detection method. Figure 6b shows the results of the improved SLIC superpixel segmentation method. Then Figure 6c represents the segmented results that are detected with image fusion effectively. In the segmented binary result, the crystal intensity can be 1 and the background can be 0. After image processing, the crystal sizes are measured by using the fast minimum enclosing circle method as shown in Figure 6d, and the CSD model is obtained with the probability density function of the Gaussian. To show the superiority of the proposed segmentation method, Figure 7 illustrates the segmentation results by using well-recognized multi-scale segmentation algorithm [15] and the popular threshold algorithm [11]. In Figure 7, most overlapping crystals were not able to be segmented. It can be seen that multiple overlapped particles outlined by red circles cannot be separated with the use the two well-recognized segmentation algorithms.

Figure 6. Segmentation results: (a) Canny edge detection; (b) SLIC segmentation; (c) fusion image; (d) size measurement.

Figure 7. Visual comparison of the crystal image with different methods: (a) multi-Canny segmentation; (b) threshold segmentation.

In addition, another highly -concentrated image captured from the experiment is analyzed as shown in Figure 8, in order to demonstrate the efficiency of the proposed method. Figure 8 shows that most of crystals can be properly separated from the background, together with a good result of size measurement.

Figure 8. Analytic result of another image: (a) the in situ captured image; (b) size measurement.

In order to validate the proposed imaging measurement method, a complementary experiment was performed using off-line verification method. The CSDs of α-form LGA crystals were obtained using two different methods: the proposed in situ imaging method and off-line measurement by using digital microscope with measurement software, which has the ability to measure size. Figure 9 shows the CSDs are produced by using the in situ method and the off-line method, respectively. Note that about 600 crystals were extracted from 15 images captured within one minute to produce a CSD model using in situ image measurement. In the off-line measurement, the CSD was also calculated with approximately 600 crystals extracted from 50 off-line images, which were captured with the off-line microscope. The crystal sizes were obtained by using manual length measurement with LAS software. To show the similarity, the mean value $\mu$ and standard deviation $\sigma$ for the Gaussian of CSD in size are used. For the quantitative comparison, the relative error of $\mu$ is 1.61% and the relative error of $\sigma$ is 1.08%. The comparison results show that the results given by the proposed in situ method are very close to those of the off-line method.

Figure 9. CSDs of LGA crystals: (a) CSD using in situ image measurement; (b) CSD using off-line measurement.

6. Conclusions

In this work, a synthetic image-measurement method to measure CSD was developed by combining superpixel segmentation and edge detection. The analysis capabilities of α-form LGA crystals at high-solid concentrations were investigated with on-line invasive imaging system. The raw images were denoised before segmentation. The segmentation algorithm successfully extracted the crystals from the on-line images with a mass of overlapping crystal blocks. Furthermore, the estimation of CSD was produced by probability density function. In the end, the experimental results on α-form LGA were performed to show that the in situ image measurement method was effective. The proposed method can measure crystal sizes during crystallization at high-solid concentrations. It can also potentially assist with the evolution modeling of crystal size distributions. Our future work will focus on studying image-based agglomeration degree and solid concentration detection for multiple morphologies.