Three-Dimensional Autofocusing Visual Feedback for Automated Rare Cells Sorting in Fluorescence Microscopy

Sorting rare cells from heterogeneous mixtures makes a significant contribution to biological research and medical treatment. However, the performances of traditional methods are limited due to the time-consuming preparation, poor purity, and recovery rate. In this paper, we proposed a cell screening method based on the automated microrobotic aspirate-and-place strategy under fluorescence microscopy. A fast autofocusing visual feedback (FAVF) method is introduced for precise and real-time three-dimensional (3D) location. In the context of this method, the scalable correlation coefficient (SCC) matching is presented for planar locating cells with regions of interest (ROI) created for autofocusing. When the overlap occurs, target cells are separated by a segmentation algorithm. To meet the shallow depth of field (DOF) limitation of the microscope, the improved multiple depth from defocus (MDFD) algorithm is used for depth detection, taking 850 ms a time with an accuracy rate of 96.79%. The neighborhood search based algorithm is applied for the tracking of the micropipette. Finally, experiments of screening NIH/3T3 (mouse embryonic fibroblast) cells verifies the feasibility and validity of this method with an average speed of 5 cells/min, 95% purity, and 80% recovery rate. Moreover, such versatile functions as cell counting and injection, for example, could be achieved by this expandable system.


Introduction
Low-abundance cells with samples containing less than 1000 target cells/ml are considered as rare cells [1]. Research on sorting and isolating a specified number of rare cells, such as circulating tumor cells (CTCs), circulating fetal cells, and stem cells from complex and heterogeneous mixtures, are vitally important in biology and medicine [2]. This practice often serves as the preparatory work in many diagnostic and therapeutic practices to enhance the research efficiency [3]. CTCs, for example, are recognized as the biomarker for various cancers, such as breast, prostate, ovarian, colon, etc. [4,5]. Sorting and detecting CTCs from cancer patients is viable for cancer prognosis and treatment. Meanwhile, the rapid and accurate sorting of rare cells is highly demanded in personalized medicine. The tailored medicine improves the precision and effectiveness of therapy [6]. Figure 1. The microrobotic manipulation system is mounted in the inverted fluorescence microscope. The fluorochrome enhances the clarity and contrast of cells and micropipette clear. The visual information is obtained by a camera and transmitted to the main computer simultaneously. In visual processing, the FAVF algorithm mainly consists of four parts: preprocessing, planar locating, depth detection, and object tracking. Planar locating relies on the scalable correlation coefficient (SCC) matching and watershed segmentation algorithm. Depth detection is achieved by the improved multiple depth from defocus (MDFD) algorithm, which derives from traditional depth from defocus (DFD). The neighborhood search based algorithm is adopted for object tracking. Assisted by FAVF, real-time 3D locating of micropipette and cells are achieved. With the visual feedback, automated rare cells sorting is carried out by microrobotic system with high positioning accuracy and high operating efficiency. The system is evaluated by sorting DiI (cell membrane red fluorescent probe) stained NIH/3T3 cells with red fluorescence from DiO (cell membrane green fluorescent probe) stained NIH/3T3 cells with green fluorescence. The experiment demonstrates the system has considerable sorting speed, purity and recovery rate. Due to its simplicity, durability, cost-effectiveness and expandability, the system is viable for the isolation of most rare cell types and promising for biological and medical research. neighborhood search based algorithm is adopted for object tracking. Assisted by FAVF, real-time 3D locating of micropipette and cells are achieved. With the visual feedback, automated rare cells sorting is carried out by microrobotic system with high positioning accuracy and high operating efficiency. The system is evaluated by sorting DiI (cell membrane red fluorescent probe) stained NIH/3T3 cells with red fluorescence from DiO (cell membrane green fluorescent probe) stained NIH/3T3 cells with green fluorescence. The experiment demonstrates the system has considerable sorting speed, purity and recovery rate. Due to its simplicity, durability, cost-effectiveness and expandability, the system is viable for the isolation of most rare cell types and promising for biological and medical research.

Scalable Correlation Coefficient Matching
Template matching algorithm has been widely used in micromanipulation for its accurate and fast performance. For cell sorting, target cells are scattered in suspension with different spatial distributions. Due to the shallow DOF of microscope, cells have various sizes and the degrees of blur. In traditional template matching method, template is preselected with fixed size, which is inapplicable in this case. The scalable correlation coefficient (SCC) matching algorithm for planar locating is proposed. According to the correlation coefficient equation:

I (x + x , y + y )) I(x + x , y + y ) -1 / (w h)
I(x + x , y + y ) where x and y are coordinate value of pixels, T is grayscale value of template, I is grayscale value of source image, w and h are the width and height of template, R is correlation coefficient. The correlation coefficient value and matching degree are positively correlated. As shown in Figure 2, focused cell is preselected as the original template. It is enlarged to a series of templates with Gaussian blur. In rough locating, two blurred target cells are detected by scalable templates with optimal sizes and are marked by a bounding box. The result of rough locating

Scalable Correlation Coefficient Matching
Template matching algorithm has been widely used in micromanipulation for its accurate and fast performance. For cell sorting, target cells are scattered in suspension with different spatial distributions. Due to the shallow DOF of microscope, cells have various sizes and the degrees of blur. In traditional template matching method, template is preselected with fixed size, which is inapplicable in this case. The scalable correlation coefficient (SCC) matching algorithm for planar locating is proposed. According to the correlation coefficient equation: and where x and y are coordinate value of pixels, T is grayscale value of template, I is grayscale value of source image, w and h are the width and height of template, R is correlation coefficient. The correlation coefficient value and matching degree are positively correlated.
As shown in Figure 2, focused cell is preselected as the original template. It is enlarged to a series of templates with Gaussian blur. In rough locating, two blurred target cells are detected by scalable templates with optimal sizes and are marked by a bounding box. The result of rough locating generates the region of interest (ROI) for subsequent depth detection. The focused target cells are matched by optimal template and their precise locations are marked with tow cross lines.

Watershed Algorithm Based Instance Segmentation
In the microscopic scene, cells are easily overlapped, due to the electrostatic force and adhesive force. Instance segmentation of target cells is essential for precise locating. The newly developed deep learning algorithm provide satisfactory results. However, data collecting and training is time consuming. Here, we present a watershed algorithm-based instance segmentation method. The process of this method is shown in Figure 3. Three target cells are stained red and stuck together ( Figure 3a). The overlapped cells are sketchily marked with bounding boxes by SCC matching. In overlapping process, the presence of overlapping regions can trigger the subsequent segmentation ( Figure 3b). Red color channel is remained by setting threshold (Figure 3c). Markers are created as water holes through a series of visual transforms (Figure 3g). An additional marker is set in the top left as background water hole. Markers are applied to Figure

Neighborhood Searching Based Micropipette Tracking
To feed back the location of the micropipette tip in real-time, tracking algorithm is crucial. As shown in Figure 4a, when the micropipette gets focused, its tip is precisely located. Centering on the micropipette tip, the original tracking template and neighborhood searching area are created. To decrease the running time of program, the size of neighborhood searching area is limited to three

Watershed Algorithm Based Instance Segmentation
In the microscopic scene, cells are easily overlapped, due to the electrostatic force and adhesive force. Instance segmentation of target cells is essential for precise locating. The newly developed deep learning algorithm provide satisfactory results. However, data collecting and training is time consuming. Here, we present a watershed algorithm-based instance segmentation method. The process of this method is shown in Figure 3. Three target cells are stained red and stuck together (Figure 3a). The overlapped cells are sketchily marked with bounding boxes by SCC matching. In overlapping process, the presence of overlapping regions can trigger the subsequent segmentation ( Figure 3b). Red color channel is remained by setting threshold (Figure 3c). Markers are created as water holes through a series of visual transforms (Figure 3g). An additional marker is set in the top left as background water hole. Markers are applied to Figure

Watershed Algorithm Based Instance Segmentation
In the microscopic scene, cells are easily overlapped, due to the electrostatic force and adhesive force. Instance segmentation of target cells is essential for precise locating. The newly developed deep learning algorithm provide satisfactory results. However, data collecting and training is time consuming. Here, we present a watershed algorithm-based instance segmentation method. The process of this method is shown in Figure 3. Three target cells are stained red and stuck together ( Figure 3a). The overlapped cells are sketchily marked with bounding boxes by SCC matching. In overlapping process, the presence of overlapping regions can trigger the subsequent segmentation ( Figure 3b). Red color channel is remained by setting threshold (Figure 3c). Markers are created as water holes through a series of visual transforms (Figure 3g). An additional marker is set in the top left as background water hole. Markers are applied to Figure

Neighborhood Searching Based Micropipette Tracking
To feed back the location of the micropipette tip in real-time, tracking algorithm is crucial. As shown in Figure 4a, when the micropipette gets focused, its tip is precisely located. Centering on the micropipette tip, the original tracking template and neighborhood searching area are created. To decrease the running time of program, the size of neighborhood searching area is limited to three

Neighborhood Searching Based Micropipette Tracking
To feed back the location of the micropipette tip in real-time, tracking algorithm is crucial. As shown in Figure 4a, when the micropipette gets focused, its tip is precisely located. Centering on the micropipette tip, the original tracking template and neighborhood searching area are created. To decrease the running time of program, the size of neighborhood searching area is limited to three times the size of the template. When the micropipette moves, as shown in Figure 4b, the original template searches in the neighborhood search area, and the match window is obtained by the correlation coefficient matching algorithm. The best match is indicated in the match window, and the location of micropipette tip is updated. Due to the fluorescence quenching and impurity adsorption, the color and shape of the micropipette can change slightly from the initial appearance. As shown in Figure 4c, the updated template has better performance to adapt subsequent tracking. times the size of the template. When the micropipette moves, as shown in Figure 4b, the original template searches in the neighborhood search area, and the match window is obtained by the correlation coefficient matching algorithm. The best match is indicated in the match window, and the location of micropipette tip is updated. Due to the fluorescence quenching and impurity adsorption, the color and shape of the micropipette can change slightly from the initial appearance. As shown in Figure 4c, the updated template has better performance to adapt subsequent tracking.

Depth Detection
Autofocus technology is one of the key technologies in computer vision and various imaging systems. It has a wide range of applications such as cameras, endoscopes and microscopes. In microscope imaging, depth from focus (DFF) and depth from defocus (DFD) are two widely used methods, which are essential for the acquisition of depth value and clear observation [34,21]. The DFD method has a great detection efficiency by comparing two defocused images. In this paper, to solve the symmetrical problem in traditional DFD, the multiple depth from defocus (MDFD) method is presented.

Imaging Model of Multiple Depth from Defocus (MDFD)
The convex lens imaging model of MDFD shown in Figure 5a. Through the lens, the object is focused as a focal point (P') on the focal plane (FP). Adjusting the imaging plane (IP) away from FP, a defocused blur circle formed, where dn is the object depth value. As shown in Figure 5b, in the imaging model, a cell is focused on the focal plane with focused diameter D. Changing IP successively with a fixed step interval d, and the blur diameter Dn acts like a linear shift. According to geometry: where object depth value dn is obtained when D and Dn are known. The point spread function (PSF, h(x,y)) is introduced to explain this phenomenon-it is modeled as two dimensional Gaussian: x y h x y (5) where spread parameter σ is positive correlated to Dn. As shown in Figure 5c, the observed image g(x, y) is the result of the convolution between the focused image im(x, y) and PSF: When IP moves away from FP, brightness energy of the cell is dispersed and the observed Dn is getting larger. For a certain cell, the energy distribution curves on different IP have the same two intersections. The width of two intersections is the focused diameter D of the cell.

Depth Detection
Autofocus technology is one of the key technologies in computer vision and various imaging systems. It has a wide range of applications such as cameras, endoscopes and microscopes. In microscope imaging, depth from focus (DFF) and depth from defocus (DFD) are two widely used methods, which are essential for the acquisition of depth value and clear observation [21,34]. The DFD method has a great detection efficiency by comparing two defocused images. In this paper, to solve the symmetrical problem in traditional DFD, the multiple depth from defocus (MDFD) method is presented.

Imaging Model of Multiple Depth from Defocus (MDFD)
The convex lens imaging model of MDFD shown in Figure 5a. Through the lens, the object is focused as a focal point (P') on the focal plane (FP). Adjusting the imaging plane (IP) away from FP, a defocused blur circle formed, where d n is the object depth value. As shown in Figure 5b, in the imaging model, a cell is focused on the focal plane with focused diameter D. Changing IP successively with a fixed step interval d, and the blur diameter D n acts like a linear shift. According to geometry: where object depth value d n is obtained when D and D n are known. The point spread function (PSF, h(x,y)) is introduced to explain this phenomenon-it is modeled as two dimensional Gaussian: where spread parameter σ is positive correlated to D n . As shown in Figure 5c, the observed image g(x, y) is the result of the convolution between the focused image im(x, y) and PSF: Micromachines 2019, 10, x 6 of 15 However, the symmetrical problem exists. As shown in Figure 5b, in symmetrical position (IP1 and IP'1) on either side of the plane, objects have the same blur diameter (D'1 and D1). They cannot be distinguished by the traditional two-image DFD method.

Algorithm Strategy of Multiple Depth from Defocus (MDFD)
To solve the symmetrical problem, MDFD captures additional images and possesses optimized performance. The flow chart of MDFD is shown in Figure 6a, preprocessing and sharpness judgement of target cell are made in the region of interest (ROI). If the cell is defocused, setting a step interval d to capture a series of images along the z axis. D and Dn are calculated by visual algorithm, and error data can be screened out by analysis the change of Dn. The cell depth value dn is calculated by equation (4). The above process may be performed multiple times, until a target gets focused.
The visual processing of MDFD is shown in Figure 6b. Preprocessing consists of grayscale transformation and Gaussian filtering. The k-means clustering algorithm is applied to precisely separate the cell from background. Cluster number k is set to 3 to represent background, cell blur edge, and body respectively. Depending on the grayscale value, the pixels will be tagged with three different labels. According to labels, the regions of cell blur edge and body are combined to represent entire cell by image binarization. Morphological operation is applied to remove smaller areas smooth the cell contour. Cell contour is optimized by convex hull algorithm to search minimum area encirclement lines. Then cell center is obtained by calculation of image moments and blur diameters of cells Dn are obtained. Extracting pixels across the horizontal line of the cell center (PAHLCC) from convex hull image. For grayscale images, the horizontal lines through the center of the objects are extracted and pixel grayscale distribution curves (PGDC) are obtained. Based on the PGDC, a sharpness judgement algorithm is proposed. For each PGDC, the sharpness of an image is evaluated by the difference between the maximum and minimum, which is considered as image contrast c. The larger the value of c, the clearer the image. According to the law of c change, error data (symmetrical case) is screened out. Besides, for different PGDC, the length of two intersections is focused diameter D of cell. Finally, d is calculated through Equation (4). When IP moves away from FP, brightness energy of the cell is dispersed and the observed D n is getting larger. For a certain cell, the energy distribution curves on different IP have the same two intersections. The width of two intersections is the focused diameter D of the cell.
However, the symmetrical problem exists. As shown in Figure 5b, in symmetrical position (IP 1 and IP' 1 ) on either side of the plane, objects have the same blur diameter (D' 1 and D 1 ). They cannot be distinguished by the traditional two-image DFD method.

Algorithm Strategy of Multiple Depth from Defocus (MDFD)
To solve the symmetrical problem, MDFD captures additional images and possesses optimized performance. The flow chart of MDFD is shown in Figure 6a, preprocessing and sharpness judgement of target cell are made in the region of interest (ROI). If the cell is defocused, setting a step interval d to capture a series of images along the z axis. D and D n are calculated by visual algorithm, and error data can be screened out by analysis the change of D n . The cell depth value d n is calculated by Equation (4). The above process may be performed multiple times, until a target gets focused.   The visual processing of MDFD is shown in Figure 6b. Preprocessing consists of grayscale transformation and Gaussian filtering. The k-means clustering algorithm is applied to precisely separate the cell from background. Cluster number k is set to 3 to represent background, cell blur edge, and body respectively. Depending on the grayscale value, the pixels will be tagged with three different labels. According to labels, the regions of cell blur edge and body are combined to represent entire cell by image binarization. Morphological operation is applied to remove smaller areas smooth the cell contour. Cell contour is optimized by convex hull algorithm to search minimum area encirclement lines. Then cell center is obtained by calculation of image moments and blur diameters of cells D n are obtained. Extracting pixels across the horizontal line of the cell center (PAHLCC) from convex hull image. For grayscale images, the horizontal lines through the center of the objects are extracted and pixel grayscale distribution curves (PGDC) are obtained. Based on the PGDC, a sharpness judgement algorithm is proposed. For each PGDC, the sharpness of an image is evaluated by the difference between the maximum and minimum, which is considered as image contrast c. The larger the value of c, the clearer the image. According to the law of c change, error data (symmetrical case) is screened out. Besides, for different PGDC, the length of two intersections is focused diameter D of cell. Finally, d is calculated through Equation (4).

System Framework
The microrobotic manipulation system is shown in Figure 7. The robot consists of a linear translation stage (M-461-XYZ, Newport, Irvine, CA, USA) and three micro-stepping motors (NSA 12, Newport, Irvine, CA, USA) with a resolution of 0.2 µm and a motion range of 11 mm for each axis. A glass micropipette (B100-50-10, Sutter Instrument, Novato, CA, USA) is heat pulled by a micropipette puller (PC-10, Narishige, Tokyo, Japan) with an inner diameter of 19 µm. It is fixed on an end actuator and connected to the syringe pump (Legato 111, KD Scientific, Ringoes, NJ, USA) via a soft rubber tube. An inverted optical microscope (OM) (IX83, Olympus, Tokyo, Japan) is connected to a CCD camera (DP22, Olympus, Tokyo, Japan) with maximum 2.76 megapixels. The lamp (U-HGLGPS, Olympus, Tokyo, Japan) and filter block (U-FNU/FNB/FNG, Olympus, Tokyo, Japan) ensure multi-colored fluorescence observation. A motorized X-Y translational stage (ProScan, Prior Scientific, Cambridge, UK) mounted on the OM can transform field of view (FOV) rapidly. A computer configured with CPU (Core i7, Intel, Santa Clara, CA, USA) and GPU (TITAN X, NVIDIA, Santa Clara, CA, USA) is utilized for visual processing and automated control. (v/v) penicillin-streptomycin solution (Solarbio, Beijing, China). Furthermore, the micropipette tip is pre-dyed with DiI. Under the fluorescence microscope, the red and green lights are excited. The overall process of automatic microrobotic manipulation strategy is shown in Figure 8. The camera transmits microscopic images to the main computer for visual processing. For FAVF, impurities filtering and images smoothing are carried out in preprocessing. Sharpness judgement distinguishes blurred and focused targets. The depth value of the micropipette tip and target cells are obtained through the MDFD method. According to the depth value, the micropipette moves to the same focal plane as the target cells. Then, the micropipette and target cells are located by precise planar locating algorithm and the overlapped targets are separated by segmentation algorithm. During the manipulation, the position of micropipette tip is tracked simultaneously. With the visual feedback, robots are controlled to move and aspirate target cells until all are screened in the FOV. The new group of cells are brought into the FOV by the translation stage. Sorting will continue until the number of cells collected is sufficient. (v/v) penicillin-streptomycin solution (Solarbio, Beijing, China). Furthermore, the micropipette tip is pre-dyed with DiI. Under the fluorescence microscope, the red and green lights are excited. The overall process of automatic microrobotic manipulation strategy is shown in Figure 8. The camera transmits microscopic images to the main computer for visual processing. For FAVF, impurities filtering and images smoothing are carried out in preprocessing. Sharpness judgement distinguishes blurred and focused targets. The depth value of the micropipette tip and target cells are obtained through the MDFD method. According to the depth value, the micropipette moves to the same focal plane as the target cells. Then, the micropipette and target cells are located by precise planar locating algorithm and the overlapped targets are separated by segmentation algorithm. During the manipulation, the position of micropipette tip is tracked simultaneously. With the visual feedback, robots are controlled to move and aspirate target cells until all are screened in the FOV. The new group of cells are brought into the FOV by the translation stage. Sorting will continue until the number of cells collected is sufficient.
transmits microscopic images to the main computer for visual processing. For FAVF, impurities filtering and images smoothing are carried out in preprocessing. Sharpness judgement distinguishes blurred and focused targets. The depth value of the micropipette tip and target cells are obtained through the MDFD method. According to the depth value, the micropipette moves to the same focal plane as the target cells. Then, the micropipette and target cells are located by precise planar locating algorithm and the overlapped targets are separated by segmentation algorithm. During the manipulation, the position of micropipette tip is tracked simultaneously. With the visual feedback, robots are controlled to move and aspirate target cells until all are screened in the FOV. The new group of cells are brought into the FOV by the translation stage. Sorting will continue until the number of cells collected is sufficient.

Planar Locating
The scalable correlation coefficient (SCC) matching algorithm were adopted for both rough locating and precise locating. The algorithm was evaluated at 9 groups of depth value. For each group, 20 targets were tested. A target should be detected with the correct number and location, otherwise it will be considered as an error detection. As shown in Figure 9, the correct detection rate is 100% in the precise locating section. In the rough locating section, the correct detection rate slightly declines when the depth value exceeded 20 µm. The average correct detection rate is 95% and the average processing time is 0.8 s, which guarantees real-time feedback.

Planar Locating
The scalable correlation coefficient (SCC) matching algorithm were adopted for both rough locating and precise locating. The algorithm was evaluated at 9 groups of depth value. For each group, 20 targets were tested. A target should be detected with the correct number and location, otherwise it will be considered as an error detection. As shown in Figure 9, the correct detection rate is 100% in the precise locating section. In the rough locating section, the correct detection rate slightly declines when the depth value exceeded 20 µm. The average correct detection rate is 95% and the average processing time is 0.8 s, which guarantees real-time feedback.

Micropipette Tip Tracking
The neighborhood search based tracking algorithm was used for micropipette tip tracking. The algorithm was evaluated by the movement of the micropipette at five groups of speed. The CCD camera has a frame rate of 60 fps and 1500 frames were tested at each group. The location of micropipette tip was marked by bounding box in each frame of image. Between consecutive frames, the ratio of overlapped region of the bounding boxes is defined as overlapping rate. When overlapping rate is greater than 70%, it is considered as accurate tracking. As shown in Figure 10, the accurate tracking rate drops significantly when speed is over 300 µm/s. To ensure the tracking accuracy, the moving speed of micropipette was set to 300 µm/s, which allowed a high accuracy (90%) and a sorting speed of 2 s/cell.

Micropipette Tip Tracking
The neighborhood search based tracking algorithm was used for micropipette tip tracking. The algorithm was evaluated by the movement of the micropipette at five groups of speed. The CCD camera has a frame rate of 60 fps and 1500 frames were tested at each group. The location of micropipette tip was marked by bounding box in each frame of image. Between consecutive frames, the ratio of overlapped region of the bounding boxes is defined as overlapping rate. When overlapping rate is greater than 70%, it is considered as accurate tracking. As shown in Figure 10, the accurate tracking rate drops significantly when speed is over 300 µm/s. To ensure the tracking accuracy, the moving speed of micropipette was set to 300 µm/s, which allowed a high accuracy (90%) and a sorting speed of 2 s/cell. micropipette tip was marked by bounding box in each frame of image. Between consecutive frames, the ratio of overlapped region of the bounding boxes is defined as overlapping rate. When overlapping rate is greater than 70%, it is considered as accurate tracking. As shown in Figure 10, the accurate tracking rate drops significantly when speed is over 300 µm/s. To ensure the tracking accuracy, the moving speed of micropipette was set to 300 µm/s, which allowed a high accuracy (90%) and a sorting speed of 2 s/cell.

Depth Detection with Multiple Depth from Defocus (MDFD)
For the sample cell, the blur diameters (Dn) on different image planes (IP) were calculated by previously mentioned algorithm (Figure 6b). To verify the reliability of Dn, the depth value of focal planes (FP) was set to 0, which was determined by sharpness judgement algorithm. The diameters of cells on the FP was considered as the actual focused diameter (Da), as shown in Figure 11. The IP was adjusted at unequal step intervals (d) and Dn were obtained. As shown in Figure 12, scatter plot for depth values and Dn was drawn and one-time polynomial was applied for data fitting. A total of 10 sample cells were tested in the experiment and the result showed that the average coefficient of Figure 10. Accurate tracking rate for micropipette tracking.

Depth Detection with Multiple Depth from Defocus (MDFD)
For the sample cell, the blur diameters (D n ) on different image planes (IP) were calculated by previously mentioned algorithm (Figure 6b). To verify the reliability of D n , the depth value of focal planes (FP) was set to 0, which was determined by sharpness judgement algorithm. The diameters of cells on the FP was considered as the actual focused diameter (D a ), as shown in Figure 11. The IP was adjusted at unequal step intervals (d) and D n were obtained. As shown in Figure 12, scatter plot for depth values and D n was drawn and one-time polynomial was applied for data fitting. A total of 10 sample cells were tested in the experiment and the result showed that the average coefficient of determination (R-square) is 0.9953. It indicated that scatter points are generally linear and the algorithm for D n detection was almost feasible.   Then, to evaluate the accuracy of depth detection of MDFD for the sample cell, the IP was set to a known initial depth (di), as shown in Figure 11. Moreover, each IP was adjusted at a certain d with the image recorded. The experimental depth value (dn) was calculated by Equation (4). To evaluate the deviation of this method, the relative error formula is presented:   Then, to evaluate the accuracy of depth detection of MDFD for the sample cell, the IP was set to a known initial depth (di), as shown in Figure 11. Moreover, each IP was adjusted at a certain d with the image recorded. The experimental depth value (dn) was calculated by Equation (4). To evaluate the deviation of this method, the relative error formula is presented: Then, to evaluate the accuracy of depth detection of MDFD for the sample cell, the IP was set to a known initial depth (d i ), as shown in Figure 11. Moreover, each IP was adjusted at a certain d with the image recorded. The experimental depth value (d n ) was calculated by Equation (4). To evaluate the deviation of this method, the relative error formula is presented: where e is the experimental error. The experimental focused diameter (D) is introduced as correction factor. Set d i from 4 mm to 36 mm and d from 2 mm to 12 mm, respectively. When the depth value is more than 40 µm, the fluorescence is very weak which cannot be recognized. Therefore, the valid test data d n are limited within the range of 40 µm, otherwise it is considered as out of range (OOR). A total of 10 sample cells were tested in the experiment, and the average e are shown in Table 1. According to the table data, error curve for different d i were drawn, as shown in Figure 13a. In curve 4, 5 and 6, e is all lower than 5%. However, in curve 2 and 3, e increases sharply and exceeds 10% at d of 10 µm and 12 µm, respectively. In curve 1, the average e is greater than 20%. The data showed that when d n are within 20 µm the results are highly precise. However, when d n exceeds 20 µm, the result is unreliable. Such a large deviation was derived from the calculation of D. Due to the imaging noise, D was always larger than D a which is inevitable. Then, multiple calculations were applied, as shown in Figure 11, and the depth values can finally converge to be accurate. In a precise range, e can decrease significantly when d is greater than 4 µm, as shown in Figure 13b. However, too large d can make d n into the large error range and it was set to 6 µm. To solve the symmetry problem, the algorithm worked based on the three blur images, and one of them was used as a reference. The results showed that the depth values were obtained after approximately two or three calculations with an almost small error of 3.21%. The entire focusing process took a short 850 ms averagely and the real-time feedback is guaranteed.

Automated Fast Autofocusing Visual Feedback (FAVF) Cell Sorting
The performance of the FAVF cell sorting method was evaluated using heterogeneous mixtures under the fluorescence microscope. The volume of the mixture was 1 mL and 100 target NIH/3T3 cells were contained. The rough planar locations of micropipette and target cells were detected by SCC matching algorithm (Figure 14a). FOV was focused after acquiring depth by MDFD and precise planar locations were obtained (Figure 14b). The micropipette was controlled to move to target cells with tip was tracked. When all targets were sorted, the FOV was transformed by regular scanning on the surface of a Petri dish with a diameter of 35 mm. Scanning stopped until the sufficient number of cells was collected (see Supplementary Video S1). The result of 10 experiments showed that the proposed method achieved the average speed of 5 cells/min, a purity of target cells of 95%, and recovery rate of 80%. The introduction of a small number of heterogeneous cells is mainly due to the deformation of the soft rubber tube-which is why a hard rubber tube will be utilized in the future research. In the entire process, FOV switching takes up the bulk of time and limits the sorting speed greatly. When additional target cells were added in the mixture, the speed got obviously increased, as shown in Figure 15. When the concentration was raised to 1000 target cells mL −1 and the sorting speed increased obviously. However, with an increase of target number, an upward trend gradually decreases in intensity. The optimal concentration was about 600 target cells/mL, with an average speed of 11 cells/min, where the sorting efficiency was mainly limited by the increment of computational complexity for 3D locating, segmentation and tracking. Meanwhile, FOV switching took approximately 10 min, scanning the entire surface of the Petri dish with a diameter of 35 mm. Therefore, the optimized the switch strategy and algorithm will be carried out in the future work to improve system performance. Cell viability before and after sorting was calculated based on statistic data by using calcein AM fluorescent stains (Molecular Probes, Eugene, OR, USA). Three independent experiments were carried out and the average cell viability were 98% (before sorting) and 90% (after sorting), which demonstrated the non-invasiveness of our method. recovery rate. Besides, this approach also shows some advantages over the FACS and IMACS methods such as a short time sample preparation, the requirement of convenient equipment, and non-invasive features. In addition, compared with microfluidic cell sorting, the robotic system is durable and reusable. Moreover, in comparison to traditional bio-manipulation methods, it realizes automated 3D locating with optimized depth detection algorithm without extra devices introduced.
The proposed method is likely universalizable for the isolation of most rare cell types, such as CTCs, stem cells, and circulating fetal cells, which could make further contributions to medical diagnosis and tailored medicine. Moreover, the open-ended system means the addition of multiple visual algorithms and end actuators. It may achieve more functions like cell counting and injection for versatile applications across biology, biotechnology, and medicine.

Conclusions
In this study, a novel FAVF algorithm for the automated sorting of rare cells under the fluorescence microscope was proposed. The algorithm is performed on the microrobotic manipulation system with visual feedback control. In FAVF, the accurate 3D locations of objects are obtained in real time. SCC matching is presented for both rough and precise planar locating. The watershed segmentation algorithm is introduced to separate the overlapped cells. The improved MDFD algorithm is used in depth detection. The neighborhood search based algorithm is applied for tracking the movement of the micropipette. With visual feedback, robots are precisely controlled to move and aspirate target cells automatically. The experiment of screening NIH/3T3 cells demonstrates the good performance of the proposed system. Compared with other existing methods, this system has many advantages, such as relatively high purity and recovery rate, universalizability, simplicity, durability, and cost-effectiveness.  Compared with the DGC method, our system allows the markedly improved purity and recovery rate. Besides, this approach also shows some advantages over the FACS and IMACS methods such as a short time sample preparation, the requirement of convenient equipment, and non-invasive features. In addition, compared with microfluidic cell sorting, the robotic system is durable and reusable. Moreover, in comparison to traditional bio-manipulation methods, it realizes automated 3D locating with optimized depth detection algorithm without extra devices introduced.
The proposed method is likely universalizable for the isolation of most rare cell types, such as CTCs, stem cells, and circulating fetal cells, which could make further contributions to medical diagnosis and tailored medicine. Moreover, the open-ended system means the addition of multiple visual algorithms and end actuators. It may achieve more functions like cell counting and injection for versatile applications across biology, biotechnology, and medicine.

Conclusions
In this study, a novel FAVF algorithm for the automated sorting of rare cells under the fluorescence microscope was proposed. The algorithm is performed on the microrobotic manipulation system with visual feedback control. In FAVF, the accurate 3D locations of objects are obtained in real time. SCC matching is presented for both rough and precise planar locating. The watershed segmentation algorithm is introduced to separate the overlapped cells. The improved MDFD algorithm is used in depth detection. The neighborhood search based algorithm is applied for tracking the movement of the micropipette. With visual feedback, robots are precisely controlled to move and aspirate target cells automatically. The experiment of screening NIH/3T3 cells demonstrates the good performance of the proposed system. Compared with other existing methods, this system has many advantages, such as relatively high purity and recovery rate, universalizability, simplicity, durability, and cost-effectiveness.

Conflicts of Interest:
The authors declare no conflict of interest.