#### 2.1. Robotic System

The designed robot for multidirectional image sensing at a small scale is illustrated in

Figure 1. The robot is mainly constructed from three nano-positioners: two linear positioners (ECS3030, Attocube Inc., Muenchen, Germany) and one rotary positioners (ECR3030, Attocube Inc., Muenchen, Germany). The first linear positioner (LP_1) is mounted on the rotary positioner (RP) and the second linear positioner (LP_2) is mounted on the first linear positioner LP_1. LP_1 and LP_2’s movement directions are mutually perpendicular. Each nano-positioner of the robot is responsible for one independent movement, thereby the robot has three degrees of freedom (DOFs) in total: two mutually perpendicular translational movements and one rotation. Additionally, we also designed a T-shape sample stage to hold the sample, a connector to connect RP and LP_1, and a holder to support the whole robot. The sample holder consists of a hole to hold a sample and a screw to fasten the sample.

**Figure 1.**
The nanorobotic system for the multidirectional image sensing at small scale, and zoomed picture for the sample holder. The sample holder consists of a hole to hold sample and a screw to fasten the sample.

**Figure 1.**
The nanorobotic system for the multidirectional image sensing at small scale, and zoomed picture for the sample holder. The sample holder consists of a hole to hold sample and a screw to fasten the sample.

The travel range, resolution and repeatability of the two linear positioners LP_1 and LP_2 are 20 mm, 1 nm and 50 nm, respectively. The travel range, resolution and repeatability for the rotary positioner RP are 360° endless, (1 × 10^{−6})° and 5% over the full range, respectively. Therefore, the positioning accuracy of the robot can be guaranteed during the image sensing process.

#### 2.2. Sample Alignment

During the experiment, the to-be-imaged micro object is assembled on the sample stage of the robotic system, which is put underneath the lens of the microscope, as shown in

Figure 2a. To image the sample from multiple directions, the sample should be able to rotate endlessly over 360°. However, one serious problem is that the field of view (FOV) of the microscope is limited, which means the sample may move out of the FOV during the rotation, as illustrated in

Figure 2b. To avoid this, sample alignment must be considered before imaging.

Unlike the traditional macro scale alignment, the microscope can only provide 2D image information. Because the imaging direction is perpendicular to the rotation axis, it’s very difficult to obtain the positon of the sample directly based on the microscope image. To address these challenges, we propose an automatic forward-backward alignment strategy.

As illustrated in

Figure 2a, two coordinate systems are established first,

i.e., the microscopy coordinates {M} and the robot coordinates {N}. The microscope image coordinates {M} are established on the imaging plane of the microscope with the origin being the lower left corner of the microscope images. The X

_{M} axis and Z

_{M} axis are along with microscope images’ two sides, and the Y

_{M} axis is parallel to the OM’s optical axis. The robot coordinates {N} are established on the robot with the origin being on the rotation axis of RP and also on the outer surface of LP_2. The directions of the X

_{N} axis and Y

_{N} axis are parallel with the movement directions of LP_2 and LP_1 respectively. As RP rotates, the X

_{N} axis and Y

_{N} axis would also rotate.

**Figure 2.**
Illustration of the system setup for sample alignment and imaging. (**a**) Two coordinates {M} and {N} are established to represent the microscope coordinates and the robot coordinates and (**b**) shows that the sample may move out of the FOV before it is aligned to the rotation axis of the robot.

**Figure 2.**
Illustration of the system setup for sample alignment and imaging. (**a**) Two coordinates {M} and {N} are established to represent the microscope coordinates and the robot coordinates and (**b**) shows that the sample may move out of the FOV before it is aligned to the rotation axis of the robot.

Given a to-be-aligned point P on the sample with initial coordinates in {N}. If we rotate the robot by an angle clockwise and anticlockwise, respectively, the coordinates of point P in {M} before rotation (

Figure 3a), after rotating a certain number of degrees clockwise forward and anticlockwise backward can be expressed by Equation (1) [

28]:

where

$i=o,f,b$, represent point P in {M} before rotation (

Figure 3a), after rotating clockwise forward and after rotating anticlockwise backward, respectively.

${n}_{o}=0$,

${n}_{f}=\alpha $,

${n}_{b}=-\alpha $,

u is the represented physical distance (µm) of each pixel in the microscope image; RX, RY and RZ represent the rotation matrix of {N} relative to {M} about the X

_{M}-axis, Y

_{M}-axis, and Z

_{M}-axis, respectively; T

_{MN} is the translational matrix between the two coordinate systems. We define

$\Delta {x}_{f}$ and

$\Delta {x}_{b}$ (

Figure 3b) which represents the position shift of point P on the microscope images after rotating clockwise and rotating anticlockwise, respectively (

$i=f,b$):

**Figure 3.**
Illustration of the forward-backward alignment approach. (**a**) The position of point P in {N} at the initial position, after forward rotation (clockwise) and backward rotation (anticlockwise); (**b**) The coordinates of point P in {M} during the alignment procedure.

**Figure 3.**
Illustration of the forward-backward alignment approach. (**a**) The position of point P in {N} at the initial position, after forward rotation (clockwise) and backward rotation (anticlockwise); (**b**) The coordinates of point P in {M} during the alignment procedure.

Angles ${\theta}_{x0}$, ${\theta}_{y0}$, and ${\theta}_{z0}$ represent the initial rotation angle of {N} relative to {M} about the X_{M} axis, Y_{M} axis and Z_{M} axis, respectively. The three angles (${\theta}_{x0}$, ${\theta}_{y0}$, ${\theta}_{z0}$) are manually compensated to be zero before the automatic alignment experiments. Angle ${\theta}_{x0}$ is compensated to be zero by adjusting the Y_{N} axis to be parallel with the Y_{M} axis. Angle ${\theta}_{y0}$ is compensated to be zero by adjusting the robot’s holder so that robot’s edges are parallel with the image plane’s edges. Angle ${\theta}_{z0}$ is compensated to be zero by rotating the rotary positioner to make the X_{M} axis parallel with the X_{N} axis.

Then, Equations (4) and (5) can be simplified to:

By solving the above two Equations (6) and (7), the to-be-aligned point P’s coordinate

${\left({x}_{n0},{y}_{n0},{z}_{n0}\right)}^{T}$ in {N} can be expressed by:

In Equation (8), $\Delta {x}_{f}$ and $\Delta {x}_{b}$ can be measured from the microscope by image processing; α is the rotation angle defined by the users; µ (um/pixel) represents the physical distance of each pixel, which can be calibrated based on the microscope’s magnification. After the to-be-aligned P’s coordinates ${\left({x}_{n0},{y}_{n0}\right)}^{T}$ in {N} are obtained, the sample can be aligned to the rotation axis of RP by moving LP_1 by $-{y}_{n0}$ and LP_2 by $-{x}_{n0}$, respectively.

The proposed alignment approach is based on the “to-be-aligned point”. Therefore, it can theoretically successfully align the “point of interest” to the rotation axis of the robot regardless of the installation position of the sample. However, in real practice, we always expect to view more regions in a single image. In this case, the sample needs to be aligned carefully to make most of the surfaces have a similar focus depth during the movement. Thus, in this paper, a T-shape sample stage is developed to hold the sample in a direction vertical to the microscope lens, by which a wide region of the sample can be imaged clearly during the rotation.

Finally, to verify the viability and robustness of this alignment approach, we implement a sample alignment experiment based on the method proposed above (see the

Supplementary Material for details). The results prove that this alignment approach has high repeatability regardless of the rotation speed, and the alignment accuracy can meet the requirement for multi-directional imaging under a microscope.

#### 2.3. Multidirectional Imaging

After the sample is aligned to the rotation axis of the robot, the sample is able to remain in the FOV of the microscope during the rotation. Thus, a series of images can be taken from different directions by continuously rotating the robot.

In this experiment, the travel speed of the linear positioner and the rotary positioner are set to 667 µm/s and 5 °/s, respectively; the rotation angle α is chosen as 5°. After the sample is aligned to the rotation axis of the robot, the focus of the microscope should be manually adjusted once to focus on the sample. After that, we can rotate the robot endlessly to view the sample from multiple directions freely, during which there is no need to adjust the focus any more.

Next, to further illustrate the image information that can be obtained, we reconstructed a panoramic image by automatic panoramic image stitching. We developed a new multidirectional imaging software system, which combines a serial of input images (the small images at the bottom of the

Figure 4) into a panoramic image (the image at the top of the

Figure 4), and creates a new projection of the panoramic image. The core contribution of the software is the new designed image stitching algorithm according to the characteristics of the robotic system. The stitching process can be divided to three steps [

29],

i.e., feature matching, image matching, and bundle adjustment.

**Figure 4.**
Developed software system to create panoramic image from a series of images taken from different rotation angles.

**Figure 4.**
Developed software system to create panoramic image from a series of images taken from different rotation angles.

#### 2.3.1. Feature Matching

The first step in our algorithm is to extract and match scale-invariant feature transform (SIFT) features between all of the images. SIFT features are located by using maxima and minima of the result of different Gaussian functions applied in scale space to a series of smoothed and re-sampled images. In our system, the SIFT features are extracted directly by using functions for detecting SIFT image features, which are written based on [

30]. Here, low contrast candidate points and edge response points along an edge are discarded, and dominant orientations are assigned to localized SIFT features. These steps ensure that the SIFT features are robust to changes in illumination, noise, and minor changes in viewpoint. In addition to these properties, they are highly distinctive, relatively easy to extract and allow for correct object identification with low probability of mismatch. Thus, our algorithm is more stable for matching and can handle images with varying illumination and noise. Once features have been extracted from all n images, robust correspondences are required in order to estimate the necessary transformation to align an image with the next image.

In our robotic system, the robot rotates the axis of OZ, and a series of images of the sample can be taken continuously with a fixed rotation angle α. Under this condition, only neighboring images are searched for matching features, and the distances between all matching features in the neighboring images have the similar values of z. According to these characteristics, a new constraint condition is added to help the feature matching stage.

Assuming that any SIFT features detected in the current image is defined as

${u}_{o}=[{x}_{mo},{z}_{mo}]$, we search the matching SIFT feature in the next image by using a k-d tree algorithm [

31], which finds the nearest neighbour of the given input point in the tree. Here, the searching range of k-d tree is limited as

${x}_{mf}\in [0,width],{z}_{mf}\in [{z}_{mf}-\sigma ,{z}_{mf}+\sigma ]$, and the result is define as

${u}_{f}=[{x}_{mf},{z}_{mf}]$. In this way, the result of the search is more accurate and execution of the comparison is faster.

#### 2.3.2. Image Matching

Assuming that the robot rotates the axis of RP, which is equivalent to the microscope rotates about its optical center, the group of transformations the images may undergo is a special group of homographies. We denote each robot’s location as 3 rotation angles

$\mathsf{\text{\alpha}}=\left[{\alpha}_{1},{\alpha}_{2},{\alpha}_{3}\right]$ and the work distance of the microscope as

$\text{f}$. For small changes in image position, this gives pairwise homographies:

where

${H}_{of}={K}_{o}{R}_{o}{{R}_{f}}^{T}{{K}_{f}}^{-1}$,

${u}_{o}=[{x}_{mo},{z}_{mo}]$,

${u}_{f}=[{x}_{mf},{z}_{mf}]$,

${K}_{o}=\left[\begin{array}{ccc}{f}_{o}& 0& 0\\ 0& {f}_{o}& 0\\ 0& 0& 1\end{array}\right]$${R}_{o}={e}^{[{\alpha}_{o}]},\text{}[{\alpha}_{o}]=\left[\begin{array}{ccc}0& -{\alpha}_{o3}& {\alpha}_{o2}\\ {\alpha}_{o3}& 0& -{\alpha}_{o1}\\ -{\alpha}_{o2}& {\alpha}_{o1}& 0\end{array}\right]$After feature matching, the homography can be calculated by the matches between the images. For each image, we consider

$\text{m}$ candidate matching images, which have the greatest number of feature matches to this image. Then we find geometrically consistent feature matches using RANSAC algorithm [

32] to estimate the homography. Here, we select sets of

$\text{r}=4$ feature correspondences and compute the homography

${H}_{of}$ between them using the direct linear transformation (DLT) method [

33]. And we repeat this with n = 500 trials and find the correct solution that has the maximum number of inliers. Furthermore, to obtain multi-directional imaging with a set of images and reject noise images which match to no other images, the probabilistic model proposed in [

29] is also applied to verify the match.

#### 2.3.3. Bundle Adjustment

For each connected component, bundle adjustment [

34] is performed to solve for the rotation

${\alpha}_{1}$,

${\alpha}_{2}$,

${\alpha}_{3}$ and the working distance

$\text{f}$ of the microscope. That is, each feature is projected into all the images in which it matches, and the sum of squared projection error is minimized with respect to the camera parameters. Given a correspondence

${u}_{o}^{k}\leftrightarrow {u}_{f}^{l}$,

${u}_{o}^{k}$ denotes the position of the

kth feature in image

o, the residual is:

In order to get the parameters of the microscope, the sum of the squared projection error is defined as follows:

where

$\mathsf{\text{\sigma}}$ is a distance threshold. We use an outlier distance

$\mathsf{\text{\sigma}}=\infty $ during initialization and

$\mathsf{\text{\sigma}}=2$ pixels for the final solution. Then the parameters are updated using Levenberg-Marquard algorithm. Each iteration step is of the form:

where

$\Phi $ are all the parameters,

$r$ is the residual and. We

$J=\frac{\partial r}{\partial \Phi}$ encode our prior belief about the parameter changes in the covariance matrix

${C}_{p}$:

Considering the rotate angle of the robot, we set the standard deviation of angles is

${\sigma}_{\alpha}=\frac{\pi}{18}$,

${\sigma}_{f}=\frac{f}{10}$. In addition, we compute

${J}^{T}J$ and

${J}^{T}r$ in Equation (12) as follows:

After solving the above parameters, we render panorama using multi-band blending to filter the some image edges.