#### 3.1. Concept

According to the constraints stated in the previous section, the camera-driven frame-by-frame intermittent tracking method cannot always derive the maximum performance of a high-frequency response actuator, and the frame rate of a high-speed vision system should be lowered so as to maintain the linear trajectory of the actuator during the time the camera shutter is open. The very flexible controllability of the high-speed vision system, whose frequency response is much higher than that of the actuator, was not fully utilized in the frame-by-frame intermittent tracking.

Thus, in this study, we propose an improved frame-by-frame intermittent tracking method that can reduce motion blur in video shooting by controlling the camera shutter timings in synchronization with the resonant vibration of a free-vibration-type actuator such as a resonant mirror. Its high-frequency vibration with a large amplitude enables the ultrafast gaze control to track fast-moving objects during the time the camera shutter is open.

Figure 2 shows the concept of our proposed actuator-driven frame-by-frame intermittent tracking method.

When the camera’s viewpoint moves unidirectionally, the viewpoint’s position

$x\left(t\right)$ at time

t vibrates at a cycle time of

$T=1/{f}_{0}$ on the following sinusoid trajectory,

where

${f}_{0}$ is the resonant frequency of the free-vibration-type actuator and

$A\left(t\right)$ is the amplitude of the vibration at time

t, assuming

$x\left(t\right)=0$ when

$t=0$. In the actuator-driven tracking approach, the exposure start and end times to capture the image at frame

k, which are expressed as

${t}_{k}^{O}$ and

${t}_{k}^{C}$, respectively, are controlled so that the camera shutter is open when the viewpoint is located in the highly linear range within the sinusoid trajectory. In parallel with the shutter timing control, the slope of the approximate line to the sinusoid trajectory when the camera shutter is open, which indicates the speed of the camera’s viewpoint, is controlled for motion blur reduction so as to coincide with the apparent speed of the target object on the image sensor. In the frame-by-frame intermittent tracking with a free-vibration-type actuator, the resonant frequency, which is a fixed value peculiar to the actuator, is not controllable, and the speed of the camera’s viewpoint can be controlled with the amplitude of the vibration, as well as the exposure start and end times, which determine the time range for the linear approximation to the sinusoid trajectory.

Figure 3 shows the control scheme of our actuator-driven tracking approach. Compared to the performance-limited mechanical actuator control in the camera-driven tracking approach, the actuator-driven tracking approach can derive the maximum mechanical performance of a free-vibration-type actuator enables motion-blur-free video shooting of faster moving objects at a higher frame rate, whereas a free-vibration-type actuator is plagued by the following limitations:

(1) Unresponsive amplitude control in resonant vibration

A free-vibration-type actuator tends to move on a periodic trajectory with a certain hysteresis caused by friction, and it is largely deviated from the ideal sinusoid trajectory in the case of the resonant vibration with a small amplitude. In the actuator-drive tracking approach, such properties may degrade the tracking performance in video shooting a target object whose speed is either very low or varies with time.

(2) Limited time aperture ratio

In the camera-driven tracking approach, the time aperture ratio, which is the ratio of the frame interval and the exposure time in video shooting, can be programmably determined by designing the target trajectory of the camera’s viewpoint freely, whereas the high-frequency response actuator cannot move on the target trajectory with a large amplitude, due to its limited movable range and speed. On the other hand, the time aperture ratio in the actuator-driven tracking approach is limited due to the sinusoid trajectory with resonant vibration. This is because the camera shutter timings are automatically determined so as to guarantee the linear motion of the camera’s viewpoint when the camera shutter is open, whereas the percentage of the linear range on the sinusoid trajectory decreases as the exposure time increases.

#### 3.2. Camera Shutter Timings and Vibration Amplitude

In motion-blur-free video shooting with actuator-driven frame-by-frame intermittent tracking, the nonlinear sinusoid trajectory with resonant vibration of a free-vibration-type actuator, which is segmented in the time range when the camera shutter is open, deviates from its approximate straight line more extensively as the camera exposure time increases. For motion-blur-free video shooting without lowering the incident light, it is important to determine a larger camera exposure time with consideration of the permissible deviation error in straight-line approximation, which corresponds to the degree of motion blur. In this subsection, we discuss how to determine parameters for camera shutter timings in actuator-driven frame-by-frame intermittent tracking on the basis of the numerical relationship between the segmented sinusoid trajectory and its approximate straight line.

As illustrated in

Figure 4, the input image is captured at frame

k with an exposure time

$\tau $ by opening and closing the camera shutter at times

${t}_{k}^{O}={t}_{k}-\tau /2$ and

${t}_{k}^{C}={t}_{k}+\tau /2$, respectively. In this study, we assume that the center time of the camera exposure is set to

${t}_{k}=2n\pi $ (

n: integer) to synchronize with the sinusoid trajectory

$x\left(t\right)=Asin(2\pi /T)t$ so that the slope of a tangent to the sinusoid trajectory is maximum at time

${t}_{k}$. To track a target object moving at a speed of

v in images when the camera shutter is open, we assume that the amplitude

A of the sinusoid trajectory is so controlled that the straight line

$y\left(t\right)=vt$ approximates the segmented sinusoid trajectory in the range of time

${t}_{k}^{O}$ to time

${t}_{k}^{C}$. Here, we assume that the open and close times for camera exposure are

${t}_{k}^{O}=-\tau /2$ and

${t}_{k}^{C}=\tau /2$, respectively, by setting the center time to

${t}_{k}=0$ for simplification, and the

y-intercept of the approximate line is zero because the segmented sinusoid trajectory in the range of time

${t}_{k}^{O}$ to time

${t}_{k}^{C}$ is symmetric about the center time

${t}_{k}$. To estimate the amplitude

A, we consider a minimization problem for the following squared-error loss function that can evaluate the deviation of the segmented sinusoid trajectory from the straight line where the target object moves,

Solving the following equation such that the partial derivative of

$E\left(A\right)$ with respect to

A is zero,

the amplitude

${A}_{min}$ can be derived as follows:

where

$r=\tau /T$ is the temporal aperture ratio that indicates the ratio of the exposure time

$\tau $ to the cycle time

T of the sinusoid trajectory.

Figure 5 shows the relationship between the temporal aperture ratio

r and the amplitude ratio of

${A}_{min}$ to

${A}_{0}$;

${A}_{0}$ is the slope of the tangent line of the sinusoid trajectory at time

${t}_{k}$ when the exposure time

$\tau $ approaches zero as follows:

Thus, the minimum value

${E}_{min}$ of the squared-error loss function is obtained as follows:

Without actuator-driven frame-by-frame intermittent tracking, the squared-error loss

${E}_{NT}$ in the range of time

${t}_{k}^{O}$ to time

${t}_{k}^{C}$ when observing a target object moving at speed

v can be described as the value of the loss function when the amplitude of the sinusoid trajectory is

$A=0$, corresponding to no camera motion for tracking, as follows:

Considering the roots of the squared-error losses of

${E}_{min}$ and

${E}_{NT}$, the relative error ratio

$\epsilon $ is defined as follows:

where

$\epsilon $ indicates the degree of motion blur reduction in video shooting with frame-by-frame intermittent tracking, compared with the deviation error in video shooting without tracking;

$\epsilon =f\left(r\right)$ is a monotonically increasing function of the temporal aperture ratio

r, and motion blur is largely canceled when

$\epsilon $ approaches zero. Thus, the temporal aperture ratio

r can be expressed as a monotonically increasing function of the relative error ratio

$\epsilon $, which is independent of the cycle time

T of the sinusoid trajectory and the target speed

v, as follows:

Figure 6 shows the relationship between the temporal aperture ratio

r and the relative error ratio

$\epsilon $. Using the relationship between

r and

$\epsilon $ in

Figure 6 as a look-up table, the camera shutter timings can be automatically determined in actuator-driven frame-by-frame intermittent tracking when the permissible degree of motion blur is initially given. For example, the relative error ratio

$\epsilon $ is permissible up to 1%, 5% and 10%, respectively, and the upper-limit values of the allowable temporal opening ratios are

$r\left(0.01\right)=0.151$,

$r\left(0.05\right)=0.333$, and

$r\left(0.1\right)=0.460$, respectively. Especially when the exposure time is constant, the open and close times for camera exposure can be determined independently from the time-varying amplitude

A of the sinusoid trajectory, which is controlled so as to cancel the apparent speed of the target object when the camera exposure is open, and these signals are generated in synchronization with the external synchronization signal from a free-vibration-type actuator.