Performance of Very Small Robotic Fish Equipped with CMOS Camera

Abstract

Underwater robots are often used to investigate marine animals. Ideally, such robots should be in the shape of fish so that they can easily go unnoticed by aquatic animals. In addition, lacking a screw propeller, a robotic fish would be less likely to become entangled in algae and other plants. However, although such robots have been developed, their swimming speed is significantly lower than that of real fish. Since to carry out a survey of actual fish a robotic fish would be required to follow them, it is necessary to improve the performance of the propulsion system. In the present study, a small robotic fish (SAPPA) was manufactured and its propulsive performance was evaluated. SAPPA was developed to swim in bodies of freshwater such as rivers, and was equipped with a small CMOS camera with a wide-angle lens in order to photograph live fish. The maximum swimming speed of the robot was determined to be 111 mm/s, and its turning radius was 125 mm. Its power consumption was as low as 1.82 W. During trials, SAPPA succeeded in recognizing a goldfish and capturing an image of it using its CMOS camera.

1. Introduction

Undersea robots are often used to investigate marine animals [1,2,3,4]. For example, URASHIMA has the ability to sail long distances and take clear pictures on the ocean floor [1,2]. AUVs like TUNA-SAND, etc., succeeded in taking videos of natural phenomena or aquatic animals and gathering data about the sea [3,4]. However, such robots are generally impractical in narrow spaces such as rivers and ponds, particularly if they are driven by a screw propeller that can become entangled in algae and aquatic plants. In addition, when investigating aquatic animals such as fish, it would be necessary for a robot to go unnoticed by the target animals. For the above reasons, robotic fish have attracted attention and has been developed [5,6]. Although robotic fish have been developed, their swimming speed has been significantly lower than that of real fish. For such a robot to monitor live fish, it is therefore necessary to improve its propulsive performance. Also, the robot should be as small as possible to allow it to navigate in shallow waters. In addition, it must have the ability to follow real fish in order to study them. The small robotic fish FOCUS developed in a previous study [7,8] had the ability to track a moving object based on color information (red) obtained using a CMOS camera. However, it moved too slowly to capture images of live fish due to the single-joint mechanism used to drive its tail fin.

In the present study, in an attempt to develop a more practical robotic fish, the present model was downsized in order to improve its performance and to avoid frightening small fish. The robot is equipped with a wide-angle CMOS camera in its head to allow it to photograph freshwater fish and recognize the red color of goldfish. It is modeled on the big-eyed herring (Sardinella zunasi), known in Japan as a SAPPA. It is designed to swim in a freshwater environment such as a river, although big-eyed herring actually swim in the sea. Its propulsive performance is investigated through a series of swimming experiments. SAPPA is not an autonomous robot, but a robot operated by remote control. Therefore, a purpose of the present study is not following goldfish autonomously. We just confirmed that the robotic fish can recognize goldfish as goldfish.

2. Robotic Fish SAPPA

2.1 Structure of SAPPA

Figure 1 shows a photograph of SAPPA, and a schematic of its internal structure is shown in Figure 2. It has a total length of 170 mm, a height of 65 mm, a width of 40 mm, and a weight of 140 g as determined by aerial gravimetry. Its outer shell is constructed from acrylic parts. The tail fin is connected to the main body by a two-joint driving mechanism containing two servomotors (Nine Eagles, NE-53070017, Shanghai, China). The driving force produced by each motor moves the tail fin from side to side. Since a real fish can move upwards or downwards by changing its direction of travel, the robot has a posture control system to change its direction of propulsion, as shown schematically in Figure 3. A weight is moved by a DC motor, and the position of the weight is measured with a proximity sensor (AsakusaGiken, AS-PROX, Chiba, Japan) to control the center of gravity of the robot. When the weight moves forward, the robot swims downwards, and when it moves backward, the robot swims upwards. The robot has an RGB CMOS camera (NIPPON CHEMI-CON, NCM03-S, Tokyo, Japan) in its head to take pictures of red goldfish, and a USB interface to connect to a PC in order to check the images.

Figure 1. Robotic fish SAPPA (BL = 170 mm).

Figure 1. Robotic fish SAPPA (BL = 170 mm).

Figure 2. Structure of SAPPA.

Figure 2. Structure of SAPPA.

Figure 3. Posture control system for changing depth.

Figure 3. Posture control system for changing depth.

A lithium polymer battery (3.7 V, 240 mAh) is used as the electric power source. In order to remote control SAPPA, the robot is equipped with a receiver (Futaba, R124H, Chiba, Japan). The receiver has a range of up to 3 m. Commands to control and perform image processing are issued by a field-programmable gate array (FPGA) board (HUMANDATA, XP68-04-LX45, Osaka, Japan) containing a Spartan-6 XP68-03-LX45 (Xilinx).

A circuit diagram of the electronic system is shown in Figure 4. Data from the RGB camera (DATA0 to DATA7) are received from the input terminals (IOB14 to IOB7). The data are analyzed in the FPGA and the output signals are sent to the USB interface board via the output terminals (IOA9 to IOA2). The tail fin is driven by Servo motor 1 and Servo motor 2. The weight in the posture control system is driven by the DC motor and power MOSFET circuit.

Figure 4. Circuit diagram for SAPPA.

Figure 4. Circuit diagram for SAPPA.

Figure 5. Control flowchart of posture control system.

Figure 5. Control flowchart of posture control system.

SAPPA can change its altitude to move upwards or downwards by a posture control system. The posture control system is equipped with a weight and a DC motor which moves the weight back and forth. The weight position changes the center of gravity. Figure 5 shows the flowchart to control the DC motor in the posture control system. The position of the weight is measured with a proximity sensor. An analog-digital conversion element (Texas Instruments Japan Limited, ADS7883, 12 bit, 3 MS/s, Tokyo, Japan) receives a signal from the proximity sensor and converts the analog signal to a digital signal. The FPGA compares the value of the digital signal and a target value. According to the difference of these values, the FPGA determines a signal to control the DC motor. The way to control the DC motor is ON or OFF. To avoid the vibration of the weight, a parameter concerning the sensitivity for the difference of the proximity sensor value is prepared in the VHDL code. Here, the target value and sensitivity are determined three ways according to the signal received from the receiver;

(1)

Target value is 4.50 V and sensitivity is 0.64 V,

(2)

Target value is 0.25 V and sensitivity is 0.30 V,

(3)

Target value is 0.13 V and sensitivity is 0.00 V.

The ON or OFF position of the motor is determined according to these target value and the value of the proximity sensor through the flowchart shown in Figure 5. When IOB16 = “0” and IOB17 = “1”, the weight moves forward. Then when IOB16 = “1” and IOB17 = “0”, the weight moves back. If IOB16 and IOB17 are “0”, then the weight does not move.

2.2 Swimming Experiments

In order to investigate the propulsive performance of SAPPA, a pool was prepared to conduct swimming experiments. The pool was 2390 mm long, 1500 mm wide, and 580 mm deep (the water depth was 250 mm). Figure 6 shows a schematic of the tail of the robot, which was constructed from polypropylene (thickness: 0.5 mm) and natural rubber (thickness: 0.3 mm). In the experiment, the phase difference between the angles of the two servomotors was varied in steps of 45°. Since the robot requires a certain distance to reach full speed, it was allowed to swim for 1000 mm, and then the time taken for it to swim a further 500 mm was measured. This experiment was conducted six times for each phase difference, and the average speed was obtained, as shown in Figure 7. The vertical axis of Figure 7 on the left and right side show measurements in mm/s and BL/s, respectively. The maximum speed was 0.653 BL/s (111 mm/s) for a phase difference of 0°. Figure 8 shows a series of snapshots taken at one-second intervals when the robot was swimming at maximum speed. It can be seen that several goldfish followed SAPPA during the experiment, showing that it does not frighten live fish.

Figure 6. Schematic of robot tail.

Figure 6. Schematic of robot tail.

Figure 7. Dependence of swimming speed on phase difference.

Figure 7. Dependence of swimming speed on phase difference.

Then, the turning radius of the robot and its ability to move upwards and downwards were evaluated. The experimental results are shown in Figure 9 and Figure 10. The turning radius was found to be 0.735 BL (125 mm) at 0.305 BL/s (51.9 mm/s). The robot was capable of rising at up to 60° and sinking at up to 50° from the horizontal position, respectively.

Figure 8. Propulsive performance measurements (phase difference 0°).

Figure 8. Propulsive performance measurements (phase difference 0°).

Figure 9. Turning radius.

Figure 9. Turning radius.

Figure 10. SAPPA moving upwards and downwards. (a) Rising (60°); (b) Sinking (50°).

Figure 10. SAPPA moving upwards and downwards. (a) Rising (60°); (b) Sinking (50°).

2.3 Evaluation of Propulsion Performance

Nagai et al. proposed a propulsive performance index S_w for real or robotic fish [9]. It is referred to as the “swimming number”, and it is the ratio of the distance swum during one period of the tail fin’s reciprocating motion to the total length of the fish. It is calculated using Equation (1), where f (Hz) is the movement frequency of the tail fin, L (m) is the total length of the fish, and u (m/s) is the swimming speed. Larger values of S_w are an indication of more efficient swimming.

$$S_w=\frac{u}{fL}$$

(1)

In the case of SAPPA, the total length is 170 mm, the frequency of the tail fin is 2 Hz, and the maximum propulsion speed is 111 mm/s. Therefore, S_w has a value of 0.326, which is considerably larger than that for the previously developed robotic fish FOCUS [7]. The swimming number of FOCUS is 0.146.

To estimate the propulsive efficiency of SAPPA, the power consumption and propulsive force were measured. The power consumption was determined by measuring the voltage and the electric current from the battery in the robot when it was swimming. Figure 11 shows a scheme of measuring the voltage and the electric current. If the Li-Po battery is in SAPPA, it is difficult to measure the voltage and the current. We supplied electricity to SAPPA from outside to measure the voltage and the current. The lead wires were slackened to prevent tension from influencing propulsion of the robot. We used the voltmeter PC510 and ammeter DCM400AD (SANWA ELECTRIC INSTRUMENT CO., LTD, Tokyo, Japan). However, it was difficult to measure the propulsive force when the robot was swimming. Therefore, it was assumed that the drag force when the robot was held motionless in circulating water was equal to the propulsive force [10,11,12,13] that would be exerted by the robot when it was swimming at the same speed as the circulating water. Equation (2) gives the propulsive efficiency η, where the drag force is D (N), the velocity is u (m/s), and the power consumption is P (W).

$$\eta =\frac{Du}{P}\times 100$$

(2)

The power consumption of SAPPA was measured while varying the phase difference between the two tail-fin motors in steps of 45°, and the results are shown in Figure 12.

Figure 11. Experimental setup for measuring power consumption.

Figure 11. Experimental setup for measuring power consumption.

Figure 12. Dependence of electricity consumption on phase difference.

Figure 12. Dependence of electricity consumption on phase difference.

Figure 13 shows SAPPA when the drag force was measured, and Figure 14 shows a schematic of the experimental setup. A shaft was installed on the head of the robot to attach it to the experimental system, which was composed of a circulating water tank (WEST JAPAN FLUID ENGINEERING LABORATRY CO., LTD, P-70, Nagasaki, Japan), a load cell (A&D Co., Ltd., LC4001-G120, Tokyo, Japan), a load cell amplifier (TEAC, TD-300A, Tokyo, Japan), a power supply, and a personal computer (PC). The PC was equipped with an analog-to-digital convertor board with a PCI interface, allowing it to read the output of the load cell amplifier. The electric current and the voltage of the battery in the robot were simultaneously measured.

Figure 13. Robotic fish held by shaft.

Figure 13. Robotic fish held by shaft.

Figure 14. Experimental setup for drag force measurement.

Figure 14. Experimental setup for drag force measurement.

The circulating water tank was an impeller type and was driven by a three-phase 200 W AC motor. The water speed in the circulating water tank can be controlled in 0 mm/s to 570 mm/s by a small inverter (VfnC1-2002P). The drag of each propulsive speed was measured as we changed the water speed in the circulating water tank according to the propulsive speed of the robotic fish. An overview of the apparatus for measuring the drag force is shown in Figure 15. The robot was fixed to the measuring apparatus so that it could not move, allowing the drag force in the circulating water to be measured by the load cell. Since the robot was fixed, the drag force varied with the water flow speed. The drag force was measured for water flow speeds of 20 to 200 mm/s in steps of 20 mm/s. In addition, the drag force was also measured for water flow speeds corresponding to the actual measured speeds of the robot during the swimming experiments. These were 21, 77, 80, 102, 107, 108, 110, and 111 mm/s, and were obtained by changing the phase difference between the tail-fin motors. The results are shown in Figure 16.

Figure 15. Measurement apparatus for drag force.

Figure 15. Measurement apparatus for drag force.

Figure 16. Dependence of drag force on water flow speed.

Figure 16. Dependence of drag force on water flow speed.

Figure 17. Dependence of propulsive efficiency on phase difference.

Figure 17. Dependence of propulsive efficiency on phase difference.

For a phase difference of 0°, a swimming speed of 111 mm/s, an estimated drag force of 3.9 mN, and a power consumption of 2.26 W, the propulsive efficiency was calculated to about 0.0756%. Figure 17 shows the dependence of the propulsive efficiency on the phase difference. The highest value was 0.0756% for a phase difference of 270°. Therefore, for a drag D = 0.0048 N, a density of water ρ = 999.97 kg/m³, a frontal projected area of A = 6540 mm², and a swimming speed of u = 111 mm/s, the drag coefficient C_D can be obtained as 0.382 using Equation (3). It is thought that the propulsive efficiency is low. We think that a cause is the motors used for SAPPA. The motor that was used for SAPPA is NE-53070017 (Nine Eagles, Shanghai, China). It is very a small servomotor with a size of 15 mm × 20 mm × 5 mm. It generates heat because of its large electrical current. It seems that a large amount of energy is lost as thermal energy in the servomotor.

$$D=\frac{\rho }{2}{C}_{D}A{V}^2$$

(3)

3. Capturing Images of Live Freshwater Fish

Table 1 and Table 2 show the specifications of the FPGA board and the wide-angle CMOS camera that are installed in the robot. Because of the wide field of view of the camera, SAPPA can find live fish more easily than the previous robotic fish FOCUS. The forward-facing CMOS camera has a resolution of 640 pixels × 480 pixels (VGA). The image data in RGB565 format are sent to the FPGA at 30 frames per second. Because of the limited RAM capacity of the FPGA, only the central 480 pixel × 320 pixel region of the image is recorded.

Table 1. Specifications of FPGA board.

Device Name	Spartan-6 (XC6SLX45-2CSG324C)
Board size	25.3 mm × 25.3 mm × 5.0 mm
Logic cells	43,661
Addressable RAM	2088 kbit
Number of user I/O ports (Board)	50
Supply voltage	2.8, 3.3 V
Clock frequency	50 MHz

Table 1. Specifications of FPGA board.

Device Name	Spartan-6 (XC6SLX45-2CSG324C)
Board size	25.3 mm × 25.3 mm × 5.0 mm
Logic cells	43,661
Addressable RAM	2088 kbit
Number of user I/O ports (Board)	50
Supply voltage	2.8, 3.3 V
Clock frequency	50 MHz

Table 2. Specifications of CMOS camera.

Optical Format		1/4 Inch
Field of view	Horizontal	105°
	Vertical	78°
	Diagonal	130°
Structure		Double lens

Table 2. Specifications of CMOS camera.

Optical Format		1/4 Inch
Field of view	Horizontal	105°
	Vertical	78°
	Diagonal	130°
Structure		Double lens

3.1 Color Information for Goldfish

In the present study, goldfish were selected as the target fish because they have a characteristic red color and are easily obtainable in Japan. In order to ensure that SAPPA can identify goldfish based on their color, it is necessary to obtain detailed color information for such fish. Their color may appear different depending on the CMOS camera used. Therefore, the robot was made to record images of goldfish swimming in the water tank. The experimental setup is shown in Figure 18. The RGB data in the RAM were sent to the PC through the USB connection, and the colors were analyzed. An example of a captured image is shown in Figure 19, and Table 3 lists the RGB values at the points indicated in the image. Although goldfish were found to have a wide range of RGB values, the R value was always larger than the G or B value.

Figure 18. Experimental setup for measuring fish color.

Figure 18. Experimental setup for measuring fish color.

Figure 19. Captured image stored in RAM.

Figure 19. Captured image stored in RAM.

Table 3. Color information for goldfish (R, G, B).

① (136, 67, 0)	② (145, 39, 0)	③ (128, 63, 57)
④ (138, 66, 7)	⑤ (170, 65, 14)	⑥ (141, 46, 29)

Table 3. Color information for goldfish (R, G, B).

① (136, 67, 0)	② (145, 39, 0)	③ (128, 63, 57)
④ (138, 66, 7)	⑤ (170, 65, 14)	⑥ (141, 46, 29)

Based on the obtained color information, the following expression was used by SAPPA to identify a goldfish.

(R > 2G) and (R > 2B) and (R > 50)

(4)

3.2 Procedure for Recognizing the Target Color

The procedure used to recognize the color of the target in order to distinguish a goldfish is as follows. First, the pixels satisfying Equation (4) above are extracted from the camera image. Second, the coordinates of the center of the extracted red pixels are calculated in the FPGA. Finally, vertical and horizontal white lines intersecting at these coordinates are added to the image in RAM. The experimental result is shown in Figure 20. SAPPA could correctly identify goldfish by recognizing their red color.

Figure 20. Successful identification of goldfish.

Figure 20. Successful identification of goldfish.

4. Conclusions

In the present study, a very small robotic fish referred to as SAPPA was developed. The purposes of the study were to downsize the robot, improve its propulsive performance, and evaluate its ability to capture images of live goldfish using a wide-angle CMOS camera. The total length of the robot was just 170 mm, which is extremely small considering that it contains a camera and a two-joint tail fin. Its maximum swimming speed was 0.653 BL (111 mm/s) and its swimming number was 0.326. Its turning radius was only 0.735 BL (125 mm), and it could climb and dive at up to 60° and 50°, respectively. To the best of our knowledge, this represents the first report on such a compact, high-performance, robotic fish equipped with a camera and driven by a tail fin. The robot was also shown to be capable of instantly identifying goldfish based on images recorded by the CMOS camera. It is therefore highly promising for use in studying aquatic animals without disturbing them.