FishSeg: 3D Fish Tracking Using Mask R-CNN in Large Ethohydraulic Flumes

To study the fish behavioral response to up- and downstream fish passage structures, live-fish tests are conducted in large flumes in various laboratories around the world. The use of multiple fisheye cameras to cover the full width and length of a flume, low color contrast between fish and flume bottom and non-uniform illumination leading to fish shadows, air bubbles wrongly identified as fish as well as fish being partially hidden behind each other are the main challenges for video-based fish tracking. This study improves an existing open-source fish tracking code to better address these issues by using a modified Mask Regional-Convolutional Neural Network (Mask R-CNN) as a tracking method. The developed workflow, FishSeg, consists of four parts: (1) stereo camera calibration, (2) background subtraction, (3) multi-fish tracking using Mask R-CNN, and (4) 3D conversion to flume coordinates. The Mask R-CNN model was trained and validated with datasets manually annotated from background subtracted videos from the live-fish tests. Brown trout and European eel were selected as target fish species to evaluate the performance of FishSeg with different types of body shapes and sizes. Comparison with the previous method illustrates that the tracks generated by FishSeg are about three times more continuous with higher accuracy. Furthermore, the code runs more stable since fish shadows and air bubbles are not misidentified as fish. The trout and eel models produced from FishSeg have mean Average Precisions (mAPs) of 0.837 and 0.876, respectively. Comparisons of mAPs with other R-CNN-based models show the reliability of FishSeg with a small training dataset. FishSeg is a ready-to-use open-source code for tracking any fish species with similar body shapes as trout and eel, and further fish shapes can be added with moderate effort. The generated fish tracks allow researchers to analyze the fish behavior in detail, even in large experimental facilities.