PMOT2023: A Large-Scale Multi-Object Tracking (MOT) Dataset with Application to Phytoplankton Observation

Abstract

Phytoplankton play a critical role in marine food webs and biogeochemical cycles, and their abundance must be monitored to prevent disasters and improve the marine environment. Although existing algorithms for automatic phytoplankton identification at the image level are available, there are currently no video-level algorithms. This lack of datasets is a significant obstacle to the development of video-level automatic identification algorithms for phytoplankton observations. Deep learning-based algorithms, in particular, require high-quality datasets to achieve optimal results. To address this issue, we propose the PMOT2023 (Phytoplankton Multi-Object Tracking), a multi-video tracking dataset based on 48,000 micrographs captured by in situ observation devices. The dataset comprises 21 classes of phytoplankton and can aid in the development of advanced video-level identification methods. Multi-object tracking algorithms can detect, classify, count, and estimate phytoplankton density. As a video-level automatic identification algorithm, multi-object tracking addresses trajectory tracking, concentration estimation, and other requirements in original phytoplankton observation, helping to prevent marine ecological disasters. Additionally, the PMOT2023 dataset will serve as a benchmark to evaluate the performance of future phytoplankton identification models and provide a foundation for further research on automatic phytoplankton identification algorithms.

1. Introduction

Phytoplankton are critical to the ocean ecology, as they serve as primary producers that initiate the marine food chain and significantly influence marine ecosystems and fisheries [1]. These microorganisms also participate in the biogeochemical cycles of biogenic elements such as carbon, nitrogen, and phosphorus. Their carbon sequestration process is an important marine carbon sink that regulates atmospheric CO${}_2$ concentrations, affecting global climate. Spatial and temporal variations in phytoplankton community characteristics, such as species composition and cell abundance, provide essential data for studying phytoplankton activity. However, non-in situ observation methods are limited by time and financial constraints, providing only short-term and temporary access to this information, which hinders our understanding of phytoplankton processes at various spatial and temporal scales [2]. In contrast, in situ observations of phytoplankton provide long-term, continuous access to information on phytoplankton species composition and cell abundance, enabling researchers to understand the causes and effects of changes in phytoplankton populations [3,4,5,6,7].

Among the various in situ plankton observation methods, acoustic-based observation techniques, such as Acoustic Doppler Current Profilers, Multifrequency Hydroacoustic Probing System, and broadband sonar, have outstanding advantages in terms of observation frequency, spatial range, and long-term observation capabilities. However, these techniques have limitations in their ability to observe the detailed features of plankton, their discrimination ability of different species of plankton is poor, and their ability to provide accurate quantitative and positional information is limited. Consequently, the reliability of their observation results is insufficient [8]. Chlorophyll fluorometry is the most mature, diverse, and widely used in situ observation device for marine organisms for phytoplankton detection [9]. Direct image recording of plankton using underwater imaging systems provides more intuitive observation. The Video Plankton Recorder (VPR) is an underwater video microscope system that can capture images of plankton at micron and centimeter scales [10]. The Underwater Video Profiler (UVP) uses high-resolution cameras and a powerful illumination system to record videos of zooplankton and macrophytes as the profiler passes through the water column [11]. For in situ observation of full-grained phytoplankton, FlowCytobot [12], FlowCAM [13], CytoSense [14], CytoBuoy [15], and CytoSub [16] are capable of acquiring information on phytoplankton from micro to pico scales and provide an effective method for in situ observation. However, the traditional methods require manual analysis of observation data, including manual counting and identification, which is labor intensive.

In recent years, deep learning has emerged as a powerful tool for various image processing tasks, including image classification, object detection, segmentation, and tracking. Compared with traditional observation and analysis methods, deep learning can significantly reduce knowledge and labor costs for researchers. Multi-object tracking (MOT) is a crucial task in computer vision that involves automatically identifying and tracking multiple objects in a video sequence. This technology has been successfully applied in areas such as autonomous driving and pedestrian tracking. Recently, multi-object tracking methods such as DeepSORT [17] and BYTEtrack [18] have emerged, which have demonstrated the ability to accurately localize, classify, track, and count objects in a video stream. These algorithms have the potential to assist researchers in analyzing and measuring the concentration of phytoplankton in a given time period. Despite their potential, researchers have not yet applied these methods to phytoplankton observation. The main reason for this is the lack of available datasets that meet the training requirements of video-level automatic recognition algorithms. As in Figure 1, existing phytoplankton datasets such as WHOI-Plankton [19], Kaggle-Plankton [20], etc., suffer from low resolution, grayscale maps, and a lack of video-level annotations, which make it difficult to extract effective phytoplankton features and localize the objects within the images. Furthermore, the individual-level annotations currently available, such as those provided by PMID2019 [21], do not include video-level data.

In this work, we introduce PMOT2023, a synthetic dataset of high-resolution video frame sequences of phytoplankton that can be used to develop and evaluate video-level automatic identification methods. As in Figure 2, it labels the position of the object in the picture and represents it in the video as a track with the same ID. We collected phytoplankton samples from the nearshore waters of Jiaozhou Bay, Qingdao, Shandong, China, and classified each phytoplankton in all images. Using the density and frequency of phytoplankton occurrence in the ocean as a guide, we generated four videos with varying densities and sample movement trajectories that reflect actual observation scenarios. The dataset contains a total duration of 1920s and 48,000 frames of images, each labeled with detailed positioning information, category, and movement trajectory for each phytoplankton. PMOT2023 is a high-resolution synthetic multi-object tracking (MOT) dataset that can facilitate the investigation of video-level automatic identification methods for phytoplankton. Furthermore, PMOT2023 can also be used for other deep learning algorithms, including video object detection, single object tracking, image object detection, image classification, and more, with only minimal preprocessing. We evaluated the performance of different automatic video identification methods on PMOT2023 and experimentally tested the feasibility of using a multi-object tracking approach for phytoplankton counting and concentration estimation. This work lays the foundation for the development of standardized, high-quality datasets for video-level automatic recognition algorithms in phytoplankton observation.

The remainder of this paper is organized as follows: Section 2 describes multi-object tracking and the current status of its application in phytoplankton; Section 3 describes the construction process of the PMOT2023 dataset; Section 4 performs data analysis on PMOT2023 and compares it with other datasets; Section 5 shows the performance performance and visualization results of the multi-object tracking method on the PMOT2023 platform; Section 6 presents conclusions and future research perspectives.

2. Phytoplankton Visual Tracking

Multi-object tracking is the challenge of automatically identifying multiple objects in a video and representing them as a set of highly accurate trajectories. In phytoplankton microscopic observation, multi-object tracking methods can help researchers to observe more intuitively and facilitate quantitative operations, such as counting and calculating phytoplankton density. With the rapid development of object detection [22,23,24,25,26,27], the accuracy of multi-object tracking has been greatly improved. Currently, tracking by detection is the most effective paradigm for MOT tasks. Many multi-object tracking efforts are based on SORT [28] (Simple Online And Realtime Tracking), DeepSORT [17], and JDE [29]. IOU-Tracker [30] (Intersection over Union-Tracker) directly computes the overlap between the tracklet of the previous frame and the detection. SORT first uses a Kalman filter to predict the future position and then associates it with the detection boxes to achieve tracking. The method of SORT is also used in other multi-object tracking efforts. DeepSORT uses a ReID model to generate appearance features for the object. JDE (joint det and embedding) combines detection and appearance feature learning into one model, reducing computational effort. FairMOT [31] has achieved a balance of speed and accuracy. Recently, SORT-based methods have been revitalized due to the improvement of object detection accuracy. BYTEtrack [18] incorporates low score detection results into the association process and achieves good results using only IOU association. BoT-SORT [32] (Bag of Tricks-SORT) improves Kalman filtering, boosts the results using camera motion compensation (CMC) [33], and balances the extent of the role of IOU and ReID [34,35,36,37,38,39] in the correlation to conduct state-of-the-art work on multiple datasets at MOT. However, currently, it is difficult to apply relevant multi-object tracking methods to phytoplankton observations, and the most significant obstacle is the lack of high-quality datasets. Multi-object tracking is a field that has specialized datasets in various scenarios. However, most of these datasets are focused on pedestrian and vehicle tracking. PETS2009 [40] is a relatively early pedestrian dataset, while MOT15 [41], MOT17 [42], and MOT20 [43] are the most popular datasets in the field of multi-object tracking, but they are limited to pedestrians and vehicles only. There are alternative datasets proposed for specific tasks, such as WILDTRACK [44] for multi-camera tracking and Youtube-VIS [45] for both tracking and semantic segmentation data. For autonomous driving, there are also specialized datasets, including KITTI [46], BDD100K [47], Waymo [48], and KITTI360 [49]. Some of these datasets focus on more diverse object categories than humans and vehicles, and ImageNet-Vid [50] provides trajectory annotations for 30 object categories in over 1000 videos. TAO [51] contains 833 categories and can be used to study the performance of algorithms with long-tailed distributed datasets. However, these datasets are only domain specific and not well-suited for direct application to phytoplankton observations.

Several phytoplankton datasets have been created. WHOI-Plankton [19] is a large-scale dataset for plankton classification, containing over 3.4 million images of 70 categories labeled by experts. The images were collected over almost 8 years by the Imaging FlowCytobot (IFCB) at the Martha’s Vineyard Coastal Observatory (MVCO), but only have image-level annotations and are only suitable for classification tasks. Kaggle-Plankton [20] is a plankton classification dataset consisting of 30,336 low-resolution grayscale images from 121 different classes; it also only contains image-level annotations. Researchers at Xiamen University created an information retrieval database of algal cells morphology by collecting 704 images from different perspectives. These images contained 144 species of common Chinese phytoplankton, mainly diatoms (93 species) and methanogens (40 species), and were annotated with main characteristic parameters and ecological distribution information. They also created a digital microscopic image database of phytoplankton containing 3239 microscopic images of 241 species. The PMID2019 [21] dataset contains 10,819 microscopic images of phytoplankton from 24 different categories, each annotated with bounding boxes and category information for each object. To extend the dataset to in situ applications, the makers of PMID2019 also used Cycle-GAN [52] to implement domain migration between dead and live cell samples. Although PMID2019 can be used to train and evaluate object detection methods for phytoplankton, it does not annotate video sequences and the annotation only distinguishes different categories, not different individuals, which makes the evaluation of tracking methods impossible.

We have produced the PMOT2023 dataset for the training and evaluation of multi-object tracking algorithms for phytoplankton. We utilized SORT, deepSORT, Bytetrack, and other methods to conduct experiments on the PMOT2023 dataset. In the results section, the paper presents the test results of different methods on both the PMOT2023 dataset and other multi-object tracking datasets.

3. Procedure for Constructing PMOT2023

3.1. Phytoplankton Sampling

To produce a high-quality synthetic video phytoplankton dataset, we collected an abundant sample of phytoplankton microscopic images. Due to the influence of environmental factors such as temperature, salinity, light, nutrient salts, and zooplankton feeding, the phytoplankton species composition and cell abundance show significant spatial and temporal variations. To obtain more comprehensive and realistic data, we collected phytoplankton samples of 21 species at different times and seasons in the nearshore waters of Jiaozhou Bay, Shandong, China, over a 2-year period. These included Ceratium furca, ceratium fucus, ceratium trichoceros, chaetoceros curvisetus, coscinodiscus, curve thalassiosira, guinardia delicatulad, helicotheca, lauderia cleve, skeletonema, thalassionema nitzschioides, thalassiosira nordenskioldi, sanguinea, thalassiosira rotula, Protoperidinium, Eucampia zoodiacus, Guinardia striata, and so on. During the collection, we included four zooplankton in the dataset simultaneously according to the frequency of occurrence, cladocera, tintinnid, and two classes of copepoda. The species collected include the most frequent phytoplankton in the waters of Jiaozhou Bay, as well as four categories of the most frequent zooplankton totaling 21 categories, which can meet the daily automatic observation of the waters of Jiaozhou Bay. Limited by the sampling means, we have no way to cover all the categories. We used the phytoplankton in situ observation equipment developed in our laboratory to capture and collect microscopic images of phytoplankton samples, during which we collected different individuals of a variety of phytoplankton to simulate real phytoplankton in situ measurement scenarios and provide sufficient knowledge for the automatic phytoplankton identification algorithm. These microscopic images are characterized by high resolution, and we also collected sufficient different morphologies of phytoplankton movement in the current. We supplemented this by taking microscopic images of phytoplankton cultured in the laboratory to generate our original data.

3.2. Dataset Design and Production Process

Observing phytoplankton can help prevent disasters, such as oceanic red tides, and improve the marine environment. Although various automatic phytoplankton identification algorithms are available [53,54,55,56], most of them operate at the image level, and there is a lack of video-level phytoplankton identification algorithms. Deep learning-based automatic identification algorithms rely on high-quality datasets, and the PMOT2023 dataset is a multi-video tracking dataset based on micrographs captured by phytoplankton in situ observation devices, which can assist deep learning models in producing better results. The PMOT2023 dataset contains 12 videos and 48,000 micrographs, encompassing 21 classes of phytoplankton. The PMOT2023 dataset will provide a tool for evaluating the performance of advanced video-level phytoplankton automatic identification methods.

As shown in Figure 3, we first collected seawater samples at different times and seasons in the nearshore waters of Jiaozhou Bay, Shandong, China. We collected microscopic images of phytoplankton samples in seawater with the help of phytoplankton in situ observation devices. The device contains a coaxial lens, imaged with the Hivision MV-CH20-10UC, and uses a peristaltic pump to extract seawater for observation. We cropped and segmented the collected images to produce sample sets. In the process of producing video data, first, phytoplankton samples were randomly selected in the sample set and randomly rotated, randomly scaled, and motion blur was added; second, random motion trajectories were generated using spline curves (as explained in next section); finally, selected phytoplankton samples were mapped onto the background according to the motion trajectories to create video frame sequences and generate annotations.

Extremes in phytoplankton density in the ocean are often accompanied by ecological anomalies, such as red tides, also known as harmful algal blooms (HABs), which are a global marine ecological anomaly caused by an overgrowth or accumulation of phytoplankton in the water column, resulting in discoloration of seawater. When red tide occurs, the average density of phytoplankton can reach 102–106 cells/mL, and because phytoplankton often appear in clumps, the instantaneous density of phytoplankton passing through the field of view of the microscope head at a given moment during the observation can even reach 5–10 times the average density. We set 5 $\mathsf{\mu }$L/min and 3 $\mathsf{\mu }$L/min of water flow through the microscope head for the observation. We produce the PMOT2023 dataset to fully consider the real situation and simulate more challenging data that better represent the varying conditions observed in situ, rather than just extreme situations such as red tide. The algorithm can be trained with this dataset and thus perform greater stability and generalization. We have designed four video models.

Firstly, these four video models represent different water velocities, phytoplankton densities, trajectories, and degrees of motion blur. These differences are determined by the various parameters used in generating random trajectories, as described in Section 3.2. In situ observations of phytoplankton often involve varying conditions such as differing phytoplankton densities which can lead to obscuring situations. The trajectory and speed of phytoplankton under the microscope can also vary, and fast movement can cause motion blurring. Our aim was to create a dataset that adequately simulates the original observations; hence, we created four video models that represent different observation scenarios, which are described in the article. Our purpose in constructing this dataset is to assist in the study of multi-object tracking algorithms. Each video model presents unique challenges to automatic recognition algorithms, which helps in the training and evaluation of different methods. Additionally, this dataset can be used as a basis for selecting automatic recognition methods for different situations in practical applications.

• DENSE-FAST: We designed a 8 min video; in the extreme case, we assumed a control water flow rate of 5 $\mathsf{\mu }$L/min flowing through the microscope head; the 8 min synthetic video contains 950 phytoplankton tracks; we set the video frame rate to 25; the whole video contains a many of categories switching and phytoplankton individual occlusion. The DENSE-FAST represents the most challenging situation for the automatic identification algorithm during in situ phytoplankton observation.
• DENSE-SLOW: The 8 min video contains 405 phytoplankton trajectories, simulating a 3 $\mathsf{\mu }$L/min flow rate through the microscope head during the observation process. The phytoplankton in the video have a slower movement speed than DENSE-FAST.
• SPARSE: The 8 min video contains 630 phytoplankton tracks, in which the phytoplankton also have a slower movement speed. It represents the general situation of phytoplankton occurrence during in situ observation of phytoplankton.
• CHALLENGE: This video sequence is more challenging. Eight minutes of video contain 1160 phytoplankton tracks. The speed of movement of each individual has greater variability and is accompanied by a change in appearance and size. PMOT2023-CHALLENGE was designed to train and evaluate the stability and generalization of the automatic phytoplankton identification algorithm for the specific case of in situ phytoplankton observation.

3.3. Video Data Production and Annotation Generation

Before making the PMOT2023 dataset, we first processed and expanded the collected sample sets. We cropped each sample set to the sample size, and then, made alternative samples by OTSU [57] threshold segmentation algorithm. We designed the program to synthesize the video, and in the process of synthesis, we ensured full randomness while considering the real situation. First, we ensured that each algal sample passing through the observation field was randomly selected and the composite video was randomly data-enhanced. These random data enhancements include random rotation and random scale resizing, which simulate the different poses of phytoplankton. Figure 4a shows the sample set images after threshold segmentation. These sample set images were used to produce the PMOT2023 dataset, as shown in Figure 4b.

In order to ensure the randomness of the phytoplankton motion trajectory in the synthetic video, we design a random function to simulate the real motion trajectory of the phytoplankton in the observation. We first select key points of the phytoplankton in the field of view every n-th frame and generate the trajectory curve by using the key points and the spline curve formula. Using $X_t,Y_t$ to denote the horizontal position and vertical position of sample i at frame t, we calculate the position $X_{t+n},Y_{t+n}$ where this sample will appear in the image at frame $t+n$ by the random function.

$$X_{t+n}=X_t+Rand(-1,1)\times Rand(a,b)+C_x$$

(1)

$$Y_{t+n}=Y_t+Rand(c,d)+C_y$$

(2)

where $Rand$ is a function of randomly selected integers in a certain range. a and b are two constants that control the range of random left–right motion of the object in the field of view. In the process of making the dataset, a, b were set randomly, which can ensure the randomness of phytoplankton movement. Of course, we added some boundary rules at the same time. For instance, a and b, in order to control the phytoplankton movement, should not exceed the width of the video interface (4096 pixels). c, d are responsible for controlling the random range of vertical movement of the phytoplankton. During the production process, we give c and d values of −25 pixels and 25 pixels, respectively. $Rand(-1,1)$ in Equation (1) controls the overall direction of the phytoplankton moving left and right in the video. $C_x$ and $C_y$ are constants, responsible for controlling the average motion pattern of the phytoplankton throughout the video. We default $C_x$ to 0 during the production of PMOT2023. $C_y$ is used to control the velocity of the phytoplankton movement in the vertical direction and is a positive value (water velocity from top to bottom). In the four video modes DENSE-FAST, DENSE-SLOW, SPARSE, CHALLENGE, we set different $C_y$, which are $n\times 55$, $n\times 35$, $n\times 35$, and $n\times 65$. These parameters are convenient for quick setting when synthesizing new data.

After the above calculation, m key points are randomly selected and the horizontal and vertical coordinates are noted as $[x_0,x_1,x_2\dots x_m]$ and $[y_0,y_1,y_2\dots y_m]$. These key points are used as control points to generate trajectories via B spline curves [58]. We assume that the current target stays in the field of view for f frames, and the final result is to calculate the position of the trajectory in each frame of the picture, denoted as $[X_0,X_1,X_2\dots X_f]$ and $[Y_0,Y_1,Y_2\dots Y_f]$. Obviously, $f<m$.

$$[X_0,X_1,X_2\dots X_f],[Y_0,Y_1,Y_2\dots Y_f]=BSplineInter([x_0,x_1,x_2\dots x_m],[y_0,y_1,y_2\dots y_m])$$

(3)

where $BSplineInter$ is the B spline curve interpolation function (finds the B-spline representation of a 1-D curve). As in Figure 5, we finally obtain the position of the randomly generated object trajectory at each frame.

While synthesizing the videos, we recorded the position of the phytoplankton in the field of view and recorded the trajectory of the phytoplankton. With the help of this information, we wrote script programs to generate multi-object tracking annotations. Accurate annotation is essential for effective deep learning, and sufficient data for supervised learning are the basis for deep learning algorithms to exhibit good performance. We performed detailed annotation on each image frame. These annotations include the corner points of the bbox where the phytoplankton are located, and the category of the phytoplankton encompasses four different zooplankton categories. The same object has the same ID in all frames in which it appears in the video, forming a track. The PMOT2023 annotation also includes the visibility rate. When an object is occluded by another object or is occluded at the edge of the field of view, we calculate the percentage of the portion that is not visible, labeled with the visibility rate. In the process of training the detector, the researcher can use the occlusion factor to filter out objects with high occlusion rates when training the detector. Figure 6 shows the annotation details, where each object is tightly surrounded by an annotation box, and its category and occlusion factor are shown.

4. PMOT2023 Statistics and Analysis

4.1. Dataset Setup and Statistics

The PMOT2023 dataset contains a total of 1920 s of video, with a total of 48,000 consecutive high-resolution video frames. We divided these data into a training set, a validation set, and a test set. During the training process, the algorithm adjusts its parameters based on the patterns and features observed in the training data. The validation set is used in the training process to see if the model training is going in a bad direction and to indicate the training direction, but is not directly involved in the training. The test set is used in the testing process to evaluate the generalization ability of the final model. However, it cannot be used as a basis for algorithm-related choices such as tuning, selecting features, or other algorithmic decisions. PMOT2023 prepares a training set, validation set, and test set for the algorithm in the following proportions.

$$Se{t}_{train}:Se{t}_{val}:Se{t}_{test}=15:3:2$$

To ensure the fairness of using the test set to evaluate the algorithm, we construct the test set with a different sample set from the validation set and the training set. This ensures that the knowledge in the two datasets is not confused and the test data are not leaked.

Table 1. Dataset size of PMOT2023.

$\mathit{Set}$	DENSE_FAST			DENSE_SLOW			SPARSE			CHALLENGE
	Duration	Frames	Tracks	Duration	Frames	Tracks	Duration	Frames	Tracks	Duration	Frames	Tracks
train_set	360 s	9000	680	360 s	9000	420	360 s	9000	300	360 s	9000	800
val_set	72 s	1800	140	72 s	1800	90	72 s	1800	60	72 s	1800	160
test_set	48 s	1200	130	48 s	1200	120	48 s	1200	100	48 s	1200	200

PMOT2023’s four video, DENSE-FAST, DENSE-SLOW, SPARSE, CHALLENGE, data volume statistics, where CHALLENGE has the most tracks.

indicates the detailed data set settings of PMOT2023. We produced 12 videos in the PMOT2023 dataset, using four 360 s videos as the training set, four 72 s videos as the validation set, and four 48 s videos as the test set by default. The training set, test set and validation set keep the distribution of corresponding video patterns close in terms of length, number of bounding boxes, scenes, and motion diversity.

After capturing the video, we meticulously labeled each frame with the ID, location of the bbox, and phytoplankton category. We calculated the number of bboxes for each category in the PMOT2023 dataset. As we randomly selected samples when creating the video, the number of samples for each category in the PMOT2023 dataset is relatively balanced. This helps mitigate the loss caused by the long-tail distribution of data during the training of automatic identification algorithms. Previous phytoplankton datasets, such as PMID2019 (a phytoplankton microscopic image dataset), have shown highly unbalanced category data, with a ten-fold difference between the classes with the highest number of samples and those with the lowest number. The same problem was observed in our initially collected sample set. As shown in Figure 7, the number of categories in the PMOT2023 dataset is more balanced, and the original sample set collection data categories are not balanced. It is important to ensure the balance of PMOT2023 category data; unbalanced data can mislead the training of deep learning models and bias the models towards the overrepresented categories. Due to the unbalanced nature of the collected sample data, in the process of synthesizing the video data, we ensure that each category is sampled with the same probability to try to avoid the impact of unbalanced PMOT2023 data on model training.

4.2. Comparison of Existing Datasets

For deep learning, a dataset needs to have sufficient data volume to support both training and evaluation of the model. In the field of multi-object tracking, the MOT17 [42] and MOT20 [43] datasets are considered large-scale. In

Table 2. PMOT2023 compares the multi-object tracking datasets, MOT17 and MOT20.

Dataset	MOT17 [42]	MOT20 [43]	PMOT2023 (Proposed)
Videos	14	8	12
Avg. tracks	96	432	267
Total tracks	1342	3456	3200
Avg. len. (s)	35.4	66.8	154
Total len. (s)	463	535	1920
FPS	30	25	25
Total images	11,235	13,410	48,000
resolution	2k or <2k	2k or <2k	4K
number of classes	12	13	26

, we compare PMOT2023 to MOT17 and MOT20 in terms of data volume, including the number of video frame images, bounding boxes, tracks, and included categories. PMOT2023 has a greater number of images and categories compared to both datasets, while having a similar track count to MOT20. Additionally, we compare PMOT2023 to PMID2019, a large-scale target detection dataset for phytoplankton, in

Table 3. Data volume comparison between PMOT2023 and PMID2019.

Dataset	PMID2019 [21]	PMOT2023 (Proposed)
lotal images	11,089	48,000
number of bbox	13,865	170,041
number of classes	24	21

.

Phytoplankton are widely distributed in the ocean with a large number of categories. The accuracy of the automatic identification algorithm classification is a very important index in the process of in situ observation of phytoplankton. The multi-object tracking algorithm suited for in situ observation of phytoplankton should not only accurately locate the phytoplankton and represent the movement trajectory of phytoplankton, but also should accurately classify the phytoplankton. Therefore, the most important feature of the phytoplankton multi-object tracking dataset that distinguishes it from the pedestrian-vehicle multi-object tracking dataset is the number of categories. The commonly used multi-object tracking dataset MOT20 contains 13 categories that classify pedestrians and vehicles at fine granularity, such as people in cars, pedestrians, etc. PMOT2023 contains 21 categories of object, with up to 14 categories of object in a single frame. The videos in PMOT2023 have more category variations, and previous multi-object tracking methods rarely consider category information into the matching strategy of multi-object tracking. PMOT2023 can help researchers improve multi-object tracking methods under the challenge of videos with rich category variations.

PMOT2023 has some other differences compared to other datasets. As shown in Figure 8, PMOT2023 foreground and background are more difficult to distinguish due to the small color difference, which also presents a greater challenge for the detection branch in multi-object tracking of phytoplankton, Figure 8a exemplifies the challenge of PMOT2023. Figure 8b is the PMID2019 [21] phytoplankton object detection dataset, which has clearer appearance features for detection in comparison.

PMOT2023 contains sufficient information on phytoplankton localization and classification and requires only simple processing to be used as a object detection dataset. We counted the PMID2019 dataset as a phytoplankton object detection dataset and compared the number of bboxes, categories, and other information contained in the dataset. As shown in Table 3, compared with other object detection datasets, the PMOT2023 without data cleaning has sufficient data as a object detection dataset, and even after data cleaning, PMOT2023 can help researchers train excellent phytoplankton image level automatic recognition and object detection algorithm models.

Previous phytoplankton datasets have not been annotated at video level, and PMOT2023, as a dataset for video-level annotation of phytoplankton, facilitates the development of deep learning-based video-level automatic identification algorithms. Video-based phytoplankton auto-identification algorithms help researchers to obtain useful information from the large number of videos collected continuously and for long periods of time during in situ phytoplankton observations. The algorithm of real-time video automatic identification can help researchers to obtain dynamic information of phytoplankton instantly and continuously for studying seasonal changes of phytoplankton populations. At the same time, the real-time automatic identification algorithm helps researchers effectively predict the prevention of marine ecological disasters caused by phytoplankton blooms by monitoring the abundance and distribution of phytoplankton and predicting the occurrence of harmful algal blooms.

5. Result

5.1. MOT Evaluation Criteria

To evaluate the performance of a deep learning model in various tasks, quantitative metrics are required to enable a side-by-side comparison. Various evaluation metrics exist to assess the performance of multi-objective tracking algorithms. The confusion matrix is a common evaluation tool for supervised learning and is mainly used to compare classification results with actual measured values. False positives (FP) represent instances where the model incorrectly identifies a negative sample as positive, while true positives (TP), true negatives (TN), and false negatives (FN) are also commonly used indicators. In MOT, the most critical metrics include Multiple Object Tracking Accuracy (MOTA), which measures the performance of detecting objects and maintaining trajectories by considering the recall and accuracy of ID switching to detection frames. Equation (4) shows how MOTA is calculated, where $F{P}_t$ denotes the number of false detections (false positives) in frame t, $F{N}_t$ denotes the number of missed detections (false negatives), $IDS{W}_t$ denotes the number of ID Switches (where IDs are incorrectly reassigned to another target), and $G{T}_t$ denotes the number of true targets (ground truth). Identification F-Score (IDF1), which incorporates ID recall and accuracy into the calculation and places more emphasis on the accuracy of trajectory and detection frame association. Equation (5) shows how IDF1 is calculated, where N denotes the number of targets, $w_i$ denotes the intersection ratio of the i-th target, and $\widehat{w_i}$ denotes the intersection ratio of the i-th target matched to.

$$MOTA=1-\frac{{\sum }_t(F{P}_t+F{N}_t+IDS{W}_t)}{{\sum }_{t}G{T}_t}$$

(4)

$$IDF1=\frac{2·{\sum }_{i=1}^N{w}_i}{{\sum }_{i=1}^N{w}_i+{\sum }_{i=1}^N\widehat{w_i}}$$

(5)

A successful multi-object detection algorithm should achieve good results and balance across multiple evaluation metrics.

5.2. Comparison of Different Multi-Object Tracking Methods

Most of the current multi-object tracking follows the tracking by detection paradigm, which takes certain association strategies to associate the detection frames of different frames of the video to form trajectories after the detector output results. SORT, DeepSORT, BYTEtrack, BoT-SORT are excellent matching strategies, which we introduced in Section 2. DeepMOT [59] incorporates a differentiable Hungarian matching module that optimizes the tracker’s association output in a continuous, end-to-end manner, enabling efficient training and joint optimization of the feature extractor and the association module. StrongSort [60] upgrades and improves the classic tracker DeepSORT, including enhancements to detection, embedding, and association. Its authors also propose the appearance information-independent global linking model (AFLink) and the Gaussian smooth interpolation (GSI) method to compensate for missed detections. As shown in

Table 4. Results of PMOT2023. We bolded the best results to show.

Methods	MOTA↑	IDF1↑	IDSW↓	Recall↑	IDRecall↑	Prcn↑
BYTEtrack [18]	52.98	67.88	39.0	58.70	55.90	91.70
SORT [28]	35.80	45.43	502.0	42.45	38.18	95.45
DeepSORT [17]	43.60	58.30	112.3	52.48	47.18	84.33
BoT-SORT [32]	44.60	31.80	174.5	37.40	29.20	91.45
DeepMot [59]	36.15	22.82	27.5	51.20	21.30	71.03
StrongSORT [60]	39.33	53.88	141.3	48.56	40.40	82.48

, we conducted some experiments to verify the performance of different association strategies on the PMOT2023 dataset, and the experiments mainly used MOTA, IDF1, and IDSW as evaluation metrics. We use Recall and IDRecall to denote the recall of bounding box and trajectory ID and Prcn to denote the accuracy of bounding box. Tracking by detection paradigm in multi-object tracking uses detectors to detect objects in each frame and then correlates these detections to form trajectories through tracking strategies to achieve multi-object tracking. For the sake of fair experiments, we kept the detector fixed and used the current state-of-the-art YOLOv7 object detection algorithm to obtain detection frame results. We also maintained the same parameters in the detector component to control variables and assess the performance of various association strategies.

5.3. Phytoplankton Counts and Density Estimates

Multi-object tracking methods perform counting functions while indicating different objects in the video as different trajectories. There is often a demand for counting the number of different phytoplankton species passing through the field of view during in situ observations. It is more efficient to adopt an algorithm for automatic identification than to collect data in situ and screen them manually in the laboratory. As in

Table 5. Counting results of different tracking methods.

	Test Set	Total	Ceratium furca	Skeletonema	Helicotheca
groundtruth	DENSE-FAST	130	7	9	5
	DENSE-SLOW	120	5	8	3
	SPARSE	100	3	4	10
yolov7 [27]	DENSE-FAST	141	5	8	7
+Bytetrack [18]	DENSE-SLOW	116	3	8	4
	SPARSE	93	3	2	10
yolov7 [27]	DENSE-FAST	153	4	8	10
+DeepSORT [17]	DENSE-SLOW	142	3	9	5
	SPARSE	118	3	2	10

, we show the results of Bytetrack and DeepSORT counts, including the total number of phytoplankton and the total number of the three species, Ceratium furca, skeletonema, and helicotheca. During the experiment, the detector outputs the position of each object in the current detection frame, including the coordinates of the bounding box’s top-left and bottom-right corners, as well as the category of each object. The tracking process utilizes this information and generates the final tracking result by comparing it with the tracking results of past frames and comparing it with the predicted results. We use the Yolov7 object detection algorithm in conjunction with the tracking algorithm to produce trajectories from the detection results. We initially tallied the number of phytoplankton in the PMOT2023 test sets DENSE-FAST, DENSE-SLOW, and SPARSE, along with the count of each phytoplankton category marked as groundtruth. We then compared the tracking count results to ground truth with errors all within acceptable limits. We found that the counts of Bytetrack were more stable than DeepSORT. The Yolov7 was used as the detector during the experiment.

A more important indicator in phytoplankton in situ observations is the density of phytoplankton. We can estimate the density of phytoplankton from the flow of water and the count results of the tracking algorithm. This can help researchers monitor phytoplankton population changes and serve as an early warning of marine ecological disasters, etc. By the residence time of phytoplankton in the field of view, DENSE-FAST, DENSE-SLOW, and SPARSE in PMOT2023 represent the current velocity passing through the field of view at 5 $\mathsf{\mu }$L/min, 3 $\mathsf{\mu }$L/min, and 3 $\mathsf{\mu }$L/min, respectively. We calculated the density for all phytoplankton in the test set, as well as the densities of phytoplankton Ceratium furca, skeletonema, and helicotheca, labeled as ground truth.The results of Bytetrack density estimation are shown and compared with ground truth as in

Table 6. Results of density estimation for different tracking methods, in cells/$\mathsf{\mu }$L.

	Test Set	Total	Ceratium furca	Skeletonema	Helicotheca
groundtruth	DENSE-FAST	32.50	1.75	2.25	1.25
	DENSE-SLOW	50.00	2.08	3.33	1.25
	SPARSE	41.67	1.25	1.67	4.17
yolov7 [27]	DENSEFAST	35.25	1.25	2.00	1.75
+Bytetrack [18]	DENSESLOW	48.33	1.25	3.33	1.67
	SPARSE	38.75	1.25	0.83	4.17
yolov7 [27]	DENSEFAST	38.25	1.00	2.00	2.50
+DeepSORT [17]	DENSESLOW	59.17	1.25	3.75	2.08
	SPARSE	49.17	2.08	0.83	4.17

, with the unit of measurement being the number of phytoplankton per microliter.

5.4. Visualization Results

We have presented visualizations of the results of our multi-object tracking algorithm, providing an intuitive way to observe how this approach can be applied to identify and track phytoplankton. Figure 9 displays the multi-object tracking results, including the target class, ID, location, and historical tracks. Specifically, Figure 9a–d represent the four segments of the test set. We use different colors to indicate different individuals appearing in the video, with each ID representing a unique target. The ID in the visualization result represents the first phytoplankton appearing in the video, enabling the counting of phytoplankton. These count results, together with water velocity or flow data, can be used to calculate phytoplankton concentration over time.

6. Conclusions

In conclusion, this work presents the PMOT2023 dataset, specifically designed for a video-level automatic identification algorithm for phytoplankton. By collecting phytoplankton samples and simulating real situations, we produced 48,000 frames of video with detailed annotations, making it a valuable resource for other video-level and image-level deep learning algorithms. With only minimal preprocessing, the dataset can be utilized for various applications, such as video object detection, single object tracking, image object detection, and image classification.

Moreover, PMOT2023 enables the implementation of video-level automatic phytoplankton identification algorithms, including multi-object tracking, which can be utilized for phytoplankton detection, classification, counting, and density estimation. By evaluating different tracking methods applied to phytoplankton based on PMOT2023, we have demonstrated the effectiveness of our dataset in advancing the field of phytoplankton research. Finally, we have implemented phytoplankton counting and density estimation using multi-object tracking methods, and the results demonstrate the potential of our dataset in enabling more accurate and efficient monitoring of phytoplankton populations in aquatic environments. Overall, PMOT2023 has the potential to advance the study of phytoplankton populations and their impacts on aquatic ecosystems. Relevant codes and data will be organized in https://github.com/yjainqdc/PMOT2023 (accessed on 21 April 2023).