Towards More Effective Ship Ballast Water Monitoring: Evaluating and Improving Compliance Monitoring Devices (CMDs)

Abstract

For accurate and reliable monitoring, compliance monitoring devices (CMDs) in Port State Control must meet strict and uniform quality standards. This study evaluates how effectively CMDs, using variable fluorescence (VF) and fluorescein diacetate (FDA) technologies, detect live organisms in the 10–50 μm size range. Employing a detailed analytical framework, we analyzed key performance indicators, including accuracy, precision, sensitivity, specificity, trueness, detection limits, and reliability by comparing CMD outputs to those of traditional microscopic methods. Reliability assessments revealed that VF-type CMD and FDA-type CMD performed robustly, with a stability rate of 99% for both, surpassing the 90% verification threshold. Precision analysis indicated an average CV exceeding 0.25; however, some samples, especially those below the D-2 standard, achieved a CV of less than 0.25. Concordance evaluations revealed that VF-CMDs and FDA-CMDs achieved rates of 63% and 55%, respectively, falling short of the 80% verification standard and underscoring the need for further calibration or optimization. Structural equation modeling shows that organism density significantly influences CMD performance. These findings underscore the challenges of accurately detecting low organism concentrations, further complicated by biological diversity and environmental variability. Despite their limitations in assessing ballast water compliance, CMDs are effective initial screening tools.

1. Introduction

Globally, managing invasive aquatic species through ballast water discharge control from ships is a critical environmental and economic issue. Ships transport large volumes of ballast water across diverse marine ecosystems to maintain stability and structural integrity during voyages [1]. Ballast water, taken from one ecological region and discharged into another, often contains non-native species that can become invasive, causing significant ecological disruption and economic losses [2,3].

In response to this challenge, the International Maritime Organization (IMO) adopted the International Convention for the Control and Management of Ships’ Ballast Water and Sediments in 2004 [4], which has been enforced since September 2017. According to IMO regulations, all ships must use approved Ballast Water Management Systems (BWMS) to effectively treat their ballast water. This treatment ensures that the concentration of living organisms in discharged ballast water complies with the D-2 discharge standard (D-2), which explicitly limits their allowable concentration [5]. Consequently, a range of Compliance Monitoring Devices (CMDs) has been developed to enable inspectors to verify and demonstrate compliance with the discharge standards. CMDs significantly advance beyond traditional microscopy by enabling rapid, automated detection and quantification of viable organisms in ballast water, thus supporting real-time compliance and operational efficiency [6]. Despite rigorous D-2 regulations on ballast water discharge, non-compliant emissions persist without substantial improvement in compliance over time. Given the persistent non-compliance, enhancing Port State Control inspections and advancing CMD technology are essential to effectively address this challenge [2,7,8]. This capability is vital for balancing maritime traffic demands with the need to protect marine ecosystems from invasive species [9].

Commercially available CMDs include various technologies, many validated for their efficacy in rapidly testing ballast water [10,11]. Key technologies include ATP assays, fluorescence-based methods like variable fluorescence (VF) and fluorescein diacetate pulse (FDA), motility and fluorescence assays (MFA), and Flow Cytometry (FCM). Additionally, emerging microfluidic chip technology enhances the precision of these evaluations. When selecting CMDs, a comprehensive assessment of their accuracy, operational efficiency, and cost-effectiveness is crucial. Each technology has unique advantages and limitations, requiring a balanced evaluation to determine the most suitable approach for effective ballast water management monitoring.

Continuous research and development aim to advance CMD technologies and methodologies [11,12,13,14,15,16,17,18,19,20,21], ensuring they are comprehensive and practical for widespread use in the shipping industry. A recent study employed a multidimensional testing protocol in a comprehensive evaluation of CMDs for ballast water, ensuring regulatory compliance across diverse maritime environments [15]. In 2022 and 2023, Casas-Monroy et al. conducted investigations delineating the fluctuating performance of CMDs under varied experimental conditions [12,13]. Furthermore, the impact of variable environmental factors on detection outcomes is underexplored, highlighting a critical area for future research. Advancing this field requires developing robust, universally applicable evaluation criteria and conducting comprehensive field trials that account for environmental variability, thereby enhancing CMDs’ detection reliability.

To address the challenges posed by the complexity and variability of ballast water in CMDs testing, this study specifically employed real ballast water samples obtained from type-approval land-based tests of BWMS. These samples included untreated water and water treated with three distinct BWMS technologies: filtration-ultraviolet, filtration-electrochlorination, and filtration-membrane separation-deoxygenation. This study primarily focused on the performance evaluation, including accuracy, trueness, reliability, sensitivity, and specificity of compliance monitoring devices (CMDs) based on FDA and VF detection principles. Additionally, the study explored the water quality and biological characteristics of these samples to determine how these factors affect CMD efficacy. The objective was to elucidate how these factors, in conjunction with the various treatment methods employed, influenced the efficacy of CMDs. Rather than focusing solely on the performance of CMDs in isolation, we have endeavored to unravel the intricate interplay between water quality, biological characteristics, treatment methods, and CMD performance. This comprehensive analysis provides a nuanced understanding of the multifaceted factors that influence ballast water management, thereby contributing to a more robust and informed approach to addressing this critical environmental challenge. The research aimed to provide comprehensive technical insights to standardize CMD evaluations, supporting their selection during Port State Control (PSC) inspections, thereby enhancing the overall effectiveness of ballast water management.

2. Method

2.1. Experimental Design

Between 2020 and 2021, we conducted land-based biological efficacy tests for Ballast Water Management Systems (BWMS) type approval, adhering to the dual standards set by the International Maritime Organization (IMO) and Environmental Technology Verification (ETV), at the Ballast Water Testing Laboratory facilities of Shanghai Ocean University. To ensure the representativeness of the ballast water used in our experiments, a total of 34 land-based biological efficacy tests were performed, encompassing three distinct water types: freshwater (FW), marine water (MW), and brackish water (BW).

The abiotic and biotic characteristics of the ballast water employed in our experiments met the requirements specified by both IMO and ETV protocols. Key parameters included salinity, Particulate Organic Carbon (POC), Total Suspended Solids (TSS), Dissolved Organic Carbon (DOC), temperature, as well as the quantity and diversity of organisms within the size ranges of ≥10–50 μm and ≥50 μm. The detailed methodologies for these land-based tests closely followed the procedures previously described by Dong et al. [22].

During the land-based testing phase, we collected ballast water samples that had not undergone any treatment by BWMS, alongside samples discharged after processing by BWMS utilizing various treatment technologies. These samples were specifically gathered to evaluate the detection performance of the CMDs under investigation, focusing on their ability to accurately identify living organisms (exhibiting vital characteristics such as motility and membrane integrity) within the size range of ≥10–50 μm (

Table 1. Summary of ballast water samples.

Water Type	Number of Measurement Samples for Each Group
	Untreated Ballast Water (U)	Treated Ballast Water				Total
		D	UV&D	EC&D	MD&D
	>D-2 1 (≈100 DS 2)	>D-21 (≈10 DS 3)	<D-2			>D-2	<D-2
Fresh water	N = 18	N = 7	N = 10	N = 4	N = 2	N = 25	N = 16
Brackish water & Marine water	N = 9	N = 9	N = 1	N = 3	N = 8	N = 18	N = 12

Note: ¹. The D-2 Ballast Water Discharge Standard mandates that the concentration of viable organisms (the size range of ≥10–50 μm) in discharged ballast water must not exceed 10 cells/mL. ². ≈100 DS indicates that the sample contains approximately 100 times the D-2 standard limit, equating to roughly 1000 cells/mL of ≥10–50 μm viable organisms. ³. ≈10DS signifies that the sample contains approximately 10 times the D-2 standard limit, corresponding to about 100 cells/mL of ≥10–50 μm viable organisms.

).

Each complete land-based testing cycle encompassed a full simulation of the ballast water management operations aboard a vessel, involving both ballasting and deballasting procedures. During the ballasting phase, untreated ballast water was pumped via ballast pumps into the BWMS for treatment, after which the processed water was transferred into the ballast tank. Following a storage period in the ballast tank, the deballasting phase entailed the discharge of the ballast water back into the sea, either with or without subsequent BWMS treatment, depending on the specific technology employed by each BWMS.

A control group was incorporated into the experimental design, wherein the water samples underwent the identical ballasting procedure but without any BWMS treatment. Throughout both the ballasting and deballasting processes, a continuous sampling method was utilized to collect 100 L samples of ballast water. Untreated ballast water samples obtained during the ballasting phase were labeled as “U”. During deballasting, 100 L samples of treated discharge water from the BWMS were consistently collected, with samples processed via ultraviolet irradiation, electrochlorination, and membrane filtration and deoxygenation marked as “UV&D”, “EC&D”, and “MD&D”, respectively. The control group’s discharge water, subjected solely to dark storage without BWMS treatment, was designated as “D”.

2.2. Sample Analysis

For the quantification and viability assessment of ≥10~50 μm living organisms, we employed the fluorescein diacetate (FDA) (Thermo Fisher Scientific Inc., Shanghai, China) and 5-chloromethyl fluorescein diacetate (CMFDA) (Thermo Fisher Scientific Inc., Shanghai, China) staining methods, as recommended in the ETV Protocol 2010, which are well-established techniques for assessing cell viability by detecting intracellular esterase activity. The sample analysis procedure was as follows: After thorough mixing of the samples to ensure homogeneity, a 1 mL aliquot was carefully transferred into a 2 mL polyethylene (PE) tube using a micropipette (Eppendorf Research Plus, Leipzig, Germany). Subsequently, Add a final concentration of 1 mM/L of FDA and 250 mM/L of CMFDA to the sample tube. The samples were then gently mixed and incubated in the dark for 10 min to allow for complete staining of viable cells. To guarantee reliable and accurate statistical results, 3 duplicate 1 mL of sub-sample are needed to count for each untreated ballast water sample, and 6 for each treated and discharged one. For microscopic analysis, the stained sample was loaded into a 1 mL Sedgewick Rafter Counting Chamber (Wildlife Supply Company, Yulee, FL, USA), which provides a standardized grid for accurate counting of organisms. Counting and species identification were performed under a fluorescence microscope (Olympus BX53, Tokyo, Japan). The detailed description of the methodologies employed is referred to the procedure in Wang et al. [3].

The same ballast water sample was analyzed concurrently using microscopy and two types of CMDs. The BOS Clean-Plankton Version 3.8 utilizes fluorescein diacetate (FDA-type) pulse technology to detect living cells based on fluorescence intensity exceeding predefined thresholds. Meanwhile, the FastBallast leverages a variable fluorescence (VF-type) technique, which uses weak light pulses to initiate algae chlorophyll fluorescence, then strong pulses to maximize this effect. Changes in fluorescence intensity are analyzed to assess algae’s photosynthetic capacity and physiological state. For microscopy, six 1 mL sub-samples were examined. The FDA-type CMDs and VF-type CMDs analyzed three 10 mL and three 20 mL parallel subsamples, respectively. The microscopic results were calculated as the average of the six aliquots, and the CMD results were reported as the average of the three parallel samples, with both expressed in cells/mL.

2.3. Data Analysis

2.3.1. Accuracy

In evaluating the accuracy of CMDs, two statistical methods are commonly employed: Kendall’s correlation analysis [12] and regression analysis [12,14,18]. Kendall’s correlation analysis offers a robust understanding of the relationship between microscopic examination results and CMD outcomes, providing insights into the degree, direction, and strength of their association through a non-parametric measure of ordinal association. Moreover, regression analysis is utilized to investigate how CMD results are influenced by the quantity of live organisms in ballast water samples, quantified using microscopic examination.

2.3.2. Precision

To evaluate the precision of detection results obtained from CMDs, the coefficient of variation (CV) was employed as a measure of the dispersion among measurement values [12,13]. A lower CV value indicates a higher consistency in measurements, thereby reflecting greater precision. Conversely, a higher CV denotes substantial variability among measurements, potentially reducing the accuracy of the detection results. Additionally, to assess the reliability or stability of these measurements, the standard deviation of the CV (CV SD) was calculated. The final results are presented as “mean CV ± CV SD”.

2.3.3. Sensitivity and Specificity

The confusion matrix is a commonly employed tool that elucidates the sensitivity and specificity of CMDs [12,23]. It aligns CMDs’ detection outcomes with those from detailed microscopic inspections, thereby distinguishing between true positives (TP, where both CMDs and microscopic analysis agree that the quantity of living organisms exceeds the D-2 standard), false positives (FP, where microscopic analysis complies with the D-2 standard but CMDs do not), true negatives (TN, where both analyses agree on compliance with the D-2 standard), and false negatives (FN, where microscopic analysis indicates non-compliance with the D-2 standard, but CMDs indicate compliance). Sensitivity is defined as the ability of CMDs to correctly identify samples that exceed the D-2 standard, while specificity refers to their ability to accurately recognize samples that do not surpass this threshold. The sensitivity (True positive rate, TPR) and specificity (True negative rate, TNR) of CMDs are calculated based on these definitions using the following formulas:

$$\mathrm{T}\mathrm{P}\mathrm{R}={\displaystyle \frac{TP}{TP+FN}}$$

(1)

$$\mathrm{T}\mathrm{N}\mathrm{R}={\displaystyle \frac{TN}{TN+FP}}$$

(2)

2.3.4. Trueness

Trueness can objectively assess the degree of agreement between CMDs and the microscopic method, providing a reliable measure of their consistency. In this assessment, we transform both the values generated by CMDs and the microscopic results into binary outcomes: TP, TN, FP, and FN. Cohen’s Kappa statistic is then utilized to quantify the categorical agreement (k) between the binary results obtained from both methods. A higher value of k, approaching 1, indicates a stronger consistency between the two methods. Cohen’s Kappa is calculated using the observed agreement (P_o) and the expected agreement (P_e). n is the total number of analytical measurements. The formula for Cohen’s Kappa is as follows:

$$k={\displaystyle \frac{{\mathrm{P}}_0-P_e}{1-P_e}}$$

(3)

$$P_0={\displaystyle \frac{(TP+TN)}{n}}$$

(4)

$$P_e={\displaystyle \frac{(\frac{\left(TP+FP\right)·\left(TP+FN\right)}{n}+\frac{\left(FN+TN\right)·\left(FP+TN\right)}{n})}{n}}$$

(5)

2.3.5. Detection

We gathered and analyzed CMDs detection results for samples both exceeding D-2 and meeting D-2. Data was quantified in terms of variation range (minimum and maximum), mean, and standard deviation (SD) to assess the detection of CMDs in the compliance analysis of ballast water. Ballast water samples meeting the D-2 standard were further subdivided into three groups based on treatment technology: UV&D, EC&D, and MD&D, evaluating the influence of various BWMS treatment technologies on the lower detection limits of CMDs. Utilizing the mean, standard deviation, and sample size derived from each set of CMDs output values, we compute the 95% confidence interval, which subsequently defines the lower and upper limits of the limit of detection (LOD) range for that specific set.

2.3.6. Reliability

To comprehensively assess the reliability of CMDs, we recorded occurrences of test data anomalies caused by CMD malfunctions such as automatic shutdowns and system issues. These malfunctions may be due to equipment defects, improper operation, or external factors. This helped us understand potential CMD issues in real-world use. Secondly, we analyzed the frequency of these anomalies as a proportion of total tests. This metric indicates the extent of interference and impact on CMDs during testing. A higher anomaly proportion suggests lower CMD reliability.

2.4. Statistical Analysis

In our analysis, we employed SPSS 21.0 software to establish a linear fit between the values detected by compliance monitoring devices (CMDs) and those observed under a microscope. Furthermore, the relationship between CMD detection values and both species composition and water quality parameters was statistically examined for significance. A p-value of less than 0.05 was considered indicative of statistically significant associations, thereby confirming meaningful correlations between these variables. To identify the key factor and the potential effect mechanisms on the performance of CMDs, we estimated the structural equation modeling (SEM) using the R package 4.5 lavaan [24], which applies the maximum likelihood (ML) method to calculate the fit of the model. Structural equation modeling (SEM) was conducted by R (4.2.1) based on the lavaan package. Prior to the modeling analysis, the “min–max” normalization formula was applied. Non-significant chi-square test (p > 0.05), goodness-of-fit index (GFI > 0.90), comparative fit index (CFI > 0.95), root mean square error of approximation (RMSEA < 0.06), and standardized root mean square residual (SRMR < 0.09) were used to show the overall goodness of fit for SEM.

3. Results

3.1. The Performance of CMDs

The present study sampled the collection of 71 ballast water samples. Employing fluorescence microscopy, an enumeration of viable organisms within the 10–50 μm size range was conducted. The results were then utilized to assess the compliance of the ballast water samples with the D-2 standard. 43 of the samples exceeded the prescribed D-2 threshold, exhibiting concentrations of 10–50 μm viable organisms ranging from 246 to 1587 cells/mL. 28 samples adhered to the D-2 standard, containing 0.20 to 8.0 cells/mL of 10–50 μm viable organisms.

3.1.1. Accuracy

The plots highlight the differences in performance between FDA-type and VF-type CMDs in mimicking microscopy results under varied testing conditions in ballast water analysis (Figure 1). VF-type CMDs generally show a trend towards better correlation in specific conditions. The FDA-type CMD shows varying degrees of correlation with microscopy across plots, with R² values ranging from 0.10 to 0.40. Conversely, the VF-type CMD demonstrates stronger correlation, with R² values typically above 0.25, up to 0.75. This higher correlation suggests the VF-type CMD may be more robust across different conditions, making it potentially more reliable for general use in compliance monitoring. An in-depth analysis across five distinct groups reveals that the variability in R² values for both CMD types underscores the significant influence of group-specific or test conditions on their reliability and accuracy. These variations could be attributed to differing ranges of organism concentrations or the presence of diverse organism types, both factors that could influence the CMDs’ performance.

3.1.2. Precision

In analyzing the ballast water samples, a notable phenomenon emerges fluorescence microscopy, while exhibiting unparalleled precision and minimal variability in the >D-2 subset, shows a marked increase in variability within the <D-2 group (Figure 2c). This trend is also observable in the performance of CMDs, yet a comparative analysis reveals that VF-type CMDs surpass microscopy in this lower variability subset. Specifically, data indicate that in the <D-2 group, the lowest coefficient of variation and standard deviation for fluorescence microscopy are 0.34 and 1.6, respectively. In contrast, VF-type devices demonstrate significantly lower values of 0.10 and 0.20.

When comparing the FDA-type and VF-type CMDs, we observe notable disparities in the average coefficients of variation and standard deviations across different groups. FDA-type CMDs displayed higher variability in measurements. For instance, in the NS group, the FDA-type device shows an average coefficient of variation of 0.38 with a standard deviation of 0.31, and in the UD group, these values rise to 0.79 and 0.51, respectively. VF-type CMDs manifested generally lower variability and more consistent results. For example, in the >D-2 condition, the VF-type device records an average coefficient of variation of 0.19 and a standard deviation of 0.20, considerably lower than those of the FDA-type. This trend continues across other conditions, such as in UD, where the VF-type shows coefficients of 0.10 and a standard deviation of 0.20, suggesting more stable and reliable performance.

3.1.3. Sensitivity, Specificity, and Trueness

In a comparative assessment of the efficacy of CMDs in detecting deviations from the D-2 standard in ballast water, VF-type devices have consistently demonstrated superior sensitivity over their FDA-type counterparts. This heightened sensitivity is particularly notable in the NS and D groups, where VF-type CMDs achieved sensitivities of 1.0 and 0.88, respectively. These values indicate an adeptness in identifying non-compliance, crucial for enforcing environmental standards effectively. Conversely, FDA-type CMDs exhibit variably lower sensitivity, with particularly diminished performance noted in the D group (sensitivity at 0.50). This suggests a potential limitation in the FDA-type’s capability to detect samples that exceed the regulatory thresholds, which could compromise regulatory compliance efforts. However, the FDA-type CMDs compensate for this by demonstrating higher specificity across all experimental groups, ranging from 0.82 to 1.00. This high specificity indicates a robust ability to minimize false positives; thus, when FDA-type CMDs certify a sample as compliant, the result is reliably accurate. Moreover, VF-type CMDs not only surpass FDA-type devices in sensitivity but also display more consistent accuracy across various testing conditions, underlining their reliability in diverse operational settings. The overall accuracy of VF-type CMDs stands at 0.83, compared to 0.78 for FDA-type devices, underscoring the former’s enhanced reliability in ensuring environmental compliance.

3.1.4. Detection

The results of the fluorescence microscopy method generally show higher means and more consistent ranges compared to both types of CMDs (Figure 2a). Microscopy tends to provide higher detection capability, particularly noticeable in the >D-2 group, where organism counts are higher. VF-type CMDs usually exhibit higher means (539, N group) with a very wide range (33–2677, N group) and broader ranges than FDA-type, suggesting they are potentially more sensitive to varying organism counts but with greater variability. FDA-type CMDs, while showing lower means and narrower ranges, indicate a more conservative detection pattern, possibly leading to fewer false positives but also a higher risk of false negatives.

3.1.5. Reliability

In this study, the VF-type CMDs were initially tested with 72 samples, but one was excluded due to exceeding the maximum detection threshold of ≥5000 cells/mL, thereby qualifying it as an outlier. The FDA-type CMDs were tested with 65 samples; however, one sample was discounted due to an unforeseen device shutdown during data collection, leaving 64 samples for evaluation. Reliability indices were computed as 99% (71/72) for the VF-type CMDs and 99% (64/65) for the FDA-type CMDs, indicating high reliability in detecting ballast water contaminants. Nonetheless, each device type encountered specific challenges: the VF-type CMDs with handling high cell concentrations and the FDA-type CMDs with mechanical stability.

3.2. Water Quality and Biological Parameters of Ballast Water Samples

A total of 29 species of 10–50 μm biota, encompassing the classes of Dinoflagellate, Bacillariophyta, Chlorophyta, Xanthophyta, and Cryptophyta, were identified in 71 ship ballast water samples. The Simpson diversity index within these samples ranged from 0 to 0.96 (Table S1).

The analysis of water quality across various ballast water samples reveals significant differences in parameters such as temperature, salinity, dissolved oxygen (DO), turbidity, total suspended solids (TSS), particulate organic carbon (POC), and dissolved organic carbon (DOC) (Table S2). There is a wide range in salinity measurements, from as low as 0.20 PSU to as high as 31 PSU, reflecting the diverse sources of ballast water, including fresh water, brackish water, and marine water. Turbidity varied from 1.0 to 71. The >D-2 group’s extremely high turbidity is statistically different from all other groups.

The correlation analysis indicates that while some water quality parameters (like turbidity and POC/DOC) have some not significant influence on the readings from CMDs, many other parameters (e.g., temperature, salinity) do not affect the performance of these devices. On the contrary, the values exported from CMDs (both FDA-type and VF-type) show a stronger correlation with biological parameters compared to water quality parameters, particularly displaying significant positive correlations (Table S3) with species diversity as measured by the Simpson’s diversity index and living organisms’ number as measured by microscopy.

3.3. Relationships Between Factors and the CMDs Performance

Structural equation modeling (SEM) was used to further explain biological factors of ballast water samples, including species composition and total living number properties (i.e., diversity and abundance), in influencing the CMDs performance properties (i.e., testing value, precision, and accuracy) (Figure 3). The SEM model shows the association between two latent and five observed variables and predicts the changes in our measured variables with a change in the latent variables. SEM results of VF-type CMDs and FDA-type CMDs indicated adequate fit (GFI (VF-type) = 0.990, CFI (VF-type) = 1.000, RMSEA (PVF-type) = 0.000, and SRMR (PVF-type) = 0.034; GFI (FDA-type) = 0.979, CFI (FDA-type) = 1.000, RMSEA (FDA-type) = 0.000, and SRMR (FDA-type) = 0.039). The SEM indicates significant effects of the biological properties of ballast water on the performance of both VF-type CMD and FDA-type CMD (Figure 3). The performance of CMDs exhibited better accuracy when the ballast water sample contained a higher number of living organisms. While a positive correlation exists between the species diversity in ballast water samples and the performance of CMDs, the impact of this diversity is indirect and relatively minor compared to the direct influence exerted by the abundance of living organisms. Biological properties of ballast water samples had indirect effects on the performance of CMDs through affecting the biological community composition (p < 0.01). Notably, the precision of the two types of CMDs exhibited contrasting responses to the biological properties of the samples. Specifically, the VF-type CMD demonstrated a significant positive correlation with the sample properties, whereas the FDA-type CMD exhibited a significant negative correlation. This suggests that the VF-type CMD achieves high precision in samples with high biological concentrations (Figure 3a), whereas the FDA-type CMD exhibits better precision in samples with lower biological abundance (Figure 3b).

4. Discussion

This study evaluates the performance of CMDs utilizing FDA and VF principles in determining whether living organisms of 10–50 μm in ships’ ballast water comply with the D-2 discharge standards. Our methodology emphasized a multi-faceted analytical framework, examining accuracy, precision, sensitivity, specificity, trueness, detection, and reliability. We employed comprehensive water quality analytical techniques to determine the physicochemical properties of these ballast water samples. Our results clearly demonstrate that the detection performance of CMDs is significantly shaped by both the density and diversity of live organisms within the specified size range (10–50 μm), with the density of organisms proving to be a more decisive factor. The findings from this study could inform international policy, aiding regulatory bodies like the International Maritime Organization (IMO) in refining CMDs testing protocols and standards. This is pivotal for enhancing global maritime safety and environmental conservation efforts.

4.1. Evaluation of the CMDs with the Verification Criteria

Non-native species transported in ship ballast water pose concerns, leading to global agreements & regulations [4,5]. Effective ballast water discharge monitoring is critical to minimize invasive species risks. Various compliance monitoring devices (CMDs) exist, but rigorous, standardized testing is needed for global adoption. A framework has been developed to validate a compliance monitoring device’s (CMD) capability in assessing non-compliance with the D-2 standard of the BWM Convention, while also confirming its operational functionality in line with the manufacturer’s claims regarding detectable non-compliance levels and intended use [25]. As of 13 May 2023, the Ballastwater Equipment Manufacturers’ Association (BEMA)’s official list indicates that no CMDs have been officially verified. This study presents, for the first time, the results of the evaluation on the compliance of FDA-type and VF-type CMDs with the verification criteria.

The complexity and variability inherent in ballast water samples necessitate a comprehensive multi-metric evaluation system for CMDs. Relying solely on correlation with microscopic methods is insufficient, as it fails to capture the full range of factors influencing CMD performance. In this study, the correlations between CMD outputs and microscopic counts were relatively low, significantly lower than in previous research (

Table 2. Summary of the correlation between results from various compliance monitoring devices (CMDs) and microscopic examination (VF = Variable fluorescence; FDA = Fluorescein diacetate pulse technology; ATP = Adenosine Triphosphate).

Compliance Monitoring Devices (CMDs)	Technical Principle	Correlation Coefficient (r)	Correlation Coefficient (R)	Determinate Coefficient (R2)	Reference
FastBallast	VF	\	0.59	0.26	This study
BOS Clean-Plankton	FDA	\	0.47	0.17	This study
Walz WATER	VF	0.85	\	\	[26]
Hach BW680	VF	0.9	\	\	[26]
BBE 10cells	VF	0.82	\	\	[26]
Turner Designs Ballast-CheckTM 2	VF	0.85	\	\	[26]
BW680	VF	\	0.34	\	[12]
Ballast-Check 2	VF	\	0.21	\	[12]
FastBallast	VF	\	0.52	\	[12]
DCMU derived PSII	VF	\	\	0.73	[19]
Automated fluorescence intensity detection device (AFIDD)	VF	\	\	0.89	[18]
Viable organism analyzer	FDA	\	\	0.99	[27]
B-QUA	ATP	\	0.34	\	[12]
QuenchGone Aqueous	ATP	\	\	0.995	[11]
ATP swab	ATP	\	\	0.64	[19]
BallastWISE	Motility and fluorescence assay (MFA)	\	\	0.91	[21]
FlowCAM	Imaging-in-flow	\	\	0.941	[20]

), indicating that CMDs do not accurately replicate the quantitative results obtained through microscopy when testing ballast water samples. Despite the low correlation, CMDs and microscopy showed relatively good trueness in determining whether ballast water samples met the D-2 discharge standard. The trueness values of 0.63 for VF-type CMD and 0.55 for FDA-type CMD fall within the range reported by previous studies (0.37–0.72) [12]. This indicates that the methods used for compliance assessment are reasonably reliable in terms of accurately representing the true state or condition being measured.

The trueness results from our study and previous studies are both less than 80%, which does not meet the requirements of the verification standard. While CMDs show reasonable trueness in binary compliance decisions, their current performance does not meet the stringent verification standards required for sole reliance. The FDA-type CMDs exhibit CV values (0.38 to 0.79, Figure 2c) exceeding the 25% acceptable range, indicating poor repeatability and high measurement variability, particularly in the <D-2 group, while the VF-type CMDs largely satisfy the criteria for acceptable variability, with CVs within the 25% threshold and nearing the 10% threshold for excellent repeatability in lower concentrations. The high variability observed in FDA-type CMD readings for low organism concentrations can be attributed to several factors: sensitivity issues in detecting and quantifying low concentrations [26], environmental interference from particulate matter and dissolved substances [19,28], and technological limitations inherent in the fluorescein diacetate hydrolysis method, particularly its sensitivity and specificity at low organism densities [13]. Non-specific esterases in living organisms hydrolyze colorless fluorescein diacetate (FDA). This reaction produces fluorescein, a green fluorescent marker. The intensity of fluorescein emission directly reflects viable cell abundance [18]. FDA-based cell metabolic activity detectors (CMDs) exhibit greater measurement variability. This variability stems from their reliance on enzymatic activity, a biological parameter highly sensitive to species differences, cellular physiological states, and environmental conditions [29]. The core advantage of VF-type CMDs lies in their ability to directly and accurately assess the activity of photosynthetic plankton through non-destructive testing technology. This technique uses specific light pulses to excite chlorophyll a in samples, measures its variable fluorescence (Fv), and calculates the maximum photochemical quantum yield (Fv/Fm), thereby directly reflecting the photosynthetic activity and survival status of the organisms. This method requires no reagent addition, is easy to operate, and can quickly output standardized activity indices (such as BWI) [30,31].

In the rigorous operational environment of ballast water Port State Control (PSC) inspections, the stability of CMDs is paramount, as it critically impacts the continuity, integrity, accuracy, and reliability of the testing process. Reliability serves as a crucial metric for assessing their performance [20,32]. Despite reliability meeting verification criteria (above 90%), both 99% reliability indices for VF-type and FDA-type CMDs, respectively, may still spark regulatory concerns about their absolute reliability for PSC inspections due to the lack of perfect stability.

The verification criteria analysis reveals that while the reliability of FDA-type and VF-type CMDs meets required standards, their trueness and precision do not fully comply with verification standards. This indicates the need for targeted technical optimization and calibration to enhance CMD performance. By addressing these shortcomings through technological advancements and rigorous calibration protocols, CMDs can be made more accurate and reliable, ensuring better compliance with ballast water management standards and more effective prevention of invasive species.

4.2. The Variability in CMDs Performance Due to Organism Density and Diversity

To enhance the availability of CMDs in the complex conditions of real ballast water detection, it is critical to identify and quantify the influence of various factors. This understanding is essential for developing a performance evaluation framework for CMDs and ensuring the effectiveness of board PSC inspection. While prior research has indicated that the detection performance of CMDs is subject to numerous influencing factors [21], empirical evidence substantiating this claim has been sparse. This study provides the first experimental validation of how water quality, the biological properties of ballast water samples, and differences in BWMS treatment methodologies impact CMD performance.

Water quality parameters serve as critical characteristics of ballast water samples, with turbidity potentially masking chlorophyll fluorescence [33] and thereby influencing the outcomes of device measurements. However, correlational analyses in this study reveal that, although certain water quality factors may have a potential impact, they are not the predominant determinants of CMDs’ detection performance. Variations in treatment processes of BWMS have a certain effect on the viability of organisms in ballast water [31], but this distinction does not significantly alter the detection capabilities of CMDs (Figure 1 and Figure 2d). It suggests that CMDs used in the study likely have high specificity and sensitivity designed to detect a broad range of changes in biological activity, regardless of the treatment method.

Our study reveals strong correlations between the performance of CMDs and biological parameters in ballast water samples, such as organism density and the Simpson’s diversity index. SEM highlights that the density of living organisms in ballast water is a crucial factor influencing CMD performance, supporting findings by Bradie et al. [26], who noted a significant impact of organism density on CMD outputs. However, our research uniquely contributes by demonstrating the significant influence of species diversity on CMD performance, an aspect not extensively explored in prior studies. The SEM results suggest that the primary function of CMDs is to detect and quantify biological entities, with a strong correlation observed between CMD performance and the Simpson’s index. This indicates that CMDs not only detect the presence of organisms but are also sensitive to changes in community structure and species composition. Despite high biodiversity in ballast water, which includes a variety of organism types that differ in size, shape, and color, the impact on CMD detection capabilities may be limited. Smaller or irregularly shaped organisms might be less detectable by CMDs employing optical detection methods like fluorescence or light scattering. BWMSs mitigate the presence of invasive and harmful aquatic organisms in ship ballast water using physical or chemical methods. After these treatments, the number of viable organisms is significantly reduced. Consequently, the minimal number of live organisms remaining post-treatment restricts the effect of species variability on CMD performance in analyzing ballast water samples.

4.3. Future Directions for Technological Enhancement of CMDs

The findings from our comparative analysis of FDA-type and VF-type CMDs underscore the necessity for technological advancements that focus on improving the trueness and reliability of detecting low levels of biological organisms in ballast water [12,13]. The density of organisms significantly influences CMD performance, with higher organism densities generally leading to more reliable CMD operation (Figure 2).

Sensitivity, specificity, and detection reflect the technical limitations of CMDs in more specific aspects. The analysis of FDA-type CMDs’ sensitivity shows moderate overall performance at 0.72, with lower sensitivity in the D group (0.50) and higher in the U group (0.85) (Figure 2d). VF-type CMDs, however, exhibit higher sensitivity, scoring 1.0 in the U group and 0.88 in the D group, with an overall sensitivity of 0.95. The detection outcomes further illustrate the variability in CMD performance. VF-type CMDs displayed higher means and broader ranges, indicating greater sensitivity but also higher variability (Figure 2b,d). These indicate VF-type CMDs’ strong ability to detect non-compliant samples across conditions, outperforming FDA-type CMDs. FDA-type CMDs, with lower precision and sensitivity, struggle with consistent detection at low densities, leading to higher false negative rates. FDA-type CMDs may underperform in high-density samples because the pulse counting mechanism can miss individual fluorescence pulses from each cell, resulting in underestimation of organism counts [33]. In dark storage conditions simulating ballast tanks, the metabolic activity of algal cells decreases, leading to reduced hydrolysis of the FDA stain and weaker fluorescence signals, increasing the likelihood of false negatives. The biodiversity in ballast water samples may also affect the sensitivity of FDA devices, as different biological species may respond differently to FDA staining.

The ongoing enhancement of Compliance Monitoring Devices (CMDs) is essential for improving ballast water management and safeguarding marine environments. To achieve this, the integration of advanced biosensors capable of detecting low concentrations of biological materials stands as a critical upgrade. These biosensors should be highly sensitive to subtle biological signals, such as fluorescence, which are indicative of live organisms. This sensitivity is vital for reducing false readings, particularly under challenging conditions that typically yield high rates of false negatives. To support the capabilities of these advanced sensors, it is crucial to refine the algorithms that process and interpret the data they collect. Improvements should focus on enhancing the software’s ability to distinguish between background noise and legitimate biological signals. This refinement will improve the accuracy of CMDs, making them more reliable in diverse operational conditions. Further, the application of machine learning techniques offers substantial benefits for the classification and quantification of detected organisms. By training machine learning models on comprehensive datasets filled with known organism signatures, CMDs can be equipped to more precisely identify and quantify a wide array of species, including those present in low concentrations. Moreover, machine learning models that adapt to changing environmental conditions will enhance the CMDs’ functionality. These adaptive models can continuously refine their parameters to ensure consistent performance across the variable characteristics of ballast water encountered during different voyages and seasons. Looking ahead, future improvements should also concentrate on boosting the detection accuracy and sensitivity of CMDs towards species that frequently occur or dominate in ship ballast water discharges. This targeted approach not only refines CMD technology but also aligns with strategic calibration methods to ensure more precise monitoring of ballast water. By focusing on species commonly present in higher proportions, CMDs can be tailored to better address ecological risks associated with maritime transport, thereby enhancing both regulatory compliance and ecological safety. Collectively, these enhancements will not only bolster the practical applications of CMDs but also emphasize the necessity for continuous improvement and recalibration. Keeping pace with environmental changes and regulatory demands is crucial for maintaining the effectiveness of CMDs in protecting the marine environment.

5. Conclusions

This research reveals that while CMDs demonstrate promising advances in ballast water detection, their current density-dependent performance and taxonomic bias necessitate a dual-strategy regulatory framework. We propose classifying CMDs as preliminary risk-assessment tools rather than standalone compliance instruments, with samples that exceed or approach the d-2 standard, mandatory microscopic verification to confirm live organism counts, while those below require no further action. Concurrently, our data highlight the need for technological advancements accounting for species-specific detection challenges on signal reliability. Future CMD development should prioritize multi-modal sensing capabilities. This feature needs to integrate multiple technologies, such as fluorescence and light scattering, to obtain species composition and activity indicators, and standardized calibration plans should be developed for different taxonomic groups. Implementing this risk-based monitoring strategy with parallel technological innovation will enhance regulatory compliance while reducing invasive species risks through more accurate, ecologically representative ballast water assessments.