Sensor Fusion and Autonomy as a Powerful Combination for Biological Assessment in the Marine Environment

1. Introduction

Biological processes are a focal point of many mapping and monitoring efforts in the ocean. Whether the focus is oil and gas production, minerals and mining, aquaculture, habitat mapping, or fisheries, all ocean exploration activities have a connection with biological activity as they seek understanding while minimizing detrimental impacts to marine ecosystems. However, assessing biological processes and/or impacts is a challenge for many reasons. Biological systems are dynamic in time and space, operate at horizontal scales smaller that ship board sampling allows, and the organisms being studied are often directly disturbed by the sampling platforms themselves. Sensors that measure of biological processes have also generally not kept pace with the development of physical sensors. On this last point, biological information is not obtained from a single, simple proxy, rather often the synthesis of information from a suite of sensors and some degree of sensor data fusion is required.

There are a number of examples of single data stream sensors and multi-sensor systems on autonomous packages. Alkire et al. [1], for example, integrated oxygen sensors, optical backscatter, beam attenuation, a chemical nitrate sensor, and a standard conductivity and temperature sensor on an autonomous profiling float. These data were used to estimate net community production and export and the mechanisms for bloom development in the North Atlantic [2], processes that are critical for understanding carbon cycling and climate variability. The data synthesis or data fusion in this example was a post-processing endeavor after the field effort. While this kind of integrated information is the outcome of the majority of projects/programs, post-processing of the data to reveal results after a collection effort critically misses the opportunity to optimize sampling.

This leads to the primary question of what is require to optimally assess biological processes in the marine environment? Here, we advocate four elements when aggregated provide a powerful means of future investigations in the marine environment. First, and demonstrated frequently, is the need for multi-sensor systems. Multiple sourced information provides additional layers of information which often are at the core of understanding a process. Williams [3] provides a review of early accounts of these efforts in assessing biological activity in large ecosystems using optics and acoustical approaches. There are also recent examples of this in remote sensing where multiple satellite sensors have been combined for an enhanced understanding of biological production [4]. Second is a platform that addresses the spatial and temporal scales of the process [5]. Traditional sampling approaches (i.e., moorings, ships) do not capture many of these scales [6], have reduced capacity in severe weather [7], and in the case of ships can often influence the biology targeted for study [8,9,10]; all problems that can be overcome to various degrees with autonomous platforms. There is an additional benefit with multiple networked platforms or vehicles [11], which has been demonstrated to significantly decrease the uncertainties in predictive oceanography [12]. Given the diversity of vehicle types, speeds, and durations, blending these overlapping capabilities further captures the scale of ocean processes [13,14]. Third is the need for real-time information synthesis. As highlighted above, it is combination of variables that provide new knowledge and developing real-time capabilities allow for reaction to the environment. Kirkpatrick et al. [15] developed a system whereby full spectral characteristics of water samples are taken, an algorithm is applied and compared with a spectral library to determine an index for identifying plankton species in real time, targeting harmful algal species, important for aquaculture, fisheries, and tourism. This sensor was then incorporated into AUVs [16] and gliders [17] for monitoring blooms. In another example, Baumgartner and Fratantoni [7] integrated digital acoustic recorders in three gliders. An automated detection algorithms was used onboard to simultaneously process sound to detect and localize Sei whales. In neither of these examples, however, was the processed data product coupled with the vehicle behavior, leading to the fourth and last requirement. In combination with the previous three requirements, is the need for real-time decision making, either by the operator, or ideally, autonomously by the vehicle based on the real-time synthesized data, again to address issues of scale and relevant sampling. It is this last requirement which is largely absent when reviewing the use of autonomous platforms with integrated biological sensors. There are a number of examples of single or multiple sensor feedback into vehicle decision-making in biological applications such as plume tracking [18] and targeted sampling [19], but here data are used to provide improved probability of a target being present, not used to identify the target itself. One example of where this has previously been implemented is in the use of visual image processing to identify specific fish species, which in turn informed path planning [20].

Taking all four of these requirements together, the assessment of biological processes in the marine environment is a grand challenge for a number of disciplines. For biologists, development of the synthesized products is critical in addition to defining features to target for sampling. For computer scientists and engineers, the onboard data analysis, decision-making structure and navigation interface is key. While tremendous progress has been made in the development and proliferation of autonomous underwater vehicle, there is also the ongoing engineering challenges of platform development, navigation enhancements, power sources, endurance, and sensor integration. Finally, the dynamic marine environment and specifically addressing biological problems provides for countless challenging applications for the robotics field. There are examples for these, such as localization [21,22], simultaneous localization and mapping (SLAM) [23], autonomy [24], and artificial intelligence [25] using underwater vehicles, but the end goal or real-world application has often been missing.

Here, we will focus on one example system with a biological sensor payload that results in complex data that must be interpreted and synthesized to inform control systems and navigation for optimized sampling. While there are numerous autonomous underwater vehicle (AUV) studies demonstrating data feedbacks to inform biological sampling, this study is uncommon in that a processed data product is used to identify a specific target real time and improve sampling density of that target by a simple autonomous response. In this case example, dual frequency echosounders are integrated into a commercial AUV to target a suite of organisms in the ocean’s water column between 0 and 1200 m. The system is briefly described followed by field results and concludes with a discussion of implications and additional applications for this approach.

2. Experimental Section

Highlighting a new integrated system addressing all four development requirements mentioned above requires details of the sensor payload, the platform, the data processing and synthesis, and the integration of autonomous decision-making. Some of the development effort also improved the sensor measurements themselves and warrants brief attention. Finally, we describe the implementation of this system in the field, demonstrating the power of this integrative approach.

2.1. Integration of Scientific Echosounders into an Autonomous Underwater Vehicle

Simrad EK-60 general purpose tranducers/transceivers (38 and 120 kHz) are routinely used on ships to [26] estimate fish abundance and other marine research. Sound pulses are generated from these tranducers and are differentially absorbed and reflected off organisms in the water. Because of differences in organism densities (tissue, bone, air pockets), different frequencies of sound interact differently with the type of organism. For example, squid reflect sound in high frequencies (38 kHz) while mammals produce a relatively even response across the frequency spectrum (38–200 kHz) [27,28]. By using two frequencies, organisms can be discriminated throughout the water column within the range of the sensors. Higher frequencies however have physical range limitations, and shipboard measurements by these echosounders are effectively restricted to collecting multi-frequency data in the upper 600 meters of the ocean. The motivation to push our ‘view’ deeper led to the integration of these sensors into an AUV capable at operating at the limit of shipboard sensors. The AUV selected was a REMUS 600 capable of operating at depth, large enough to accommodate the scientific echosounders, with a duration long enough to capture behavioral changes in the targeted organisms (Figure 1). The motivation for building the platform, a general vehicle description, and the physical integration of the echosounders are detailed in Moline et al. [29], but here we focus on the data, data analysis, and integration with navigation.

Figure 1. The REMUS 600 AUV on deck ready for deployment. The bulge in the forward wet section was to accommodate for the dual echosounder transducers (inset). The section aft houses the electronics for data acquisition and processing.

Figure 1. The REMUS 600 AUV on deck ready for deployment. The bulge in the forward wet section was to accommodate for the dual echosounder transducers (inset). The section aft houses the electronics for data acquisition and processing.

2.2. Advantages of Using an AUV as a Platform for Echosounders

There are a number of advantages for deploying echosounders on AUVs. Simrad EK-60 general purpose tranducers/transceivers (38 and 120 kHz) are routinely used on ships to quantify the distribution of marine organisms such as fish and krill. There are disadvantages to deploying from ships as they inherently produce sound which interferes with the very target measurements being attempted. This results in the need to post-process data to remove the background ship noise. After removing sources of noise from within the vehicle during the integration of the echosounders (Figure 2), we conducted tests off the coast of California in 2013 to compare the impact of having the echosounders on the AUV rather than a ship. The AUV echosounders had a lower and more stable noise floor than the ship-based instruments even in the calmest seas. The advantage of the AUV-based data relative to data collected in the same experiment from a ship was 17 and 19 dBW at 38 and 120 kHz, respectively. From a practical perspective, this means that when a −75 dB threshold, a common choice, is applied to our volume scattering data, background noise is not detectable at 38 kHz to a range of 1530 m from the AUV sensors rather than 1140 m from the ship and 420 m from the AUV rather than 305 m from the ship at 120 kHz. It was also noted that the ship also had unexplained high noise outliers that were absent from the AUV.

Figure 2. While integrating the echosounders a number of avenues were used to eliminate noise in the system including adjusting the switching frequency of the AUV’s drive motor, shielding cables, installing filters on power lines, modifying internal grounding paths, isolating electronics, and altering cable positions. (A) Shows the transducers mounted in the forward wet section and (B) Shows the transceivers and two computers on a rail mount for the dry electronics section aft of the transducer wet section.

Figure 2. While integrating the echosounders a number of avenues were used to eliminate noise in the system including adjusting the switching frequency of the AUV’s drive motor, shielding cables, installing filters on power lines, modifying internal grounding paths, isolating electronics, and altering cable positions. (A) Shows the transducers mounted in the forward wet section and (B) Shows the transceivers and two computers on a rail mount for the dry electronics section aft of the transducer wet section.

Another advantage to integrating the echosounders into the AUV is increasing the effective resolution of the targets. The AUV is able to travel to the depths of interest and close the distance between the sensors and target. For example, in a mission south of Catalina Is., CA in September, 2013, there was an acoustic scattering layer at approximately 300 m. At the range this feature is sampled, the ship-based sensor had a horizontal footprint of 37–40 m while the AUV sensor, flying 4 m above the layer, had a footprint of 0.6–3.7 m. The increased resolution means one can now resolve individuals rather than simply the layer at these depths (Figure 3). The multiple frequencies also allow for classification of the individuals into taxonomic categories and size estimates [30,31]. This system, therefore, makes it possible to examine the biology of animals in the mesopelagic zone in ways previously only possible in the upper ocean.

Finally, by integrating this sensors into an AUV system, the platform itself can respond to the local environment. While this scenario could be set up on a ship, the ability of the ship to maneuver and respond to the environment is limited relative to an AUV system, especially relative to the size of the organisms of interest. By incorporating autonomy into this AUV, one can increase sampling resolution, improve quantification of targets of interest, and ultimately better understand fundamental biological processes. Details of decision-making approach used is discussed later, but it is first important to understand the steps between raw data acquisition and the synthetic data used in affecting navigational control.

Figure 3. Example of output from the dual-frequency echosounders integrated into the AUV. Both panels show the same slice of water within an acoustic scattering layer off California. Evident is the ability of this system to discriminate individual targets and the clear differences in the raw volume scattering between the two frequencies. The difference in raw scattering allows for individuals to be classified into taxonomic categories.

Figure 3. Example of output from the dual-frequency echosounders integrated into the AUV. Both panels show the same slice of water within an acoustic scattering layer off California. Evident is the ability of this system to discriminate individual targets and the clear differences in the raw volume scattering between the two frequencies. The difference in raw scattering allows for individuals to be classified into taxonomic categories.

2.3. Data Processing and Analysis

Data are first collected by the echosounder transceivers, which are on a rail mount in an air-sealed section aft of the wet transducer section (Figure 1 and Figure 2). The transceivers are connected via ethernet to the two PC-104 form factor computer stacks attached to the same rails as the transceivers (Figure 2B). Each computer stack is a VersaLogic Leopard based, 2.26 GHz commercial temperature Intel Core 2 Duo processor with 4 GB RAM, dual gigabit Ethernet, and two solid state hard drives running the Windows 7 operating system. Both computers are coupled to the vehicle’s network via Ethernet, allowing the computers to be viewed and controlled remotely through the vehicle’s wired and wireless connections. One computer stack runs Simrad’s ER60 data acquisition software along with the operating system on one hard drive and acquires data directly to its second hard drive. This computer stack also has a serial connection to the vehicle computer which provides time, vehicle depth, and position which is automatically merged into the acoustic data stream utilizing existing navigation input options within the ER60 software. The second computer stack is responsible for processing data and providing synthesized results to the vehicle to modify navigation. As processing the echosounder data is computationally demanding, separation of data acquisition and data processing ensures the robustness of the system. The processing computer runs Echoview software (Echoview Software Pty Ltd, Hobart Tasmania, Australia), as well as a custom, stand-alone Windows-based application written in C++ that manages the software and passes processed information to the vehicle’s computer via Ethernet at the frequency determined within the data processing program implemented within Echoview. Echoview provides robust near real-time analysis that can incorporate basic data processing along with tools for combination of the two frequencies of acoustic data, analysis of solitary targets, volume scattering integration, and more. These analyses are incorporated into a visually programmed “data flow” that is saved as a distinct file that can easily be replaced as analysis needs change.

The data processing and synthesis example used in this effort targeted squid and will be describe here. Data from the two frequencies (Figure 3) of the entire water column are processed (removal of the seafloor, correcting data depth as the vehicle dives, removal of noise, etc.) to identify individual targets using the Live Viewing module of Echoview software (Echoview Pty Limited, Tasmania, Australia) (Figure 4). The target fields of the two frequencies are then combined, retaining only targets detected by both frequencies. Here, a target filter was applied based on the differences in the volume scattering between expected for squid [28]. From the echoes consistent with squid, target strength is converted to an estimate of squid length and the target density within a depth interval are produced and combined to provide an estimate of squid biomass (Figure 4). Importantly, this analysis can easily be altered to target fish, krill, or marine mammals either by the user between missions or predetermined internally. While not attempted, these multiple filters could be applied simultaneously to provide combined target scenarios if desired.

Figure 4. Diagrammatic flow of data processing, synthesis and data product generation with squid as the target organisms. From the raw data taken off California in 2013 (Figure 3), targets are determined for each frequency, combined with a target filter to achieve a set of data products to be used in vehicle decision-making and autonomy.

Figure 4. Diagrammatic flow of data processing, synthesis and data product generation with squid as the target organisms. From the raw data taken off California in 2013 (Figure 3), targets are determined for each frequency, combined with a target filter to achieve a set of data products to be used in vehicle decision-making and autonomy.

2.4. Autonomous Decision-Making

Autonomy implemented thus far has been based on the simplest case, a binary signal. The key point however outlined in the introduction is that the response signal is based on a processed data product and a target group with the goal of improving sampling density. Here, after the three data products are generated, we determine a threshold for providing a positive signal. Figure 5 illustrates the decision-making threshold. Integrated over a particular time (here 1 min) and over the full depth range (600 m) the system identifies the target number and the size of the target. If the criteria set in the data flow for size and number (here 100 squid > 20 cm long) is met, the custom application running on the stack sends a “1” to the vehicle’s RECON (Remote Control) computer, otherwise it sends a zero. A 1 or 0 flag is encapsulated and transmitted as a user datagram packet (UDP) data packet through the vehicle network. These outputs occurred within 30 s of the acquisition of the data, providing close to real time feedback to the vehicle. The RECON computer was programmed to respond when the UDP data packet sends a series of “1”, taking control over from the vehicle’s primary navigation computer, pausing the primary mission, and executing a secondary mission for either a set amount of time, until completing the secondary mission, or until additional sensor input met some prescribed condition. The autonomy or vehicle response to the simple binary system is highlighted in the Results section from work recently completed in the Bahamas.

Figure 5. Example of threshold determination for autonomous decision-making based on real-time data products from data taken off California in 2013 (Figure 4). Here, squid number and length are used to determine changes in vehicle navigation.

Figure 5. Example of threshold determination for autonomous decision-making based on real-time data products from data taken off California in 2013 (Figure 4). Here, squid number and length are used to determine changes in vehicle navigation.

3. Results

Application of the complete integrated system first took place off the coast of California in 2013 with 28 mission traveling over 650 km at depths to 550 m with the longest mission of ~17 h. Flight metrics showed robustness in maintaining depth target (±0.05 m), heading (±0.1°), pitch (±0.3°) and roll (±0.3°). Level flight is especially important for these acoustic sensors making measurements 600 m away from the vehicle [29]. In these same initial tests we determined the noise floor of the integrated system and robustness in delineating single targets. While the vehicle autonomy was not implemented during the field efforts off of California as there were prescribed transects needed to complete the intended experimental design, the data processing and threshold determination performed as intended.

A second field effort was recently completed within the Tongue of the Ocean (TOTO), Bahamas in July 2015 (Figure 6). Here we completed 22 missions over 11 d with over 700 km underwater time (Figure 7). These missions like the previous ones off California were designed to quantify distributions of whale prey (fish, squid, and crustaceans) in the mesopelagic (200–1000 m). Because of the need to broadly cover the area and duplicate the experimental design, as was set up off the California coast, most of the missions were long (20–30 km) transects. For one mission however, we implemented a response mission whereby real-time data processing and the detection of aggregated squid triggered the vehicle to initiate a secondary mission to resolve finer scale distributions and increase the data density of the same target organism (Figure 8). On the positive trigger, the vehicle started a pre-programed fine grid mission and completed the entire mission prior to additional commands or the continuation of the larger mission. In order to best identify the distribution of squid, it was important for the vehicle not to react or respond to an isolated squid signal (a single “1” from the data analysis software) but be in aggregation or cluster. Therefore, the threshold for initiating the fine scale grid mission was the first instance where greater than half of the most recent 10 bins were positive (Figure 8). The vehicle approached from the east and began the secondary mission. The grid was designed as an expanding box starting in the center with 100 m separation between boxes, growing to 200 m, and finally one at 500 m separation with dimensions of the largest box of 750 m (Figure 8). On completing the mission, the vehicle surfaced, acquired a GPS position and corrected for the offset by ocean currents ending the secondary mission in the northeast quadrant (intended box–pink) and continuing with the original mission in the center and then follow the trackline to the west. As the vehicle does not have intertial navigation and only relies on compass heading, ocean currents pushed the vehicle off course to the south. Even though the mission data was not symmetrical, the mission goal of sampling over smaller scales was achieved and demonstrated in challenging field conditions. Data collected here showed a number of spatial features in the organism distributions that would have otherwise not been possible by other means, both in responsiveness and of a depth beyond the ships echosounder range. A ‘hotspot’ of squid was detected over the eastern edges of the 200 m spaced boxes. This scale was within a general transition to higher squid abundance to the southeast. While the contour resolution chosen for Figure 8 is to illustrate these two scales, the binary output from a section of the southern line of the 500 m box also illustrates that there are additional higher spatial resolutions to consider for this target organism.

Figure 6. The Tongue of the Ocean area where the AUV was tested in the Bahamas in July, 2015. Highlighted is the AUV transect and the location where the high resolution grid was run.

Figure 6. The Tongue of the Ocean area where the AUV was tested in the Bahamas in July, 2015. Highlighted is the AUV transect and the location where the high resolution grid was run.

Figure 7. REMUS AUV (A) being deployed and (B) underway beginning a mission in July 2015 in the Bahamas.

Figure 7. REMUS AUV (A) being deployed and (B) underway beginning a mission in July 2015 in the Bahamas.

Figure 8. Mission designed to resolve smaller scale distributions of organisms that was run on 10 July 2015 in the Bahamas. The intended track is in pink with the actual track in black. Offset is due to currents (see text). The rate of positive detections for squid targets per 100 m is shown in red. In the yellow box is a binary stream from the processed data at the beginning of the mission and the sequence in italics that met the threshold for initiating the finer grid mission. In contrast, the gray box is the real-time onboard binary output from a portion of the southerly leg showing higher squid densities throughout the water column. Binary data for the mission are shown as smoothed surface color plot. Apparent are two features that would not have been identified or appropriately scaled from the linear transect: a strong local gradient and a small ‘hotspot’ of squid detection across several of the nested sampling boxes.

Figure 8. Mission designed to resolve smaller scale distributions of organisms that was run on 10 July 2015 in the Bahamas. The intended track is in pink with the actual track in black. Offset is due to currents (see text). The rate of positive detections for squid targets per 100 m is shown in red. In the yellow box is a binary stream from the processed data at the beginning of the mission and the sequence in italics that met the threshold for initiating the finer grid mission. In contrast, the gray box is the real-time onboard binary output from a portion of the southerly leg showing higher squid densities throughout the water column. Binary data for the mission are shown as smoothed surface color plot. Apparent are two features that would not have been identified or appropriately scaled from the linear transect: a strong local gradient and a small ‘hotspot’ of squid detection across several of the nested sampling boxes.

4. Discussion

Results here present a new tool for quantifying the abundance and distribution of nektonic organisms in an area of the bathypelagic previously unattainable by ship board measurements. Of equal importance to the sensor integration and the depth range are the onboard data processing and synthesis, and the ability of the vehicle to independently respond to varied products; different organisms, size, number, depth distribution and biomass. While this example is one of the simplest possible, it demonstrates the powerful combination of multiple data sources, a platform that can improve time/space sampling, real-time data synthesis, and autonomous decision-making. The mission was able to address the science objective of providing information on the horizontal distribution of squid of a particular size and the scales of their organization. Additionally, observations indicated that the squid target distribution was not equally displaced by the currents as the vehicle (550 m), suggesting these deep-dwelling organisms were is a more quiescent environment when sampled. Given the number of product variables output from processing on this vehicle, there are a host of scenarios that one could employ to improve understanding of the biology in the system. Not only could the data combination triggering the behavior be more complex, but the autonomy could also increase in complexity. While only one ‘decision’ cycle has being demonstrated here, it does not preclude the possibility for much faster cycle times for decision-making. These options for autonomy will be explored and tested in future deployments. While the possible applications of this system are numerous, it is also important to place these possibilities in the context of the scientific goals. Even the basic understanding of the horizontal distributions of these organisms in this depth region are unknown and is what motivated the first step presented here.

There are many applications for closed loop decision-making. For example in oil and gas production, there is a need for constant monitoring of the marine environment for both chronic problems and acute effects such as drill cuttings, seeps and leaks. Vehicles could be configured to measure these plumes with fluorometric sensors and internally process data to build a real-time map of a given plume, constantly updating its navigational goals. In aquaculture, low oxygen, harmful algae and disease are constant threats and integrated monitoring with long endurance vehicles, such as gliders outfitted with the appropriate sensors suite is possible [17,32]. Fisheries assessment is difficult and a vehicle similar to the one presented here could allow for improved quantification of biomass, school size, and migration patterns of commercial species (i.e., herring, pollock, salmon, krill). Vehicles configured to image the seafloor could be used similarly to track populations of bottom fish and habitat [33]. If fact, this approach has been used by Clarke and colleagues [20] to identify and track rockfish off the coast of California. By image identification of bottom fish species and coincidently imaging the seafloor, they were able to assess associations and identify usable habitat. While the example presented here focused on layers and schools of organisms, a similar system could be used to track individual organisms, such as marine mammals and top other predators without the use of tagging as has been demonstrated [21]. More challenging than sampling larger organisms is applying these closed-loop approaches to smaller organisms as these targets are much more difficult to identify in real time. Novel and innovative AUV autonomy work has used proxy measures of the environment to improve the likelihood of sampling larval organisms and harmful algae in vertical layers and fronts [14,19,34,35,36]. While clearly effective, this approach may produce false positives and must be locally tuned based on target species (i.e., Pseudo-nitzschia spp.). New imaging technologies for phytoplankton, larval invertebrates and larval fishes are currently available [37,38] and if integrated into autonomous platforms could directly identify these targets and influence both sampling and navigation. While the sensors, platforms, and data fusion need to vary to match the diversity of marine organisms, we advocate a strategy towards on-board product generation to drive navigation and sampling as demonstrated here could be similar.

5. Conclusions

Biological sampling in aquatic environments pose significant challenges to science and engineering primarily due to the dynamic nature of the environment and the complex behavior of the organisms of interest. Autonomous platforms begin to address these scales, especially when incorporating real-time data to influence the vehicle navigation. We present a new tool for examining one class of organisms with the used of acoustics, and although simplistic in its demonstration illustrates the power of on board integrated product generation in improving sampling. While the development of underwater, surface, and aerial vehicles continues, it is the multi-sensor integration and synthesis coupled to autonomous navigation and novel path planning techniques [39] that will provide new awareness. Today, as oceanographers, we are still largely planning our sampling on geometric patterns (transect lines, squares, and grids) because we are just beginning to understand the dynamic distributions and abundances of marine organisms. Adaptable intelligent sampling based on real-time data products will provide decision autonomy [40] and revolutionize the way we sample the ocean and in turn transform our assessment of the marine environment, elucidate the processes that govern its physical and biological dynamics, and provide insights to predicting future change.