Human-Centered Explainable Artiﬁcial Intelligence for Marine Autonomous Surface Vehicles

: Explainable Artiﬁcial Intelligence (XAI) for Autonomous Surface Vehicles (ASVs) addresses developers’ needs for model interpretation, understandability, and trust. As ASVs approach wide-scale deployment, these needs are expanded to include end user interactions in real-world contexts. Despite recent successes of technology-centered XAI for enhancing the explainability of AI techniques to expert users, these approaches do not necessarily carry over to non-expert end users. Passengers, other vessels, and remote operators will have XAI needs distinct from those of expert users targeted in a traditional technology-centered approach. We formulate a concept called ‘human-centered XAI’ to address emerging end user interaction needs for ASVs. To structure the concept, we adopt a model-based reasoning method for concept formation consisting of three processes: analogy, visualization, and mental simulation, drawing from examples of recent ASV research at the Norwegian University of Science and Technology (NTNU). The examples show how current research activities point to novel ways of addressing XAI needs for distinct end user interactions and underpin the human-centered XAI approach. Findings show how representations of (1) usability, (2) trust, and (3) safety make up the main processes in human-centered XAI. The contribution is the formation of human-centered XAI to help advance the research community’s efforts to expand the agenda of interpretability, understandability, and trust to include end user ASV interactions.


Introduction
Artificial Intelligence (AI) is being increasingly used in maritime applications. This is perhaps most clearly seen in Autonomous Surface Vehicles (ASVs), a category of maritime vessels that emerged in oceanographic and marine biological data collection [1][2][3] and has recently branched out into urban mobility [4][5][6] (Figure 1). A related category is the Maritime Autonomous Surface Ship (MASS), now formally acknowledged by the International Maritime Organization (IMO) [7], the world regulatory agency for safety and environmental protection at sea. MASS are distinct from ASVs primarily in scale and by virtue of their reliance upon a hybrid format of AI-human collaborative seafaring coordinated from a 'Shore Control Center' (known alternatively as 'Remote Control Station/Center') [8][9][10]. However, trends toward the development of ASVs with passengers and automated shipboard navigation systems, such as auto-crossing and auto-docking [11,12], are blurring the line between the two categories. Work towards scaling MASS and ASVs into widespread use raises new challenges related to ensuring that AI system goals are aligned with the values of those who will be interacting with them. This is broadly the motivation behind the growing field of Explainable AI (XAI), characterized, as expressed by [13], by its mission to 'create AI systems whose learned models and decisions can be understood and appropriately trusted by end users' (p. 44). This mission is necessarily multi-disciplinary, meeting at the crossroads of fields as diverse as cognitive science, human-computer interaction, cybernetics, safety engineering, computer science, human factors, sociology, and others.
Regulations (COLREGS) extend the aim of AI system interpretability and accountability to operational and regulatory oversight domains.
This work investigates the issues surrounding the explainability and trust of AI systems from the lens of a practice-based, human-centered design approach tailored specifically to the needs of ASVs. From this perspective, efforts towards automation transparency aiming to improve model understandability are just one important factor among several others, including the affordances necessary for understanding its use and establishing trust among a broader stakeholder base emerging from more widespread deployment. Our research question is: can a human-centered approach to XAI contribute to building trust among real-world ASV users?

Method
Our aim in this paper was to introduce the concept of human-centered XAI for ASV applications. In this aim, methods describing the process of creating scientific concepts were particularly helpful. We drew inspiration from the cognitive and social sciences, and used the principles behind model-based reasoning [30] to structure our findings. Modelbased reasoning posits that concept formation in science occurs through analogy, visualization, and mental simulation. We organized the findings accordingly, with analogy, visualization, and mental simulation each allotted their own sub-section. These sub-sections contain examples from our own research and from recent research of our peers at our university. The examples illustrate an important point: they are not the findings of the article in themselves; rather, they show how, through the lens of model-based reasoning, they can be used to construct the concept of human-centered XAI for ASVs.
The social sciences have placed great importance on the process of concept formation, possibly even more so than the natural sciences. We thus drew inspiration from this field, too. Weber, in unequivocal terms, described the concept as 'one of the great tools of all scientific knowledge' ( [31], p. 151), highlighting their role in allowing research to proceed through their capacity 'to establish knowledge of what is essential' ( [32], p. 213). He also pointed out that existing concepts can be used as building material for new concepts, which is the case for human-centered XAI in our work: an amalgamation of human-centered design and Explainable AI. Swedberg [33] described the process of creating new concepts as an essential part of building theory in science. Observation, according to Swedberg, leads to naming of a phenomenon, which then 'often needs to be turned into a concept… to a get a firm grip on the phenomenon' ( [33], pp. [58][59]. The empirical data we present were drawn from practice-based research activities over the period 2018-2021 at the Norwegian University of Science and Technology (NTNU). Lacking the space to include everything, we selected examples that we think helped to formulate the concept of human-centered XAI for ASVs in the framework of model-based reasoning. While the research we presented stemmed from a range of projects, one is featured predominantly: the design and construction of an autonomous passenger ferry called the milliAmpere2, which is described in Section 2.1.

The milliAmpere2 Autonomous Passenger Ferry
Up until the mid-1960s, there was a century-old service available in central Trondheim for transporting passengers across a 100-m-long urban canal ( Figure 2). Locally

Method
Our aim in this paper was to introduce the concept of human-centered XAI for ASV applications. In this aim, methods describing the process of creating scientific concepts were particularly helpful. We drew inspiration from the cognitive and social sciences, and used the principles behind model-based reasoning [30] to structure our findings. Model-based reasoning posits that concept formation in science occurs through analogy, visualization, and mental simulation. We organized the findings accordingly, with analogy, visualization, and mental simulation each allotted their own sub-section. These sub-sections contain examples from our own research and from recent research of our peers at our university. The examples illustrate an important point: they are not the findings of the article in themselves; rather, they show how, through the lens of model-based reasoning, they can be used to construct the concept of human-centered XAI for ASVs.
The social sciences have placed great importance on the process of concept formation, possibly even more so than the natural sciences. We thus drew inspiration from this field, too. Weber, in unequivocal terms, described the concept as 'one of the great tools of all scientific knowledge' ( [31], p. 151), highlighting their role in allowing research to proceed through their capacity 'to establish knowledge of what is essential' ( [32], p. 213). He also pointed out that existing concepts can be used as building material for new concepts, which is the case for human-centered XAI in our work: an amalgamation of human-centered design and Explainable AI. Swedberg [33] described the process of creating new concepts as an essential part of building theory in science. Observation, according to Swedberg, leads to naming of a phenomenon, which then 'often needs to be turned into a concept . . . to a get a firm grip on the phenomenon' ( [33], pp. [58][59]. The empirical data we present were drawn from practice-based research activities over the period 2018-2021 at the Norwegian University of Science and Technology (NTNU). Lacking the space to include everything, we selected examples that we think helped to formulate the concept of human-centered XAI for ASVs in the framework of model-based reasoning. While the research we presented stemmed from a range of projects, one is featured predominantly: the design and construction of an autonomous passenger ferry called the milliAmpere2, which is described in Section 2.1.

The milliAmpere2 Autonomous Passenger Ferry
Up until the mid-1960s, there was a century-old service available in central Trondheim for transporting passengers across a 100-m-long urban canal ( Figure 2). Locally called the Fløttmann, this canal-crossing service was administered by a person in a rowboat (Figure 3a). Today the service is available in the summer months as a tourist attraction. In 2018 the local municipality proposed to construct a pedestrian bridge at the location where the Fløttmann crosses, meeting resistance among stakeholders. Out of these discussions emerged the idea of an automatic passenger ferry, put forward by an associate professor at NTNU. Under the theme of 'digital transformation', the Autoferry Project was kicked off in the same year [34]. Before long, the first prototype was ready. It was called milliAmpere, named after the first electric ferry in Norway, the Ampere. This prototype continues to be used by students, PhDs, and postdocs as a research platform for testing and development of sensor fusion, powering, maneuvering [35], safety, cyber-security, and automated collision avoidance [36]. Meanwhile, work started in 2019 to design an operational version of the automated ferry, capable of carrying up to twelve passengers for public use at the same location as the Fløttmann. As of late-2021, the milliAmpere2 (Figure 3b) has been commissioned and is undergoing field testing.
From a design perspective, the milliAmpere2 offers a unique opportunity to investigate human-centered XAI because it poses the hypothetical question: which would you choose given a choice of the human-operated Fløttmann and the autonomous mil-liAmpere2? As we approach the operational stage of milliAmpere2, this question will soon represent a real choice. The question motivates our research question: can a humancentered approach to XAI contribute towards building trust among real-world ASV users?
called the Fløttmann, this canal-crossing service was administered by a person in a boat (Figure 3a). Today the service is available in the summer months as a tourist a tion. In 2018 the local municipality proposed to construct a pedestrian bridge at the tion where the Fløttmann crosses, meeting resistance among stakeholders. Out of discussions emerged the idea of an automatic passenger ferry, put forward by an asso professor at NTNU. Under the theme of 'digital transformation', the Autoferry Pr was kicked off in the same year [34]. Before long, the first prototype was ready. It called milliAmpere, named after the first electric ferry in Norway, the Ampere. This totype continues to be used by students, PhDs, and postdocs as a research platform testing and development of sensor fusion, powering, maneuvering [35], safety, cybe curity, and automated collision avoidance [36]. Meanwhile, work started in 2019 to d an operational version of the automated ferry, capable of carrying up to twelve passen for public use at the same location as the Fløttmann. As of late-2021, the milliAmp ( Figure 3b) has been commissioned and is undergoing field testing.
From a design perspective, the milliAmpere2 offers a unique opportunity to inv gate human-centered XAI because it poses the hypothetical question: which would choose given a choice of the human-operated Fløttmann and the autonomous mill pere2? As we approach the operational stage of milliAmpere2, this question will represent a real choice. The question motivates our research question: can a human tered approach to XAI contribute towards building trust among real-world ASV use    called the Fløttmann, this canal-crossing service was administered by a person in a ro boat (Figure 3a). Today the service is available in the summer months as a tourist attr tion. In 2018 the local municipality proposed to construct a pedestrian bridge at the lo tion where the Fløttmann crosses, meeting resistance among stakeholders. Out of th discussions emerged the idea of an automatic passenger ferry, put forward by an associ professor at NTNU. Under the theme of 'digital transformation', the Autoferry Proj was kicked off in the same year [34]. Before long, the first prototype was ready. It w called milliAmpere, named after the first electric ferry in Norway, the Ampere. This p totype continues to be used by students, PhDs, and postdocs as a research platform testing and development of sensor fusion, powering, maneuvering [35], safety, cybercurity, and automated collision avoidance [36]. Meanwhile, work started in 2019 to des an operational version of the automated ferry, capable of carrying up to twelve passeng for public use at the same location as the Fløttmann. As of late-2021, the milliAmpe ( Figure 3b) has been commissioned and is undergoing field testing. From a design perspective, the milliAmpere2 offers a unique opportunity to inve gate human-centered XAI because it poses the hypothetical question: which would y choose given a choice of the human-operated Fløttmann and the autonomous milliA pere2? As we approach the operational stage of milliAmpere2, this question will so represent a real choice. The question motivates our research question: can a human-c tered approach to XAI contribute towards building trust among real-world ASV users     Figure 4 depicts the XAI audience we considered in this work, tailored specifically to the ASV application case. Figure 4 also illustrates the scope of this work in terms of what segments of the XAI audience we did not consider. Continued efforts are needed to expand the XAI audience towards organizational stakeholders, such as managers and owners, as well as regulatory agencies and non-governmental organizations.  Figure 4 depicts the XAI audience we considered in this work, tailored specifica the ASV application case. Figure 4 also illustrates the scope of this work in terms of segments of the XAI audience we did not consider. Continued efforts are needed pand the XAI audience towards organizational stakeholders, such as managers and ers, as well as regulatory agencies and non-governmental organizations. To illustrate the distinct XAI needs for different user groups, consider a visualiz of an object classification algorithm ( Figure 5). While a developer needs to know the ability that a given classification of an identified object is correct, a ship operator ne know practical navigation details (e.g., the ship's name, current speed and heading destination). In contrast, a passenger may only need to know whether the ASV ha covered it.

Methodological Considerations
At the outset of our research, we did not intend to create the concept of human tered XAI for ASVs, nor, once we stumbled upon the idea, did we immediately beg ing model-based reasoning to systematically give the idea structure. Model-based re ing, as a method, was used a posteriori, as a way of lending our reasoning, which occ 'abductively' [37], a kind of organization that is only possible in hindsight. Thus m based reasoning has a decidedly historical perspective, adding up to an account o ceptual change grounded in cognitive phenomena. It represents what Nersessian ca cognitive-historical method: a kind of 'bootstrapping procedure commonly used ence' where the 'range of historical records, notebooks, diaries, correspondence, d publications, and artifacts, such as instruments and physical models, serves as the s of empirical data on the scientific practices' ( [30], p. 7). Such a historical perspectiv even be a necessary part of concept formation because, as pointed out by Swedberg often not possible to create a concept until the research is well underway' ( [33], p. 6 To illustrate the distinct XAI needs for different user groups, consider a visualization of an object classification algorithm ( Figure 5). While a developer needs to know the probability that a given classification of an identified object is correct, a ship operator needs to know practical navigation details (e.g., the ship's name, current speed and heading, and destination). In contrast, a passenger may only need to know whether the ASV has discovered it.  Figure 4 depicts the XAI audience we considered in this work, tailored specifically to the ASV application case. Figure 4 also illustrates the scope of this work in terms of what segments of the XAI audience we did not consider. Continued efforts are needed to expand the XAI audience towards organizational stakeholders, such as managers and owners, as well as regulatory agencies and non-governmental organizations. To illustrate the distinct XAI needs for different user groups, consider a visualization of an object classification algorithm ( Figure 5). While a developer needs to know the probability that a given classification of an identified object is correct, a ship operator needs to know practical navigation details (e.g., the ship's name, current speed and heading, and destination). In contrast, a passenger may only need to know whether the ASV has discovered it.

Methodological Considerations
At the outset of our research, we did not intend to create the concept of human-centered XAI for ASVs, nor, once we stumbled upon the idea, did we immediately begin using model-based reasoning to systematically give the idea structure. Model-based reasoning, as a method, was used a posteriori, as a way of lending our reasoning, which occurred 'abductively' [37], a kind of organization that is only possible in hindsight. Thus modelbased reasoning has a decidedly historical perspective, adding up to an account of conceptual change grounded in cognitive phenomena. It represents what Nersessian calls the cognitive-historical method: a kind of 'bootstrapping procedure commonly used in science' where the 'range of historical records, notebooks, diaries, correspondence, drafts, publications, and artifacts, such as instruments and physical models, serves as the source of empirical data on the scientific practices' ( [30], p. 7). Such a historical perspective may even be a necessary part of concept formation because, as pointed out by Swedberg, 'it is often not possible to create a concept until the research is well underway' ( [33], p. 60).

Methodological Considerations
At the outset of our research, we did not intend to create the concept of humancentered XAI for ASVs, nor, once we stumbled upon the idea, did we immediately begin using model-based reasoning to systematically give the idea structure. Model-based reasoning, as a method, was used a posteriori, as a way of lending our reasoning, which occurred 'abductively' [37], a kind of organization that is only possible in hindsight. Thus model-based reasoning has a decidedly historical perspective, adding up to an account of conceptual change grounded in cognitive phenomena. It represents what Nersessian calls the cognitive-historical method: a kind of 'bootstrapping procedure commonly used in science' where the 'range of historical records, notebooks, diaries, correspondence, drafts, publications, and artifacts, such as instruments and physical models, serves as the source of empirical data on the scientific practices' ( [30], p. 7). Such a historical perspective may even be a necessary part of concept formation because, as pointed out by Swedberg, 'it is often not possible to create a concept until the research is well underway' ( [33], p. 60).
The scientific literature on concept formation is extensive, covering not just the cognitive and social sciences, as we have mentioned, but also the philosophy of science, linguistics, mathematics, and other fields. Defining a concept also raises the question of defining other concepts, as pointed out by Wittgenstein: 'What should we gain from a definition, as it can only lead us to more undefined terms?' ( [38], p. 26). Similarly, it is widely recognized that even the word 'concept' has no formal definition. This circular problem is exemplified by the concept of 'mental model,' a useful concept we invoke throughout the paper, but that nonetheless has no agreed-upon definition across the many fields of science using it. This has led to, as expressed by [39], a 'muddling' of the term. Here, we use the definition offered by Nersessian, whose model-based reasoning framework for concept formation we also adopted, and whose definition is agnostic to the myriad fields applying it: 'a mental model is a structural, behavioral, or functional analog representation of a real-world or imagined situation, event, or process' ( [30] p. 95).

Results
Here we present observations of research activities as they relate to three elements of concept formation for human-centered XAI for ASVs. These three elements are listed below and originate from the model-based reasoning framework for concept formation (Section 2):

1.
Analogy (representation of an unfamiliar concept in terms of a familiar one); 2.
Visualization (representation of internal states through external imagistic processes); 3.
Mental simulation (representation of behavior through 'mental models' and thought experiments, including numerical simulation).

Analogy
During the early-stage design of the human-machine interface for the milliAmpere2 prototype ferry, it emerged that trust was important for establishing an interaction relationship among passengers. The ASV technology represented a new concept, after all-one that without a human operator broke with convention. The interaction relationship, we reasoned, could be designed into the process of introducing the ferry to passengers, including an explanation of what the ferry was and how it worked. The prototype information post, depicted in Figure 6, contains such an introductory message: 'World's first driverless passenger ferry. The service is free and open for everyone. The ferry works like an elevator. You press the Call button, and it calls the ferry. You can take aboard everything from your pets to your bike and stroller. The ferry goes every day from 07:00 to 22:00. The ferry crosses between Ravnkloa and Venstre Kanalkai.' The information post in Figure 6 contained the phrase: 'The ferry works like an elevator.' The analogy with an elevator promoted the desired change in passenger trust by enabling understanding one new representation in terms of another, more familiar one. The elevator analogy also served to encode a 'mental model' of how the service works (see Section 2.3 for our definition of 'mental model'). To illustrate this, consider a new passenger mentally simulating pressing a button to call the ferry, understanding that one must wait for its arrival after being called. Then, stepping inside once the doors open, this passenger can press another button inside to close the doors and initiate the crossing. Finally, upon arrival, the doors open on the other side, and they can disembark. An interaction relationship is thus established based on encoding a representation of its use in a mental model and reinforcing that mental model with the expected user interfaces. Buttons for calling the ferry and for initiating it upon entry also reinforce to the user that they are using it correctly and have control. This example shows that analogy plays an important role in explaining ASV tionality to end users. In the milliAmpere2 example, the analogy worked by transfe the representation of a familiar concept (an elevator) to an unfamiliar concept (ASV nology). In this example, the analogy enhanced the usability of the ASV. Analog much in common with mental simulation, a method of concept formation we return Section 3.3.

User Displays
As in the previous example where functional representation was transferred t ASV through an analogy (Section 3.1), so could useability be transferred by affirmin resulting 'mental model' with expected design affordances. In this example, thos pected affordances consisted of elevator-like inputs and user displays explaining th ternal representation of the ASV. For example, referring to the example in Figure 6, pressing the 'Go' button, the 'Closing doors' screen is displayed to the passenger, an 'Go' button begins indicating the extent to which the ferry has reached its destinatio ing an animated radial dial. If passenger count exceeds twelve then the expectation i that the ferry cannot depart, and the 'Go' button is greyed out with a warning me that passenger count has been exceeded. The screen also displays an avatar of the along with safety information and the expected time of arrival for different stops. O objects, including land, coastal infrastructure, and other ships and leisure craft, ar picted on the screen, displaying to passengers what the autonomous system has det in the environment. Such visualizations serve two purposes: they support usabili affirming the users' mental model of how the ferry works, and they transform int representations of the ferry ('what is it thinking?') to external imagistic representatio This example builds the case for human-centered XAI because it shows how vi zation on user displays can influence user understanding, interpretability, and tru this example, the visual representation of 'what the ferry is thinking' enhanced the A usability. Rather than trying to accomplish this in traditional XAI terms, a distinctl man-centered approach involved the representation of internal states to end through visual means. This example shows that analogy plays an important role in explaining ASV functionality to end users. In the milliAmpere2 example, the analogy worked by transferring the representation of a familiar concept (an elevator) to an unfamiliar concept (ASV technology). In this example, the analogy enhanced the usability of the ASV. Analogy has much in common with mental simulation, a method of concept formation we return to in Section 3.3.

User Displays
As in the previous example where functional representation was transferred to the ASV through an analogy (Section 3.1), so could useability be transferred by affirming the resulting 'mental model' with expected design affordances. In this example, those expected affordances consisted of elevator-like inputs and user displays explaining the internal representation of the ASV. For example, referring to the example in Figure 6, upon pressing the 'Go' button, the 'Closing doors' screen is displayed to the passenger, and the 'Go' button begins indicating the extent to which the ferry has reached its destination using an animated radial dial. If passenger count exceeds twelve then the expectation is met that the ferry cannot depart, and the 'Go' button is greyed out with a warning message that passenger count has been exceeded. The screen also displays an avatar of the ferry along with safety information and the expected time of arrival for different stops. Other objects, including land, coastal infrastructure, and other ships and leisure craft, are depicted on the screen, displaying to passengers what the autonomous system has detected in the environment. Such visualizations serve two purposes: they support usability by affirming the users' mental model of how the ferry works, and they transform internal representations of the ferry ('what is it thinking?') to external imagistic representations.
This example builds the case for human-centered XAI because it shows how visualization on user displays can influence user understanding, interpretability, and trust. In this example, the visual representation of 'what the ferry is thinking' enhanced the ASV's usability. Rather than trying to accomplish this in traditional XAI terms, a distinctly human-centered approach involved the representation of internal states to end users through visual means.

Design, Form and Aesthetic
During the early-stage design of the milliAmpere2, several data collection efforts were launched to gain insights into how the ferry design affected users' perception of the new technology. For example, in [40], surveys, interviews, and workshops were used to understand how design, form, and aesthetics conveyed human-centered values, such as 'safety, stability, user-friendliness.' This culminated in the design of a physical model ( Figure 7) and eventually the full-scale ferry ( Figure 3b). As described in [40], the design portrayed the familiar curvature and materials of a sailboat-its broad, open deck, the elements of a bridge. These visual representations transferred meaning in a similar way to direct analogy (Section 3.1). In [41], researchers interviewed pedestrians and gained insights into how people interacted with the design as a service. The latter found, for example, that 'older users enjoyed teaching one another how to use new technology,' suggesting that actions involved in discovery and learning were highly valued in the technology interaction. This suggested that enhancing reflexive experiences like discovery and learning promoted overall interpretability, explainability, and trust in the ASV.
This example showed that design, form, and aesthetics played an important role in a human-centered approach to XAI because they helped to build trusting interaction relationships between end users and the ASV. The mechanism worked in a similar way to the analogy (Section 3.1) by transferring representations of familiar concepts (sailboats, bridges, interaction with 'new technology') to the ASV representation through visual means. During the early-stage design of the milliAmpere2, several data collection e were launched to gain insights into how the ferry design affected users' perception new technology. For example, in [40], surveys, interviews, and workshops were u understand how design, form, and aesthetics conveyed human-centered values, su 'safety, stability, user-friendliness.' This culminated in the design of a physical mode ure 7) and eventually the full-scale ferry ( Figure 3b). As described in [40], the desig trayed the familiar curvature and materials of a sailboat-its broad, open deck, th ments of a bridge. These visual representations transferred meaning in a similar w direct analogy (Section 3.1). In [41], researchers interviewed pedestrians and gain sights into how people interacted with the design as a service. The latter found, for ple, that 'older users enjoyed teaching one another how to use new technology,' su ing that actions involved in discovery and learning were highly valued in the techn interaction. This suggested that enhancing reflexive experiences like discovery and ing promoted overall interpretability, explainability, and trust in the ASV.
This example showed that design, form, and aesthetics played an important ro human-centered approach to XAI because they helped to build trusting interaction tionships between end users and the ASV. The mechanism worked in a similar way analogy (Section 3.1) by transferring representations of familiar concepts (sail bridges, interaction with 'new technology') to the ASV representation through means.

Sensor Data
We observed developers using visualizations of the autonomous navigation sy aboard the milliAmpere prototype ferry with the aim of understanding and impr how the system made decisions. The milliAmpere has been used extensively for t the sensor fusion algorithms driving the navigation and collision avoidance system example, Figure 8 shows a visualization of sensor data aboard the milliAmpere Robotic Operating System (ROS). For an ASV like the milliAmpere to apprehend its ronment and minimize the risk of detecting false positives and false negatives aff motion planning, a vast amount of real-time data is processed 'under the hood.' Sy sizing this data into a single input for motion planning underpins efforts in the fi sensor fusion. Explainability in these efforts represented a major challenge fo milliAmpere given the large volume of data. In Figure 8 we see no fewer than thre data sources overlayed onto a navigation map: Lidar, radar, and Infrared (IR) vide visualization also includes real-time data processing, including radar edge-tracin object tracking and classification.
Such visualizations point to a human-centered XAI process in that they represe internal state of the ASV ('what it is sensing') in an external imagistic representation external representation is used by developers in mental modeling processes to ev trust in ASV decision-making in the context of end user interactions.

Sensor Data
We observed developers using visualizations of the autonomous navigation systems aboard the milliAmpere prototype ferry with the aim of understanding and improving how the system made decisions. The milliAmpere has been used extensively for testing the sensor fusion algorithms driving the navigation and collision avoidance systems.
For example, Figure 8 shows a visualization of sensor data aboard the milliAmpere in the Robotic Operating System (ROS). For an ASV like the milliAmpere to apprehend its environment and minimize the risk of detecting false positives and false negatives affecting motion planning, a vast amount of real-time data is processed 'under the hood.' Synthesizing this data into a single input for motion planning underpins efforts in the field of sensor fusion. Explainability in these efforts represented a major challenge for the milliAmpere given the large volume of data. In Figure 8 we see no fewer than three raw data sources overlayed onto a navigation map: Lidar, radar, and Infrared (IR) video. The visualization also includes real-time data processing, including radar edge-tracing and object tracking and classification.
Such visualizations point to a human-centered XAI process in that they represent the internal state of the ASV ('what it is sensing') in an external imagistic representation. This external representation is used by developers in mental modeling processes to evaluate trust in ASV decision-making in the context of end user interactions. Another example of XAI visualizations stemmed from ML-based autonomous ing and undocking maneuvers. In [42], several XAI visualizations were created f developer that display the changing values of azimuth forces and angles (f1, f2, a and tunnel thruster forces (f3) of a simulated ship docking. One such visualization, form of an action plot (Figure 9), helps the developer to monitor the training status algorithm, identify bugs, and improve the autonomous system. Visualizations like the plot in Figure 9 have a limited XAI audience because meaning is decipherable only to experts of ML-based techniques for the motion con ASVs. In [42], the same scientists who generated this developer-centric plot recast t formation to a user-centric interface ( Figure 10), visually conveying which features model are weighted most heavily, as well as an avatar showing corresponding th power and direction. The display, intended to explain an automated docking proces navigator, also depicts green lines representing target destination and red lines repr ing the distance to the nearest object.
This example shows how visualization of AI models can enhance trust in AI de making among end users in a real-world context. This example, while starting with cialized visualizations intended for developers to improve model interpretability, ev towards broader user-centered interpretability when confronted with user intera This evolution supported the case that human-centered XAI processes were presen ing the design of ASVs that involved end user interactions. Another example of XAI visualizations stemmed from ML-based autonomous docking and undocking maneuvers. In [42], several XAI visualizations were created for the developer that display the changing values of azimuth forces and angles (f1, f2, a1, a2) and tunnel thruster forces (f3) of a simulated ship docking. One such visualization, in the form of an action plot (Figure 9), helps the developer to monitor the training status of the algorithm, identify bugs, and improve the autonomous system. Another example of XAI visualizations stemmed from ML-based autonomous docking and undocking maneuvers. In [42], several XAI visualizations were created for the developer that display the changing values of azimuth forces and angles (f1, f2, a1, a2) and tunnel thruster forces (f3) of a simulated ship docking. One such visualization, in the form of an action plot (Figure 9), helps the developer to monitor the training status of the algorithm, identify bugs, and improve the autonomous system. Visualizations like the plot in Figure 9 have a limited XAI audience because their meaning is decipherable only to experts of ML-based techniques for the motion control of ASVs. In [42], the same scientists who generated this developer-centric plot recast the information to a user-centric interface (Figure 10), visually conveying which features in the model are weighted most heavily, as well as an avatar showing corresponding thruster power and direction. The display, intended to explain an automated docking process to a navigator, also depicts green lines representing target destination and red lines representing the distance to the nearest object.
This example shows how visualization of AI models can enhance trust in AI decision making among end users in a real-world context. This example, while starting with specialized visualizations intended for developers to improve model interpretability, evolved towards broader user-centered interpretability when confronted with user interaction. This evolution supported the case that human-centered XAI processes were present during the design of ASVs that involved end user interactions. Visualizations like the plot in Figure 9 have a limited XAI audience because their meaning is decipherable only to experts of ML-based techniques for the motion control of ASVs. In [42], the same scientists who generated this developer-centric plot recast the information to a user-centric interface (Figure 10), visually conveying which features in the model are weighted most heavily, as well as an avatar showing corresponding thruster power and direction. The display, intended to explain an automated docking process to a navigator, also depicts green lines representing target destination and red lines representing the distance to the nearest object.
This example shows how visualization of AI models can enhance trust in AI decision making among end users in a real-world context. This example, while starting with specialized visualizations intended for developers to improve model interpretability, evolved towards broader user-centered interpretability when confronted with user interaction. This evolution supported the case that human-centered XAI processes were present during the design of ASVs that involved end user interactions. Figure 10. Visualization showing the same information presented in Figure 9 but int operator (adapted from [42], image used with permission).

Data Visualization for Shore Control Center Operators
Central for the milliAmpere2, with its goal of operational functionali monitoring and control supervision. These aims were at the core of the Sho [43,44] (Figure 11), a research platform developed in parallel with the mill ject. The Shore Control Lab is a Shore Control Centre equipped for research designed with active engagement from various stakeholders and expert gro to be able to support a fleet of ASVs from one location, it will start with the supporting operation of just the one vessel. Having freed up the attention th mally have been allocated to mundane control tasks on the ferry, the cap hend problematic out-of-the-ordinary events is enhanced at the Shore C (e.g., handling emergencies, rescues, and special weather events). Operato first line of support to the milliAmpere2 via an onboard video and audio lin a call button. Of central importance to Shore Control Center work is the 'control threshold' that demarcates where AI control ends and where hum gins (for takeovers), and vice versa (for handover). Understanding whe threshold lies relative to the AI system limitations defines the operators' pr cause it defines when they need to intervene. This control threshold also de textual and local factors (it may, for example, be lowered in adverse weath more frequent interventions, and raised after AI system improvement fewer).
This example points to an underlying human-centered XAI process w pretability of the AI system is necessitated by virtue of the operators' bein of ASV operations. Because there is a level of collaboration between the A Shore Control Centre operator, the success of this teamwork hinges on the of the system's constraints and of relevant safety-critical data. Human-ce this aim, continues to play a role in designing the Shore Control Centre to e decision-making in contextually nuanced, out-of-the-ordinary events that Figure 10. Visualization showing the same information presented in Figure 9 but intended for a ship operator (adapted from [42], image used with permission).

Data Visualization for Shore Control Center Operators
Central for the milliAmpere2, with its goal of operational functionality, was remote monitoring and control supervision. These aims were at the core of the Shore Control Lab [43,44] (Figure 11), a research platform developed in parallel with the milliAmpere2 project. The Shore Control Lab is a Shore Control Centre equipped for research activities and designed with active engagement from various stakeholders and expert groups. Designed to be able to support a fleet of ASVs from one location, it will start with the milliAmpere2, supporting operation of just the one vessel. Having freed up the attention that would normally have been allocated to mundane control tasks on the ferry, the capacity to apprehend problematic out-of-the-ordinary events is enhanced at the Shore Control Center (e.g., handling emergencies, rescues, and special weather events). Operators are also the first line of support to the milliAmpere2 via an onboard video and audio link activated by a call button. Of central importance to Shore Control Center work is the presence of a 'control threshold' that demarcates where AI control ends and where human control begins (for takeovers), and vice versa (for handover). Understanding where the control threshold lies relative to the AI system limitations defines the operators' primary role because it defines when they need to intervene. This control threshold also depends on contextual and local factors (it may, for example, be lowered in adverse weather, resulting in more frequent interventions, and raised after AI system improvements, resulting in fewer).
This example points to an underlying human-centered XAI process whereby interpretability of the AI system is necessitated by virtue of the operators' being 'in the loop' of ASV operations. Because there is a level of collaboration between the ASV system and Shore Control Centre operator, the success of this teamwork hinges on the explainability of the system's constraints and of relevant safety-critical data. Human-centered XAI, in this aim, continues to play a role in designing the Shore Control Centre to enhance human decision-making in contextually nuanced, out-of-the-ordinary events that challenge the constraints of the ASV system. Figure 11. The NTNU Shore Control Lab is designed to enhance human perception and decision-making for ASV opera tion in out-of-the-ordinary situation handling (see [43,44] for details).

Visual Signals to Predict Future States
Visual signals can be useful in explaining the behavior of ASVs and enhancing coordination and usability. Figure 12 depicts a 1:10 scaled model of milliAm mounted with light-emitting diodes (LEDs) programmed to light up in different c intensities, and light patterns. Early-stage designs of the milliAmpere2 involved t how light signals explained the ASV's future state to observers. Preliminary experim results showed that light signals improved understanding and predictability of a d range of states, including docking, crossing, vessel detection, autonomous or m mode, direction changes, speed reductions, emergency stops, and distress signals.
In the broader aim of enabling understandability, interpretability, and trust a ASV end users, the use of visual signals showed how the prediction of future states be represented as 'mental models' of behavior. Aside from enhancing general usa this is especially important for other vessels seeking to coordinate traffic in confine terways.

Path Planning
Humans make predictions about certain entities using simulated enactment type of 'mental model' representing that entity's behavior. To explore how this rela processes in human-centered XAI, we presented an illustrated example of a co avoidance maneuver in Figure 13. In this example, different motion paths of a target represent different behaviors for an observing vessel, affecting the observer's predi of the target vessel's planned path. In Figure 13, the milliAmpere2 (target) is the sta vessel, and the R/V Gunnerus (observer) is the give-way vessel, and according to R in the COLREGS must milliAmpere2 give 'ample time' to initiate a collision avoi maneuver. It is assumed that the crew onboard Gunnerus is carefully watching f Figure 11. The NTNU Shore Control Lab is designed to enhance human perception and decisionmaking for ASV operation in out-of-the-ordinary situation handling (see [43,44] for details).

Visual Signals to Predict Future States
Visual signals can be useful in explaining the behavior of ASVs and enhancing traffic coordination and usability. Figure 12 depicts a 1:10 scaled model of milliAmpere2 mounted with light-emitting diodes (LEDs) programmed to light up in different colors, intensities, and light patterns. Early-stage designs of the milliAmpere2 involved testing how light signals explained the ASV's future state to observers. Preliminary experimental results showed that light signals improved understanding and predictability of a diverse range of states, including docking, crossing, vessel detection, autonomous or manual mode, direction changes, speed reductions, emergency stops, and distress signals.
In the broader aim of enabling understandability, interpretability, and trust among ASV end users, the use of visual signals showed how the prediction of future states could be represented as 'mental models' of behavior. Aside from enhancing general usability, this is especially important for other vessels seeking to coordinate traffic in confined waterways. J. Mar. Sci. Eng. 2021, 9, 1227 1 Figure 11. The NTNU Shore Control Lab is designed to enhance human perception and decision-making for ASV opera tion in out-of-the-ordinary situation handling (see [43,44] for details).

Visual Signals to Predict Future States
Visual signals can be useful in explaining the behavior of ASVs and enhancing t coordination and usability. Figure 12 depicts a 1:10 scaled model of milliAm mounted with light-emitting diodes (LEDs) programmed to light up in different c intensities, and light patterns. Early-stage designs of the milliAmpere2 involved te how light signals explained the ASV's future state to observers. Preliminary experim results showed that light signals improved understanding and predictability of a di range of states, including docking, crossing, vessel detection, autonomous or m mode, direction changes, speed reductions, emergency stops, and distress signals.
In the broader aim of enabling understandability, interpretability, and trust am ASV end users, the use of visual signals showed how the prediction of future states c be represented as 'mental models' of behavior. Aside from enhancing general usab this is especially important for other vessels seeking to coordinate traffic in confined terways.

Path Planning
Humans make predictions about certain entities using simulated enactments type of 'mental model' representing that entity's behavior. To explore how this rela processes in human-centered XAI, we presented an illustrated example of a col avoidance maneuver in Figure 13. In this example, different motion paths of a target v represent different behaviors for an observing vessel, affecting the observer's predic of the target vessel's planned path. In Figure 13, the milliAmpere2 (target) is the stan vessel, and the R/V Gunnerus (observer) is the give-way vessel, and according to R in the COLREGS must milliAmpere2 give 'ample time' to initiate a collision avoid maneuver. It is assumed that the crew onboard Gunnerus is carefully watching fo

Path Planning
Humans make predictions about certain entities using simulated enactments in a type of 'mental model' representing that entity's behavior. To explore how this relates to processes in human-centered XAI, we presented an illustrated example of a collision avoidance maneuver in Figure 13. In this example, different motion paths of a target vessel represent different behaviors for an observing vessel, affecting the observer's predictions of the target vessel's planned path. In Figure 13, the milliAmpere2 (target) is the stand-on vessel, and the R/V Gunnerus (observer) is the give-way vessel, and according to Rule 8 in the COLREGS must milliAmpere2 give 'ample time' to initiate a collision avoidance maneuver. It is assumed that the crew onboard Gunnerus is carefully watching for the signs of such a maneuver. The observing vessel is, in other words, mentally simulating the future path of the milliAmpere2 based on interpretation of its current motion characteristics. In Figure 13 one could imagine that different parameters programmed in the onboard milliAmpere2 navigation system would result in different turning radii around the standon vessel. Furthermore, one parameter might appropriately explain that 'ample time' is being accounted for in collision avoidance initiation, whereas another may not. This is an example of human-centered XAI because it links qualitative properties (in this case, 'ample time' as expressed in COLREGS) to computational decision-making (control parameters to execute a collision avoidance maneuver) that characterize the AVS's behavior. In doing so, the AI system embodies interpretable motion behavior whose actions are simulated in the minds of observers as a type of 'mental model.' Making motion interpretable in this way is important in shared waterways where normal traffic is expected to interact with ASV traffic, especially for collision avoidance in constrained waterways.
J. Mar. Sci. Eng. 2021, 9,1227 12 of 23 signs of such a maneuver. The observing vessel is, in other words, mentally simulating the future path of the milliAmpere2 based on interpretation of its current motion characteristics. In Figure 13 one could imagine that different parameters programmed in the onboard milliAmpere2 navigation system would result in different turning radii around the stand-on vessel. Furthermore, one parameter might appropriately explain that 'ample time' is being accounted for in collision avoidance initiation, whereas another may not. This is an example of human-centered XAI because it links qualitative properties (in this case, 'ample time' as expressed in COLREGS) to computational decision-making (control parameters to execute a collision avoidance maneuver) that characterize the AVS's behavior. In doing so, the AI system embodies interpretable motion behavior whose actions are simulated in the minds of observers as a type of 'mental model.' Making motion interpretable in this way is important in shared waterways where normal traffic is expected to interact with ASV traffic, especially for collision avoidance in constrained waterways.

Trustworthiness
Apprehending subtle cues in how the milliAmpere2 moves, passengers will form subjective notions of trustworthiness of the automated system's decision-making capacity. Here we assume that the passengers are on board the ASV and therefore feel its motions in a kinesthetic sense. To illustrate this, Figure 14 compares two motion paths of the ASV Telemetron developed in the AutoSea Project [45]. While the first motion path accomplishes the task of collision avoidance, it is characterized as jagged and disjointed (Figure 14, left) when compared to that of a human operator or a better algorithm ( Figure  14, right). Were the milliAmpere2 to navigate in this way, it is likely that passengers on board would conflate its motion with unpredictability, thus undermining trust despite what was objectively a successful collision avoidance.
This example shows that human-centered XAI for ASVs involves interpretating system trustworthiness through its motion. This is a type of 'mental model' that represents the inner workings of the navigation system as a kinesthetic experience. The process of aligning ASV motion to kinesthetic experience suggests a decidedly human-centered approach to XAI in the broader aim of aligning ASV actions with human-centered values like predictability, comfort, and trustworthiness.

Trustworthiness
Apprehending subtle cues in how the milliAmpere2 moves, passengers will form subjective notions of trustworthiness of the automated system's decision-making capacity.
Here we assume that the passengers are on board the ASV and therefore feel its motions in a kinesthetic sense. To illustrate this, Figure 14 compares two motion paths of the ASV Telemetron developed in the AutoSea Project [45]. While the first motion path accomplishes the task of collision avoidance, it is characterized as jagged and disjointed (Figure 14, left) when compared to that of a human operator or a better algorithm (Figure 14, right). Were the milliAmpere2 to navigate in this way, it is likely that passengers on board would conflate its motion with unpredictability, thus undermining trust despite what was objectively a successful collision avoidance.
This example shows that human-centered XAI for ASVs involves interpretating system trustworthiness through its motion. This is a type of 'mental model' that represents the inner workings of the navigation system as a kinesthetic experience. The process of aligning ASV motion to kinesthetic experience suggests a decidedly human-centered approach to XAI in the broader aim of aligning ASV actions with human-centered values like predictability, comfort, and trustworthiness. J. Mar. Sci. Eng. 2021, 9,1227 12 of 23 signs of such a maneuver. The observing vessel is, in other words, mentally simulating the future path of the milliAmpere2 based on interpretation of its current motion characteristics. In Figure 13 one could imagine that different parameters programmed in the onboard milliAmpere2 navigation system would result in different turning radii around the stand-on vessel. Furthermore, one parameter might appropriately explain that 'ample time' is being accounted for in collision avoidance initiation, whereas another may not. This is an example of human-centered XAI because it links qualitative properties (in this case, 'ample time' as expressed in COLREGS) to computational decision-making (control parameters to execute a collision avoidance maneuver) that characterize the AVS's behavior. In doing so, the AI system embodies interpretable motion behavior whose actions are simulated in the minds of observers as a type of 'mental model.' Making motion interpretable in this way is important in shared waterways where normal traffic is expected to interact with ASV traffic, especially for collision avoidance in constrained waterways.

Trustworthiness
Apprehending subtle cues in how the milliAmpere2 moves, passengers will form subjective notions of trustworthiness of the automated system's decision-making capacity. Here we assume that the passengers are on board the ASV and therefore feel its motions in a kinesthetic sense. To illustrate this, Figure 14 compares two motion paths of the ASV Telemetron developed in the AutoSea Project [45]. While the first motion path accomplishes the task of collision avoidance, it is characterized as jagged and disjointed ( Figure 14, left) when compared to that of a human operator or a better algorithm ( Figure  14, right). Were the milliAmpere2 to navigate in this way, it is likely that passengers on board would conflate its motion with unpredictability, thus undermining trust despite what was objectively a successful collision avoidance.
This example shows that human-centered XAI for ASVs involves interpretating system trustworthiness through its motion. This is a type of 'mental model' that represents the inner workings of the navigation system as a kinesthetic experience. The process of aligning ASV motion to kinesthetic experience suggests a decidedly human-centered approach to XAI in the broader aim of aligning ASV actions with human-centered values like predictability, comfort, and trustworthiness.

Human-AI Collaboration
Shared control between the ASV and remote operator can be simulated in a virtual environment. In Section 3.2.4, we introduced the Shore Control Centre as a safety-critical element in the overall ASV system because it ensured that humans were 'in the loop' and able to take over control when needed. To design the collaborative system and train operators in its use, waiting for such an intervention situation to arise was not practical. Instead, virtual simulations at the Shore Control Lab were used to reenact them ( Figure 15). The virtual simulator is custom-built on top of the Gemini open-source platform [46] and, in effect, lets the researcher carry out thought experiments about ASVs in an immersive environment. For example, by having operators run simulated operations of the milliAmpere2 (as depicted in Figure 15), researchers can investigate what levels of operator involvement in control induce sufficient readiness for safety-critical interventions.
This example showed that virtual simulation can be used to design for human-AI collaboration for ASVs. Virtual simulation is, in this sense, a type of immersive thought experiment whose aim is to represent real-world operations where collaborative control is required to ensure safety. This type of simulation points to a human-centered XAI approach aiming to align the ASV system with real-world operational conditions and expectations of safety. Figure 14. Collision avoidance of the ASV Telemetron: (Left) The vessel avoids collision, but the path is jagged, disjointed and unpredictable. This undermines the trust of the crew, operators, passengers, and other vessels in the environment (Right) Telemetron avoids collision with a smooth and predictable path. Crew, operators, passengers, and other vessel trust the autonomous vessel.

Human-AI Collaboration
Shared control between the ASV and remote operator can be simulated in a v environment. In Section 3.2.4, we introduced the Shore Control Centre as a safety-cr element in the overall ASV system because it ensured that humans were 'in the loop able to take over control when needed. To design the collaborative system and trai erators in its use, waiting for such an intervention situation to arise was not practica stead, virtual simulations at the Shore Control Lab were used to reenact them (Figur The virtual simulator is custom-built on top of the Gemini open-source platform [46 in effect, lets the researcher carry out thought experiments about ASVs in an imme environment. For example, by having operators run simulated operations of the mill pere2 (as depicted in Figure 15), researchers can investigate what levels of operat volvement in control induce sufficient readiness for safety-critical interventions.
This example showed that virtual simulation can be used to design for huma collaboration for ASVs. Virtual simulation is, in this sense, a type of immersive tho experiment whose aim is to represent real-world operations where collaborative co is required to ensure safety. This type of simulation points to a human-centered XA proach aiming to align the ASV system with real-world operational conditions and e tations of safety.

Summary
In Table 1 we summarize the Findings, tracing the various processes of human tered XAI in the examples above.

Analogy
Represents ASV as a familiar concept to explain AI interaction (enhances usability)

Visualization
Represents ASV sensory perception and numerical models visually to explain AI decisionmaking (enhances trust) Represents ASV functionality and affordances through user displays, design, form, and aesthetics to explain AI interaction (enhances usability and trust) Represents ASV system constraints visually to explain human-AI collaboration (enhances usability and safety) Represents ASV be ior through visual nals to explain AI d sion-making (enha safety) Figure 15. The virtual simulator based on Gemini open-source platform [46] allowed for immersive thought experimentation of operational design of milliAmpere2.

Summary
In Table 1 we summarize the Findings, tracing the various processes of humancentered XAI in the examples above.

Developers Passengers Operators
Other Vessels

Analogy
Represents ASV as a familiar concept to explain AI interaction (enhances usability)

Visualization
Represents ASV sensory perception and numerical models visually to explain AI decision-making (enhances trust) Represents ASV functionality and affordances through user displays, design, form, and aesthetics to explain AI interaction (enhances usability and trust) Represents ASV system constraints visually to explain human-AI collaboration (enhances usability and safety) Represents ASV behavior through visual signals to explain AI decision-making (enhances safety)

Mental simulation
Represents ASV behavior in terms of 'mental models' (often synthesized as numerical simulations) to explain AI decision-making (enhances trust) Represents ASV behavior using 'mental model' of kinesthetic experience to explain AI decision-making (enhances trust) Represents ASV behavior in terms of 'mental models' (often synthesized as immersive virtual simulations) to explain human-AI collaboration (enhances safety) Represents ASV behavior in terms of 'mental models' of motion characteristics to explain AI decision-making (enhances safety)

Discussion
In this section we expand upon the findings, examining how the concept of humancentered XAI can be applied to address end user trust in ASVs and contribute to the broader multi-disciplinary discussions around XAI. To help frame this discussion, we start with a definition of human-centered XAI as it applies to ASVs (see Box 1). We also present a table listing human-centered XAI processes compared to corresponding technology-centered XAI processes (Table 2). Box 1. Human-centered Explainable AI defined.
Human-centered Explainable AI is the process of making AI technology understandable, interpretable, explainable, and trustworthy for end user interactions. For ASVs, this involves representing capabilities and constraints of the AI system in line with developers', primary users', and secondary users' interaction needs. Table 2. Technology-centered XAI compared to human-centered XAI.

Mission is to ensure and improve model accuracy and efficiency
Mission is to establish and maintain an interaction relationship

Provides data-driven visualizations of models based in mathematics
Uses mental models based on analogy-making to explain technology use 3.

Frames increases in machine autonomy as subsequent reductions in human autonomy
Frames increases in machine autonomy as independent of human autonomy 4.

Considers the 'black box' interpretability problem as a barrier to AI development
Accepts that user does not need to know inner working of AI to use it successfully

5.
Considers humans as a source of error in overall system safety Considers humans as a source of resilience in overall system safety 6.
Often assigns AI human-like qualities, leading to over-selling and not meeting user expectations Explains limitations to users and actively manages expectations 7.
Seeks to improve the performance of human task by prioritizing computation Seeks to enhance the performance of human tasks by prioritizing collaboration

Explainability Needs for Different End User Interactions
One of the features of human-centered XAI is its ability to orient the explaining to the specific needs of the end user. To illustrate this point, consider that AI developers work primarily in a simulated environment without any direct risks for crew or equipment. They also have plenty of time and several tools at their disposal with which to present and analyze data visualizations of the AI model (e.g., Figures 8 and 9). If the algorithm fails, they will not intervene but use it to learn and to improve the model. Operators, on the other hand, either onboard the vessel or in a Shore Control Center, are faced with direct risks and responsibility for crew, vessel, and equipment. They do not have time to analyze complex data visualizations but need simpler visualizations mapped to the ship's layout to be able to quickly decide whether to intervene ( Figure 10).
In Section 2, three end user groups were identified for investigation in this research: 1. Developers (researchers, engineers); 2.
In Table 3 we characterize the various XAI needs of these different groups. A similar effort was presented in [42], where the authors analyzed the differences between developers and vessel operators in terms of characteristics, the context of use, and XAI needs. Here, we expand the analysis to a broader range of end users, such as passengers and other vessels. Note that in Table 3 we refer to Automatic Information Service (AIS), which is a globally standardized information-sharing platform for marine traffic, broadcasting information like ship name, flag, location, and speed (https://www.marinetraffic.com/, accessed on 22 October 2021). Vessel Traffic Services (VTS) refer to land-based coordination centers for aiding mariners in safe and efficient use of navigable waterways [47].

XAI to Establish Interaction Relationships and Build User Trust
Interacting with an ASV involves interpretability, understandability, explainability, and trust-phenomena that are also at the core of the XAI field. While developers focus on explaining AI techniques using mathematics and data visualization, a generalized approach based on leveraging more universal interaction elements, such as analogy-making, design, aesthetics, form, and motion characteristics, was used to reach a wider audience of end users in the case studies we examined. These elements were particularly important in establishing an interactive relationship with new end users during their first encounter with an ASV.
Explaining AI can be done by comparing it to familiar, 'human-friendly,' concepts. Recently, the use of 'human-friendly' concepts has seen increasing use among developers testing the interpretability of deep learning models [48]. Similarly, we observed that explaining ASV functionality to potential passengers played a central role in trust (Section 3.1), suggesting that explaining using familiar concepts plays an important role in human-centered XAI. For the milliAmpere2, for example, we observed the use of an information post containing a useful analogy that helped relate the abstract concept of ASV technology into something more concrete: an elevator (we return to this theme in Section 4.3 in the context of using metaphors). Using familiar concepts may also be useful for explaining safety. Typically, in the risk sciences, safety measures are represented using probabilities, expressed, for instance, as a percent likelihood for false positives and false negatives. However, as shown by [49], interpretation of what such error rates mean in practice, especially when related in terms of base rate information (such as total prevalence of all errors) runs counter to intuition. Explaining the ASV's safety to users may be more effectively accomplished by comparing it to an existing baseline; for example, whether it can be shown to be significantly safer than a corresponding manned surface vessel.
Explaining AI can also be done by rending activities visible. In computer vision, saliency-based XAI techniques that visualize the pixels that a network is fixated upon for its predictions have made important strides towards model interpretability [50]. In [51], the authors showed how similar visualization techniques can lead to insights into how image classification models function in intermediate layers, leading to insights about model improvements. A similar discussion about 'where to look' arises for the shore control center operators whose tasks are to apprehend and respond to out-of-the-ordinary events (Section 3.2.4). In the context of XAI for collaborative systems, this raises the need for explaining actions taken during coordinated work. In [52], the authors demonstrated the importance of 'rendering activities visible' in control room work, observing, for example, the central role of out-loud explanations, verbal exclamations, pointing, and other cues and gestures in coordinated action among operators handling a situation. XAI for collaborative AI-human work, such as in the Shore Control Centre, should, in the light of such 'mutual monitoring' practices, avoid the pitfall of rendering activities in AI systems invisible.
The 'rendering actions visible' corollary applies in equal measure to secondary users. These users, as defined by [53], are those who are not interacting with a system directly and yet are affected by the system's and primary users' actions. In the context of autonomous ships, these users are indirectly influenced by the decisions of the AI system and include marine traffic, recreational boats, and bystanders. Motion planning characteristics (Section 3.3.1) and visual signals (Section 3.3.3) were shown to be especially useful for conveying an understanding of ASV state and intention.

XAI by Encoding Abstract Concepts as Mental Models to Support User Interaction
Humans have the remarkable ability of understanding concepts through abstraction, even when faced with something entirely new and with very little information. In [54], the authors studied the use of metaphors and framed them as a central mechanism by which humans make sense of the world. Metaphors go beyond just natural language, guiding the most fundamental interactions in our daily lives, usually without our even being aware. For example, the authors describe 'orientational metaphors' that lend abstract concepts of spatial orientation relative to what we know best: our bodies. Consciousness, for instance, is up; unconsciousness is down: 'Get up. Wake up. He rises early in the morning. He fell asleep. He's under hypnosis. He sank into a coma' (p. 15, authors' original emphasis). The human-centered approach to XAI leverages humans' ability to understand abstract concepts via metaphors using 'mental models.' There are many definitions for mental models, and we presented a working definition in Section 2.3. A 'mental model' is, in this context, a type of abstraction that encodes in the mind of an end user an explanation of how to use a technology. For the milliAmpere2, we observed that the 'elevator mental model' was especially useful (Section 3.1). In this application, the simplicity of the mental model belied its power in establishing and maintaining a trusting interaction relationship. Simply providing a mental model, though, was not enough to maintain a trusting interaction relationship. Interface inputs were thus designed to affirm the mental model, and outputs were mapped to actions that reinforced it.

XAI That Frames Human Autonomy Independent of Machine Autonomy
Adopting Level of Automation (LOA) taxonomies, such as of the Norwegian Forum for Autonomous Shipping (NFAS) framework [55], was useful for explaining the extent to which the milliAmpere2 system was automated. For example, the LOA called 'Constrained Autonomy' fits the description of the milliAmpere2: 'The ship [or ASV] can operate fully automatic in most situations and has a predefined selection of options for solving commonly encountered problems... It will call on human operators to intervene if the problems cannot be solved within these constraints. The SCC or bridge personnel continuously supervises the operations and will take immediate control when requested to by the system. Otherwise, the system will be expected to operate safely by itself.' [55] (pp. [11][12] However, [16] showed that conventional taxonomies for LOA imply that introducing machine autonomy comes at the cost of human control: a 'one-dimensional' zero-sum relationship. In a comprehensive review, [56] compared twelve taxonomies for automation, all of which presumed such a one-dimensional LOA, composed of an incremental scale from full human control to full autonomous control. Since their review, several new taxonomies have been proposed specifically for ASVs that adopt a similar one-dimensional LOA format [7,55,57]. Scholars have warned about the loss of human autonomy such taxonomies might unintentionally introduce, proposing instead LOA frameworks that consider joint human and machine autonomy as two-dimensional 'stages of automation' [16]. As explained by [58], examples of highly reliable, automated systems already exist in cars, including Anti-lock Braking Systems (ABS), Traction Control Systems (TCS), and Electronic Stability Control (ESC). The success of these technologies lies in their ability to enhance, rather than supplant human decision-making when braking and maneuvering in safety-critical situations.
Inspired by the 'stages of automation' framework proposed by [16], which considers joint human and machine autonomy in AI systems, we mapped several prominent maritime surface vessels onto four quadrants of AI-human control ( Figure 16). The basis for positioning these vessels in Figure 16 is described in Table 4, which characterizes their respective levels of human and machine control. The results show a trend moving from the upper left quadrant (high human control; low machine control) toward the bottom right quadrant (high machine control; low human control). A necessary adjustment for achieving trust in ASVs is to shift toward the upper right quadrant (high machine control; high human control).

XAI That Accepts That Users Do Not Need to Know Inner Workings of AI to Use It Successfully
To illustrate the motivation behind this guiding principle, we can represent 'black box' AI techniques with a technology we are familiar with: a dishwasher. The dishwasher user has control over several inputs in the form of loading dishes and pressing buttons to control the setting and start the machine. When the machine is activated, it runs automatically and, after a time, produces clean dishes. While it is possible to implement a transparent door on the dishwasher so that users could see what is happening, it is unlikely that this level of transparency will contribute to an enhanced user experience. We are, in this case, simply projecting the value of transparency onto the user, possibly at the risk of a less efficient and more expensive product. This may be what [22] observed when finding that an ML algorithm's end user trust levels were not affected by model transparency. Similarly, we observed in the milliAmpere2 user research that transparency of analogous 'black box' AI techniques may be less of an influencing factor than design, aesthetics, form, and motion characteristics for building trust (Section 3.2.2).
Product developers have known for a long time that predicting user experiences is notoriously difficult [63]. Part of the reason is that user experience is entangled with higher-order values and ambitions of which the user themselves may be only dimly aware. In the dishwasher example, these ambitions could be related to the higher-order value of preserving harmony in the household rather than the more obvious benefit of skipping kitchen chores. In the milliAmpere2 case, high-order ambitions are more likely linked to discovery and learning (Section 3.2.2) than to simply crossing a canal. Since the early days of AI development, scholars have discussed the dissonance that can result from automated machine decision-making in the context of societal values. More than half a century ago, Norbert Wiener wrote that 'should we create a mechanical agency with whose operation we cannot efficiently interfere once we have started it . . . then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it' [64] (p. 1358). The issue remains elusive today and has been recast into a discussion about 'value alignment in AI' [23]. A new field of AI ethics has also emerged, where scholars cast questions about morals in decision-making, previously considered theoretical, into a new practical light [65,66]. A human-centered approach to XAI does not assume that making AI decisions transparent will necessarily benefit an interaction relationship. Instead, it eschews explanations that risk projecting the values of the developers onto users, steering the user instead towards reflexive and subjective experiences.
J. Mar. Sci. Eng. 2021, 9,1227 18 of 23 Figure 16. Two-dimensional 'stages of automation' plot adapted from [16] showing human-AI system control for various vessels. These are described further in Table 4. Table 4. Stages of automation described for the example shown in Figure 16. For operators in the Shore Control Centre who had specific roles and safety responsibilities, this approach led to designing interactions that enhanced experts' ability to apprehend out-of-the-ordinary events the AI system is unable to handle (Section 3.2.4). As [67] observed from experts' prediction making when compared to simple regression models, the experts, despite not producing as accurate predictions as the models, demonstrated an ability to 'know what to look for' in making those predictions [67] (p. 573). In other words, experience has taught experts to identify which factors are linked to future events if not able to accurately synthesize that information into an accurate prediction. Well-aligned collaboration of AI-human systems, such as Shore Control Centers, should, in this light, combine the synthesizing ability of computation with experts' semantic knowledge of which factors are meaningful to decision-making. For such collaborative systems, understanding the respective roles of the AI system and its human expert backup underpinned this emerging 'hybrid' format of collaborative work. More work is needed in this area, and the field of XAI may contribute to techniques and strategies for rendering collaborative work more visible to operators.

Conclusions
As more ASVs are deployed, and AI-based navigation systems become more widespread, trust is emerging as a barrier to acceptance. This is especially the case as we move towards passenger ASVs, where unintended consequences of model errors can result in real harm to people. The tenants of the field of Explainable AI (XAI)-with its emphasis on interpretability, understandability, explainability, and trust-represent common values for multiple disciplines interested in addressing the challenge of building trust in ASVs. Traditionally, XAI has been composed of computer scientists with the aim of improving model efficiency and accuracy by better interpretation of opaque 'black box' Machine Learning (ML) models. However, as downstream consequences of AI deployment in real-world maritime applications become clearer, more disciplines are contributing to the field. This includes human-computer interaction and interaction design, two fields positioned at the crossroads of humans and technology. In this article, we constructed the concept of 'human-centered XAI' to address the distinct needs of end user interactions for ASVs. We did this by drawing from examples of ongoing research that pointed to processes of analogy, visualization, and mental simulations that, according to model-based reasoning [30], are signatures of formation for new concepts. Our investigation was guided by the research question: can a human-centered approach to XAI contribute towards building trust among real-world ASV users?
The findings showed that the main aims of human-centered XAI processes were to enhance (1) usability, (2) trust, and (3) safety. Usability was enhanced through the representation of AI functionality; trust was enhanced through the representation of AI decision-making, sensory perception, and behavior; and safety was enhanced through the representation of AI constraints and human-AI collaboration. All representations were enabled by interrelated processes of analogy making, visualization, and mental simulation that sought to align AI goals and actions with the values held by end users during interaction with ASVs.
Several important limitations exist. Firstly, given that the nature of concepts is to lack permanence, the processes that we define as 'human-centered XAI' may change over time, across cultures, and with shifting societal values. Secondly, all the examples presented were drawn from either our own research or from the research of our collaborators at our university. This entailed a high degree of reflexivity in the research process that risked entangling the processes observed with the formation of the concept we constructed.
For the future direction of research, we hope to see lively discussion about how the concept can be taken further in practice-based research. Could a 'Handbook of humancentered XAI' or similar synthesize the processes we underlined for developers, engineers, designers, and other practitioners? Furthermore, can we expand the XAI audience to include organizational stakeholders and even regulatory agencies?
This work has demonstrated that as ASVs scale for widespread deployment, design practices are orienting towards end user interaction design. However, this is occurring without the guidance of a fully formulated conceptualization of what this design practice entails. The main contribution of this work is the formation and definition of human-centered XAI for this purpose, helping to advance ASV design toward wide-scale deployment.

Acknowledgments:
The authors acknowledge the support of the Department of Design, the Department of Electronic Systems, and the Department of Engineering Cybernetics at NTNU. The authors gratefully acknowledge the support of several former Master's students from the Department of Design who contributed to this work: Petter Mustvedt, Henrik Ellingsen, and Peter Glesaaen. We would also like to thank Erik Scott Hognestad, Maia Sløveren Andressen, and Jonas Klev Selvikvåg (Department of Design) for their contributions to the interaction design visualizations and to doctoral candidates Øystein Helgesen, Emil Thyri, and Vilde Gjaerum for their contributions to XAI visualizations and to Thomas Kaland, Kjetil Vasstein, Mikael Hanssen, and Felix Petermann for their contributions to visualizations of the Shore Control Lab. Finally, we would like to thank the whole team behind milliAmpere and milliAmpere2 for supporting this work.