Evolution of Socially-Aware Robot Navigation

Abstract

In recent years, commercial and research interest in service robots working in everyday environments has grown. These devices are expected to move autonomously in crowded environments, maximizing not only movement efficiency and safety parameters, but also social acceptability. Extending traditional path planning modules with socially aware criteria, while maintaining fast algorithms capable of reacting to human behavior without causing discomfort, can be a complex challenge. Solving this challenge has involved the development of proactive systems that take into account cooperation (and not only interaction) with the people around them, the determined incorporation of approaches based on Deep Learning, or the recent fusion with skills coming from the field of human–robot interaction (speech, touch). This review analyzes approaches to socially aware navigation and classifies them according to the strategies followed by the robot to manage interaction (or cooperation) with humans.

1. Introduction

Society 5.0 represents a new paradigm in which people and artificial beings cooperate in routines, environments, and the interactions of everyday life [1]. This cooperation is intended to be natural and intuitive, and the new artificial actors are expected to behave appropriately. Service robots are one of the technologies with the greatest number of potential applications in this new social paradigm [2]. They are also one of the most profoundly affected by the new technical and contextual complexities arising from these new requirements [3]. Service robots, being inevitably social when used in everyday life scenarios, face the most challenging technical, ethical, social, and legislative demands. In particular, Socially Aware Robotics (SAWR) is an emerging area of research that seeks to understand how cognitive robots can be aware of their social context and use this capability to behave as more accessible, accepted, and useful devices, being able to establish more appropriate and effective interactions to assist humans [4]. A robot aiming to exploit socially enhanced autonomous capabilities needs to perceive its environment and reach a certain level of understanding of its context. However, to be truly socially aware, a robot must not only react intentionally to perceived changes, but it must also be able to predict or learn what the behavior of the humans surrounding it will be, anticipating consequences and selecting the best possible and most comprehensible action, while respecting social conventions [5,6].

One of the essential capabilities of most robotic solutions, more deeply affected by these new social requirements, is navigation. Service robots working in daily life environments cannot just search for the shortest collision-free path. They should also solve a multi-variable optimization problem that considers, for example, human comfort and social rules. As a result, traditional navigation approaches are no longer adequate due to their limited flexibility, and new proposals arise. The growing importance of the topic has given rise to several review articles analyzing these proposals in different ways. Concepts regarding the human factor were highlighted in the survey papers on proxemics ([7,8]), and semantic and social mapping ([9]). The review paper by Gao and Huang [10] focused on the evaluation methods, scenarios, datasets, and metrics commonly used in previous socially-aware navigation research. Zhu and Zhang [11] reviewed Deep Reinforcement Learning (DRL) methods and DRL-based navigation frameworks. They differentiated between several typical application scenarios: local obstacle avoidance, indoor navigation, multi-robot navigation, and social navigation. Chik et al. [12] focused on robot navigation as a hierarchical task, involving a collection of sub-problems that can range from the high-level decision to reactive avoidance of low-level obstacles. The review highlighted how the whole navigation stack should evolve to address the problem of dealing with dynamic human environments, including human detection, tracking, and predictive modeling, at a more local level, and considering social costs at a higher level.

The paper by Kruse et al. [13] discusses human-aware navigation. In this paper, the authors state that ‘the robotics and human–robot interaction (HRI) communities have not yet produced a holistic approach to human-aware navigation’ [13], even though they identified 77 citations between 1995 and 2012 closely related to this topic. They also defined two axes on which to classify these papers. First, they established four categories: comfort, naturalness, sociability, and others. In the second axis, the technologies on which the articles focus are considered. The categories in this second axis are pose selection, global planning, behavior selection, and local planning.

The present survey extends the previous work of Kruse et al. [13] in two dimensions. On the one hand, it updates the analysis considering new approaches and research conducted in the last ten years in the field of social navigation. On the other hand, this survey focuses on those methods in which the robot modifies its behavior in the presence of another mobile agent, or traces a path (e.g., avoiding disturbing a group of people talking to each other). Hence, it differs from the work of Kruse et al. [13] as the basis of our categorization will not be functionality, but the degree to which the method employed for reacting in the presence of humans is able to learn from previous observations or predict the near future. Prediction and Learning will be the terms that will guide the literature search, establishing subgroups in the more global group of social navigation. Moreover, this survey would like to draw attention to papers that focus on social comfort and explore how social navigation methods are evolving to equip robots with more human-like skills. As robots become increasingly similar to humans, understanding the complexities of social norms and adapting to them becomes more important. These papers shed light on how robots can be designed to better interact with humans in social situations.

The rest of the paper is organized as follows: Section 2 explains the literature review process. Results are presented in Section 3. Discussion is provided in Section 4. Finally, conclusions and future work are drawn in Section 5.

2. Methodology

This section describes the criteria used both to select the set of articles considered for this review article (Section 2.1) and to organize them into different groups (Section 2.2).

2.1. Article Selection Criteria

We carefully curated a collection of papers on the topics of social navigation, with a special interest in two keywords, prediction and learning. Our selection process involved a thorough review of the literature, drawing from the works of Kruse et al. [13], Chik et al. [12], Gao and Huang [10] and Zhu and Zhang [11], and extended with recent citations. We selected the most relevant papers that explored the areas of social navigation, comfort, prediction, and learning. These papers were further narrowed down by evaluating the quality and relevance of the references cited in each article. Finally, we prioritized the papers that were most frequently cited in previous works.

To gain insight into the number of papers published on our focus topics, we can refer to the graphs in Figure 1, Figure 2 and Figure 3. These graphics were generated using data from the Web of Science (https://clarivate.com/webofsciencegroup/solutions/web-of-science/, accessed on 13 February 2023). For the topic (TS = (social AND navigation)) AND TS = (robot), we obtain 723 results in the range from 1994 to 2022 (data from the current year, 2023, were not included as the year has not finished and these data could disturb the statistics). Adding TS = (learning), the number of citations reduces to 207 results (Figure 3). With TS = (prediction), the number is reduced to 80 results (Figure 2). From this large dataset, we have covered in this survey 100 papers. Figure 4 compares the papers covered by the survey by Kruse et al. [13] and the current review.

2.2. Article Classification Criteria

As mentioned above, the goal of socially aware navigation is not only to find a collision-free path from a starting point to a destination. This navigation process also requires carefully taking into account the movement of people around the robot, and the interactions processes between the robot and these people. When the number of people is small, few interactions are required to avoid possible collisions. People tend to move along rectilinear trajectories over long periods of time. However, when the density of people increases, problems arise [14]. In these scenarios, people have to frequently change their motion states (direction of travel, but also velocity or acceleration) in real time, to avoid collisions while trying to reach their destinations. Linear models are no longer correct for modeling interactions (human–robot, but also human–human). The perception of the robot’s movement by others becomes particularly relevant. Thus, in addition to being safe, it is important that the motion of the robot is legible, allowing people in the vicinity of the robot to easily understand its movement intention [15].

Analyzing the evolution of socially aware robot navigation addresses two major topics: (i) evaluating how robot social skills have made human interactions more comfortable and natural; (ii) evaluating how algorithms solve navigation in crowded scenarios. Following on previous work from several researchers on robot navigation in dense crowds, we will group in this survey works on robot navigation into three categories [14,16]: (1) reactive approaches; (2) proactive approaches; and (3) learning-based approaches. Moreover, we added a fourth category: (4) multi-behaviour navigation approaches. In reactive approaches, the robot reacts to other mobile agents through one-step look ahead strategies [14]. These approaches are typically very efficient (e.g., the social force model [17]). Proactive approaches predict the behavior of the human and then plan a suitable path. Predictions can be based on reasoning (assumptions of how agents behave in general), or learning (justified by observations of how agents behave) [13]. Strategies for prediction can deal with human motion [18] or intentions [14]. Learning-based approaches aim towards the robot learning the navigation policy, and adapting it to the target scenario. Deep reinforcement learning (DRL) has been extensively employed for solving this problem [11,19]. Finally, we added multi-behaviour navigation approaches as a separate subsection. Significantly, these methods address the question of whether it can be useful to consider interaction actions, such as touching, gesturing, or speaking, for the sake of allowing robots to navigate in dynamic, crowded environments. The importance of being able to coordinate different robotic functionalities (navigation, dialogue) to solve a navigation goal will grow in the coming years as service robots actually share the environment with people.

3. Socially Aware Navigation Approaches

This section develops the proposed taxonomy for classifying socially aware robotic navigation methods. A table illustrating the methods used as examples of each category in a concise manner is included at the end of each subsection.

3.1. Reactive Approaches

Robots that navigate in a dynamic or unknown environment can use path planners based on heuristic grid search, or a sampling-based approach, to obtain high-level routes. However, while these path planners can provide low-risk routes [20], they must always be combined with a reactive collision avoidance system. This reactive navigation layer must operate at a high frequency, to guarantee fast enough obstacle detection and avoidance. Moreover, the presence of different mobile agents in the environment (people or other autonomous vehicles) makes this reactive, local navigation scheme a basic element for maintaining a safe environment. These two factors (speed requirements and safety issues) set the onboard processor of the robot as the adequate device to be executing this reactive layer.

This section collects articles that deal with reactive, local navigation when there are other mobile agents in the environment. Being reactive approaches, they may not take into consideration either prediction or learning.

Table 1. Approaches considered in Section 3.1. The table covers the methods they use for solving the local navigation problem.

Reference	Methods
Fiorini and Schiller [17]	Velocity Obstacle (VO)
Shiller et al. [21]	Non-Linear Velocity Obstacle (NLVO)
Kluge and Prassler [22]	Recursive Probabilistic Velocity Obstacles
Fulgenzi et al. [23]	Probabilistic Velocity Obstacle (PVO)
	Dynamic occupancy grid provided by a general sensor system.
Palm and Driankov [28]	Velocity potential of an incompressible fluid
Babinec et al. [27]	Vector Field Histogram (VFH*)
Yao et al. [25]	BHPF. Calibrated using the Black-hole potential field deep Q-learning (BHDQN)
Zheng et al. [29]	Improved Social Force Model Based on Emotional Contagion and Evacuation
	Assistant
Reddy et al. [30]	Extend Social Force Model to incorporate the social cues by adding new social forces
	Extends the Geometric approach to incorporate the social cues by selecting the
	geometric gap as per the social reference
	Hybrid approach combining the social potential field and geometric method

assesses the approaches covered in this section. Most of these methods are based on the concept of Velocity Obstacle (VO), Artificial Potential Fields (APF), Vector Field Histogram (VFH), or Social Force Model (SFM). The VO concept was introduced by Fiorini and Shiller [17], and has been widely used to ensure safe navigation (e.g., Shiller et al. [21], Kluge and Prassler [22], and Fulgenzi et al. [23]). The VO concept of a moving obstacle $A_j$ to an agent $R_i$ is the set of all those velocities for $R_i$ that will result in a collision, at some moment in time, with the obstacle. Hence, in each planning cycle, the agent should choose a velocity different from any of the VOs induced by the moving obstacles (see Figure 5 (Right)). The APF is a virtual force field approach initially proposed by Khatib [24]. Briefly, the robot’s motion is controlled by a gravity force, generated by the target to reach, and repulsive forces, generated by the obstacles. Yao et al. [25] proposed improving the traditional APF with reinforcement learning in order to deal with dynamic scenarios. The VFH approach uses a two-dimensional Cartesian histogram grid as a world model, which is updated continuously with data sampled by onboard sensors [26]. Babinec et al. [27] modify the VFH* to deal with both static and moving obstacles. As a relevant disadvantage, it cannot handle non-linear motion.

The SFM simulates agent dynamics using interaction forces (Figure 5 (Left)). It allows expressing the collision-avoiding behavior through a function. The inputs for this function are the relative and absolute positions and velocities of the mobile agents [31]. The SFM is a powerful approach for crowd navigation, as discussed in the next section, which presents several types of modified SFMs proposed for robot navigation and human modeling. For instance, Zheng et al. [29] propose an SFM based on emotional contagion for evacuation assistant (ecaSFM). Reddy et al. [30] propose a novel hybrid approach extending the SFM with geometric constraints. This proposal maintains the proactive nature of the geometric approach and retains the reactive nature of the SFM.

Finally, it is interesting to include in this list the proposal by Palm and Driankov [28], as a representative example of reactive proposals not based on the previous approaches. Palm and Driankov [28] use the behavior of a fluid in the presence of obstacles as a simile. Then, they propose a local navigation method in which a set of streamlines is continuously updated. The method is applicable in an environment where there are other mobile elements such as robots or people.

3.2. Proactive Approaches

Purely reactive methods are not very common in social navigation. When navigating between people, it is interesting for the robot to consider these people as cooperative partners. Hence, the robot can interact with people to jointly avoid collisions [32]. This cooperative approach requires the robot (i) to predict the behavior (i.e., trajectories) of the people, and then (ii) proactively plan the path to follow. There is a multitude of methods to predict the behavior of other mobile agents that share the environment with the robot: from those that assume certain assumptions about how these agents behave in general (reasoning-based prediction), to those that are justified on the basis of observations of how these agents behave, usually in a specific environment or under specific conditions (learning-based techniques) [13]. Subsequent path planning methods include, among others, modified versions of the SFM [33] or the VO concept [34]. The next subsections describe the state-of-art of predictive models and path-planning algorithms for proactive approaches.

3.2.1. Predictive Models

Table 2 lists the approaches described in this section. Regarding predictive models, initial efforts to model the behavior of other mobile agents, and thus avoid routes that could lead to collisions, worked with highly deterministic models of motion. No weight was given to uncertainty, and they typically considered a multi-robot framework. In two proposals (van den Berg et al. [35], Snape et al. [34]) the concept of Velocity Obstacle was modified to deal with a scenario where several robots coexist, but where each of them works independently. Assuming that all robots employ the same technique for navigation, the aim of the Reciprocal Velocity Obstacle (RVO) [35] is to avoid the oscillatory behavior they may adopt when approaching or crossing another robot. The Hybrid Reciprocal Velocity Obstacle [34] extends the previous approach, setting priorities in the interaction between robots.

Many methods suggest modifications to the SFM [36,37,38]. The aim of all these approaches is to map out a collision-free, smooth route, but one that takes into account the presence of mobile agents—people or other autonomous vehicles. To achieve this, they include some interesting techniques. For instance, Zanlungo et al. [36] assume that deviations of the robot from the straight path leading to its goal are only due to avoidance of collisions with other moving agents, but they predict the time in the future at which the relative distances to each approaching agent will be at a minimum. Then, they assume that forces at that time depend on the distance between the mobile agents (circular specification, CS). Figure 6 schematizes the situation where two approaching agents interact. Forces will be circular symmetric forces as those used in CS, but based on this future situation, which is assumed to be the most interesting for the agent since it is when a collision can happen. Shiomi et al. [38] use a specific version of the SFM, called CP-SFM, to simulate human-like collision-avoidance behavior in robots for low-density environments, such as shopping mall corridors. Paths generated by the planner maintain a social distance from people and respect their personal space.

Engines, such as the Kalman filter, could be used to predict the position of mobile agents surrounding the robot by incorporating the uncertainty dimension. The problem is that, when there are several agents, the use of such engines leads to an uncertainty explosion [39], which can make it impossible for the robot to find a safe path to a target. To try to control this growth in uncertainty, different models of human motion have been proposed [40,41]. For instance, Du Toit and Burdick [42] propose directly limiting the predictive uncertainty of each individual agent. Joseph et al. [43] describe a more complex motion model: a Gaussian process mixture with a Dirichlet Process prior over mixture weights. This non-parametric model presents a high computational cost. The problem is solved in the RR-GP algorithm [44], a clustering-based trajectory prediction solution that uses Bayesian non-parametric reachability trees to improve the Gaussian prediction. Although for dealing with the position uncertainty, more sophisticated individual models are proposed, this line of research does not consider agent interaction in their models [39].

As previously mentioned, the uncertainty about the person’s future behavior makes it difficult for a robot to determine the appropriate actions for avoiding collisions. In addition, non-collision can only be a partial solution, and the robot can be asked to avoid situations where its behavior disturbs people, for example, due to excessive proximity. In a scenario where the robot shares the environment with several people, instead of formulating the behavior of each person, the robot must be able to predict how each person’s behavior will vary over time. Common approaches for predicting trajectories, such as the Kalman filter or Particle filters, exhibit different disadvantages for our scenario [45], being more adequate to those methods that consider in the model a goal-based policy. Thus, they assume that human behavior is captured in the previously observed trajectories, and the problem is then to determine to which group of trajectories the current one belongs. For instance, Bennewitz et al. [46] use models of human motion patterns, which can be learned using an Expectation-Maximization (EM) algorithm. With this information, the robot can predict where the person is or where s/he will be. Ziebart et al. [45] proposed using maximum entropy inverse optimal control to model the goal-directed trajectories of people. Maximum entropy is also used by Kuderer et al. [47]. Representative features of the people’s trajectories are analyzed to find the probability distributions that drive their navigation behaviors. Several prediction functions using the minimum curvature variation concept were described in Ferrer and Sanfeliu [48]. Built over the Curvature Length Predictor (CLP), the wCLP weights an average of previous predictions in a limited time window. Experiments showed that it can quickly recognize the new intention of motion when the destination changes or unexpected behaviors happen. Kabtoul et al. [49] proposed using a quantitative time-varying function to model the human–robot cooperation in an interaction scenario. Using this cooperation estimation, the human motion trajectory is predicted by a cooperation-based trajectory planning model. Figure 7 provides an overview of the approach proposed by Ikeda et al. [50]. The approach performs an offline analysis for estimating sub-goals (and a probabilistic transition model) in the environment (Figure 8). Then, at run-time, the approach allows estimating the future positions of people based on the sequence of previously traversed sub-goals and the current velocity.

Instead of predicting trajectories, Luber et al. [51] take advantage of the place-dependency of human behaviour for building a spatial affordance map. The problem of learning this spatial model of human behavior is posed as a parameter estimation problem of a non-homogeneous spatial Poisson process. The spatial affordance map is learned using Bayesian inference from observations of track creation, matching, and false alarm events, gained by introspection of a laser-based multi-hypothesis people tracker. The framework is described for people detection and tracking, but it was for instance integrated into the navigation scheme proposed by Ferrer et al. [52].

With the aim of taking into consideration human–robot interaction, several authors have demonstrated that the same proxemic zones that exist in human–human interaction can be useful to explain human–robot interaction scenarios [53,54,55]. Sisbot et al. [15] proposed including in the model safety- and visibility-related criteria to control the distance between the robot and human and keep the robot mostly in the human’s field of view. The safety, visibility, and hidden zone grids are combined into a single grid to find the most cost-efficient path. In the work by Svenstrup et al. [56], the behavior of the robot is based on adaptive potential functions that are adjusted accordingly such that social spaces are respected. Castro-Gonzalez et al. [57] proposed a method for predicting people’s positions in crossing behaviors using proxemics. The modified social force model (MSFM) [58] integrates social components (body pose, face orientation, and personal space during motion) into the classical SFM based on human position. In the MSFM, the short-term intended direction is described by body pose, and a supplementary force-related face orientation is added for intention estimation. Face orientation is employed as the best indication of the direction of personal space during motion. To endow robots with the ability for navigating dynamic human environments in a socially acceptable manner that would guarantee human comfort and safety, Truong et al. [33] proposed extending SFM with the Hybrid Reciprocal Velocity Obstacle technique. The result is the so-called proactive Social Motion Model (PSMM), which considers not only human states (position, orientation, motion, field of view, and hand poses) relative to the robot but also social interactive information about human-object and human group interactions. A survey paper describing the social concepts of proxemics theory applied in the context of human-aware autonomous navigation was provided by Rios-Martinez et al. [7], and more recently by Samarakoon et al. [8]. In general, these navigation algorithms model human–robot interaction, emphasizing on maintaining the proper safety distances, but they do not consider human–robot cooperation.

In dense crowds, a common problem of those proactive approaches that take into consideration the uncertainty in the position of humans or robots is the so-called Freezing Robot Problem (FRP) [14,39] (Figure 9). Briefly, the problem is that the robot may not be able to find even one feasible route due to the difference between the predicted and the real motion of mobile agents. To deal with this problem, the interaction of all agents with the remaining static and dynamic obstacles must be considered. This strategy is addressed by SFM-based approaches [59,60] or the so-called optimal reciprocal collision avoidance (ORCA) approaches [14,61,62,63]. Farina et al. [60] proposed merging the SFM with Laumond’s human locomotion models. The resulting Headed SFM is able to reliably reproduce human motions both in free space and in highly crowded environments. In the proactive kinodynamic planner, Ferrer and Sanfeliu [59] propose using the Extended Social Force Model (ESFM) to simplify both the prediction model and the planning system under differential constraints. The main problem with SFM-based approaches is that parameter tuning depends on the specific scenario [64]. For computing free routes, ORCA conducts an optimized search in the feasible geometric space. In the reciprocal n-body collision avoidance [61], the problem of avoiding collisions between multiple robots is reduced to solve a low-dimensional linear program. The approach is based on the VO concept. In the GLMP (Global and Local Movement Patterns) approach [62], the characteristics of agents’ motion and movement patterns are learned from 2D trajectories using Bayesian inference. Motion patterns consider local movement ones, corresponding to the current and preferred velocities, and global characteristics such as entry points and movement features. The Pedestrian Optimal Reciprocal Collision Avoidance (PORCA) proposed by Luo et al. [63] is a human motion prediction model that takes into account the human’s global navigation intention and the local interactions with the robot and other people. Chen et al. [14] proposed the intention-enhanced ORCA (iORCA). The iORCA employs a naive Bayesian classifier for estimating the most possible pedestrian destination, and then it can compute the human velocity. Moreover, to deal with possible changes of intention, iORCA updates these destinations at each time step. Inspired by the RVO model, the eRVO model integrates the emotional effect into velocity decision [65]. Generally, ORCA approaches have shown to be more stable than SFM in low sampling rates and in dense crowd scenarios [65,66].

Data-driven approaches have been also proposed for capturing agent motion considering interactions with static and dynamic obstacles. Thus, inspired by the success of Long-Short Term Memory networks (LSTM) in other research tasks, Alahi et al. [67] proposed the Social-LSTM, a data-driven architecture for predicting human trajectories in future instants. For capturing the dependencies between multiple correlated sequences, a social pooling layer is introduced. This allows the associated LSTMs to spatially close sequences to share their hidden states with each other. Thus, the model can automatically learn typical interactions that take place among trajectories that coincide in time. Considering that every person within a crowd implicitly cooperates with each other to avoid collisions, the Social Attention by Vemula et al. [68] captures the relative importance of each person when navigating, without emphasizing the proximity. The problem with these approaches is that they are computationally expensive, and therefore difficult to incorporate into mobile robots.

Table 2. Approaches considered in Section 3.2.1. The table covers the methods they use for modeling human motion.

Reference	Methods
van den Berg et al. [35]	Reciprocal Velocity Obstacle
Snape et al. [34]	Hybrid Reciprocal Velocity Obstacles
Zanlungo et al. [36]	SFM extended to the near future
Ferrer et al. [37]	Social Force Model (SFM)
Shiomi et al. [38]	Collision Prediction Social Force Model (CP-SFM)
Trautman et al. [39]	Multiple Goal Interacting Gaussian processes algorithm
Large et al. [40]	Velocity Obstacle (VO) & Obstacles motion prediction
Thompson et al. [41]	Probabilistic Model of Human Motion
Du Toit and Burdick [42]	Thresholding the uncertainty
Joseph et al. [43]	Gaussian processes (GP) & Dirichlet process (DP) prior over mixture weights
Aoude et al. [44]	RR-GP—Learned motion pattern model by combining the flexibility of Gaussian
	processes (GP) with the efficiency of RRT-Reach
Ziebart et al. [45]	Maximum entropy inverse optimal control
Bennewitz et al. [46]	Learned human motion patterns
Kuderer et al. [47]	Maximum entropy
Ferrer and Sanfeliu [48]	CLP-Time Window Predictor
Kabtoul et al. [49]	Quantitative time-varying function to model HR cooperation
Ikeda et al. [50]	Sub-goals to retrieve useful information not only for prediction but also for robot navigation, environment modeling and pedestrian simulation.
	Sub-goals used as the nodes of the robot global path planner, and as the nodes of the planner in the pedestrian simulator
Luber et al. [51]	Non-homogeneous spatial Poisson process
	Bayesian inference from observations of track creation
	Matching and false alarm events
	Gained by introspection of a laser-based multi-hypothesis people tracker
Vega et al. [53]	Adaptive Spatial Density Function
	Asymmetric Gaussian representation for personal space
	Inclusion of the space affordances
	Probabilistic Road Mapping (PRM)
	Rapidly-exploring Random Tree (RRT)
	Elastic band algorithm
Mead and Mataric [54]	Probabilistic framework for proxemic behavior production in HRI
Mead et al. [55]	Heuristic-Based vs. Learned Approaches
Sisbot et al. [15]	Safety and visibility related criteria
Svenstrup et al. [56]	Adaptive potential functions respecting social spaces
Castro-González et al. [57]	Hidden Markov Models for predictions
Ratsamee et al. [58]	Modified SFM (MSFM) considering body pose, face orientation and personal space
Truong and Ngo [33]	Proactive Social Motion Model (PSMM)
Ferrer and Sanfeliu [59]	Extended SFM (ESFM)
Farina et al. [60]	SFM with Laumond’s human locomotion models
van den Berg et al. [61]	Reciprocal n-body collision avoidance
Bera et al. [62]	GLMP approach-Global and Local Movement Patterns
	Pedestrian trajectory data using Bayesian Inference
	Ensemble Kalman Filters (EnKF) and Expectation Maximization (EM)
Luo et al. [63]	Pedestrian Optimal Reciprocal Collision Avoidance (PORCA) combines with a Partially Observable Markov Decision Process algorithm (POMDP)
Chen et al. [14]	Interactive MPC (iMPC) framework
	Intention Enhanced ORCA (iORCA)
Xu et al. [65]	Emotional Reciprocal Velocity Obstacles (eRVO)
Alahi et al. [67]	Social Long-Short Term Memory networks (Social-LTSM)
Vemula et al. [68]	Social Attention
	S-RNN architecture

Table 2. Approaches considered in Section 3.2.1. The table covers the methods they use for modeling human motion.

Reference	Methods
van den Berg et al. [35]	Reciprocal Velocity Obstacle
Snape et al. [34]	Hybrid Reciprocal Velocity Obstacles
Zanlungo et al. [36]	SFM extended to the near future
Ferrer et al. [37]	Social Force Model (SFM)
Shiomi et al. [38]	Collision Prediction Social Force Model (CP-SFM)
Trautman et al. [39]	Multiple Goal Interacting Gaussian processes algorithm
Large et al. [40]	Velocity Obstacle (VO) & Obstacles motion prediction
Thompson et al. [41]	Probabilistic Model of Human Motion
Du Toit and Burdick [42]	Thresholding the uncertainty
Joseph et al. [43]	Gaussian processes (GP) & Dirichlet process (DP) prior over mixture weights
Aoude et al. [44]	RR-GP—Learned motion pattern model by combining the flexibility of Gaussian
	processes (GP) with the efficiency of RRT-Reach
Ziebart et al. [45]	Maximum entropy inverse optimal control
Bennewitz et al. [46]	Learned human motion patterns
Kuderer et al. [47]	Maximum entropy
Ferrer and Sanfeliu [48]	CLP-Time Window Predictor
Kabtoul et al. [49]	Quantitative time-varying function to model HR cooperation
Ikeda et al. [50]	Sub-goals to retrieve useful information not only for prediction but also for robot navigation, environment modeling and pedestrian simulation.
	Sub-goals used as the nodes of the robot global path planner, and as the nodes of the planner in the pedestrian simulator
Luber et al. [51]	Non-homogeneous spatial Poisson process
	Bayesian inference from observations of track creation
	Matching and false alarm events
	Gained by introspection of a laser-based multi-hypothesis people tracker
Vega et al. [53]	Adaptive Spatial Density Function
	Asymmetric Gaussian representation for personal space
	Inclusion of the space affordances
	Probabilistic Road Mapping (PRM)
	Rapidly-exploring Random Tree (RRT)
	Elastic band algorithm
Mead and Mataric [54]	Probabilistic framework for proxemic behavior production in HRI
Mead et al. [55]	Heuristic-Based vs. Learned Approaches
Sisbot et al. [15]	Safety and visibility related criteria
Svenstrup et al. [56]	Adaptive potential functions respecting social spaces
Castro-González et al. [57]	Hidden Markov Models for predictions
Ratsamee et al. [58]	Modified SFM (MSFM) considering body pose, face orientation and personal space
Truong and Ngo [33]	Proactive Social Motion Model (PSMM)
Ferrer and Sanfeliu [59]	Extended SFM (ESFM)
Farina et al. [60]	SFM with Laumond’s human locomotion models
van den Berg et al. [61]	Reciprocal n-body collision avoidance
Bera et al. [62]	GLMP approach-Global and Local Movement Patterns
	Pedestrian trajectory data using Bayesian Inference
	Ensemble Kalman Filters (EnKF) and Expectation Maximization (EM)
Luo et al. [63]	Pedestrian Optimal Reciprocal Collision Avoidance (PORCA) combines with a Partially Observable Markov Decision Process algorithm (POMDP)
Chen et al. [14]	Interactive MPC (iMPC) framework
	Intention Enhanced ORCA (iORCA)
Xu et al. [65]	Emotional Reciprocal Velocity Obstacles (eRVO)
Alahi et al. [67]	Social Long-Short Term Memory networks (Social-LTSM)
Vemula et al. [68]	Social Attention
	S-RNN architecture

3.2.2. Navigation Strategies Using Agent Motion Models

Once a model of human motion is available, the next step in a proactive approach is to define a planner able to find an optimal navigation policy.

Table 3. Approaches considered in Section 3.2.2. The table covers the methods they use for solving the navigation problem.

Reference	Methods
Svenstrup et al. [70]	Rapidly-exploring Random Tree (RRT)
	Trajectory Generation Problem
	Model Predictive Control (MPC)
	Dynamic Potential Field
Foka and Trahania [69]	Predictive navigation performed in a global manner with the use of a POMDP
	Polynomial Neural Network (PNN)
	Future motion prediction
	Robot Navigation-HPOMDP (RN-HPOMDP)
Rios-Martinez et al. [73]	RISK-RRT algorithm navigation
	Learned Gaussian Processes
	Personal Space
	Model of o-space in F-formations
Park and Kuipers [71]	The formulation of the kinematic control law
	The pose-following algorithm for smooth and comfortable motion of unicycle-type robots
Park et al. [72]	Model Predictive Equilibrium Point Control (MPEPC) framework
Du Toit and Burdick [42]	Motion Planning
Ferrer et al. [52]	SFM and prediction information
Ferrer and Sanfeliu [74]	Bayesian Human Motion Intentionality Prediction
	Sliding Window BHMIP (BHMIP)
	Two variants: the Sliding Window BHMIP and the Time Decay BHMIP
	Expectation-Maximization method
Palm et al. [75]	Recognize the human intention with relative speeds
	Collision avoidance by extrapolation of human intentions and heading angle
	Compass dial
	Fuzzy rules for Human-Robot interactions
Ferrer et al. [76]	Socially-aware navigation framework for allowing a robot to navigate accompanying the person
Khambhaita and Alami [77]	Cooperative navigation planner
	Trajectory Optimization: Elastic band
	Expectation-Maximization method
	Optimization framework
	Graph-based optimal solver
	Time-to-collision and directional constraints during optimization
Kabtoul et al. [78]	Cooperative navigation planner

lists methods that aim towards this objective. Foka and Trahania [69] suggested a unified model for considering global and local planning (as well as localization). The model is a specific instantiation of the hierarchical Partially Observable Markov Decision Process (POMDP), called Robot Navigation-HPOMDP (RN-HPOMDP). The framework estimates the final destination of all mobile agents, and this information is employed for effective obstacle avoidance. Svenstrup et al. [70] proposed an algorithm for robot trajectory planning in dynamic human environments, using a potential field generated from the perceived positions and motions of people. The problem is solved using a Rapidly Exploring Random Tree (RRT) algorithm enhanced with a robot motion model and controller, and using a Model Predictive Control (MPC) approach to execute only a short segment of the planned trajectory. The method minimizes the cost of traversing the potential field, resulting in comfortable and natural robot trajectories. Dutoit and Burdick [42] describe an MPC framework for planning in a dynamic scene. As described in Section 3.2.1, the predicted motion uncertainties of both the robot and people are set as a hard threshold. Park and Kuipers [71] and Park et al. [72] proposed combining MPC and the equilibrium point control to provide a model predictive equilibrium point control (MPEPC) for a wheelchair robot navigating crowds. Taking into consideration human intention and human–robot interactions, the interactive MPC (iMPC) framework was proposed by Chen et al. [14]. The iMPC framework applies the iORCA model in the state transition function to predict human states, and extends this interactive model with robot constraints.

Based on the SFM, Ferrer et al. [37,52] put the emphasis on the design of socially aware navigation frameworks, where topics, such as human comfort and safety are of vital relevance. Ferrer et al. [52] integrated pedestrian intention and interaction into a scheme for a robot’s human-awareness navigation based on the social forces concept. Their experiments show that socially aware navigation is well suited for a robot companion task in open spaces.

To respect people’s personal space, while also avoiding collisions, Rios-Martinez et al. [73] propose an extension of the RRT algorithm for navigation. The Risk-RRT approach uses a Gaussian procedure learning for estimating the area occupied by the person (o-space).

Although human intentionality was predicted, in some sense, by Ikeda et al. [50], the topic is the core of the proposals by Ferrer and Sanfeliu [74] and Palm et al. [75]. The Bayesian Human Motion Intentionality Prediction (BHMIP) employs the Expectation-Maximization method for estimating destination points [74]. The method has a simple formulation, low computational complexity and outperforms existing methods such as the ones proposed by Foka and Trahania [69]. Figure 10 provides a snapshot of the robot Dabo, used for testing the BHMIP. The proposal by Palm et al. [75] focuses on the recognition of human intentions in a human–robot interaction scenario. The framework includes a method for predicting and avoiding collisions by extrapolating human intentions.

Taking into consideration the human–robot interaction factor, Ferrer et al. [76] propose a socially-aware navigation framework for allowing a robot to navigate, accompanying the person in a safe and natural way. In this scenario, the robot companion has to deal with two goals at the same time: to navigate toward the person’s predicted destination, and to approach the person who accompanies them. The prediction model is based on Extended SFM (ESFM).

The cooperative navigation planner by Khambhaita and Alami [77] is a tool designed to plan cooperative trajectories for robots and humans while respecting the robot’s kinematic constraints and avoiding other non-human dynamic obstacles. The planner could adapt the robot’s trajectory and propose co-navigation solutions even in confined spaces. It predicts a plausible trajectory for humans, and plans a corresponding robot trajectory that satisfies social constraints. The planner generates both human and robot trajectories using a graph-based optimal solver and balances the efforts between both agents to solve the co-navigation task. The approach includes proxemics, time-to-collision, and directional constraints during optimization. The trajectory optimization uses an elastic band approach and a least-squares problem is mapped into a hyper-graph representation to adjust the position and orientation of nodes and minimize the imposed constraints.

In the same way that we describe in the previous subsection, ignoring the cooperation between the mobile agents in the path planning step can lead to the freezing of the robot. In the proposal by Kabtoul et al. [78], proactive and natural maneuvering is suggested for navigation around people. The approach consists of two main steps. First, the space is explored and dynamically divided into a set of channels using a local segment of the global path. The optimal channel is found using a fuzzy cost model, and its center line provides the goal path to the local navigation module. To convey a human-like steering behavior, a smooth lane change maneuver is adapted to travel between channels using a Quintic transition path. In the final stage, the exact tracking control commands are derived using a sliding mode control method.

3.3. Learning-Based Approaches

In recent years, many researchers studied methods by which a robot learns the navigation policy adapted to the target environment. Deep reinforcement learning (DRL) was often used to learn this interaction policy. Approaches such as Collision Avoidance with DRL (CADRL) [79], Socially-Attentive RL (SARL) [80], and Socially-Attentive Object-Aware DRL (SOADRL) [81] have been proposed to address this problem. However, the exact positions of pedestrians can be difficult to estimate in real-life situations. To avoid this computation, end-to-end (E2E) learning approaches directly map raw sensory inputs to the desired steering commands. On the other hand, the problem of automating the computation of the reward function in DRL-based approaches has been addressed using inverse reinforcement learning (IRL). By learning the reward function directly from the data, IRL can improve the learning rate and performance of the system. Several IRL approaches were used to learn social navigation behaviors for robots, including optimizing reward function parameters using maximum likelihood estimation [82], and modeling IRL from a Bayesian perspective [83].

3.3.1. Deep Reinforcement Learning and End-to-End Approaches

In recent years, Deep reinforcement learning (DRL) has emerged as a successful tool for solving those tasks where it is not easy to engineer a direct solution. Briefly, DRL introduces deep neural networks to solve reinforcement learning problems. A multitude of approaches for training a collision avoidance policy based on DRL has been proposed in the last decade, some of which take social awareness into account. Table 4 enumerates some of these approaches, that are described in this section.

Chen et al. [79] use DRL to train a navigation strategy in a multi-agent scenario (Collision-Avoidance with DRL, CADRL). The hand-crafted reward function positively benefits reaching the desired goal and penalizes collisions. This proposal was extended for considering humans and social norms in Chen et al. [84] (SA-CADRL). Specifically, in this work, a socially compliant behavior (e.g., passing by the right side) was learned using a reward function that depends on situational dynamics. The scheme was improved by Everett et al. [85], who leveraged GPU processing to train multiple agents in parallel, and Long Short-Term Memory (LSTM) to convert the variable size state of the crowd into the fixed-size vector. Although these approaches were successful for dealing with multi-agent navigation, they failed to account for complex interactions among humans [86]. To improve the comfort of people sharing the crowd with a robot, Hu et al. [87] use social stress indexes in the reward function and value network of the DRL framework. A multi-layer perceptron is employed to extract local features and social-attention scores. As in the rest of the approaches cited in this paragraph, they make use of exact pedestrian positions in the input. This will be not an easy task to address for a real robot deployed in a real scenario [88]. Gil et al. [89] proposed computing robot actions by a combination of robot velocities learned by a RL model (AutoRL [90]) and robot velocities computed using an SFM [36].

Previous approaches typically address robot navigation in a crowded environment as a one-way human–robot interaction problem, experiencing problems when the crowd grows. To avoid this situation, Chen et al. [80] proposed explicitly modeling the crowd-robot Interaction (Socially Attentive RL approach, SARL). Self-attention is used to discover the collective importance of the crowd by considering human–human and human–robot interactions. Obstacles are however left out of the policies learned by the SARL approach. This issue is addressed by the SOADRL [81]. This approach extends the SARL by separately processing information related to static and dynamic objects. Moreover, the SOADRL addresses the problem of navigation in crowds with sensors that only offer a limited field of view. As with the SA-CADRL, the SARL and SOADRL make use of the exact mobile agents’ position in the input. To deal with the problem of a large-sized crowd, Chen et al. [91] suggested enabling the system to identify those humans in the crowd that are most critical for navigation. The proposal uses a graph representation to learn this policy, which encodes information about the crowd and predicts human attention scores in the navigation task. A graph convolutional network is trained based on human gaze data, which accurately predicts human attention to different agents in the crowd as they perform a navigation task. The learned attention is integrated into the graph-based reinforcement learning architecture. The problem of partial observability (due for example to sensor limitations, occlusions, or perception uncertainty) is considered by Gao et al. [92]. To achieve this, the proposal makes use of a recurrent neural network (RNN) (the so-called gated recurrent unit (GRU)) to infer the unobservable states. To respond in real-time to human behaviors, Samsani and Muhammad [86] proposed modeling Danger Zones for the robot. These zones are formulated by taking into account real-time human behavior, and then they encode all possible actions that people can take at a given time. The robot is trained to avoid these danger zones for safe and secure navigation.

With the help of deep neural networks, DRL can apply end-to-end (E2E) learning, i.e., to learn a black box model, extract features from captured high-dimensional data and learn complex policies. Thus, they avoid the need for detection and efficient tracking of people. From a general point-of-view, where the goal is to directly map raw sensory inputs to desired steering commands, the learning process can be summarized as follows [93]:

• the robot moves according to a given action and obtains information from the environment (observations) and a reward;
• following a policy, and given the captured observation, an action is generated;
• the policy is updated by an RL-based algorithm.

The robot finally gets the optimal policy to achieve the goal by repeating the learning process. In order to use DRL in an E2E robot navigation context, the whole problem setting must be stated and translated into an RL framework [94]. For instance, to avoid collisions, Long et al. [93] proposed directly mapping raw 2D laser measurements to desired motion commands using a 4-hidden-layer neural network. Shi et al. [95] proposed an E2E navigation framework that translates sparse laser-ranging results into movement actions. The goal of using this highly abstract data as input is that robots trained by simulation can be also deployed in real environments. The proposal shows robust navigation but in relatively simple environments. The Role Playing Learning (RPL) [96] endows a robot with appropriate group behaviors when it is traveling with a human companion. This E2E proposal uses neural networks to map sensory data to velocity outputs while adhering to social norms. The RPL process is formulated under an RL framework and optimized using Trust Region Policy Optimization (TRPO). To directly learn control strategies from visual input, Mnih et al. [97] combined a convolutional neural Network (CNN) with a Q learning algorithm (DQN network model). Lee and Yusuf [98] mapped the data from an RGB-D camera to steering commands. A Deep neural network is employed for detecting the target, and the collision avoidance and navigation are addressed by a DQN model (or a double DQN (DDQN)). The approach is able to deal with static and dynamic obstacles, but it does not consider social factors.

Previously described E2E planning approaches use RL. Another possibility is to use a supervised learning scheme to imitate expert demonstrations (imitation learning, IL). IL is sample efficient and, given training data, a navigation model could be found quickly [99]. On the other hand, IL is conceptually less robust than RL [100], as RL allows the robot to learn from its own mistakes during training [101]. Pfeiffer et al. [99] proposed a data-driven E2E motion planner, where the robot learns to navigate as the human user likes. To achieve this, expert demonstrations of how to navigate in a given training environment are provided. Using this data, the aim is not only to replicate the provided demonstrations in one specific scenario, but rather to be able to learn collision avoidance policies and transfer them to previously unseen environments. As Figure 11 shows, a single model based on the TensorFlow framework [102] is used for extracting information and for estimating the steering commands. IL and RL are combined by Pfeiffer et al. [100] into a single target-driven, mapless navigation scenario. The Reinforced imitation learning (R-IL) uses expert demonstrations to pre-train the navigation policy, and then applies a Constrained Policy Optimization (CPO) [103] for incorporating constraints during RL training. The authors demonstrate that this scheme can reduce the training time, to reach the same performance level that plain RL, by a factor of 5. Pfeiffer et al. [99,100] test their solutions in a static environment. The CrowdMove approach [104] is a multi-robot, multi-scenario, and multi-stage training (3M) framework. It employs a 4-hidden-layer neural network as a nonlinear function approximator to the navigation policy, and extends the Proximal Policy Optimization (PPO) [105] to the multi-robot scenario. Experiments demonstrate that the navigation policy can achieve autonomous navigation for different mobile platforms in a large variety of crowd environments.

Long-term planning and learning for navigating within other mobile agents have been also combined as separate (but intimately tied together) modules. Instead of the E2E previous solutions, the aim here is to allow the motion planner to estimate the path and decompose it into a set of subgoals, being the learned low-level controller in charge of adapting the route to the dynamics of the current situation. Gao et al. [107] proposed combining a path planner with a neural-network motion controller (the intention-net). The intention net maps images to motion controls in an E2E scheme. The path planner uses an a priori 2D map to compute the paths. This planned path provides the intentions to the intention-net layer. The navigation system proposed by Pokle et al. [108] also combines a path planner and machine learning. Here, the global planner computes the routes toward a goal, and a deep local trajectory planner and velocity controller provides the motion commands. The low-level motion controller is responsible for avoiding obstacles and also for respecting the space of nearby pedestrians. Significantly, both approaches demonstrated that this scheme outperforms an end-to-end framework where planning guidance is not considered. Pérez-D’Arpino et al. [109] proposed a navigation stack that combines motion planning and RL. The RL component learns to handle the local interactions with other mobile agents as it pursues the globally planned trajectory. Choi et al. [110] proposed a framework where the DRL learns navigation policies that adapt to a wide range of reward weightings and other navigation parameters. Then, a Bayesian deep learning method is used for optimizing navigation parameters to human preferences.

Table 4. Approaches considered in Section 3.3.1. The table covers the methods for DRL and E2E approaches.

Reference	Methods
Chen et al. [79]	Decentralized Multi-agent Collision Avoidance algorithm based on a novel
	application of deep reinforcement learning
Chen et al. [84]	SA-CADRL, a multi-agent collision avoidance algorithm that considers and
	exhibits socially compliant behaviors
	Time-efficient navigation policy that respects common social norms
	Reinforcement learning framework
Everett et al. [85]	Collision avoidance algorithm, GA3C-CADRL, that is trained in simulation
	with deep RL without requiring any knowledge of other agents’ dynamic
	Long Short-Term Memory (LSTM)
	LSTM that enables the algorithm to use observations of an arbitrary number
	of other agents
Samsani and Muhammad [86]	Human Behavior Resemblance Using Deep Reinforcement Learning
	The Danger Zones are formulated by considering the real time human behavior
Hu et al. [87]	Deep reinforcement learning framework (DRL) and the value network
	The DRL framework incorporating these social stress indexes
Dugas et al. [88]	Reinforcement Learning of Robot Navigation in Dynamic Human Environments
	NavRepSim environment is designed with RL applications in mind
Gil et al. [89]	Social Force Model (SFM) allowing human-aware
	Two Machine Learning techniques: Social navigation and Neural Network (NN)
	RL technique
Francis et al. [90]	PRM-RL:Probabilistic road-maps (PRMs) as the sampling-based planner and
	reinforcement learning-RL method in the indoor navigation context
Chen et el. [80]	Crowd-Robot Interaction (CRI)
	Attention-based Deep Reinforcement Learning
Liu et al. [81]	Imitation learning and deep reinforcement learning approach for motion planning
	in such crowded and cluttered environments
Chen et al. [91]	Graph convolutional network (GCN) for reinforcement learning to integrate
	information
	Attention network trained using human gaze data for assigning adjacency values.
Gao et al. [92]	Learn an efficient navigation policy that exhibits socially normative navigation
	behaviors
	Convolutional social pooling layer that robustly models human–robot
	co-operations and complex interactions between pedestrians
	Partial observability in socially normative navigation
Long et al. [93]	Decentralized sensor-level collision avoidance policy for multi-robot systems
	Policy gradient-based reinforcement learning algorithm
Gromniak and Stenzel [94]	End-to-end Deep reinforcement learning
Shi et al. [95]	Navigation strategy based on deep reinforcement learning (DRL)
	Conventional A3C algorithm, an ICM A3C model was proposed
Li et al. [96]	Role Playing Learning (RPL)
	NN policy is optimized end-to-end
[97]	A deep Q-network (DQN), combine reinforcement learning with a deep neural
	networks
Lee and Yusuf [98]	Deep reinforcement learning for autonomous mobile
	The trained DQN and DDQN policies are robot navigation in an unknown
	environment evaluated in the Gazebo testing environment
	Two types of deep Q-learning agents, such as deep Q-network and double deep
	Q-network agents
Pfeiffer et al. [99]	Data-driven end-to-end motion planning approach for a robotic platform
	End-to-end model is based on a CNN
	Learn navigation strategies
Pfeiffer et al. [100]	Imitation learning(IL) and reinforcement learning (RL)
Tai et al. [101]	A map-less motion planner was trained end-to-end through continuous control
	deep-RL from scratch
	The learned planner can generalize to a real non-holonomic differential robot
	platform without any fine-tuning to real-world samples
Fan et al. [104]	Multi-robot, multi-scenario, and multi-stage training framework
Gao et al. [107]	Two-level hierarchical approach:Model-free deep learning and model-based path
	planning
	Neural-network motion controller, called the intention-net, is trained end-to-end to
	provide robust local navigation
	Path planner uses a crude map
Pokle et al. [108]	Hierarchical planning and machine learning
	Traditional global planner to compute optimal paths towards a goal
	Deep local trajectory planner and velocity controller to compute motion commands
	Combines traditional planning with modern deep learning techniques
Pérez-D’Arpino et al. [109]	Reinforcement learning to learn robot policies
	The proposed model uses a motion planner that provides a globally planned
	trajectory whereas the reinforcement component handles the local interactions
	needed for on-line adaptation to pedestrians
Choi et al. [110]	Novel deep RL navigation method that can adapt its policy to a wide range of
	parameters and reward functions without expensive retraining
	Bayesian deep learning method

Table 4. Approaches considered in Section 3.3.1. The table covers the methods for DRL and E2E approaches.

Reference	Methods
Chen et al. [79]	Decentralized Multi-agent Collision Avoidance algorithm based on a novel
	application of deep reinforcement learning
Chen et al. [84]	SA-CADRL, a multi-agent collision avoidance algorithm that considers and
	exhibits socially compliant behaviors
	Time-efficient navigation policy that respects common social norms
	Reinforcement learning framework
Everett et al. [85]	Collision avoidance algorithm, GA3C-CADRL, that is trained in simulation
	with deep RL without requiring any knowledge of other agents’ dynamic
	Long Short-Term Memory (LSTM)
	LSTM that enables the algorithm to use observations of an arbitrary number
	of other agents
Samsani and Muhammad [86]	Human Behavior Resemblance Using Deep Reinforcement Learning
	The Danger Zones are formulated by considering the real time human behavior
Hu et al. [87]	Deep reinforcement learning framework (DRL) and the value network
	The DRL framework incorporating these social stress indexes
Dugas et al. [88]	Reinforcement Learning of Robot Navigation in Dynamic Human Environments
	NavRepSim environment is designed with RL applications in mind
Gil et al. [89]	Social Force Model (SFM) allowing human-aware
	Two Machine Learning techniques: Social navigation and Neural Network (NN)
	RL technique
Francis et al. [90]	PRM-RL:Probabilistic road-maps (PRMs) as the sampling-based planner and
	reinforcement learning-RL method in the indoor navigation context
Chen et el. [80]	Crowd-Robot Interaction (CRI)
	Attention-based Deep Reinforcement Learning
Liu et al. [81]	Imitation learning and deep reinforcement learning approach for motion planning
	in such crowded and cluttered environments
Chen et al. [91]	Graph convolutional network (GCN) for reinforcement learning to integrate
	information
	Attention network trained using human gaze data for assigning adjacency values.
Gao et al. [92]	Learn an efficient navigation policy that exhibits socially normative navigation
	behaviors
	Convolutional social pooling layer that robustly models human–robot
	co-operations and complex interactions between pedestrians
	Partial observability in socially normative navigation
Long et al. [93]	Decentralized sensor-level collision avoidance policy for multi-robot systems
	Policy gradient-based reinforcement learning algorithm
Gromniak and Stenzel [94]	End-to-end Deep reinforcement learning
Shi et al. [95]	Navigation strategy based on deep reinforcement learning (DRL)
	Conventional A3C algorithm, an ICM A3C model was proposed
Li et al. [96]	Role Playing Learning (RPL)
	NN policy is optimized end-to-end
[97]	A deep Q-network (DQN), combine reinforcement learning with a deep neural
	networks
Lee and Yusuf [98]	Deep reinforcement learning for autonomous mobile
	The trained DQN and DDQN policies are robot navigation in an unknown
	environment evaluated in the Gazebo testing environment
	Two types of deep Q-learning agents, such as deep Q-network and double deep
	Q-network agents
Pfeiffer et al. [99]	Data-driven end-to-end motion planning approach for a robotic platform
	End-to-end model is based on a CNN
	Learn navigation strategies
Pfeiffer et al. [100]	Imitation learning(IL) and reinforcement learning (RL)
Tai et al. [101]	A map-less motion planner was trained end-to-end through continuous control
	deep-RL from scratch
	The learned planner can generalize to a real non-holonomic differential robot
	platform without any fine-tuning to real-world samples
Fan et al. [104]	Multi-robot, multi-scenario, and multi-stage training framework
Gao et al. [107]	Two-level hierarchical approach:Model-free deep learning and model-based path
	planning
	Neural-network motion controller, called the intention-net, is trained end-to-end to
	provide robust local navigation
	Path planner uses a crude map
Pokle et al. [108]	Hierarchical planning and machine learning
	Traditional global planner to compute optimal paths towards a goal
	Deep local trajectory planner and velocity controller to compute motion commands
	Combines traditional planning with modern deep learning techniques
Pérez-D’Arpino et al. [109]	Reinforcement learning to learn robot policies
	The proposed model uses a motion planner that provides a globally planned
	trajectory whereas the reinforcement component handles the local interactions
	needed for on-line adaptation to pedestrians
Choi et al. [110]	Novel deep RL navigation method that can adapt its policy to a wide range of
	parameters and reward functions without expensive retraining
	Bayesian deep learning method

3.3.2. Inverse Reinforcement Learning

One of the major issues to solve when designing a DRL-based scheme is the choice of the reward function. Thus, a bad choice can dramatically impact learning rate and performance [111]. One alternative to the hand-crafted setting of the reward function is to learn this function directly from the data. Inverse Reinforcement Learning (IRL) is a good mechanism for addressing this learning process.

Table 5. Approaches considered in Section 3.3.2. The table covers the reviewed methods for IRL.

Reference	Methods
Ziebart et al. [112]	Maximum Entropy Inverse Reinforcement Learning
	Inverse reinforcement and imitation learning
Henry et al. [82]	Inverse Reinforcement Learning with
	Gaussian Processes for environmental
Pérez-Higueras et al. [113]	Inverse reinforcement learning
	Global path planner- Dijkstra’s algorithm
Gerkey and Konolige [114]	DARPA Learning Applied to Ground
	Globally optimal paths on a cost map
Vasquez et al. [115]	Compare IRL based learning methods
	Motion Planning -grid-based GPU
Kretzschmar et al. [116]	Hamiltonian Markov Chain Monte Carlo (MCMC)
	Learn a model of the navigation behavior of cooperatively
	navigating agents such as pedestrians
	Voronoi graph of the environment
Wang et al. [117]	Neural Network Rapidly-exploring Random Trees
	A cost function based on neural network
Ramachandran and Amir [118]	Bayesian IRL (BIRL)
	Reward learning is an estimation task
	Markov Decision Process
	Apprenticeship learning task
Kim et al. [83]	Path planning module
	Inverse Reinforcement Learning
	Framework for socially adaptive path planning in dynamic
	environments
	Generating human-like path trajectory
Okal and Arras [119]	Graph-based representation of the continuous
	New extension of BIRL

presents some recent approaches that use IRL in the context of robot social navigation. Ziebart et al. [112] proposed a probabilistic approach based on the principle of maximum entropy that allows optimizing the parameters of a reward function using maximum likelihood estimation (MLE). The scheme was used by Henry et al. [82] to learn human-like navigation behavior in crowded environments. Specifically, the approach learns from example paths by estimating values such as crowd density on the fly using Gaussian processes. The performance of the approach was evaluated within a realistic crowd simulation and resulted in natural paths that blended seamlessly with existing crowd movements. Pérez-Higueras et al. [113] also proposed IRL for social navigation. However, instead of using the costs for path plans, they employ them to learn local execution policies and provide steering commands to the robot. The global path planner is based on Dijkstra’s algorithm and the local planner is an extension of the Trajectory Roll-out algorithm [114]. Vasquez et al. [115] compared IRL and manual tuning in the learning of the parameters of a reward function. They proposed evaluation metrics to benchmark these techniques. The results from simulations using two IRL approaches and several feature sets are reported and evaluated using objective and subjective performance metrics. Obtained results demonstrated that IRL-learned reward parameters are better than manually tuned ones. The maximum entropy approach was used by Kretzschmar et al. [116] to learn the parameters of a joint trajectory distribution of all navigating agents in an environment, including the robot itself. Hamiltonian Markov chain Monte Carlo sampling is used to compute the feature expectations over the resulting high-dimensional continuous distributions. While learning a joint distribution over all agents allows for high-quality inference, it does not scale to moderately populated settings (e.g., a few dozen agents).

Wang et al. [117] proposed a Neural Network Rapidly-exploring Random Trees (NN-RRT*) planner for generating robot paths in human–robot interaction environments. Based on this planner, they propose the NRTIRL framework. Figure 12 provides an overview of the framework. Briefly, the idea is to compare planned routes generated by the NN-RRT* and demonstration routes generated by human volunteers. To achieve this, features of demonstration and planned routes are provided to the neural network to obtain their corresponding cost. If their difference is higher than an allowable error, the feature vector of the planned route is used as input, and the cost of the demonstration route is used as the output of the neural network. The neural network is then optimized until the difference between demonstration and planned routes is lower than this allowable error. When the parameters of the neural network converge, the new NN-RRT* planner is updated and able to generate routes that are more similar to the ones provided by humans.

Ramachandran and Amir [118] proposed modeling IRL from a Bayesian perspective (BIRL) and solving reward learning and apprenticeship learning using a modified Markov Chain Monte Carlo (MCMC) algorithm. The reward is modeled as a random variable vector that determines the distribution of expert states and actions. The distribution of the rewards which best explains expert trajectories is then inferred. Kim et al. [83] use BIRL to learn a linear reward function over features extracted from RGB-D cameras on a robotic wheelchair. The proposed framework consists of three modules: feature extraction, BIRL, and path planning. The feature extraction module extracts information from a RGB-Depth sensor, the BIRL module uses expert demonstrations to learn a cost function that considers social variables, and the planning module uses a three-layer architecture to optimize the global and local paths while avoiding obstacles. Okal and Arras [119] proposed a new approach to modeling socially normative behavior in robots using MDPs and a modified version of the BIRL. They focused on spatial robot motion behaviors and use a graph-based representation to integrate task-specific constraints into the MDP. The use of this graph-based representation allows the authors to instantiate global planners such as RRT or A* using the rewards learned for local navigation.

3.4. Multi-Behavior Navigation Approaches

When people move in an environment where there are many other people, it is not easy to behave like a rigid object, whose aim is to reach a goal without touching these other people. We usually have to ask for permission to pass, sometimes even interrupting a conversation, informing with gestures of our intentions, or even giving a light touch so that a person moves slightly to let us pass. This scenario, which is complex to translate to a robotic agent as it involves mixing originally different functionalities, implies integrating the navigation stack with other modules present in the software architecture. This integration can be done by incorporating all these functionalities into the navigation framework (as we would do for a robot that follows a person, for example, by integrating the person detector into the navigation stack) or by allowing all these functionalities to cooperate in a single shared representation of the environment. In this section, we present a few examples of approaches (

Table 6. Approaches considered in Section 3.4. The table covers multi-behavior navigation approaches and multi-modal behavior navigation approaches.

Reference	Methods
Chen et al. [122]	Travel model selection according to the traffic state
Freitas et al. [123]	Self-adaptation based on QoS metrics
	Planning encoded in Behaviour Trees. CORTEX
	software architecture
Bozhinoski and Wijkhuizen [124]	Self-adaptation based on MROS framework
	Quality models for adapting the local planner
	configuration at run-time
Kamezaki et al. [16]	Proximal Crowd Navigation (PCN) approach
	(proximity and physical-touching).
	inducible SFM (i-SFM) for predicting human motion.
Dugas et al. [120]	The IAN framework
	Interaction actions (saying, touching, and gesturing)
	for navigating in crowded scenarios
Vega-Magro et al. [121]	The SNAPE framework
	CORTEX cognitive software architecture

) that concatenate actions associated with different functionalities for reaching a navigation goal (multi-modal behaviors). The first of the integration options discussed above is the one used by Kamezaki et al. [16] and Dugas et al. [120]. The second is the one proposed by Vega-Magro et al. [121].

Allowing a robot to switch behaviors depending on the context entails entering the field of self-adaptation. In the proposal by Chen et al. [122], after estimating a local trajectory for obstacle avoidance based on predicted mobile agents’ routes, the robot can choose a travel model for navigation according to the traffic state. Freitas et al. [123] proposed a framework that allows a robot to change its navigation configuration depending on the context (e.g., aborting a mission when the power autonomy level is low and the robot is redirected to the charging station). Some of these context changes are related to the presence of people in the environment. However, this work does not really focus on enabling a robot to navigate naturally and socially correctly between people. Using the MROS model-based framework, Bozhinoski and Wijkhuizen [124] presented a similar approach that adapts the local planner configuration at run-time to satisfy a set of quality requirements. None of these methods consider other action skills for the robot apart from moving through the environment.

The IAN (Interaction Actions for Navigation) approach [120] is defined as a high-level, multi-behavior, interaction-aware planning for navigation in unstructured, human-populated environments. Briefly, the approach allows the robot to choose a specific navigation behavior based on the observed state of the environment. These behaviors combine actions associated with different robotic functionalities (multi-modal behaviors). The first behavior considers a static or sparsely dynamic scenario and consists of a motion planner based on the local velocity field. RVO is used for modeling mobile agents. In the second behavior, the robot is able to verbally announce that it is moving, while indicating the direction of movement with its hand. The third behavior is employed by the robot to pass through very close people in a highly populated scenario. The Dynamic Window Approach (DWA) is employed as a local planner. When the DWA cannot find a non-zero solution, the robot moves at a very low speed, with its arm forward, reaching its own base footprint, and announcing verbally that it is passing through the people.

The SNAPE framework [121] emphasizes that a robot needs social behaviors and cooperation from humans to navigate socially. The core of the proposal is the use of the CORTEX software architecture for robotics [125,126]. This architecture deploys a set of software agents surrounding a graph-based world model. These agents are organized into five layers (perception, social, navigation, human–robot interaction, and high-level planning). Using CORTEX, the SNAPE framework manages all the information flow, from the perception of the environment to behavior planning. However, planning is considered not only at the behavior level, but also at the dialogue and task-planning levels. The decision maker coordinates the activity at all layers. For instance, when the robot detects that a person is blocking its route, it can approach this person, draw its attention with a specific dialogue, and ask permission to pass.

Based on an inducible SFM (i-SFM), Kamezaki et al. [16] propose a reactive, proactive, and inducible way-point-i-SFM fused-path planning method (the proximal crowd navigation (PCN) approach). The PCN considers proximity but also physical touching for tracing the routes, and uses i-SFM for predicting human motions. The PCN is able to predict the movement of the people in the robot’s surroundings, generate multiple paths including physical-touch paths using the way-point method, and determine the route taking into consideration the movement efficiency and the degree of crowd invasion.

4. Discussion

Robot navigation within humans has been a goal of the robotics community for decades. Reactive approaches, such as the ones presented in this study, were designed to handle dynamic scenarios, but have severe constraints (e.g., constant velocities for moving obstacles) that do not allow them to handle complex environments. These approaches do not model humans and their not always predictable behavior. Moreover, they cannot take into account multi-agent behaviors, such as joint planning. To deal with these problems, proactive approaches focus on human modeling and reciprocal planning. They have faced several problems, related to, for example, giving relevance to social comfort (practically covered by most of the proactive and learning-based approaches), the interaction with humans but also with groups of humans, or the Freezing Robot Problem. The problem of the first approaches for modeling human motion was solved by considering goal-based policies. Considering that people are moving towards certain goals, the robot can project their motion (trajectories) on a map and trace a route for avoiding them. However, these goals are not necessarily available to the robot. Given the positions in a map of all people surrounding the robot, proxemics can be used for extending this map, taking into account social norms. In addition to space, other factors need to be taken into account. Traits such as body posture and facial expressions can help the robot to be more approachable and predictable, further improving its ability to function in social environments. As we review in this work, proactive approaches have evolved to manage human–robot interaction and, subsequently, human–robot cooperation. Cooperative planning provides navigation efficiency and also human acceptability [120]. In parallel, learning-based approaches have emerged in this last decade to learn motion planners. As data-driven approaches, the major strength of these methods is that they are able to estimate a practical human model without having to specify social norms, as they are implicitly present in the data. In recent years, the major limitation was that these approaches need large data sets, which were captured from virtual scenarios. The great difference between the simulation environment and the real-world one is the major challenge to transfer the trained model to a real robot. Virtual-to-real approaches are currently able to generalize the learned planners and to satisfactorily deployed them in unseen real environments [101]. It is clear that real-world experiments involving actual people and uncontrolled environments are crucial to validating the effectiveness of social navigation. These experiments reveal the challenges and complexities that arise in real-life situations, providing valuable insights into how robots can improve their navigation skills.

The use of databases to calibrate or train the system is usually necessary in all methods examined. These datasets are usually obtained by recording pedestrian trajectories with cameras or sensors in specific environments, or are collected by experts. Relying solely on recorded datasets may not be good practice, as the goal is to generalize pedestrian behavior, which requires a variety of datasets from different sources. In the case of proactive methods, the use of real data is common. For example, the Edinburgh Informatics Forum Pedestrian Database was successfully used by Ferrer and Sanfeliu [48,74]. Others, such as Luber et al. [127] and Ferrer and Sanfeliu [74], used The Freiburg People Tracker. However, this is not the case, as mentioned above, for learning-based methods. These usually describe the tools used to generate virtual learning environments, which enable them to obtain the volumes of data needed to successfully complete the training. For these methods to be successful in real environments, the data used for prediction or learning must be as realistic and diverse as possible. Therefore, it is crucial to collect data sets from a variety of sources and experts.

Two relevant topics have begun to be taken into account in the most recent proposals. On the one hand, the possibility for the robot to handle different behavioral options, and that, depending on the context, the robot itself self-adapts its behavior. Self-adaptation is a topic widely addressed in robotics, with specific proposals related to navigation and including parameter reconfiguration, algorithm changes, or even reconfiguration of the architecture itself [123,124]. Although it has not been analyzed in depth in this survey, some references were added to Section 3.4. On the other hand, navigation has started to be considered as a task that is not only about finding a path free of obstacles, or that this path is as close as possible to the one a person would follow, but may require specific skills that we would not include a priori in a navigation stack. Thus, to avoid getting stuck in an environment densely crowded with people, the robot may need to talk to people, or even push them lightly. This multi-modal collaboration scenario can be useful for a robot to become more socially aware, allowing it, for example, to greet people it crosses paths with.

5. Conclusions and Future Work

This survey analyzes the problem of robot navigation in every day, crowded environments. The analysis of recent studies highlights the advancements in robot navigation, particularly in the area of social navigation. Although purely reactive proposals were presented, when a mobile robot is deployed in an environment where people move around freely, it becomes necessary for this robot to predict the movement of these other agents. The use of prediction enables the robot to quickly adapt and optimize its navigation, To generate these predictions, navigation algorithms have evolved from a human–robot interaction scenario to a human–robot cooperation one, where it is expected that people will proactively help the robot to find a free and safe path. However, the complexity and variety of human behavior in the real world can make this assumption fail. Recent approaches propose that the mobile robot can interact with people not only because their paths may cross on the map, but more actively, through gestures, vocalization and touch, to require their help in navigating [16,120,121]. As a signaling mechanism for conveying an intention to humans, incorporating features such as body posture and gestures also contributes to making the robot appear more friendly and predictable to humans, leading to better human–robot interactions and an overall improved experience.

Future work in this field should focus on thoroughly reviewing existing experiments and exploring ways to further improve robot performance. It is also important to keep abreast of the latest developments and advances in this field. In addition, as mentioned above, it could be interesting to analyze methods using record data and methods using test databases and to analyze the behavior of the methods in real environments.