Image Recognition-Based Architecture to Enhance Inclusive Mobility of Visually Impaired People in Smart and Urban Environments

Abstract

The demographic growth that we have witnessed in recent years, which is expected to increase in the years to come, raises emerging challenges worldwide regarding urban mobility, both in transport and pedestrian movement. The sustainable development of cities is also intrinsically linked to urban planning and mobility strategies. The tasks of navigation and orientation in cities are something that we resort to today with great frequency, especially in unknown cities and places. Current navigation solutions refer to the precision aspect as a big challenge, especially between buildings in city centers. In this paper, we focus on the segment of visually impaired people and how they can obtain information about where they are when, for some reason, they have lost their orientation. Of course, the challenges are different and much more challenging in this situation and with this population segment. GPS, a technique widely used for navigation in outdoor environments, does not have the precision we need or the most beneficial type of content because the information that a visually impaired person needs when lost is not the name of the street or the coordinates but a reference point. Therefore, this paper includes the proposal of a conceptual architecture for outdoor positioning of visually impaired people using the Landmark Positioning approach.

1. Introduction

In recent decades, we have witnessed an exponential demographic growth of cities, which substantially increases the difficulties of urban planning in terms of mobility, which presents itself as an emerging challenge in all countries [1]. Whether in terms of transport or pedestrian navigation, mobility has taken on a new dimension and represents a problem to which many researchers respond, focusing on the sustainable development of cities. Infrastructures and city design are constantly changing to respond to the daily displacement needs of millions of people from the outskirts to city centers [2]. Sustainable development will be difficult to achieve with a large number of private vehicles and the current public transport network [3]. The mobility as a service (MaaS) paradigm emerges as an innovative concept that intends to be a strong ally towards sustainability, which is user-centric and combines different means of transport to minimize CO₂ emissions and, at the same time, maximize the use of sustainable transportation and/or means of active mobility.

Navigation within cities is a daily activity inherent to all citizens, performed several times throughout the day for commuting to work, schools, shopping, and leisure purposes, among others. For all those in a well-known city, the navigation turns out to be intuitive, something simple that does not pose any significant difficulties. However, when we are in an unknown city, the aspects of orientation and mobility become more relevant. The orientation task in a city involves several elements, such as positioning, navigation, and direction. Urban spatial orientation is, in turn, intrinsically linked to the aspect of mobility, which can be described as the ability of someone to move in an environment, known or not, and which may contain several obstacles that are related to how citizens are informed about their position or how to navigate to the desired location [4]. Suppose several challenges arise for urban mobility in general terms, such as precision. In that case, additional questions are brought up when we talk about visually impaired people (VIP) that are immediately noticed by the different ways VIP use their senses to orient themselves [5]. The World Health Organization (WHO), in its report on “Visual Impairment 2010”, refers to approximately 320 thousand people per million with some visual impairment worldwide, and about 47 thousand people per million are considered blind [6]. In Portugal, the 2001 reports had values of 160 thousand visually impaired individuals [7], while the 2011 reports point to 900 thousand visually impaired individuals, 28 thousand of whom are blind [8]. Although the mobility challenges for this segment are varied, we will focus, throughout this paper, on the problem of positioning when a VIP loses track of where they are, which represents one of the main challenges they feel and experience in their daily routines. Receiving information that they are close to a particular store, near a given known intersection, etc., can be decisive and make all the difference for a VIP’s orientation [9,10].

One of the techniques that could be used would be the GPS combined with the information on Google Maps. Still, the information usually available does not have the necessary refinement and detail because the points of interests (POIs) inserted are typically thought of more globally and not with all the detail VIPs need. This paper proposes a conceptual framework for an outdoor positioning system for VIP that uses Landmark Positioning, an approach that fits inside the broad concept of Visual Positioning. The framework assumes a VIP with a smartphone camera that captures the image of surroundings as the VIP is walking and a backend server that can process those images and get information about its positioning, which is informed to the VIP. The approach mainly uses images with landmarks that represent locations in the cities and, for that reason, constitute helpful information for a VIP [11].

The paper is organized as follows. Firstly, Section 1 describes the problem context and its main implications for defining a proper solution. Section 2 presents a literature review on smart mobility and its main challenges, trends, and inclusiveness. Section 3 offers the related work regarding the VIP. Section 4 presents an architecture proposed for Outdoor Positioning for Visually Impaired People using Landmarks (OPIL) framework and model. Finally, in Section 5, the conclusions and future work are addressed.

2. Urban Mobility Challenges, Trends, and Inclusiveness

The advent of technologies, systems integration, and the era of Big Data and Artificial Intelligence, among others, gave birth to the concept of intelligent “things”. Smart cities are born from integrating all these concepts and the possibility of their application to different areas of intervention and domains. According to [12], there are six main axes/domains of interventions within a smart city: Government, Mobility, Environment, Economy, People, and Living. The world population has been growing significantly in recent years. In 1974, the world population numbered approximately 4 billion people, 6 billion in 1999, 7.8 billion in 2020, and an increasing trend according to United Nations projections, presented in Figure 1.

Figure 1. World population evolution [12].

This growth undoubtedly challenges cities at various levels in terms of urban mobility. The congestion problems arising from the high population density can be partly solved with efficient transport networks. Still, the current proposals do not stop there since the concept of smart mobility goes further and calls for more innovative solutions [13]. These include recommendations for sustainable mobility solutions, such as active transport that use environmentally friendly fuels and citizens’ accountability [14,15]. Mobility must be seen as a path that allows for increasing sustainable development. The impact of smart mobility may, at first, seem to focus on solving what appears to be the most obvious problem: traffic congestion. But the impacted dimensions are much higher and encompass sustainability, economy, and living and, therefore, with a direct impact not only on citizens but also on governmental entities [13]. The “smart” character of mobility can be seen in various ways: public access to information and making it available in real-time to make travel planning more efficient, save time and money, and reduce CO₂ emissions [16,17].

Several challenges and barriers to providing inclusive and equitable mobility services amongst vulnerable user groups, such as people with disabilities and impairments, arise. These commonly include poverty [18] budget constraints, lack of accessibility, inadequate service integration, lack of correspondence between user needs and service provision, preliminary safety measures or unsafe infrastructure, and lack of access to or understanding of technology [19,20]. Another common problem is improper parking in places reserved for people with disabilities or even in public spaces in such a way that interfere with the mobility of people with reduced mobility. An example comes from the growing use of e-scooters in the urban context, which, in the absence of the proper legal and safety framework, translates into parking on public roads and sidewalks in ways that impair the pedestrian movement of people with disabilities, with considerable impact on blind and partially sighted people who encounter unexpected obstacles [21].

It is acknowledged that challenges to providing mobility services while affecting different users are also often related to an area’s geography and economic characteristics and have additional prominence in other prioritized regions. Therefore, better communication with communities of need is necessary, particularly during planning processes. Taking a more inclusive, communicative approach to mobility planning will allow for better, more efficient mobility plans that more robustly serve the needs of the most vulnerable users [22].

Spatial and temporal coverage of services needs to be considered for all users. While often centered on ‘peak hour’ needs, shifting employment patterns, changing family demographics, and more dispersed activity-based lifestyles require that better mobility services coverage is needed in terms of both space and time to ensure that all users have equal access to opportunities that may not fit into traditional working/leisure hours.

Better service integration is needed [23] with more heavily dispersed populations, it and is unlikely that a one-seat trip provided by one operator will adequately serve the needs of all users. Integration, however, should be a multidimensional consideration, encompassing integration of services (including nontransport services), payment, and information, thus promoting the transition to a fully mobility as a service (MaaS) system [24,25,26].

A more effective information-sharing process should be developed to fully serve all users’ needs. Information should be accurate and reliable, accessible from various platforms (including digital and paper) and in different formats (including, for example, images, text, and multiple languages), and secure to guarantee its consistency and confidence to all users [27]. New service provisions should be explored, including considerations beyond traditional mobility services or systems and incorporating walking, cycling, car-share, carpool, taxis, and other services [28]. Some resistance is expected as innovation pushes towards a shared transport model. A study presented in [29] shows that car-sharing service providers have shown the greatest resistance to this adaptation, contrary to e-bike-sharing services that are more open to this change.

Adequate financial and policy support is needed from the local and national governments. Namely, financial concerns are the primary concern regarding the cost of transport for users and the cost of providing service to vulnerable populations and areas. The cost was identified as a significant barrier to mobility services and transport use, indicating that more considerations should be made for ways to offset its impact.

The transportation system must always be planned and designed to ensure maximum inclusiveness for all members of society, especially those with disabilities, seniors, and children/youth, without limiting or restricting their mobility options [30], with the final purpose to make sure all means of transport are prepared to receive every citizen without any limitation or exclusion, thus promoting a more inclusive society.

Therefore, developing solutions for all these cases should be highly prioritized, namely for ensuring the embedding of new intelligent assistive technology conceived for easing the navigation and routing experience of blind and visually impaired people throughout their daily commuting safely and independently [31,32,33].

3. Related Work on VIP

Visual assistive technology can be divided into three main categories: vision enhancement, vision substitution, and vision replacement [34,35]. Due to the fast presence of sensors and their functions in almost every activity of society, these systems are becoming increasingly available in terms of applications and devices, providing different services, such as localization, detection, and avoidance. Sensors can help navigate and orient VIP and give them a sense of their external environment. Sensors can identify any surrounding object property and facilitate VIP’s mobility tasks [36]. Vision replacement is the most complex category, as it is related to medical and technological issues. In such systems, sensor data will be sent to the brain or a specific nerve. Vision replacement and vision enhancement are comparable yet different. In the vision enhancement category, the process data that is sensed by a sensor will be displayed in a device. In the vision substitution category, the result of any data sensed by any sensor will not be displayed, and the output result will be acoustic, tactile, vibrating, or a combination. This makes the vision substitution systems very similar to intelligent transportation systems and mobile robotics.

This research work is focused on the visual substitution category, which can be further subdivided into three other categories: electronic travel aids (ETAs), electronic orientation aids (EOAs), and position locator devices (PLDs). Table 1 briefly presents each visual substitution category subcategory and its services.

Table 1. Visual substitution subcategories.

Subcategory Name	Description	Services
Electronic Travel Aid (ETAs).	Collected and sensed data about surrounding areas are sent to the user or a remote server via sensors, laser, or sonar.	Surrounding obstacles are identified. Texture and gaps on the surface can be provided. The distance between the user and an obstacle is determined. Great locations are identified. Self-orientation throughout an area is improved with obstacle avoidance information.
Electronic Orientation Aid (EOAs)	Guidelines and instructions about a path are given to the user through a device.	The best path for a particular user is determined. Clear direction and path signs are given by calculating the user’s position and tracking the path.
Position Locator Devices (PLDs)	The user’s location is identified, for example, using the global positioning system (GPS)	Guidance from one point to another point is given.

3.1. Navigation of Visually Impaired People

A study involving VIP and control groups aims to assess the ability to travel and learn routes and the consequent capacity for mobility in an urban environment [37]. The results show that VIP can learn complex routes efficiently when assisted and guided. They experience greater difficulties when they are expected to do so autonomously. In [38], a system for navigation in outdoor environments for VIP is proposed using a geographic information system (GIS) as a fundamental supporting system. Given the current position, the GIS allows the system to find a route to the desired destination, thus supporting the VIP in its movement along the route. Information about the user’s positioning is obtained through a GPS receiver, a magnetic compass, and a gyrocompass. System communication with the VIP uses a keyboard and voice messages. In [39], the authors present an app developed to help VIP move autonomously, identifying obstacles that may exist along the route through the smartphone’s camera. The Walking Stick app fulfills three objectives: (1) to identify obstacles, (2) to allow the app to run without having to open it explicitly (it is accessible through shaking movement), and (3) safety requirements by detecting falls from the smartphone and emitting a sound so it can be found. The application considers development requirements to avoid rapid battery consumption. In [40], the X-EYE app is presented to promote VIP independent mobility, using a wearable camera which reduces the cost of the solution. The application allows the detection of obstacles, people recognition, tracking and sharing of location, SMS reader, and a translation system. The authors indicate some limitations of the solution, such as the partial appearance of images and rapid changes in the background or lighting conditions. In [41], the authors introduce a navigation system for VIP with the main objective of addressing obstacle detection, which is useful for navigation in outdoor and indoor environments. The system uses ultrasonic sensors that, together with a buzzer and a vibrator, can be placed on a jacket or other garment to be easily included in the daily life of the VIP and, therefore, more easily be accepted as an accessory that is easily transported. In [42], a system for indoor navigation using the computer vision approach is presented. The user must use a camera in their hand while navigating the environment prepared with fiducial markers augmented with audio information. The system has two modes of operation. In the first mode—free mode—the user freely navigates through the pre-prepared environment and is informed of the environment around them through audio and the detected markers. In the second mode—guided mode—the user navigates from the current location to the desired destination using the shortest path (using the Dijkstra algorithm). In [43], a support system for indoor navigation is presented, consisting of a navigation system and a map information system installed on a cane. The system works on the assumption that there is a colored navigation line on the pavement. The cane has a color sensor that, when leaning against the line on the floor, manages to inform the user through a vibration system that they are moving over the line. A one-chip microprocessor controls the color recognition system. The map information system uses RFID tags along the line on the floor and RFID receivers placed on the cane, allowing a navigation system from a given source to a given destination to be made available. In [44], the authors present a navigation system that detects obstacles and guides VIPs through the most appropriate paths. Obstacles are detected using an infrared system, and feedback is sent to the user through vibration or sound to inform them of their position. Aware of the limitation of canes regarding protection at the head level, the system has incorporated a sensor in a cap (and, therefore, at the level of the user’s head area) that allows information about obstacles and navigation. In [45], the authors present a navigation system that aims to fill the gaps in the GPS (only valid in outdoor environments). Its main focus is to provide the VIP with accurate information about its spatial positioning. The proposed system can be used in an indoor or outdoor environment and uses the camera carried by the user and computer vision techniques. As the user follows a previously memorized path, the camera’s image is recognized to calculate its direction and positioning precisely. In [46], the authors propose a system that intends to complement the use of the cane when traveling in indoor environments. The system consists of a geographic information system related to a building that contains information about visual markers that allow the system to create a mental map of the environment and later allow navigation in space. In [47], a system is proposed to help VIPs move around outdoors with the possibility of detecting obstacles. The system, which aims to combat the limitations of other solutions in terms of cost, dependency, and usability, makes use of a mobile-based camera vision system to allow great usability in the use of the system in places with which the VIP is not familiar, such as parks, streets, among other places. Deep learning algorithms are used to recognize objects and interact with a mobile application which is also part of the system. In [48], a system is proposed based on a mobile application that stores important information for the VIP during the route, making their journey easier and safer simultaneously. This concept goes against generalized navigation, favoring a personalized recommendation based on the information each person prefers to receive, based on the most favorable input for them to feel located and safe. The system uses voice feedback consisting of multi-sensory clues combined with microlocation technology. In [49], the authors propose an android app to assist in indoor navigation based on transmitting information to the user via audio messages and the existence of QR codes. The system takes predesigned paths for VIPs along which QR codes are placed to infer the position and recommend the quickest and shortest route to an intended destination. The system allows detecting deviations in the route from what is expected and informs the user of the deviation that has occurred. In [50], the authors propose a navigation system based on simultaneous location and mapping (SLAM) to help VIPs navigate indoor environments. This system integrates sensors and feedback devices, including an RGB-D sensor and an inertial measurement unit (IMU) on the waist, a camera on the user’s head, and a microphone together with a headset. The authors developed a visual odometry algorithm based on RGB-D data to estimate the user’s position and orientation, refining the orientation using the IMU. The head-mounted camera recognizes port numbers, and the RGB-D sensor detects important landmarks, such as corners and corridors. In [51], the authors describe an integrated navigation system with RFID and GPS, entitled Smart-Robot. The system uses location based on RFID and GPS operating, therefore, both in indoor and outdoor environments, respectively. The system uses a portable device composed of an RFID reader, GPS, and an analog compass to obtain location and orientation information. During navigation, the system avoids obstacles using ultrasonic and infrared sensors. User feedback is made through vibration in a glove and loudspeakers. In [52], the authors propose a VIP system that can operate indoors and outdoors. The system uses an ultrasonic sensor for navigation and voice outputs to prevent obstacles. Table 2 summarizes the challenges and approaches.

Table 2. Summary of challenges and approaches of navigation systems for visually impaired people.

Reference	Challenge	Approach
[38]	Outdoor navigation	SIG, GPS receiver, magnetic compass, and gyrocompass
[39]		Smartphone camera, accelerometer
[47]		Deep learning, camera vision-based system
[48]		Voice feedback, multi-sensory clues, microlocation technology
[44]	Obstacle detection and navigation	The infrared sensor in a cap for head protection
[40]	Obstacle detection	Wearable camera connected to the smartphone
[41]		Ultrasonic sensors, vibrato rand buzzer
[42]	Indoor navigation	Webcam, Dijkstra Algorithm, Computer Vision
[50]		SLAM, inertial sensors, feedback devices
[43]		Walking stick, color sensor, colored floor, RFID tags
[49]		QR codes
[46]		GIS, visual markers
[45]	Indoor and outdoor navigation	Camera, computer vision
[51]		RFID, GPS, ultrasonic and infrared sensors
[52]		Ultrasonic sensor

3.2. Outdoor Positioning Using Landmarks

In [53], the authors present an indoor and outdoor navigation system based on landmarks and artificial intelligence. Template matching techniques are used to find patterns to estimate the distance to the landmark. Several techniques are used, such as high boost filtering and histogram equalization, to improve the images for the processing process. The dynamic exclusion heuristic converts the estimated distance into the actual space [54] and presents a system that uses QR codes displayed in registered landmarks. The system thus allows the user, based on QR codes, to identify their position and, on the other hand, to have access to the navigation functionality based on landmarks. The system includes images of each landmark, allowing users to navigate freely using these images without needing guided navigation. In [55], the authors present a system for localization in outdoor environments using landmarks based on deep learning. The proposed location method is based on the faster regional convolutional neural network (Faster R-CNN), which is implemented for the detection of landmarks in images, and the feed-forward neural network (FFNN), which is used to retrieve the location coordinates and compass directions for the device implemented in the real world based on landmarks detected from R-CNN. In [56], the authors report on how to drive a mobile robot using natural landmarks such as trees and plants on Japan’s University’s outdoor campus. An essential component of the system is the acquisition of natural landmarks being proposed for the landmark agent (LmA). This component has three states: SLEEP, WAKE UP, and TRACK. The SLEEP status means that there is no candidate for a landmark. WAKE UP means that a candidate for a reference point has been detected, and the agent begins to locate it, and finally, the TRACK state means that the object is repeatedly detected two or more times. When you make the transition order SLEEP-WAKE UP-TRACK, the reference point is detected and recorded on the LmA map. Another component of the system is autonomous navigation using the acquired natural landmarks. The navigation strategy uses the perceived route map (PRM) generated by automatically acquiring natural landmarks through the teaching of routes. In [57], the authors propose an approach to improve the information provided by most navigation systems that are based on information based on street names and step-by-step instructions. The authors suggest improving this information based on landmarks collected and inserted in the recommendation system to provide information that the user can more easily identify.

3.3. The Role of Machine Learning in Image Recognition

In this section, we highlight some of the research being carried out regarding the use of machine learning algorithms and techniques in the field of image recognition, which has been a field of research for the past two decades. These algorithms can recognize landmarks and help VIP gain positional awareness within an urban environment. Those landmark images can be acquired in real-time from a smartphone camera or other mobile devices. The identification can be read out loud with the help of accessibility tools such as Google TalkBack.

To solve this problem, Yeh et al. propose an image search approach in web search engines [58]. In this approach, location recognition is performed not only through images but also through text. The process includes three phases: image search on Google to identify similar images, collection of keywords from pages where similar images are found, and a new search on google with the keywords obtained to obtain similar images. The technique used is a hybrid search for images and keywords. Li and Kosecká present a method of probabilistic location recognition for visual navigation systems [59]. The proposed system has two phases: in the first phase, the environment is divided into several locations categorized by key points; in the second phase, the key points are used to estimate the most likely location. This process demonstrates a high localization rate using only 10% of the characteristics of an image. To perform location recognition in large datasets, Schindler et al. present a method that uses a generalized version of the vocabulary tree search algorithm [60]. This method allows using datasets with ten times the number of images. The algorithm recognized 20 km of the urban landscape, storing 100 million characteristics in a vocabulary tree. Gallagher et al. use a K-nearest-neighbor algorithm in a dataset with more than 1 million georeferenced photographs. This method proves to be more effective in determining the location of an image than using only the visual information of an image [61]. One of the biggest challenges for visual localization is the search for images similar to the original in a database with georeferenced images. For the visual location to be carried out, the objects must be identified in different poses, sizes, and background environments. Faced with this challenge, Schroth et al. use a “bag of words” quantization and indexing method based on a tree search [62]. Cao and Snavely propose a method of representing images from a database in a graph [63]. The method selects a group of subgraphs and creates a function that calculates the distance between each group. Each image is classified according to this distance function. A probabilistic method to increase the diversity of these classified images is also presented. The paper states that this method improves the performance of location recognition in image datasets compared to “bag of words” models. Wan et al. present research about using neural networks to find similar images based on the content. This work investigates a convolutional neural network with several changes in its definitions. Convolutional neural networks are very effective in recognizing images [64]. Liu et al. use a semi-supervised machine learning model using a convolutional neural network. The image dataset was created with images from surveillance cameras and smartphones and took into account the changes that variations in the weather can cause in the panorama of an image [65]. Zhou et al. present research on the methods used to obtain images based on the content. In this study [66], searches for images are grouped according to the method used: by keyword, by example image, by sketch, by color layout, and by concept layout. Zhou et al. highlight two pioneering studies carried out around the 2000s. The first is the introduction of local visual invariant characteristics capable of capturing the invariance to the rotation and resizing of images. The second work highlighted was the introduction of the “bag of words” model capable of creating a compact representation of images based on the number of local characteristics present. It has been revealed that recently the convolutional neural networks have shown an above-average performance compared to the other methods. In this work, the authors explain that convolutional neural networks are a variant of deep neural networks and are usually used in recognition and image acquisition. In [67], Tzelepi and Tefas propose a convolutional neural network model to obtain image characteristics representation. Subsequently, the neuronal network is again trained to create more effective image descriptors. In this method, three different approaches are proposed: in the case that no information is provided, in the case the classification of the images is provided, and in the case, user feedback is provided. The proposed method outperforms all other unsupervised methods used in the comparison. To reduce the time spent obtaining images based on the content, Saritha et al. present a deep learning method [68] capable of extracting the properties of an image and classifying them according to those properties. In this work, properties such as color histogram, image edges, and texture are taken into account. The extracted properties are saved in a small file for each image, where files with similar properties must belong to similar images. The images’ properties are extracted using a convolutional neural network.

3.4. Convolutional Neural Networks for Image Recognition

With recent advances in machine learning, the most used solution for image recognition is neural networks, specifically convolutional neural networks. We next explain how artificial neural networks work and their more specific subgroup of convolutional neural networks.

One of the types of machine learning algorithms, neural networks, demonstrates a remarkable ability to learn complex relationships between objects belonging to a large amount of data. Neural networks are also valuable for the analysis of unstructured data. Examples of unstructured data are text, audio files, and images [69]. However, neural networks can be difficult to program, train, and evaluate. There are several frameworks capable of facilitating this work for a programmer. Examples of these tools in Python are TensorFlow to deal with neural networks in a more particular way or Keras, which works on top of TensorFlow, offering a higher level of abstraction. Neural networks are composed of an input layer that receives the input data, an output layer that classifies the data, and one intermediate layer that processes the information. A simple neural network has only one middle layer, whereas a deep neural network has two or more layers. The unity of neural networks is called a neuron, and the connection between each neuron is called a synapse (Figure 2) [70].

Figure 2. Example of the operation of a deep neural network (adapted from [67]).

Convolutional neural networks are the most used neural networks in image processing. Like neural networks, they are also composed of “neurons” that receive input and process them using activation functions. However, unlike the layers of neural networks, convolutional neural networks use images as input, not data vectors [69]. Convolutional neural networks have several applications: they can identify pictures, group them by their similarity or recognize objects and places portrayed in the image. The components of a convolutional neural network are as follows [69]:

• Data input
• Convolutional layer;
• Pooling layer;
• Flattening.

Image input is the first component of a convolutional neural network. In this component, the image is converted into a matrix (Figure 3), where each value represents the color of each pixel in the image [69].

Figure 3. Example of inputting images into a convolutional neural network (adapted from [66]).

The convolutional layer is where image processing begins and consists of two parts:

• Filter;
• Characteristics map.

The filter is a matrix or pattern combined with the original image’s matrix to obtain the feature map. The feature map is the reduced image produced by the seizure between the original image and the filter (Figure 4). In a real convolutional neural network, several filters are used to create multiple feature maps for each image [69].

Figure 4. Example of the convolutional layer in a convolutional neural network (adapted from [66]).

The pooling layer helps the network to “ignore” the least important data in an image, further reducing the image and preserving the most critical features (Figure 5). This component consists of functions such as “max pooling”, “min pooling”, and “average pooling”.

Figure 5. Example of the pooling layer in a convolutional neural network (adapted from [66]).

Flattening is the component where the image is arranged to serve as an input to a shared neural network. In this part of the process, the matrix is “flattened” and converted into a single column (Figure 6).

Figure 6. Example of flattening in a convolutional neural network (adapted from [66]).

4. Proposed OPIL Framework

4.1. Overview and Architecture

The architecture proposed for Outdoor Positioning for Visually Impaired People using Landmarks (OPIL), presented in Figure 7, aims to allow a VIP to obtain information on the location where they are positioned, one of the scenarios that represent a challenge for this segment of people when they move autonomously in a city. The architecture comprises two main components: a mobile application that the VIP must use and a backend server that does the image recognition processing and obtains the information about a helpful description that can be transmitted to the VIP so they can orient themselves.

Figure 7. Core components of the proposed framework.

In the architecture proposed for OPIL, it is assumed that each VIP has a mobile phone and uses an app that is one of the components of the system. This app allows the capture of the image through the mobile phone’s camera, which helps collect information about where the user is. This app must consider the battery consumption and the processing power it takes from the mobile phone. The user must keep the mobile phone fixed for some time, and only when they get the result sent by the backend as a result of processing its positioning (in some cases, the location may not be identified) can they move the mobile phone to another location. Whenever the backend receives an image, a comparison is made with all the images in a prepopulated and georeferenced image database. If the image is successfully identified, a description is returned by the algorithm in such a way that it is helpful for the VIP.

4.2. Components

4.2.1. Mobile Application

The mobile application represents the process’s starting and ending point, as seen in Figure 8. The VIP triggers the process when the user opens the app and points the camera to the place it is facing, allowing the image collection. The image capture module (ICM) is responsible for presenting the UI that captures the image, collecting it, and sending it to the backend server communication module (BSCM), which is responsible for communicating over the Internet with the backend server that will process the image (this process will be described in more detail in the next section). As soon as there is a response from the backend server, the callback method module (CMM) is triggered automatically to be able to process the response and send it to the notification module (NM), which is responsible for the interaction with the VIP to inform the user of the result about their positioning.

Figure 8. The architecture of the mobile app component.

4.2.2. Backend Server and Proposed Algorithm

The proposed algorithm for image recognition is divided into three phases (Figure 9). In the initial phase, datasets are obtained where all points of interest necessary for the project (from different angles, light levels, weather conditions, etc.) are photographed and identified. The second phase trains the model using a convolutional neural network. In this phase, the identification attributes are the images of the photographed objects, and the target attribute is the identification of these objects. The final phase corresponds to the trained model’s use in identifying new images of the objects.

Figure 9. The 3 phases of the algorithm for the recognition of points of interest.

The first phase corresponds to data collection, where images are gathered. Later, the identification model is trained using those same pictures. The collection of images must have an equal number of images with points of interest and those without points of interest so that the model’s training is not biased. Images without points of interest should be similar to images with points of interest. All images must be changed to have the exact dimensions. The image distribution should be about 80% training and 20% test images. Changing image properties such as opacity, converting them to black and white, or reducing the definition may be necessary to reduce model training time at the expense of accuracy. An image marking tool may be required to identify and select points of interest in the photos manually. Some examples of these tools are LabelImg (https://github.com/tzutalin/labelImg (accessed on 1 July 2022)) and OpenLabeling (https://github.com/Cartucho/OpenLabeling (accessed on 1 July 2022)).

The second phase corresponds to the training of the convolutional neural network. In this phase, the model is trained to differentiate the landmarks based on their geometric-functional typologies, such as punctual elements (e.g., monuments, stores, schools, city hall), linear elements (e.g., rivers, roads, sidewalks), or areal elements (e.g., squares, green parks). When creating the model, there are some general guidelines to follow. The use of convolution and pooling should be alternated, with convolution initially being used and pooling at the end before connecting to the neural network (e.g., convolution-polling-convolution-pooling-neural network-output). During convolution, there is little need to use a filter more significant than a three-by-three matrix. It is advisable to use max pooling along the neural network and average pooling at the end before connecting to the neural network layer. Generally, the more layers a neural network has, the greater its accuracy and execution time. Preprocessing of the images should be done only if necessary to increase the accuracy or speed of the model. Creating new pictures with existing images (data augmentation) almost always helps improve a model’s accuracy. The number of nodes in the first intermediate layer of the neural network should be half of the nodes in the input layer, and in the second layer, half of those in the first layer. The number of nodes in the neural network’s output layer must equal the number of classes identified by the network. The number of nodes in the middle layers of a neural network must follow a geometric progression (2, 4, 8, 16, 32, …).

The third phase corresponds to the use of the model. A smartphone application must be created to obtain images from the camera and send the data to an external API. If access to the Internet via mobile data is impossible, then the pretrained model must be integrated into the application. To be able to use the android model, the pretrained model must be in the “.pb” format (TensorFlow) and must be converted to this format if it is in the “.h5” format (keras). Suppose the solution is used in more than one geographical area. In that case, it is advisable to add the GPS coordinates to the characteristics matrix of the images that serve as input data to the convolutional neural network. Suppose the solution is used in more than one geographical area. In that case, the smartphone application must also obtain the GPS position when taking images, which must be used to forecast the point of interest.

The model’s usefulness will depend on the imageability of the urban environment. This is because, and according to Lynch, an imageable city has, in principle, a better degree of legibility [71]. Lynch also states, in his 1960 article [71], that imageability is the “quality in a physical object which gives it a high probability of evoking a strong image in any given observer. It is that shape, color, or arrangement which facilitates the making of vividly identified, powerfully structured, highly useful mental images of the environment”.

4.3. Framework Security Aspects

The preferred framework is supported by mobile devices and other digital technologies for its operation, i.e., cloud computing. The technological development and the sophistication of such devices usually reveal unexpected and underestimated security vulnerabilities. As there is an increased reliance on technology, it is also necessary to increase efforts to protect the technological resources and infrastructure, ensuring that they operate safely and uninterruptedly.

As the amount of data collected increases, new systems and applications are integrated, the number of users increases, and many that lack computer skills, the potential occurrence of information security incidents, breaches, and threats also increases. Thus, it is almost mighty to have adequate security services and systems, as well as good practices, to manage all the interactions within the framework; being fundamental to have mechanisms that ensure the confidentiality, integrity, and availability (CIA) properties of the framework operation and information [72].

As a smartphone supports the proposed framework for capturing landmarks and surroundings, on the Internet for bidirectional communication, and in cloud computing for the backend server operation and image comparison, the CIA properties must also be guaranteed. To guarantee the CIA properties in the proposed framework, a cybersecurity layer in the form of a mobile threat defense (MTD) system must be considered. MTD can be viewed as sophisticated, dynamic protection against cyber threats targeted against mobile devices, and the safety is applied to devices, networks, and applications [73]. This will maintain the security information of the data generated and received by the smartphone (or another mobile device), protecting against malicious applications, network attacks, and device vulnerabilities [74].

For the backend server, a cloud service system can be used. As the information and data, the server manages, and stores are compassionate. If the CIA properties are not met, this can cause a high-level impact on the users and organizations that use the proposed framework, possibly causing a severe or catastrophic adverse effect on organizational operations, organizational assets, and individual users. It is suggested to use the security as a service (SecaaS) provided by the cloud provider. This is a package of security services offered by a service provider that offloads much of the security responsibility from an enterprise to the security service provider. The services typically provided are authentication, antivirus, antimalware/spyware, intrusion detection, and security event management [75]. SecaaS is considered part of any cloud provider’s software as a service (SaaS). A segment of the SaaS offering of a CP. In a significantly more straightforward manner, the Cloud Security Alliance defines SecaaS as the provision of security applications and services via the cloud either to cloud-based infrastructure and software or from the cloud to the customers’ on-premises systems [76]. The Cloud Security Alliance has identified the following SecaaS categories of service: Identity and access management; and data loss prevention. The combination of the previously referred systems and services will improve the level of usage confidence of the framework and can be viewed as an adequate strategy to secure both the end user, that relies on the utilization of mobile devices, and server-side data and information (at rest and being used).

Cybersecurity is critical nowadays because safety, security, and trust are inseparable. The relative insecurity of almost every connected thing is top of mind due to the heightened awareness of the impact of cyberattacks and breaches, highlighted in the media and popular culture depictions of nefarious hacking exploits. This makes achieving security from the outset extremely critical to avoid becoming any computing service and application highlighted by this media attention [77].

5. Conclusions and Future Work

Urban mobility is currently in a massive evolving period due to the digitalization of society, the effects of the COVID-19 pandemic, the global economic dynamics, and the ongoing pattern of people concentrated in the urban area. The mobility challenges are one of the most visible and notable elements that profoundly affect urban metabolism and its decurrent effects. The solutions being developed towards ensuring more inclusive and universal access to the mobility system are essential to guarantee that visually impaired people could benefit from this reality and substantially improve their daily commuting activities and overall life in the urban ecosystem.

The presented solution addresses a specific problem related to the difficulty of orientation by a blind or visually impaired person when they lose their direction in an urban environment. Making use of the massive use of mobile phones, the solution assumes the existence of one of these devices to capture an image of the environment that surrounds the user to be able, through image recognition techniques, to obtain the user’s location, by comparison to a georeferenced image data and a trained model.

Some factors need to be addressed such as reducing the time to acquire data and the performance of the increased number of users that could use such a solution simultaneously in a city. These factors are decisive for the success of the solution.

This solution aims to contribute to more inclusive mobility in urban environments. The main limitations of the solution are mainly in terms of the scalability for large areas, although the potential for historic centers or smaller cities is promising. In future terms, we intend to create a functional prototype that allows the execution of tests in a controlled environment. The evolution to an approach that contemplates the cooperation of citizens, through crowdsourcing, is a possible way to overcome the scalability problems that the solution may have now.