Making Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies

: People with visual impairment face various difﬁculties in their daily activities in comparison to people without visual impairment. Much research has been done to ﬁnd smart solutions using mobile devices to help people with visual impairment perform tasks like shopping. One of the most challenging tasks for researchers is to create a solution that offers a good quality of life for people with visual impairment. It is also essential to develop solutions that encourage people with visual impairment to participate in social life. This study provides an overview of the various technologies that have been developed in recent years to assist people with visual impairment in shopping tasks. It gives an introduction to the latest direction in this area, which will help developers to incorporate such solutions into their research.


Introduction
Visual Impairment (VI), which results from various diseases and degenerative conditions, causes significant limitations in visual capability. VI cannot be corrected by conventional means [1]. Currently, more than 253 million people live with VI, and this number is projected to increase in the coming decades [2]. People with Visual Impairment (PVI) have limitations in the function of their visual system. These limitations prevent them from seeing and doing daily activities, such as navigation or shopping [3][4][5][6][7][8][9]. For example, PVI have difficulties in reading product labels during shopping; they thus miss important information about the content of their food and sometimes make bad choices. During shopping, PVI also face navigation troubles, which encourage them to consume takeout [10,11]. Another problem is how to walk in an environment with many barriers such as walking in unknown places or crossing a street [12,13]. Moreover, the lack of support services in their surrounding environment make PVI dependent on their families and prevent them from being socially active [14,15]. Last, but not least, PVI face social barriers such as the attitudes of other people and society [16]. Therefore, it is important to develop solutions to help PVI improve their mobility, protect them from injury, encourage them to travel outside of their own environments, and interact socially [17].
Recently, mobile devices, such as smartphones, smart glasses, and notebooks have become popular. These new devices have various capabilities that are useful in developing complex software applications. Mobile devices can also be connected to cloud computing and offload tasks to be executed there, which saves power, memory, and battery [18][19][20]. The advantages of mobile technologies make them useful for accessing information from any place at any time and give PVI the opportunity to use smartphones in their daily activities [21][22][23][24][25]. In this way, smartphones are used with Assistive Technology (AT) to offer multiple solutions; this technology is called Mobile Assistive Technology (MAT). Researchers have conducted extensive investigations on using MAT to help PVI navigate from one place to another and shop without any support from people without disabilities [26][27][28][29][30][31][32][33][34]. In this study, we concentrate on the available solutions to help PVI in the shopping process. We divided the shopping process into two parts. The first part is how to prepare the shopping list, which provides assistance during shopping, before shopping. The second part helps them navigate inside shopping malls and identify products during shopping.
The purpose of this literature review is twofold. The first aim is to answer the following research questions: The second aim is to overview the available MAT solutions about helping PVI prepare shopping lists, navigate inside shopping malls and recognize products during shopping. It also discusses how the proposed solutions can help PVI in the shopping process and summarizes the challenges and drawbacks of the proposed solutions. This paper is structured as follows: Section 2 describes the research methodology. Section 3 discusses multiple solutions and how they can help PVI. Section 4 explains the main benefits and research challenges. Finally, Section 5 outlines the conclusions of this study.

Research Methodology
In order to identify most of the available MAT solutions for PVI, we searched the following databases: Springer, Science Direct, Web of Science, Institute of Electrical and Electronics Engineers (IEEE) Xplore, Google Scholar, Association for Computing Machinery (ACM) Digital Library and Microsoft Academic. We used the following keywords to search for peer reviewed journal articles: ("Assistive technology" OR "Assistive technology devices" OR "Mobile assistive technology devices" OR "navigation solution" OR "shopping") AND, ("visual impairment" OR "blind *"), ("avoiding obstacles" OR "write * notes" OR "text to speech") AND ("visual impairment" OR "blind *"). We set the search period to articles published between January 2010 and December 2018. The search query returned 8893 records. Duplicates were removed, reducing the search results to 842 articles. Then, we eliminated 433 results by restricting to articles in the English language, articles that describe research intervention for PVI based on their titles, and articles that are free and downloadable. Next, all keywords were screened, which eliminated 206 articles, because they were not technical papers, or they were literature reviews or surveys.
Abstracts of the resulting 203 papers were then screened for relevance to our research goals. One hundred and thirteen of the articles were deemed inappropriate because they did not study visual impaired or blind populations, or they were not related to MAT. Next, two different researchers conducted a full text article review of the remaining 90 articles. Forty-six articles were eliminated due to not helping PVI in the shopping process and avoiding obstacles. The resulting 44 articles met all inclusion criteria and were evaluated in this study. Figure 1

MAT Solutions for PVI Shopping
When entering into the world of PVI, one should be aware of certain obstacles PVI face while shopping alone, particularly when the support from shop assistants is limited [26,34]. Many retailers offer online shopping, but this method is difficult and time-consuming, as PVI have to listen to all the choices before choosing the product. Even worse, if any items are missing, they must re-listen to the list again [36]. Some shops offer home delivery, but this option requires the PVI to make an appointment and wait for delivery. These alternatives limit personal autonomy and make independent shopping difficult so, PVI often avoid using these services.
Buying a product in person at the grocery store is also difficult for PVI. They often wait for help from store employees, which is time-consuming. Moreover, most stores cannot assign an employee to help PVI, as hiring an assistant is too expensive and offers no privacy [34,37]. PVI also face difficulties when searching for a product on the shelves and checking their details, as the shopkeepers frequently move products around. Kostyra and co-workers found that sensory attributes are important for PVI when they select different products that have the same appearance. They performed a questionnaire, and the results showed that using mobile devices can make PVI feel independent during the shopping process, which is important to them [11].
We searched for available MAT solutions to help PVI before and during the shopping process. We found that the shopping process is divided into two parts: the first part is how to prepare the shopping list to make shopping easy and fast; the second part is dealing with how to navigate and identify products during the shopping process, as shown in Figure 2 [38][39][40][41][42][43][44][45][46][47]. It is also shown in the figure that MATs during the shopping process are classified into three categories, based on the technology that was used: tag based, Computer Vision (CV) based, and hybrid systems. The remaining part of this section gives an overview of technologies that have been developed in recent years related to each part in the shopping process. Moreover, it gives an in-depth look at these solutions and some research examples to show how they work.

Shopping Preparation
Earlier studies [8,11,48] assume that shopping list preparation is a useful activity, as most PVI prefer to follow a predetermined list. It is necessary to help PVI prepare shopping lists and store them in a database. During shopping, PVI can then retrieve and use their lists. Several applications take an image from a printed text, a written list, or Braille and send it to an Optical Character Recognition (OCR) module, which analyses and converts it to text [49][50][51]. In some cases, the system sends the image to a character recognition API in cloud computing, which identifies the words and converts it to text [52]. Other applications use Speech to Text (STT) techniques to transform PVI voice commands to a list of items [53]. After preparing the shopping lists, they are stored in a database. During shopping, PVI retrieve the shopping items from the database and listen to them by using Text-To-Speech (TTS) [51,54].

Navigation and Product Identification
Going to the store is not the only navigation challenge for PVI; in-store navigation is also a complex problem [24]. It is challenging for PVI to navigate inside shops and to reach and identify products. It is also difficult for them to get detailed information about products, such as production and expiration date. Additionally, PVI always need help from others, as most shops are not well prepared to help them [11]. As a result, researchers have developed multiple solutions to help PVI navigate and identify products. These solutions are divided into three categories, as shown in Figure 2: (1) Tag Based Systems: such as Radio-frequency identification (RFID) and Near Field Communication (NFC), which use wireless components to transfer data from a tag attached to an object for the purposes of automatic identification and tracking. (2) Computer Vision Based Systems: some of these systems require unique visual tags such as Quick Response (QR) codes, Barcode, or Augmented Reality (AR) markers to be placed on products. These tags are used for detecting and giving PVI all available details about the products. Other systems do not require tags to be placed on products. Instead, they utilize information about the objects' physical features to identify them. (3) Hybrid Systems: these take the strengths of two or more systems and combine them into a new system to deliver better accuracy and performance [9,[41][42][43][55][56][57][58][59][60][61][62].

Tag Based Systems
Tag systems use wireless communication technology to transfer data from the tag attached to an object to a tag reader for automatic identification and tracking. Developers are using several types of tags, but we will concentrate on RFID and NFC tags [63]. RFID uniquely identifies items by using radio waves. Each RFID system has three components: a tag, a reader, and an antenna. The reader sends a signal to the tag via the antenna, and the tag responds to this signal by sending its unique identification information [64]. There are two types of RFID tags: active and passive. Active tags broadcast a signal up to a range of 100 m. Passive tags use electromagnetic energy from the reader, but they do not have their own power source. They can broadcast a signal up to a range of 25 m. Passive tags are typically smaller and cheaper than active tags [65].
For shopping, RFID tags are attached to products and PVI hold tag readers to identify them. After identifying the items, PVI get details about them, such as name, price and special offers [55], [66][67][68]. These systems use a server or a local database to store product details. Other solutions use RFID tags for navigation inside shopping malls. These solutions use tag reader attached to white canes to identify RFID tags. They use a map to guide them through the store and suggest the shortest route for PVI to reach their products. They give audio messages that provide verbal directions [69,70].
NFC technology is a version of RFID that operates at a maximum range of about 10 cm. The NFC system consists of three main components: a tag, a reader, and a database. The tag is a place to store information, while the reader is an NFC mobile device that reads the content of the tag. The database stores additional information to the NFC tags. NFC technology is used for active or passive communication. In active mode, the NFC tag and NFC tag reader produce a radio field during communication. In passive mode, only the NFC tag reader generates a radio field and begins the communication session, so NFC tags do not process any information sent from other sources, and cannot connect to other passive components. The main differences are the communication distance and the use of mobile devices by NFC systems or RFID Readers by the RFID systems [33,71].
To use them for shopping, NFC tags are attached to products and identified by smartphones like RFID. After identifying the items, PVI get details about them [33,72]. These systems use a database to store product details so PVI can scan them and get information such as name, price, and special offers [37]. Some other solutions used NFC tags for navigation inside shopping malls. Like RFID, NFC tags use a map to guide PVI through the store and suggest the shortest route for PVI to reach their products. These tags also give audio messages that provide verbal directions [33,73].

Computer Vision Based Systems
CV based systems accept visual inputs from the camera and use CV techniques to extract valuable information and recognize objects in the surrounding environment. Finally, they provide information to the PVI through tactile or auditory channels [74]. Researchers classified CV based systems into tag based and non-tag based. In tag based systems, unique visual tags such as QR code, Barcode, and AR markers are placed on products to aid the recognition process. Recognition is accomplished by capturing an image of the tag and analyzing this image to determine the identity of the object based on its tag information. Then, they use a database to retrieve product details, such as name, price, and special offers [42,75,76]. Other solutions use these tags for navigation and giving the shortest route to products [77,78]. Finally, tactile or voice commands are used for warnings and providing direction commands to the PVI [79].
With non-tag based systems, developers do not attach tags to objects. They use CV techniques to analyze the images and identify objects [80][81][82]. Non-tag systems require extensive computational power to analyze images and give accurate results [44]. For example, Zientara et al. used smartglasses to accurately locate and identify objects using CV techniques [41,83]. They also used a glove with a camera, which guides hand movements to point and grasp things. Kumar and Meher used a color recognition module with a convolution neural network to recognize objects and their colors [84,85]. Jafri and co-workers processed the depth and motion tracking data obtained via a Google Tango Tablet to create and update a 3D box around the PVI to detect and avoid obstacles [43,86]. Hoang and co-workers utilized color images, depth images, and accelerometer information from a mobile Kinect and transferred them to laptop for processing and detecting obstacles [87]. Concerning the obstacle warning module, a tactile-visual substitution system uses voice commands to warn the PVI to avoid obstacles.

Hybrid Systems
In the previous two sections, several solutions to assist PVI in navigating and identifying products have been discussed. These solutions use tags such as RFID and NFC, visuals tags such as QR codes and AR Markers, or CV techniques. However, these solutions are not suitable under all situations, because each environment has specific features. For example, CV techniques cannot be used in areas with considerable light, because the quality of the taken image will be poor. In this case, it is better to use a different technology, like RFID or NFC, to improve system accuracy. Another example is when, in a shop, items, which all have needed information, already have barcodes or QR codes. Developers can use these tags for product identification and use CV techniques or non-visual tags (RFID, NFC) for navigation. Two or more such technologies can be combined, which would lead to the development of hybrid systems.
For example, McDaniel and co-authors proposed a system that integrates CV techniques with RFID Systems. This system identifies information about relevant objects in the surroundings and sends them to PVI [56]. López-de-Ipiña and co-authors integrate RFID with QR codes, to allow PVI to navigate inside a grocery store [57,88]. The system used the RFID reader to identify the RFID tags to navigate inside the store. It adopted the smartphone camera to identify QR codes placed on product shelves. Finally, Fernandes and co-authors developed a solution to help PVI identify objects and navigate in indoor locations using RFID and CV technologies [58]. This system used the RFID reader to receive the current location of PVI and CV techniques to identify objects.

Discussion
In this study we have presented different approaches, techniques, and technologies to help PVI navigate and identify products while shopping. We classified the shopping approaches into preparing a list of what to purchase and navigating inside the store to identify items on the shelves.
Solutions for shopping list preparation used some techniques, such as CV, OCR and STT, to read list details from PVI and store them in a database. During shopping, audio messages are given to PVI about shopping items stored in the database using TTS. Shopping preparation makes it easy for users to buy from a list, but it has some limitations: it assumes that the shopper already knows what they wish to buy on their trip [8], and this is not always the case. Also, shopping may take a long time. Shopping is not simply buying items on a list. For people without visual impairment, shopping also entails opportunistically exploring new products or brands, engaging in cultural learning about tastes, and making different choices based on occasional sales [89]. Some solutions require taking an image of the shopping list, but it is difficult for PVI to take a good quality image.
Researchers have also developed multiple solutions to help PVI navigate and identify products using RFID and NFC for tag based systems. RFID tags have some benefits: their signals can penetrate walls and obstacles, which cover a wide area, and reuse existing infrastructures, resulting in cost reduction [90]. Also, PVI do not need to be in a certain direction to receive messages from the tagged item [91]. Moreover, the tag does not need to be within the line of sight of the RFID reader which allows it to be placed inside items. However, there are some drawbacks: to setup the environment, hundreds of RFID tags need to be installed, which is costly. Information overload is another major problem, as it is overwhelming to receive information about all the items in the store at the same time and attempt to use this information to identify various objects [56]. Another significant limitation is that these systems are used only in a restricted environment in which objects have been tagged, and these tags need regular maintenance. Sometimes, RFID tags are attached to items like the liquid in metal cans, which reflect the radio waves during communication. Using RFID technology with glass causes a reflection of the radio waves, which affects the system outcome.
NFC tags have the following benefits: PVI can simply touch the NFC tag with an NFC reader, such as a smartphone, to begin the required service. NFC readers read information stored in the tags, which enables PVI to get product information. Moreover, by using NFC tags, researchers can build low-cost indoor navigation systems. NFC technology also has a very low response time, because the time required to transfer data from an NFC tag to a mobile device and generate the walking path to the items is short. They also provide accurate position and orientation information, so the orientation of the user to the destination is facilitated. Moreover, NFC tags work well in unclean environments and do not require a direct line of sight between the reader and the tags. Finally, PVI do not have to carry large or bulky devices-only their smartphones. However, there are some drawbacks: NFC tags are not as effective and efficient as Bluetooth or Wi-Fi Direct when it comes to data transfer rates. NFC can only send and receive very small packets of data, so real-time positioning cannot be provided in the NFC Internal system. Also, PVI must be inside the reading area in order to identify NFC tags and must have an NFC-equipped smartphone.
Moreover, researchers developed multiple solutions to help PVI navigate and identify products using CV techniques. CV techniques rely on visual tags, such as QR codes, barcodes, and AR markers, or utilize information about the objects' physical features. CV tag based systems offer several advantages: they only need to identify tags to get product details, so they need low computational power and small storage space. Many of these approaches do not need tags to be explicitly placed, as products already have unique visual tags, such as barcodes and QR codes. Such tags can be generated and printed at a very low cost compared to non-visual tags, such as RFID, and can be easily modified. CV tag based systems are ideal for tasks that require differentiating among groups of objects. They are vital for PVI when the contents of the items are different, such as a tube of glue versus a tube of eye drops, as they have the same shape and it may be dangerous if they choose the wrong one [92]. However, tag based CV techniques require a prior selection of items and the correct placement of tags on those items. Moreover, if there are many tagged items in a small area, PVI would be confused by receiving information about them all at the same time. Visual tags also must be in line-of-sight of the camera, otherwise, they will not be detected. Furthermore, visual tags cannot be placed inside items, as the appearance of these tags are important for PVI. These tags can also be damaged during movement through the supply chain or by weather. Also, it is difficult for a smartphone camera to detect CV tags if the PVI is moving fast, and the recognition rate decreases as the distance between the reader and the tags increases [75].
CV non-tag based systems have several advantages. These systems are cost-effective, as they need little or no infrastructure and most of them can be easily installed on smartphones. However, they have several limitations. Their performance may be unreliable in real-world environments because of factors like motion blur and image resolution, as well as changes in conditions, such as illumination, orientation, and scale. These systems use extensive computational power, and PVI need to take many photos. However, taking good quality photos is difficult for PVI. Finally, feedback latency must be reduced to make these systems more effective.
Finally, researchers have created hybrid systems by taking the strengths of two or more systems and combining them. Numerous attempts have been made in this area to balance the trade-offs of the combined technologies. As a result, there is a significant improvement in accuracy, robustness, performance, and usability. However, the major drawback of these systems is that they use significant infrastructure due to the combination of technologies, which results in increased complexity and cost. Figure 3 shows a complete scenario of the shopping process, from preparation of the shopping list to completion of the shopping task. To summarize, the answers to Q1, Q2, and Q3 are: A1: This study presented the main categories of MAT to help PVI shop. We divided the shopping task into two parts: 1) Preparing a list of what to buy, and 2) navigation inside a store and identification of items on the shelves. Preparing a list for shopping solutions used techniques such as CV and STT to obtain list details from PVI and store them in a database. During shopping, audio feedback is given to PVI about shopping items using TTS. The navigation and identification of products use RFID and NFC for tag based systems, visual tags like QR codes, barcodes and AR markers for CV tag based systems, or information about the physical features of items for CV approaches A2: Table 1 shows the strengths and weaknesses of the latest MAT shopping systems for PVI. A3: Table 2 summarizes the criteria for the most effective solutions. Based on the categories and solutions discussed in Section 2, we selected some criteria and compared them in Table 2. The first criterion is the cost of applying the technology to any solution. It is shown that CV tag techniques can be used without any cost except for printing the QR codes or AR markers and putting them in the correct place. When using a barcode, there is no need to print them, as they are already placed on each product. Tag based techniques can be used at a low cost, as shops only need some RFID or NFC tags to be installed in the correct places. If CV techniques are used, high-quality equipment, such as cameras, are needed for good results. The second criterion is the equipment needed to detect and identify products or places. For CV tag based solutions, PVI need only their smartphone cameras to detect and identify items. In CV techniques, some solutions only need smartphone cameras, while others need high-quality cameras to take high resolutions images and machines with powerful processors for computations. In tag based techniques, PVI need RFID reader or smartphones supporting NFC technology. The third criterion is the number of items able to be scanned at the same time. Only RFID readers, AR markers, and CV techniques can scan more than one item at the same time, which is useful in some situations, such as if PVI want to identify and count the items in their shopping cart. The fourth criterion is whether or not the PVI must be in the line of sight with the identified products. For tag based solutions, PVI do not have to be in the line of sight of items, and the PVI can identify them in any direction. In tag based solutions, tags must be within 3 m for RFID tags and within 10 cm for NFC tags, while CV solutions depend on some other parameters, like the tag size in QR codes or barcodes, and the marker or camera parameters for CV techniques. The last criterion is the storage capacity of each solution. Only some tags, such as RFID, NFC and QR codes, have a storage capacity, while others, such as AR markers and barcodes, do not have any storage capacity. Researchers can select and design new technology solutions based on specific requirements and which criteria to focus on, and how to evaluate tradeoffs. Further research is needed to develop precise, more effective, low cost, and easy to use helping systems for PVI. Finally, if scientists and engineers develop MAT for PVI, they should study and take into account the Web Content Accessibility Guidelines [95] and Section 508 standards [96].

Conclusions
PVI face many problems during the shopping process. This study has discussed the current, most prevalent, solutions to help PVI shop effectively. We conclude that all presented solutions have some advantages and disadvantages. Researchers have tried to design and evaluate hybrid solutions that exploit the main advantages, and avoid the disadvantages, of individual systems. However, these hybrid systems increased the infrastructure use, the time consumption, and the system complexity. This study provides an introduction to guide and motivate researchers towards carrying out more studies that may lead to good solutions to help PVI in the shopping process.
Author Contributions: All authors contributed to the present paper with the same effort in finding available literature resources, as well as writing the paper.

Funding:
The authors would like to thank the financial support of Széchenyi 2020 under the EFOP-3.6.1-16-2016-00015.