Investigating the Performance of the Order-Picking Process by Using Smart Glasses: A Laboratory Experimental Approach

Abstract

Background: Order picking process is critical for accurate and efficient order fulfilment. Pick-by-vision is a promising technology that may support order picking process, however there is still a limited amount of research concerning the impact of this technology on the performance of order-picking. The purpose of this paper is to investigate certain operational and technical parameters that affect the performance of pick-by-vision technology in item-level order picking via a series of laboratory tests. Methods: A systematic literature review is conducted for the identification of parameters that affect pick-by-vision performance. Subsequently, the analytical hierarchy process is adopted to rank these parameters, concerning their impact on order picking. Then, the design of experiment and NASA task load index methodology are implemented for assessing pick-by-vision efficiency and perceived workload. Results: The results reveal the parameters that significantly affect the performance of the pick-by-vision system, as well as the best configuration of parameters for the implementation of the proposed system in real environments. Conclusions: The results obtained are encouraging, showing how pick-by-vision technology can support order picking efficiency. Furthermore, practical implications are presented that deal with the organizational culture, process re-engineering, staff resistance to change, and motivation for maintaining the new way of executing order-picking processes.

1. Introduction

The increase of e-commerce sales, the globalization of trade, the customers’ demand for frequent and low-volume orders, and the need for faster response times are the main factors that increase the complexity of logistics processes [1,2]. The management of the aforementioned challenges and the optimization of warehouse operations, coupled with logistics cost reduction, are complicated tasks for warehouse managers to cope with, because most warehouses are manually or semimanually operated, resulting in delivering labor-intensive services to their customers [3]. Once the processes of a standard workflow in a warehouse has been taken into account, it can be argued that order picking contributes significantly to logistics costs and customer service [1,3,4]. Indeed, in manual warehouses, the order-picking process embraces the most labor-intensive operation, resulting from 55% to 65% of the total operational warehouse cost [5], while in automated warehouses, the order-picking cost becomes capital intensive because of the high investment cost [6]. This is the main argument of logistics professionals who prioritize warehouse improvements by focusing mainly on the order-picking process.

Focusing on the development of information and communication technology (ICT) and a number of other technologies (e.g., pick by light, pick by voice, ring scanners, augmented reality, RFID, etc.) have already been adopted for order picking. These technologies digitalize the traditional paper-based picking list, facilitating in that way the fulfillment of modern customer needs for increased efficiency and accuracy [7]. Pick by vision through smart glasses is an innovative solution that may improve both time efficiency and order-picking accuracy [8]. According to Stoltz et al. [9], pick by vision uses wearable technology and provides a fast and hands-free solution for the execution of the order-picking process. This innovative technology combines the best of vision-guided picking so as to produce a more efficient and more accurate operation beyond the conventional order-picking technologies [7].

Despite the general impression that pick by vision is a promising order-picking technology, there is still a limited amount of research concerning the impact of this technology on the performance of the order-picking process, which makes it difficult to derive solid results and make practical recommendations. Indeed, on the basis of the available literature [10,11,12,13], a significant number of studies have dealt with the optimization of the order-picking process, by considering a series of strategic, tactical, and operational parameters, but only a small number of studies have considered the parameters that affect order-picking accuracy and efficiency during the design and optimization phase of the order picking process [14]. Indeed, in [14], the authors take into account four parameters: (a) display holder, (b) field of view, (c) barcode type, and (d) existence of confirmation, without, however, considering other, equally important parameters that deal with ergonomic parameters, such as battery position, and order-profile parameters, such as order lines per order, items per order line, etc. To this end, we argue that there is still a need for further investigation in parameters that affect order picking, in order to thoroughly analyze their impact on order-picking productivity, efficiency, and operational cost.

To this end, the aim of this paper is to investigate certain parameters that affect the operational performance of a pick-by-vision system via a series of laboratory tests. Initially, 27 parameters and three performance measurement indices are identified for the pick-by-vision system design, development, and testing by adopting a systematic literature review (SLR) methodology. Six parameters are selected via an analytical hierarchy process (AHP) in order to investigate their impact on the proposed pick-by-vision system. Subsequently, a series of laboratory tests are conducted by adopting the design of experiments (DoE) methodology. The proposed pick-by-vision system is investigated to assess its order-picking time and workload. The perceived workload of the pick-by-vision system is evaluated via NASA TLX survey.

The remainder of this paper is structured as follows. Section 2 presents the findings from the literature review and describes the selected parameters for the evaluation of the proposed pick-by-vision system. Section 3 presents the necessary steps that have been adopted for the design of the laboratory experiments. In Section 4 and Section 5, the results of the laboratory tests and the discussion of the findings, coupled with theoretical contribution and practical implications, are presented. Finally, Section 6 concludes the paper.

2. Identification and Selection of Parameters

2.1. Subsection

The systematic literature review (SLR) methodology, coupled with a series of research questions (RQs), was adopted with the aim of identifying, detecting, and categorizing the parameters that affect the design and operation of the proposed pick-by-vision system. According to

Table 1. Research questions for the identification, detection, and categorization of parameters that affect the design and operation of the proposed pick-by-vision system.

No	Description of Research Question
RQ1	Which are the main parameters that should be taken into consideration for a pick-by-vision device parameterization (system design)?
RQ2	Which are the main parameters that should be taken into consideration for the evaluation and optimization of pick-by-vision systems in terms of operational performance?
RQ3	Which are the main parameters that should be taken into consideration for the evaluation and comparative assessment of pick-by-vision technology in terms of order profile?
RQ4	Which are the main performance measurement indices and side effects that should be taken into consideration for the evaluation pick-by-vision technology in terms of performance and ergonomics?

, a total of four research questions were developed. In order to answer the above RQs, we use the systematic literature review (SLR) method. More specifically, we followed a three-step protocol based on previous prominent articles [15,16,17,18,19,20,21], in order to come up with reliable and proven work. The steps of selecting protocol are described as follows.

In the first step, a series of search terms/keywords and induction criteria were determined, in order to conduct comprehensive research (

Table 2. Inclusion criteria for implementing SLR methodology.

Inclusion Criteria	Description
Search terms/Keywords	Vision picking, pick-by-vision, wearable technology, wearable computers, order picking, augmented reality, head-mounted displays, smart glasses, user interface, logistics
Source types	(a) peer-reviewed journals, (b) international conferences, (c) reports, (d) dissertations
Language	English

). The research focused on papers published in peer-reviewed journals, at international conferences, in dissertations and in technical reports on the field of logistics. The main reason for including articles from international conferences, dissertations, and reports in this work is that the number of papers in peer-reviewed journals that deal with the evaluation of pick-by-vision systems is limited.

In the second step, a review of selected articles (from Step 1) took place on the basis of the titles and abstracts of the articles. During this review, a series of articles out of the research scope was excluded from our list. More specifically, 46 studies focused on different aspects of pick-by-vision technology and fields. After the completion of this step, the remaining number of articles was 31.

In the last step, the reading of full versions of available studies led to the final selection of the list of studies to be considered. Eight studies were excluded, and by implementing the snowballing method, we were able to add to our initial list some additional papers, reports, and dissertations that met the inclusion criteria. The final corpus involved 66 studies. The latter were reviewed in order to identify the key parameters that affect the design and implementation of a pick-by-vision system.

2.2. Descriptive Analysis of the Corpus

After the results of SLR were taken into account, it was revealed that 44.6% of the reviewed studies were coming from journals, 40% from conferences, 4.6% from reports, and 10.8% from dissertations. The small number of published studies and therefore the limited number of journal articles were representative signs that the field is quite new, from a research point of view.

The latter seems to be confirmed if we take into account the time distribution of the reviewed studies. Given the results of the time distribution of the reviewed studies, it is evident that the years of publication among the identified publications vary from 2001 to 2021. The number of studies factoring in the design, development, and testing of pick-by-vision technology has rapidly grown during the past several years. Almost, half of the considered studies were published within the past four years, indicating that the area has been significantly expanding over the past few years. The peak in the number of studies is observed during the two-year period from 2018 to 2020, when 27 studies were published.

According to a geographical analysis of the study areas, 70.5% of the studies were conducted in Europe and 25.6% on the American continents. The majority of the European studies were conducted in Germany (60% of the studies). Furthermore, 7.3% of the studies were conducted in Greece, same as Sweden, and Slovenia is next, at 5.4%. The remaining 20% of the studies were conducted in various other European countries, as shown in Figure 1.

2.3. Identification of Parameters for the Design and Investigation of Pick-by-Vision Systems

After the structure of the basic research questions had been finalized, the identified parameters were classified in four categories: (a) parameters for system design, (b) parameters for system evaluation in terms of operation, (c) parameters for system evaluation in terms of order profile, and (d) performance measurement indices that can be used for the evaluation of pick-by-vision technology. All the reviewed parameters per category are presented in

Table 3. Overview of the reviewed parameters for the design and investigation of the proposed system.

Category	Subcategory	Parameter	References
A. Parameters for system design	Ergonomic aspects	Display Position	[22,23,24]
		Display Type	[24,25,26,27]
		Interaction Device	[24,25,28,29,30,31]
		Battery Position	[27]
		Display Holder	[22,30,32]
		Scanner Position	[9,32,33,34,35]
		Weight of Equipment	[27,34,36,37,38]
	Visualization aspects	Field of View	[26,30,39,40,41]
		Focal Distance	[26,39,42]
		Visualization Optics	[29,38,41,43,44]
		Information Mode	[8,22,25,26,45,46]
		Information Availability	[25,34]
		Display View	[9,33,37,47,48]
		Existence of AR	[45,46,49]
		Direction Interface	[28,32,50,51,52]
		Display Settings	[25,26,29]
	Technical aspects	Barcode Type	[9,35,53,54,55]
		Scanning Distance	[8,44,56,57,58]
		Battery Life	[27,38,59,60,61]
		Existence of Tracking System	[32,40]
		Confirmation Equipment	[28,34,36,41,62]
B. Parameters for system evaluation in terms of operation		Picking Strategy	[63,64]
		Handling Unit	[32,57]
		Existence of Confirmation	[14,28,29,37,65]
C. Parameters for system evaluation in terms of order profile		Number of Orders	[33,36,37,47,49,56,59,66]
		Lines per Order	[14,24,33,37,47,56,64,66,67,68]
		Items per Line	[14,32,33,37,47,49,56,64,66]
D. Performance measurement indices		Efficiency (Time)	[9,25,34,47,62,65]
		Accuracy	[23,25,34,39,47,65]
		Workload	[30,33,34,37,41,42,68]

.

The first category comprises the device design and development for the proposed system and includes 21 reviewed parameters. Because of the high number of parameters in this category, they were further classified into three distinctive subcategories. The first subcategory deals with the ergonomic aspects and involves seven parameters, the second one focuses on visualization aspects and includes nine parameters, and the third subcategory is associated with technical aspects and encompasses five parameters. The ergonomic aspects of device parameterization play a critical role during the design and development of the system in that they deal with parameters that define how comfortable a worker would feel while using the system. A crucial issue is that the worker has to wear the equipment needed for a pick-by-vision system during a shift. To this end, the pick-by-vision equipment must be light, ergonomically designed, and safe and must have an eight-hour battery life [32]. The visualization aspects during the design and development of pick-by-vision technology deal with the graphical user interface (GUI) of the device. Indeed, one of the most important features of a pick-by-vision system is the GUI, because the virtual information must be displayed in the lens of the glasses at the right time and at the right position [49,59]. Nevertheless, the display of necessary information (i.e., stock location, article number, goods description, required quantities, etc.) on pickers’ glasses or headbands does not always efficiently appear, because of various problems, such as eye strain, difficulties seeing the display image, eye pain, eye concentration problems, and headaches, which have been observed during the testing of pick-by-vision technology [23]. The technical aspects focus mainly on the hardware used during pick by vision. This category is vital during the design and development of pick-by-vision systems because it includes parameters that affect the ability of a system to read and recognize the products, to monitor the process and direct the order pickers, and to ensure order-picking-process accuracy.

The second category deals with three parameters that enable the operational performance of the system to be evaluated. According to the available literature [4,22,28,29,37,57,63,64,65], this category includes a series of parameters (e.g., picking strategy, handling unit, etc.) that can be used by investigators when they assess the operation performance of the system in the field.

The third category includes three parameters that are used for the comparison assessment of the proposed system with other picking technologies. This category describes a series of factors, such as the number of orders, lines per order, etc., that can be used in order to compare the pick-by-vision system with other conventional order-picking technologies and systems (e.g., light picking, voice picking, RF-scanning picking, etc.) [14,24,33,37,47,56,64,66,67,68].

Lastly, the fourth category includes three performance measurement indices and deals with the final output of the evaluation process. These indices are used by professionals and academics to evaluate pick-by-vision technology and other order-picking technologies in order to compare them [23,25,34,39,47,65].

2.4. Selection of Parameters for the Design and Investigation of Pick-by-Vision Systems

After taking into account the results of the SLR, we implemented the predefined steps of an analytical hierarchy process (AHP) [69] in order to identify the most important parameters that affect a pick-by-vision system. All the reviewed parameters were validated from experts’ opinions on five aspects: ergonomic aspects (EAs), visualization aspects (VAs), technical aspects (TAs), operational aspects (OAs), and order-profile aspects (OPAs). In addition to the 27 parameters that were identified in the SLR process, one more parameter, that is storage level, was mentioned by the experts and was taken into account during the AHP process, resulting in 28 parameters in total to be assessed. Following the methodology recommended by Saaty [69], during the first step, the construction of hierarchy structure took place. More specifically, the AHP framework of evaluating pick-by-vision parameters was structured in three levels (Figure 2).

The first level includes the goal (to prioritize critical pick-by-vision design and development parameters), the second level focuses on the dimensions of parameters (five dimensions), and the third level deals with the constructs of dimensions (28 parameters).

During the second step of the recommended methodology, the necessary questionnaire with the pair-wise comparison matrices (PWCM) was constructed. For the construction of the questionnaire, all dimensions and constructs of the AHP-based hierarchical model were taken into account, and the scale of numbers suggested by Saaty [69] was used. After the construction of the necessary questionnaire, the ranking of the selected parameters was completed by experts. In this phase, a series of interviews with logistics/warehouse managers, specialists, and executives took place. The questionnaire was completed by 15 experts who work in logistics service providers and in commercial and manufacturing companies (with in-house logistics) in Greece. The steps for completing the questionnaire were specific and the same for all participants, and they are presented below.

• Step 1: Presentation of the main aim and objective of this research.
• Step 2: Detailed description of reviewed parameters to the participants (experts).
• Step 3: Specific instructions given to participants on how to complete the questionnaire.
• Step 4: Rating of pick-by-vision system design parameters by experts (completed via questionnaire).

After the ranking was completed, a short discussion on the participants took place in order to give us their feedback about the pick-by-vision technology, the preferable operational/functional services, the challenges and inefficiencies, and the potential benefits of its implementation in real-life scenarios. After the completion of interviews and data collection (experts’ inputs), the data analysis and the calculation of consistency were accomplished according to the methodology recommended by Saaty [69].

In the last step, the priorities were calculated on the basis of the AHP methodology by taking into account the hierarchical model and the ratings achieved through the questionnaire. Furthermore, for each pair-wise comparison matrix (PWCM), the maximum eigen values (λmax), CI, and CR were calculated. Moreover, values for the consistency ratio (CR) were within an acceptable range for all the pair-wise comparison matrices, ensuring the reliability of decision makers.

Taking into account the results of the ranking, we concluded that the most important dimension for pick-by-vision technology design and development was the ergonomic aspects (EAs), followed by order-profile aspects (OPAs), visualization aspects (VAs), operation aspects (OAs), and technical aspects (TAs).

In terms of ergonomic aspects (EAs), the weight of equipment (WE) and scanner position (SP) were found to be as the most important constructs, while for the order-profile aspects (OPAs) dimension, the lines per order (LO) and number of orders (NOs) were ranked as the most important constructs. Finally, according to the experts, the technical aspects (TAs) dimension was less important than the other four dimensions. The complete ranking of critical constructs/parameters for pick-by-vision technology design and development is presented in

Table 4. Overall weighting and ranking of pick-by-vision design and development parameters.

Dimension Description	Weight of Dimensions	Rank	Parameters Description	Local Weight of Parameters	Overall Weight of Parameters	Overall Ranking of Parameters
Ergonomic aspects(EAs)	0.44	1st	Display Position	0.023	0.023	17th
			Display Type	0.069	0.069	6th
			Interaction Device	0.032	0.032	11th
			Display Holder	0.024	0.024	16th
			Weight of Equipment	0.151	0.151	1st
			Scanner Position	0.076	0.076	4th
			Battery Position	0.030	0.030	12th
			Storage level	0.034	0.034	10th
Visualization aspects(VAs)	0.16	3rd	Field of View	0.060	0.060	7th
			Focal Distance	0.009	0.009	23rd
			Visualization Optics	0.006	0.006	24th
			Information Mode	0.035	0.035	9th
			Information Availability	0.025	0.025	14th
			Display View	0.013	0.013	22nd
			Existence of AR	0.004	0.004	26th
			Direction Interface	0.003	0.003	28th
			Display Settings	0.018	0.018	20th
Technical aspects(TAs)	0.08	5th	Barcode Type	0.029	0.029	13th
			Scanning Distance	0.006	0.006	25th
			Battery Life	0.043	0.043	8th
			Ex. of Tracking System	0.004	0.004	27th
			Confirmation Equipment	0.020	0.020	19th
Operation aspects(OAs)	0.11	4th	Picking Strategy	0.069	0.069	5th
			Handling Unit	0.025	0.025	15th
			Existence of Confirmation	0.013	0.013	21st
Order-profile aspects(OPAs)	0.21	2nd	Number of Orders	0.079	0.079	3rd
			Lines per Order	0.107	0.107	2nd
			Items per Line	0.021	0.021	18th

.

For the selection of factors to be investigated via laboratory experiments, the ranking from the AHP were factored in, but the final selection of parameters was made by factoring in also various limitations of the available laboratory layout (e.g., limited space for conducting the tests), lack of availability to further develop graphical user interface (GUI) of the system, etc. After taking into account these limitations, we decided to investigate via laboratory experimentation the following six parameters: (a) battery position, (b) type of order, (c) storage level, (d) confirmation equipment, € items per order line, and (f) order lines per order. A detailed analysis for the selected parameters will take place in the following section.

3. Design of Experiments

During the stages of the development and evaluation of a process or a system, it is important to adopt a robust methodology with specific steps for the execution of the experimental procedure (e.g., planning and conducting experiments, data collection and analysis, etc.) in order to achieve reliable and valid results [70,71,72]. To this end, for the experimental design and the performance evaluation of the proposed pick-by-vision system, the design of experiment (DoE) methodology was adopted as proposed by Montgomery [68]. DoE was used in order to investigate the effect of the selected parameters in terms of order-picking efficiency.

3.1. Experimental Design

The main scope of this work was to investigate the performance of a pick-by-vision system via a series of laboratory tests. To this end, a number of experiments were designed and executed by including six parameters/factors. Figure 3 depicts the input of our experiment, which includes six parameters and their corresponding levels, and one output, which is the order-picking efficiency (time) per order line.

The first factor is the battery position. Many warehouses may have one, two, or even three 8-hour shifts per day, so there is no time on shift for charging the pick-by-vision equipment. The use of an external battery can resolve this problem; however, it is crucial to share the battery and headband weight equally in order to avoid any inconvenience that may negatively affect the picker’s performance. Because the display holder is a headband and the one side holds the arm of the headband, it would be best to avoid adding extra weight on this side. Therefore, the two positions (levels) of the battery that will be investigated are weighted back (Figure 4a) and weighted side (the opposite side of the arm) (Figure 4b).

The second factor is the type of order. This parameter includes two levels: discrete order picking and multiple order picking with six orders simultaneously. In discrete order picking, a single worker walks to pick all the necessary items to fill a single customer order [69]. On the other hand, in multiple order picking, multiple customer orders are picked simultaneously by an order picker [73].

The third factor is the storage level, which can be classified as low storage level (level 1) and high storage level (level 2). In Figure 5, the low storage level includes the first and second levels of the racks, whereas the high storage level includes the third and fourth levels of the racks. An average person has direct contact with the high storage level when scanning a barcode or picking a product, while for the low storage level, they need to bend over to achieve direct contact with the barcode and the product’s position. Bending over or kneeling in front of the rack may cause the picker annoyance and fatigue, negatively affecting their performance and their body health.

The fourth factor to be investigated is the confirmation equipment. For this factor, both scanner confirmation and RFID reader confirmation are evaluated during the laboratory tests. Confirmation equipment can maximize the order-picking accuracy (decrease in the number of incorrectly picked items) of an order-picking system or process. There are many ways of confirming the accuracy of order picking, but in our lab tests, scanner confirmation and RFID reader confirmation are selected. The wristband shown in Figure 6 is an RFID reader that reads an RFID tag (placed in a tote) in order to confirm that a picker has correctly placed a certain product in a specific tote (each tote contains the products of an order). Meanwhile, scanner confirmation is accomplished through the headband’s camera by reading the barcode of each product.

The fifth factor to be tested is items per order line. Following the designed test, one to three items per order, are used for level 1 (few) and four to six items per order line are used for level 2 (many) of the parameter. The last factor to be tested is order lines per order. Similarly, 1one to three order lines per order are used for level 1 (few) and four to six order lines per order are used for level 2 (many). “Order lines per order” is defined as the variety of different SKUs in the same order, regardless of the number of items for each SKU. The number of order lines per order is expected to give a better understanding of time-efficiency changes noticed in the experiments.

Given the available types of experiments and the objective of this work, which focuses on the identification of the factors/parameters that affect the performance of the pick-by-vision system in terms of order-picking efficiency, we conclude that the most suitable experimental design type is screening (factorial design) [70]. Accordingly, a full factorial design has been used for the experiments conducted, one that incorporates six factors at two levels (26 full factorial design = 64 runs). The combination of factors and their levels are depicted in

Table 5. The design matrix.

Run	Battery Position	Type of Order	Storage Level	Confirmation Equipment	Items per Order Line	Order Lines per Order
1	Weighted Side	Discrete	Low	Scanner	Few	Few
2	Weighted Side	Discrete	Low	Scanner	Few	Many
3	Weighted Side	Discrete	Low	Scanner	Many	Few
4	Weighted Side	Discrete	Low	Scanner	Many	Many
5	Weighted Side	Discrete	Low	RFID Reader	Few	Few
6	Weighted Side	Discrete	Low	RFID Reader	Few	Many
7	Weighted Side	Discrete	Low	RFID Reader	Many	Few
8	Weighted Side	Discrete	Low	RFID Reader	Many	Many
9	Weighted Side	Discrete	High	Scanner	Few	Few
10	Weighted Side	Discrete	High	Scanner	Few	Many
11	Weighted Side	Discrete	High	Scanner	Many	Few
12	Weighted Side	Discrete	High	Scanner	Many	Many
13	Weighted Side	Discrete	High	RFID Reader	Few	Few
14	Weighted Side	Discrete	High	RFID Reader	Few	Many
15	Weighted Side	Discrete	High	RFID Reader	Many	Few
16	Weighted Side	Discrete	High	RFID Reader	Many	Many
17	Weighted Side	Multiple	Low	Scanner	Few	Few
18	Weighted Side	Multiple	Low	Scanner	Few	Many
19	Weighted Side	Multiple	Low	Scanner	Many	Few
20	Weighted Side	Multiple	Low	Scanner	Many	Many
21	Weighted Side	Multiple	Low	RFID Reader	Few	Few
22	Weighted Side	Multiple	Low	RFID Reader	Few	Many
23	Weighted Side	Multiple	Low	RFID Reader	Many	Few
24	Weighted Side	Multiple	Low	RFID Reader	Many	Many
25	Weighted Side	Multiple	High	Scanner	Few	Few
26	Weighted Side	Multiple	High	Scanner	Few	Many
27	Weighted Side	Multiple	High	Scanner	Many	Few
28	Weighted Side	Multiple	High	Scanner	Many	Many
29	Weighted Side	Multiple	High	RFID Reader	Few	Few
30	Weighted Side	Multiple	High	RFID Reader	Few	Many
31	Weighted Side	Multiple	High	RFID Reader	Many	Few
32	Weighted Side	Multiple	High	RFID Reader	Many	Many
33	Weighted Back	Discrete	Low	Scanner	Few	Few
34	Weighted Back	Discrete	Low	Scanner	Few	Many
35	Weighted Back	Discrete	Low	Scanner	Many	Few
36	Weighted Back	Discrete	Low	Scanner	Many	Many
37	Weighted Back	Discrete	Low	RFID Reader	Few	Few
38	Weighted Back	Discrete	Low	RFID Reader	Few	Many
39	Weighted Back	Discrete	Low	RFID Reader	Many	Few
40	Weighted Back	Discrete	Low	RFID Reader	Many	Many
41	Weighted Back	Discrete	High	Scanner	Few	Few
42	Weighted Back	Discrete	High	Scanner	Few	Many
43	Weighted Back	Discrete	High	Scanner	Many	Few
44	Weighted Back	Discrete	High	Scanner	Many	Many
45	Weighted Back	Discrete	High	RFID Reader	Few	Few
46	Weighted Back	Discrete	High	RFID Reader	Few	Many
47	Weighted Back	Discrete	High	RFID Reader	Many	Few
48	Weighted Back	Discrete	High	RFID Reader	Many	Many
49	Weighted Back	Multiple	Low	Scanner	Few	Few
50	Weighted Back	Multiple	Low	Scanner	Few	Many
51	Weighted Back	Multiple	Low	Scanner	Many	Few
52	Weighted Back	Multiple	Low	Scanner	Many	Many
53	Weighted Back	Multiple	Low	RFID Reader	Few	Few
54	Weighted Back	Multiple	Low	RFID Reader	Few	Many
55	Weighted Back	Multiple	Low	RFID Reader	Many	Few
56	Weighted Back	Multiple	Low	RFID Reader	Many	Many
57	Weighted Back	Multiple	High	Scanner	Few	Few
58	Weighted Back	Multiple	High	Scanner	Few	Many
59	Weighted Back	Multiple	High	Scanner	Many	Few
60	Weighted Back	Multiple	High	Scanner	Many	Many
61	Weighted Back	Multiple	High	RFID Reader	Few	Few
62	Weighted Back	Multiple	High	RFID Reader	Few	Many
63	Weighted Back	Multiple	High	RFID Reader	Many	Few
64	Weighted Back	Multiple	High	RFID Reader	Many	Many

.

Furthermore, the design of experiment included five replicates per run, so the total number of samples was N = 320. Every run was conducted in a random order, as suggested by the DoE methodology.

3.2. Subjects’ Features, and Experimental Equipment and Set-Up

A total of 10 subjects took part in the laboratory tests: five men and five women. Their ages ranged from 22 to 41, and the average age of participants was 25.1 years old. Eight out of the 10 participants were right-eye dominant, whereas only two of them were left-eye dominant. Three subjects used prescription glasses during the tests. All the participants were native Greek speakers; thus, all instructions and survey instruments were provided in Greek. The same applied for the personal questionnaires used for capturing the experience of the participants after the experiments had been conducted. In order to avoid any kind of bias toward the pick-by-vision system, the participants were selected because they were completely inexperienced in order picking process. To equal the lack of experience and in order to minimize the learning effects, the subjects participated a short tutorial and participated in a training session, where they could use the pick-by-vision equipment and pick a series of orders in the laboratory. In this way, the subjects felt familiar with the pick-by-vision system and got ready to conduct the tests.

The testing of the pick-by-vision system took place in a dense picking laboratory environment (Figure 7 and Figure 8). The laboratory environment consisted of 16 pick bins divided into two shelving units. Each shelving unit had four rows and two columns, and each pick bin contained 6–10 items. The order cart, which was used during multiple order picking, had three storage levels, and each level contained two plastic bins (totes).

Each subject could simultaneously pick up to six orders (each plastic bin was assigned to one order) in the case of multiple order picking. While running the experiments, a series of specialized equipment was used. More specifically, when subjects interacted with the picking system, a RealWear HTM-1 headband was used (RealWear Inc. 600 Hatheway Rd #105, Vancouver, WA 98661, United States), while for the confirmation of order-picking movements, the SLS^® M-100/M-101 Wearable RFID Reader (Smart Label Solutions LLC 1100 Durant Drive, Howell, MI 48843, United States) was selected. All the aforementioned equipment was connected to a warehouse management system (WMS) installed in a computer server at the laboratory.

Each participant was assigned to run a group of 12 picking lists separated in six multiple order pickings (simultaneously) and six discrete order pickings (one by one). Every picking list consisted of one to six items per order line and one to six order lines per order, shared equally. The items on the shelves contained similar product categories with different sizes and weights. Every item could be handled with one hand.

3.3. Formulation of Research Hypothesis

As mentioned earlier, the performance of the pick-by-vision system was measured by taking into account order-picking efficiency (order-picking time per order line). Order-picking time was measured by a typical stopwatch, and the time data were presented in minutes per order line. In order to investigate whether the parameters under consideration were statistically significant, certain null hypotheses were introduced, as follows.

The first null hypothesis (H_0.1) states that the performance of the pick-by-vision system is the same when the battery position is either weighted back or weighted side:

H_0.1: t_{weighted-back} = t_{weighted-side}

(1)

The second null hypothesis (H_0.2) states that the time needed for the laboratory experiment is same when the type of order is either discrete order or multiple order:

H_0.2: t_discrete = t_multiple

(2)

The third null hypothesis (H_0.3) states that the time needed for the laboratory experiment is equal when the storage level is either low or high:

H_0.3: t_low = t_high

(3)

The fourth null hypothesis (H_0.4) states that the performance of the pick-by-vision system remains the same when the confirmation equipment is either a scanner or an RFID reader:

H_0.4: t_scanner = t_RFID

(4)

The fifth null hypothesis (H_0.5) states that the order-picking time of the laboratory experiment is the same when items per order line is either few or many:

H_0.5: t_few = t_many

(5)

The sixth null hypothesis (H_0.6) states that the order-picking time of the laboratory experiment is the same when order lines per order is either few or many:

H_0.6: t_few = t_many

(6)

3.4. Results from the Statistical Analysis

After the completion of the tests and the collection of the data, a quantitative analysis was conducted in order to evaluate the order-picking time of the pick-by-vision system. The detailed results of the ANOVA analysis on order-picking efficiency are presented in

Table 6. Results of statistical analysis (estimated effects) on order-picking time (efficiency). The symbol * relates the main effects as far as their interactions are concerned.

Source of Variation	Term	p-Value
Main Effects	Battery Position	0.55
	Type of Order	0.052
	Storage Level	0.732
	Confirmation Equipment	0
	Items per Order Line	0
	Order Lines per Order	0.173
2-way interactions	Battery Position*Type of Order	0.35
	Battery Position*Storage Level	0.314
	Battery Position*Confirmation Equipment	0.031
	Battery Position*Items per Order Line	0.925
	Battery Position*Order Lines per Order	0.256
	Type of Order*Storage Level	0.042
	Type of Order*Confirmation Equipment	0.004
	Type of Order*Items per Order Line	0.145
	Type of Order*Order Lines per Order	0.053
	Storage Level*Confirmation Equipment	0.014
	Storage Level*Items per Order Line	0.621
	Storage Level*Order Lines per Order	0.062
	Confirmation Equipment*Items per Order Line	0.008
	Confirmation Equipment*Order Lines per Order	0.219
	Items per Order Line*Order Lines per Order	0.019
3-way interactions	Battery Position*Type of Order*Storage Level	0.513
	Battery Position*Type of Order*Confirmation Equipment	0.085
	Battery Position*Type of Order*Items per Order Line	0.564
	Battery Position*Type of Order*Order Lines per Order	0.839
	Battery Position*Storage Level*Confirmation Equipment	0.411
	Battery Position*Storage Level*Items per Order Line	0.77
	Battery Position*Storage Level*Order Lines per Order	0.891
	Battery Position*Confirmation Equipment*Items per Order Line	0.374
	Battery Position*Confirmation Equipment*Order Lines per Order	0.251
	Battery Position*Items per Order Line*Order Lines per Order	0.111
	Type of Order*Storage Level*Confirmation Equipment	0.644
	Type of Order*Storage Level*Items per Order Line	0.003
	Type of Order*Storage Level*Order Lines per Order	0.932
	Type of Order*Confirmation Equipment*Items per Order Line	0.436
	Type of Order*Confirmation Equipment*Order Lines per Order	0.855
	Type of Order*Items per Order Line*Order Lines per Order	0.778
	Storage Level*Confirmation Equipment*Items per Order Line	0.386
	Storage Level*Confirmation Equipment*Order Lines per Order	0.092
	Storage Level*Items per Order Line*Order Lines per Order	0.128
	Confirmation Equipment*Items per Order Line*Order Lines per Order	0.005
4-way interactions	Battery Position*Type of Order*Storage Level*Confirmation Equipment	0.125
	Battery Position*Type of Order*Storage Level*Items per Order Line	0.815
	Battery Position*Type of Order*Storage Level*Order Lines per Order	0.729
	Battery Position*Type of Order*Confirmation Equipment*Items per Order Line	0.572
	Battery Position*Type of Order*Confirmation Equipment*Order Lines per Order	0.181
	Battery Position*Type of Order*Items per Order Line*Order Lines per Order	0.603
	Battery Position*Storage Level*Confirmation Equipment*Items per Order Line	0.552
	Battery Position*Storage Level*Confirmation Equipment*Order Lines per Order	0.797
	Battery Position*Storage Level*Items per Order Line*Order Lines per Order	0.53
	Battery Position*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.111
	Type of Order*Storage Level*Confirmation Equipment*Items per Order Line	0.171
	Type of Order*Storage Level*Confirmation Equipment*Order Lines per Order	0.963
	Type of Order*Storage Level*Items per Order Line*Order Lines per Order	0.048
	Type of Order*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.422
	Storage Level*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.43
5-way interactions	Battery Position*Type of Order*Storage Level*Confirmation Equipment*Items per Order Line	0.341
	Battery Position*Type of Order*Storage Level*Confirmation Equipment*Order Lines per Order	0.608
	Battery Position*Type of Order*Storage Level*Items per Order Line*Order Lines per Order	0.156
	Battery Position*Type of Order*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.34
	Battery Position*Storage Level*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.035
	Type of Order*Storage Level*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.942
	Battery Position*Type of Order*Storage Level*Confirmation Equipment*Items per Order Line*Order Lines per Order	0.371

.

Figure 9 presents a Pareto chart that verifies the validity of the ANOVA analysis, in that it distinguishes the factors that are statistically significant. After the obtained results have been taken into account, it can be seen that there are a number of factors and combinations of factors that significantly affect the efficiency of the pick-by-vision system under investigation. The results have shown that for cases H_0.1, H_0.2, H_0.3, and H_0.6, the null hypothesis is accepted, whereas for cases H_0.4 and H_0.5, the null hypothesis has been rejected. To this end, it can be concluded that the confirmation equipment and items per order line affect the performance of pick-by-vision system.

In Figure 10, the residual plots are presented. It can be seen that the normal probability plot follows a straight line and that the versus fits plot has randomly distributed residuals around zero. The histogram plot has a bell shape, and the versus order plot shape presents no specific pattern. Therefore, the data are highly reliable.

Now that the interactions between factors and configurations have been analyzed, we investigated the levels of the statistically significant factors and the system configuration that results in the shortest order-picking time. More specifically, according to Figure 11, it can be observed that using the scanner takes less order-picking time than using the RFID tag reader does. Furthermore, when items per order line are few, the picking efficiency is better than when the items per order line are many (Figure 12).

Another important factor that usually affects the performance of an order-picking system is the type of order. In our experiment, this factor was not considered as significant. This is because the tests were conducted in a laboratory environment where no significant travel distances exist. As it can be seen in Figure 7, the participants may reach the shelves from the area where the cart is by taking a few steps. The lack of travel distance affected the significance of the type of order parameter, and thus, the difference, in terms of picking efficiency, between discrete order picking and multiple order picking was eliminated.

Last but not least, according to the results of the statistical analysis, it can be concluded that the configuration of the investigated pick-by-vision system that provides the most encouraging results in terms of order-picking efficiency (i.e., order-picking time per order line) incorporates the following levels per parameter: battery position—weighted side; type of order—discrete order picking; storage level—high storage level; confirmation equipment—scanner”; items per order line—few; and order lines per order—few.

4. Perceived Workload Evaluation

A perceived workload evaluation can be accomplished by many means. NASA TLX is a widely used subjective multidimensional assessment tool that rates perceived workload to assess a task, system, or process [74], and according to the available literature, NASA TLX has achieved some solid goals in human-factors research, while assessing system design and development phases [75]. The NASA TLX is based on a weighted average of ratings on six subscales [74,76]. Three dimensions are related to the demands imposed on the subject (mental demand, physical demand, and temporal demand) and three to the interaction of a subject with the task (effort, frustration, and performance). According to a series of similar works [25,30,33,37,56,65], it can be seen that the NASA TLX methodology is the most suitable methodology to evaluate the perceived workload of the proposed pick-by-vision technology. According to NASA [74], the implementation of NASA TLX follows two steps. The first step deals with the source of load (weights) and the second step with the magnitude of loads (rating). Further information on the implementation step of NASA TLX methodology is available in the work of NASA [74].

Taking into account the aforementioned steps of NASA TLX methodology, we evaluated the proposed pick-by-vision system’s perceived workload. After completing a task (order picking), every participant filled the NASA TLX questionnaire, based on the aforementioned steps and the experimenter’s instructions. To this end, in Figure 13, the final results of the NASA TLX survey are presented. The pick-by-vision system scored M = 32.8 (SD = 9.1). The individual factors presented from low workload to high workload scored M = 23.4 (SD = 14.2) for performance, M = 26.1 (SD = 17.5) for mental demand, M = 28.0 (SD = 14.6) for physical demand, M = 30.4 (SD = 20.5) for temporal demand, M = 33.0 (SD = 16.7) for effort, and M = 33.6 (SD = 19.1) for frustration level.

According to these results, the overall NASA TLX score proves that the perceived workload of each participant is satisfyingly small. The overview of all the individual factors shows that the participants had no significant problems; thus, they escalated the subscales with small differences in order to distinguish them from each other. Additionally, as Casner and Gore [77] support, people who are overworking and people who are underworking exhibit similar performance as they both commit errors, have low efficiency, get frustrated, and have poor awareness of their surroundings. To this end, after the workload gauge from Casner and Gore [77] has been taken into account, it is clear that the overall NASA TLX score of the proposed system is kept on acceptable level.

More specifically, the highest perceived workload index came from the frustration level. Indeed, subjects admitted that difficulty in the scanning process was irritating them, and slight movements of the headbands caused them annoyance. Similarly, effort is the second-highest factor, as subjects concluded that compared with the rest of the factors, hard work came second. Additionally, the laboratory environment and the existence of a stopwatch forced the participants to pick intensely. As they admitted, the more familiar they got, the less intensity they felt. As far as the physical demand is concerned, the lack of travel distances and overall age avoided low scores, as no travel time existed and crouching and bending were not a problem for young people. In terms of mental demand, the subjects felt that they didn’t need to think about what to do, as they were fully guided throughout all the process. The lowest scoring factor was performance, proving that no special skills were needed to conduct the tests. Last but not least, in real warehouses, where professional pickers work for eight hours, travel distances are longer, the average age is higher, and no laboratory experiment is conducted, an increased NASA TLX score is expected to be observed.

5. Discussion

The objective of this study was to investigate certain operational and technical parameters that affect the performance of pick-by-vision technology in item-level order picking via a series of laboratory tests. The findings of this study are essential because they show the parameters (technical and operational) that affect the performance of pick-by-vision technology and may result in increased order-picking efficiency and accuracy. After the obtained results have been accounted for, it can be seen that there were a number of factors and combinations of factors that significantly affected the efficiency of the pick-by-vision system under investigation. The results showed that the confirmation equipment and items per order line affect the performance of the pick-by-vision system. In addition, it can be observed that selecting the scanner as the piece of order confirmation equipment reduces order-picking time over using an RFID tag reader. Furthermore, when items per order line are few, the picking efficiency is better than when items per order line are many.

According to the results of the statistical analysis, the configuration of the pick-by-vision system that provided the most encouraging results in terms of order-picking efficiency (i.e., order-picking time per order line) incorporated the following levels per parameter: (a) battery position—weighted side; (b) type of order—discrete order picking; (c) storage level—high storage level; (d) confirmation equipment—scanner; (e) items per order line—few; and (f) order lines per order—few.

In the perceived workload evaluation, the overall NASA TLX score proved that the workload of each participant was satisfyingly small. The overview of all the individual factors showed that the participants had no significant problems; thus, they escalated the subscales with small differences in order to distinguish them from each other.

5.1. Contribution to Theory

On the basis of these findings, it may be argued that the results provide some key insights into theory. First, a series of parameters were tested, and their role concerning the performance of pick-by-vision technology was assessed. Other studies, such as [14,34,68], have also evaluated a series of parameters, but not to the extent of the analysis that was made in this study. Furthermore, an evaluation of the parameters identified by the SLR method was made by practitioners of the logistics field. More specifically, the view of logistics managers was taken into account on practical issues (i.e., order picking process) and was analyzed by using the AHP method. In addition, although other studies have presented findings from laboratory tests concerning picking accuracy and efficiency when pick-by-vision technology was implemented, this work adopted the DoE methodology and a statistical analysis to identify correlations between parameters and the set-up of parameters that would provide the best result in pick-by-vision system performance. Lastly, this study also provided useful insights in theories on perceived workload. By adopting the NASA TLX methodology, pick-by-vision technology can be used by pickers with no significant problems. Other studies, such as [21,27], have also evaluated ergonomic parameters without, however, taking into account other technical and operational parameters that are also important during an evaluation of a pick-by-vision system. To this end, the main contribution of this work focused on the identification of key parameters that affect the performance of pick-by-vision technology and the development of a framework for the structured categorization of these parameters. Furthermore, this work presented significant results from laboratory testing and the best set-up for the pick-by-vision system for increased picking efficiency. Last but not least, the parameters tested in this work were selected on the basis of the answers received from the questionnaires (via the AHP method) completed by logistics managers who have significant experience in the order-picking process.

5.2. Practical Implications

From the obtained results and through interviews with logistics managers, a number of useful practical implications arise. The later deal mainly with (a) organizational culture, (b) process re-engineering, (c) staff resistance to change, and d) motivation for maintaining the new way of doing business. These implications are discussed below:

• Organizational culture: an effective digital transformation from typical order-picking methods to the pick-by-vision system needs more than updating the current technology. Automation tools are likely to create dissatisfaction among the workforce if not managed properly. An organizational culture is needed, the lack of which can cause the investment to fail and reduced performance.
• Process re-engineering: apart from the need for organizational culture, malfunction may be caused by a lack of necessary process re-engineering. It is thus crucial for companies to identify their needs and adjust their order-picking processes in accordance with the adopted new technology.
• Staff resistance to change: staff tend to resist to technological change because they believe that their position is in danger. Typical examples of such situations are met in the logistics sector when new order-picking techniques/systems are introduced. In order to keep the workforce and management united, continuous staff training, user-friendly systems, and technologies facilitating worker’s lives are essential.
• Motivation for maintaining the new way of doing business: after a complete and multilevel installation of an order-picking system, it is important to maintain the new way of doing business. Continuous improvement is required in order to maintain the interest of the user, as are suggestions for improvements from people working with the new system.

5.3. Limitations and Future Research

Although, the laboratory tests conducted have resulted in encouraging results on the performance of pick-by-vision technology, there are always opportunities for future research, such as in the following areas, which were not investigated in this study:

• Human factor: the human factor plays a critical role when new technologies are adopted, especially for the pick-by-vision system, where training and familiarization are necessary for its use. Despite the fact that most people can work over long periods with pick-by-vision headbands or glasses without being strained, there are still some people who find reading continuously from a smart device difficult. Issues that should be further investigated include ergonomics, mental and physical demand on users, performance, and frustration level.
• Technical issues: Another critical issue for future investigation is the technical aspect of pick-by-vision, especially when it comes to the user interface (UI). Furthermore, attention should also be paid to issues from integrating the pick-by-vision system into other systems, especially in the closer integration of augmented reality (AR) and warehouse management systems (WMSs), the increasing comfort of hardware components, and the potential connection of picking systems with automatic identification systems, such as RFID tags. Other technical issues to be further investigated may include battery life and scanning distance.
• Comparative assessment with other picking technologies: Lately, a comparative assessment of pick-by-vision with alternative picking technologies, such as voice picking and pick to light, have come up, but still there are many research opportunities in this area. Indeed, experiments should be conducted in order to assess the accuracy and efficiency of different picking technologies. Last but not least, it is worth evaluating investment costs in order to compare not only the performance but also the cost of obtaining an order-picking system.

6. Conclusions

The aim of this paper was initially to investigate certain parameters that affect the operational performance of a pick-by-vision system via a series of laboratory tests. A total of 27 parameters and three performance measurement indices were identified via a systematic literature review. Six of them were selected via an analytical hierarchy process (AHP), to be investigated further via a series of laboratory experiments by adopting the design of experiments (DoE) methodology. The proposed pick-by-vision system was investigated in terms of order-picking time and workload. The configuration of the pick-by-vision system that provided the most encouraging results in terms of order-picking efficiency (i.e., order-picking time per order line) incorporated the following levels per parameter: (a) battery position—weighted side; (b) type of order—discrete order picking; (c) storage level—high storage level; (d) confirmation equipment—scanner; (e) items per order line—few; and (f) order lines per order—few. The perceived workload of the pick-by-vision system was evaluated via a NASA TLX survey. The results are encouraging, showing that the pick-by-vision technology can be used by pickers with no significant workload inefficiencies.