Conceptual Framework for Evaluating E-Commerce Deliveries Using Agent-Based Modelling and Sensitivity Analysis

Abstract

To model the urban logistic environment, many variables need to be taken into account as this system is composed of different stakeholders with conflicting objectives and actions that influence the actions of others. Agent-based modeling (ABM) and simulations are useful tools for studying this environment with so many variables and uncertainties. Typically, urban logistic stakeholders are represented by agents who interact with each other and obey different rules. This study presents a conceptual framework for evaluating e-commerce urban freight using ABM and Design of Experiments (DoE). The simulations performed were designed to highlight the most important parameters that affect the response variable and should be taken into account in the experimentation process. The scenarios were studied to evaluate the use of Delivery Lockers (DLs) and failures in delivery attempts. The results of the DoE analysis allowed us to verify that the DL utilization rate, first attempt delivery failure rate, and truck stoppage time most impact costs and profits in the modeled system. The framework presented showed that the use of DoE in the simulation study reduced the number of simulated scenarios, which is a significant gain given the complexity of the model and the relatively large computational time required to execute a new scenario. Consequently, different studies may use the findings presented here to plan scenarios in their simulation projects.

1. Introduction

The growing popularity of e-commerce has led to an increase in the number of shipping vehicles in cities. These vehicles are responsible for transporting orders to consumers. In Brazil, e-commerce reached R$182.7 Bi in sales in 2021, representing 27% growth over 2020, growing in all regions of the country [1].

Most e-commerce orders require that consumers be physically present at their homes to receive their orders, since boxes or packages cannot always fit into mailboxes, or because some deliverers require signatures [2]. Sometimes delivery “failures” occur because consumers are not home when packages are sent out. Second trip(s) to consumer residences are often needed [3]. If delivery cannot be completed, even after a third attempt, logistics operators can offer different options to e-commerce customers. Some alternatives include leaving the order with a neighbor, leaving it outside the premises (not safe), or delivering the item to a nearby location (post office, distribution center, etc.) [4]. When this is not possible, items are returned to the sender. This results in high costs to either the logistics operators, the online retailers, or the consumers.

Regarding B2C e-commerce (Business-to-Costumer), Assisted Home Delivery (AHD) is generally preferred by consumers. This type of delivery is the most problematic solution in terms of service and programming costs [5]. In this modality, delivery workers make several stops during a trip, and count on people being home to make deliveries.

In most Latin American cities, such as in Brazil, e-commerce deliveries are all assisted, i.e., someone must be at home/work to receive deliveries [1,2]. AHD deliveries have disadvantages for consumers who require flexibility, and for delivery workers who need to optimize their distribution of goods. City logistics can minimize the inconveniences caused by this and have been the object of study in several reports [6,7,8].

These authors suggest consolidating last-mile deliveries as a solution to this problem. Delivery Lockers (DLs) have already been implemented in several countries [2,9,10,11]. DLs are automated lockers that consumers can use to retrieve and/or return online purchases.

It is important to study e-commerce deliveries given the large number of failed deliveries, high costs, and environmental impacts for home deliveries [7]. The main alternatives for home deliveries include reception boxes, delivery boxes, controlled access systems, and collection and delivery points [12,13,14,15]. According to [16], despite significant advances in freight transport modeling in recent years, there is still a lack of available tools for evaluating novel logistics solutions. In order to contribute to the recent literature, this study seeks to offer a structure to assess implementing Delivery Lockers. Agent-Based Modeling (ABM) and Design of Experiments have been used to investigate the most important e-commerce variables within the urban logistics context.

This paper is organized as follows: Section 2 presents a literature review, where the main references for the subject are presented; Section 3 describes the research method adopted and details the application of this method; Section 4 and Section 5 provide the analysis and discussion of the main results; finally, Section 6 presents the conclusions.

2. Background

2.1. E-Commerce Deliveries

E-commerce sales have been growing continuously over recent years. In 2020, an overall growth of 20.7% was forecasted [17]. This estimate has substantially increased with the COVID-19 pandemic due to changes in e-commerce patterns [18]. According to a forecast made by E-BIT (2021), online sales should grow even more in 2021, by 26%, with a revenue of R$ 110 billion more than the previous year [1].

Consumers who were already shopping via e-commerce before the pandemic felt even more secure and comfortable when using this purchase modality due to factors such as product variety, ease of price research, ease of purchase, and time savings [19]. The same seems to be true of new buyers. According to an Ebit/Nielsen survey, e-commerce peaked between April 5 and June 28, 2020, which corresponds to the height of social distancing in Brazil [1].

Growing B2C (Business-to-Customer) purchases lead to more deliveries in cities, and this leads to worsening urban freight (UF) scenarios, resulting in numerous conflicts and problems [15,20]. Many researchers have been studying city logistics measures to mitigate UF problems [8,9,10,15,20,21].

Logistics operators and online retail stores have started using DLs to minimize e-commerce delivery problems. DLs are automatic small-order delivery systems. Studies on using automated cabinets as an alternative to home deliveries have focused mainly on four topics: conceptualizing the topic, characterizing the demand for these services, developing mathematical methods and models to analyze the potential savings from using collection points, and studying Locker locations.

Table 1. Summary of papers that studied DLs.

Objective	Location	Authors
Conceptualizing the topic	-	[22,23,24]
Characterizing the demand for these services	The Netherlands	[25]
	The Netherlands	[26]
	China	[27]
	Poland	[7]
	Christchurch, New Zealand	[28]
	Belo Horizonte, Brazil	[29]
	Singapore	[30]
	China	[31]
Developing mathematical methods and models to analyze the potential savings from using collection points	China	[32]
	Rome, Italy	[33]
	Antwerp, Belgium	[34]
	Antwerp, Belgium	[35]
	Toronto, Canada	[36]
	-	[13]
	Belo Horizonte, Brazil	[15]
	Turin, Italy	[37]
	Athens, Greece	[12]
Studying Locker locations	Brisbane, Australia	[38]
	Valladolid, Spain	[18]
	Singapore	[39]

presents a summary of the works reviewed in this study, which focused on DLs.

One can see an evolution in the themes of the studies after analyzing recently published papers, which initially sought to conceptualize using lockers and have now moved on to dealing with consumer acceptance of lockers and modelling systems to economically and environmentally evaluate solutions. DL implementation, as an alternative to traditional delivery methods, is a current topic. However, according to [18], the successful use of this initiative depends on local characteristics (e.g., city street layouts, local e-commerce demands, etc.). We seek to complement previous simulation studies and contribute to general knowledge on the advantages of this delivery choice. To this end, we have established a framework using ABM and simulations along with sensitivity analysis using DoE.

The literature shows that a series of factors must be considered to simulate e-commerce deliveries, especially since this involves numerous stakeholders (agents) [40]. Modeling and simulating these processes must account for different agent behaviors and the many variables associated with the agents, which makes experimentation processes very long and costly. For this reason, we seek to highlight the most important parameters that impact the response variable, and which should be considered in the experimentation process, since not all variables impact the final result, and can even be eliminated from more detailed analyses in some cases [41]. We will use DoE to conduct this analysis, since DoE has already been used in other logistics studies outside the urban context [42].

2.2. Design of Experiments

The Design of Experiments methodology (DoE), was developed by Fisher between 1920 and 1930, and was improved upon by researchers such as Box, Hunter, and Taguchi [41,43]. It is based on statistical methods for optimizing experiments, planning, and execution, and analyzes the results using experimental error minimization and control, i.e., the effects generated for non-controllable factors [44].

DoE allows for systematic data analysis using statistical methods, leading to valid and objective conclusions. Since it is an experiment planning process, DoE allows researchers to estimate how experimental results or responses are affected from predetermined changes in factors of interest [43].

DoE allows for planning and carrying out computational experiments using simulations to obtain more reliable results, reduce variability, and allow for process improvements. DoE also helps with analysis and identifies the most important variables within a process to discover if there are interactions among the variables [45]. The authors of Ref. [42] used DoE in their simulations to assist in orderly identifying factors that impact economic performance the most.

DoE evaluates the effects of multiple factors on responses using a simultaneous study of these factors at different experimental levels. The direct consequence of this is reduced analysis time on the results and reduced costs for the entire experimental process [43]. For these authors, the following steps must be used in the statistical approach for experiment planning and analysis:

• Identify and state the problem;
• Choose the factors, the levels, and the ranges;
• Select the response variables;
• Choose the experimental design;
• Conduct the experiment;
• Make conclusions and recommendations.

After identifying the problem, defining the factors and their respective levels, and selecting the response variables, we chose the experimental design. Complete factorial, fractional factorial, response surface, and mixed planning arrangements are most commonly used [43]. We used General Full Factorial Planning in this study to completely scan the study region since this uses all factors and respective levels (at a 95% confidence level). After simulation, statistical methods were used to estimate the effects of the studied factors, such as Analysis of Variance (ANOVA). Then, the results were interpreted and analyzed.

There are many factors that can influence the response of a system when modeling and simulating, especially when this involves several agents, making the experimentation process long and costly. Sensitivity analysis should be conducted to identify the most important parameters that impact the response variable, since not all variables are of equal importance, and some can even be eliminated from the analysis [41].

Regarding using simulations and DoE together, the authors of [46] stated that DoE maximizes the usefulness of the information generated in simulations, minimizing decision-making efforts. This is because it improves simulation performance, since trial and error techniques can be avoided when seeking solutions. According to [47], the benefits of using DoE include system performance improvements when searching for good configuration solutions, avoiding trial and error approaches. DoE has been widely used in simulation studies [48,49]; however, the DoE technique applied to simulation studies for logistics has not been widely applied, especially when simulations are based on agents [50,51,52].

3. Agent-Based Simulation and Modeling

Modeling and simulation based on agents is somewhat different because the object of study is urban logistics; there are some difficulties since several agents with conflicting objectives are involved. Each agent acts differently and expects specific results. For logistics dealing with e-commerce deliveries, we identified no studies that used ABM to analyze impacts with urban Delivery Lockers combined with DoE. This study seeks to contribute to the literature by offering a framework for assessing the use of Delivery Lockers for e-commerce deliveries with ABM and DoE.

We developed a framework with known steps from modeling and simulation studies from [53,54], which were adapted to account for multiple agents within urban logistics (Figure 1).

Figure 1 shows that the proposed structure comprises three main stages: the model conceptualization or formulation, model implementation, and result analysis. The first phase (or stage) allows us to understand the system that will be simulated and the objectives, including building the conceptual model, validating this model, and documenting and modeling the input data. The second phase consists of building, implementing, and validating the computational model. During the analysis phase, the computational model carries out the experiments. The model performs several “runs”, and the simulation results are analyzed and documented.

3.1. Design

3.1.1. Building and Documenting the Conceptual Model

Information on agent behavior when performing actions was needed to build the model. Then, the outputs were generated to analyze the scenarios. The model considers interactions among the four main agents in the e-commerce delivery processes:

• The Carrier;
• The E-commerce store;
• The Delivery lockers;
• The Customer.

The conceptual construction of each agent will not be presented since the focus of this study was to combine the use of ABM and DoE. However, this can be consulted in [15].

3.1.2. Validating the Conceptual Model

The conceptual model was presented to a group of specialists in the urban transportation field, including researchers and people who are responsible for delivery and routing processes for e-commerce carriers. These experts assessed whether the actions and interactions of each modeled agent were consistent with real-life situations. The model underwent two rounds of adaptations. According to the experts, the final conceptual model adequately represented the system. After validating the conceptual model, we then proceeded to building the computational model.

3.2. Implementing the Model

3.2.1. Building the Simulation Model

The conceptual model was implemented in a computational environment. The computational model was developed and implemented in Anylogic^®. This software program is widely used in ABM studies on urban logistics [15,55,56]. It is very useful since it has a Geographic Information System (GIS) map, in addition to ABM functionalities.

In the computational model, the DLs and Distribution Center (DC) are fixed on the map. The client is randomly allocated, and can choose a route or move, just like trucks. Since agents follow the same modeled rules (interactions and actions), this model can be generalized to other regions or cities, since the GIS can be easily changed, along with the input parameters that support the model (

Table 2. Computational model input parameters.

Agent	Parameters
Client	Number of orders per year
	Location (latitude and longitude)
	% Customers receiving deliveries at home
	% Customers receiving deliveries at a DL
	Wait time for receiving orders
Order	Number of orders per day
	Volume
	Weight
	Delivery address: Home or DL
	% of orders not delivered in the 1st attempt
	% of orders not delivered in the 2nd attempt
	% of orders not delivered in the 3rd attempt
Delivery Lockers	Location (latitude and longitude)
	Number of Customers
Carrier	Number of trucks
	Truck stoppage time
	Home client unloading time
	DL client unloading time
	Associated load
	Average truck speed
	Truck capacity

).

The deterministic or stochastic values for the parameters can be defined according to the situation that will be simulated. In the case of this study, the input parameters that were not used in the sensitivity analysis such as order weight and volume were used for statistical probability provided by the carrier, as in [15].

3.2.2. Implementing the Computational Model

The computational model was implemented for the “Contorno” region of Belo Horizonte, the capital of Minas Gerais state, Brazil. The city has approximately 2.5 million inhabitants spread across 331 km². It is the sixth most populous city in Brazil [57]. The region received its name because it is limited to Contorno Ave. This area was chosen given the availability of data on it, and because it is an economically relevant area in Belo Horizonte. It has a high degree of urbanization, leading to delivery-related problems in this region [58]. Furthermore, this region is characterized by its high population density, mainly high-income individuals. Given the results from [29], this region contains a significant portion of potential e-commerce consumers that either reside or work there.

Currently, most e-commerce deliveries in Belo Horizonte occur at homes. In Brazil, these deliveries follow the three-attempt policy; i.e., when delivery cannot be made after the first attempt, two additional attempts can be made over the following days. This policy results in more trips, and consequently, increased freight costs. This study seeks to investigate an alternative for re-deliveries by implementing DLs, and to estimate the impacts of this new system. As identified in [29], 30% of the population shops online, i.e., about 28,105 customers in this region. In the model, the customer homes were located in the Contorno region. Geographically, these houses were arranged homogeneously within the region.

The number of DLs in the model was 5 for every 100,000 inhabitants. This number was based on data from cities where these systems are being used [5]. There are 93,684 inhabitants in the region, so five DLs were used in the simulation. Regarding the location of the Delivery Lockers, according to [25], supermarkets are most commonly used. E-commerce customers stated that this was preferred in [29]. We chose supermarkets in the Contorno region.

We simulated for a small cargo vehicle; these are most frequently used by couriers and e-commerce carriers for last-mile deliveries since they are not restricted from circulating in these areas (specifications = 1500 kg, volume = 10 m³). Equation (1) gives the number of trucks (N_T) necessary for making last-mile deliveries as a function of daily demand (Demand) and the number of orders/day that each truck can deliver (Load Size):

$$N_T=\frac{Demand}{Load Size}$$

(1)

The number of orders delivered by trucks varies according to the average stop time, the number of orders delivered during one stop (order density), and driver productivity, according to interviews with carriers from different cities, and based on literature. This varies between 50 to 120 orders/day per truck.

One load would contain around 300 orders if the load was determined as a function of a truck’s capacity (Volume and Weight), since orders delivered to lockers are small and light. According to the interviews, carriers do not recommend delivering this many orders because of potential cargo theft, in addition to the fact that it would be impractical to deliver so many packages during a single working period. Thus, the load size was obtained by experimenting with the software program, with the travel time (Equation (2)) taking into account working hour limits (8 h/day, with a 10% tolerance limit to this value).

From the model, we obtained the en route time (T_trip), the number of unloaded orders (n_o) (those which are actually delivered), and the number of failures (n_f) (orders that were not delivered). The sum of the failures and unloaded orders is the Load Size that a truck can deliver in a specified number of hours, considering the stop (T_stoppage), unload (T_unloading), failure (T_failure), and route (T_route) times. The first three are model input parameters.

$$T_{trip}=T_{Route}+T_{stoppage}+T_{unloading}\times n_o+T_{failure}\times n_f$$

(2)

The number of trucks (Equation (1)) needed to deliver simulated daily demand can be calculated using the Load Size. Since there can be scenarios with high re-delivery rates, it is important to emphasize that experimentation within the model must be as careful as possible. Therefore, daily demand increases as a result of re-deliveries.

An experimentation test was carried out to obtain the initial value for the load size, respecting the working hour limits, to obtain the initial number of trucks needed. Subsequently, other tests were carried out (as many as necessary) to verify whether the number of trucks could deliver all the orders. For example,

• Daily demand = 80 orders/day;
• Truck Stoppage Time = 5 min;
• Truck Unloading time = 2 min;
• Failure time = 1 min.

We determined that the load size for the demand would be 40 orders/truck using the first experiment. From Equation (1), we determined that two trucks would be needed. However, a second trial showed that two trucks were insufficient for delivering all orders, given the high number of re-delivery orders. Therefore, other experiments were carried out, and we determined that three trucks would be needed.

According to Equation (2), delivery time encompasses en route times, offload times, and Truck Stoppage Time. However, the unloading time is short compared to the Truck Stoppage Time (which includes finding a parking space and walking to the delivery location), because deliveries are individual and small. For the unloading time, we adopted an average triangular time distribution of 2 min for deliveries, and 1 min for when customers were not home.

The interviews showed us that the average truck speed (input parameter) in cities is around 12 km/h for last-mile deliveries. This value was also used by other researchers in several other studies [34].

In addition to truck speed, we also needed to account for the DL deployment rate, the delivery failure rate for the 1st, 2nd, and 3rd attempts, and the Truck Stoppage Time. We decided to carry out sensitivity analysis to observe the variations of these parameters and how they influence the model’s results, since these values are continuously changing in the literature, and these data are difficult to obtain from carriers.

3.2.3. Verifying and Validating the Computational Model

The verification process occurred in two stages, as per [59] and as suggested by [53]. First, the logic and elements of the computational model were compared with the logic and elements of the validated conceptual model. Then, the second step was performed using the debugger build model (Anylogic® assists in locating and debugging errors) in Anylogic^®. During this verification process, errors were found and corrected. Most errors were related to incorrect JAVA programming. Verification was completed when structural and logical errors were no longer identified in the constructed computational model.

After verifying the computational model, we then validated the model. The authors of [60] stated that although high complexity is an ABM differential, this complexity does not guarantee that each simulation run will follow the same sequence, which can lead to conflicts in the final output, making validations different from well-controlled experiments. The authors also stated that commonly used statistical validation procedures do not work properly for this type of simulation, and suggested face-to-face validation mechanisms for good results.

Face-to-face validation is a widely used technique suggested by [61]. Similar to the validation for this conceptual model (to ensure that the model satisfactorily represented the simulated system), face-to-face validation techniques were used to validate the computational results. According to [54], face-to-face validation involves the analyst (the person who built the model) discussing results with people who really understand the processes that were simulated.

The simulation animations and the results were presented to managers at two e-commerce carriers. These managers plan shifts and schedule deliveries. We observed that the real-life processes corresponded well with the results from our computational models, for both companies. We presented these managers with agent actions and interactions to verify if these were consistent with reality, since we were dealing with ABM. Based on the animations we presented them with, they affirmed that the agent actions and interactions were indeed consistent with real-life processes.

We discussed validating the scenarios for implementing DL with a manager of a transport company that is participating in testing this initiative in São Paulo and Rio de Janeiro. According to this manager, the delivery processes for DLs were well represented within the model, since no major construction changes were needed, only cargo consolidation, which was carried out correctly by the agents.

After validating the model, we analyzed the number of replications and simulation times that would be used in each experiment. We combined 3, 5, 10, and 15 and replications for 15, 30, 180, and 365 days. There are no large variations between 3, 5, 10, and 15 replications for experiments that simulate 365 days. Thus, we decided to use 365 days and 3 replications to run the scenarios. The model run time was a determining factor when choosing the number of replications, which for example was 12 h for 365 days, making it computationally unfeasible to run many replications, in addition to the statistical replication analysis.

3.3. Results Analysis

3.3.1. Cost Analysis

The simulation results for each agent were analyzed in terms of costs and gains in each scenario to obtain global comparable results for each scenario. The costs and gains were as follows:

• Truck Fuel Costs (F_CT);
• External Costs for the Truck (E_CT);
• Truck Time Costs (T_CT);
• Customer Fuel Costs (F_CC);
• External Customer Costs (E_CC);
• Re-delivery costs (R_C);
• Hosting Gains (G_H).

The equations used to calculate the aforementioned costs and gains are described in [15].

3.3.2. Sensitivity Analysis

Sensitivity analysis can be performed on simulations to select the most important variables, or to select variables that can be eliminated from more detailed analyses [41]. The DoE methodology aided in the analysis, and allowed us to identify the most important variables in the model, in addition to finding out whether there were interactions among variables [41]. The simulation model parameters (variables) are called factors [47]. Other terms used when designing an experiment are “level”, i.e., the possible variations for each factor, and “step”, i.e., the step or difference between one level and another [62]. Finally, the “Response Variable” measures the simulation model performance or output [63]. Combining all factor levels creates a DoE scenario that can then be simulated in our model.

The most widely used DoE techniques in the literature are Full Factorial Planning, Fractional Factorial Planning, Taguchi arrangements, and Response Surface Methodology [64]. We used General Full Factorial Planning here. Full Factorial designs completely scan the study region since they use all factors and levels (at a 95% confidence level).

Minitab^® was used to create an experimental matrix of the DoE and for the statistical analyses that were later carried out. For more information on DoE, see [43].

Table 3. ANOVA for output variables.

	Level −	Level +	Step
%DL	0	0.45	0.225
%I1	0.05	0.25	0.2
%I2	0.5	0.8	0.3
%I3	0.6	0.8	0.2
TP	5	10	5

shows the input parameters that made up the DoE and the levels of each.

Table 3 gives %DL, which is the locker use rate among clients. This is the only parameter with three DoE levels. We took a value similar to a result from [29] for “level +”, which indicates a 43% propensity for e-commerce customers in Belo Horizonte for using DLs. This was taken as the highest value, because no DL use rate exceeds this value in the literature on countries that have already employed lockers [35]. The other levels were smaller than this, and “level −” represents situations where DLs are not used. We decided to simulate scenarios using an intermediate value at 25.5%, since DL implementation is the central focus of this study.

Regarding delivery failure rates, for the first attempt (% I1), we searched for values in the literature and in interviews with carriers, both for “level +” and “level −”. Via interviews with carriers, we found that failure rates in some places may be much lower than what the literature accounts for. According to the carriers, this is the result of strategies such as calling absent customers, leaving the orders with neighbors, or leaving orders with building receptionists. They stated that in Brazil, having entrance lobbies with receptionists in buildings in large cities leads to higher delivery rates. Therefore, we used a 5% failure rate for the first attempt, as indicated by the carriers for “level −”. We used a 25% value for “level +” since this was the most commonly used value in the literature [3]. According to [3,34], first attempt failed deliveries are more likely to fail on the second or third attempt, ranging between 50% and 80%. Thus, higher failure rates were adopted for the second and third attempts. “Level −” at 50% and “level +” at 80% were used for failure rate 2 (%I2). Since failure rate 3 tends to be higher than failure rate 2, we used 60% and 80% for “level −” and “level +”, respectively.

Sensitivity analysis was performed for the stoppage time, which directly influences the total trip time. In [29], it was identified that the average stoppage time for the Contour Region was 9.5 min. According to the carriers, this time can vary depending on location. Some studies on e-commerce deliveries adopted an average Truck Stoppage Time of less than 9.5 min [34,65]. We used 10 min for “level +”, as per [29], and 5 min for “level −”.

We developed the DoE experimental matrix and simulated the scenarios after defining the factors and levels. Scenarios with demand at 80 orders/day and three delivery trucks were simulated for the experiment. A total of 48 different scenarios were created from combining all parameters and levels with each other.

4. Numerical Analysis of Results

Analysis of Variance (ANOVA) was used to estimate the effects of the factors on the response variables after running the experiments. The response variables of interest are those that were used to calculate costs in this study, i.e., the total distance traveled by trucks, the average distance traveled by customers, the average travel time, and the costs. ANOVA was calculated for these variables (

Table 4. ANOVA for output variables.

ANOVA For	Total Truck Distance	Average Trip Time	Average Distance of the Customers
DL	0.000	0.000	0.000
%I1	0.000	0.000	0.178
%I2	0.329	0.157	0.558
%I3	0.896	0.868	0.178
TP	0.014	0.000	0.332
DL*%I1	0.048	0.003	0.313
DL*%I2	0.930	0.777	0.707
DL*%I3	0.802	0.645	0.485
DL*TP	0.667	0.000	0.763
%I1*%I2	0.333	0.824	0.332
%I1*%I3	0.769	0.349	0.558
%I1*TP	0.599	0.003	0.558
%I2*%I3	0.343	0.190	0.086
%I2*TP	0.505	0.618	0.845
%I3*TP	0.551	0.439	0.086
R²	95.07	99.54	99.73
R²(adj)	91.41	99.20	99.53
R²(pred)	84.40	98.55	99.14
p-value	0.875	0.154	0.243

) and for costs (

Table 5. ANOVA for costs.

ANOVA For	FCT	ECT	TCT	RC	ECC	FCC	GH
DL	0.000	0.000	0.000	0.000	0.000	0.000	0.000
%I1	0.000	0.000	0.000	0.000	0.158	0.158	0.766
%I2	0.329	0.329	0.137	0.000	0.805	0.805	0.599
%I3	0.896	0.896	0.961	0.001	0.730	0.730	0.696
TP	0.014	0.014	0.000	0.577	0.463	0.463	0.154
DL*%I1	0.048	0.048	0.000	0.000	0.134	0.134	0.764
DL*%I2	0.930	0.930	0.742	0.022	0.883	0.883	0.293
DL*%I3	0.802	0.802	0.875	0.920	0.706	0.706	0.486
DL*TP	0.667	0.667	0.000	0.000	0.681	0.681	0.591
%I1*%I2	0.333	0.333	0.097	0.028	0.845	0.845	0.090
%I1*%I3	0.769	0.769	0.515	0.790	0.970	0.970	0.235
DL*TP	0.599	0.599	0.000	0.213	0.782	0.782	0.033
%I2*%I3	0.343	0.343	0.435	0.376	0.038	0.038	0.707
%I2*TP	0.505	0.505	0.851	0.462	0.896	0.896	0.420
%I3*TP	0.551	0.551	0.618	0.384	0.414	0.414	0.777
R²	95.0	95.07	99.76	99.91	99.42	99.42	99.99
R²(adj)	91.47	91.41	99.58	99.84	98.99	98.99	99.98
R²(pred)	84.40	84.40	99.24	99.71	98.17	98.17	99.99
p-value	0.201	0.875	0.512	0.717	0.054	0.054	0.053

).

Based on the statistical analysis, we observed that the R² values are very close to 100% and indicate that the model fit is satisfactory. The adequacy of the statistical model was verified using the Anderson Darling normality test for the residuals. The residuals of all the analyzed response variables showed adequate behavior under normal experimental conditions. The Anderson Darling test cannot reject hypotheses for residuals that follow normal distribution, since all found p-values are greater than 0.05.

According to the results given in Table 4 and Table 5, we can see that not all five factors significantly interfere with the response variables. The p-values smaller than the significance level are highlighted in the tables (p-value < α and α = 0.05), and the corresponding variables have the greatest influence on the model. Note that the %DL parameter has significant impacts on the response variables. The failure rate for the first delivery attempt (%I1) and the stop time (TP) have significance for all results that deal with the truck, i.e., the distance traveled by the truck, the average travel time, the fuel cost for the truck, the external cost for the truck, and cost for time.

The only response variable that showed significance for %I2 and %I3 was the Re-delivery Cost. This was expected, since this cost is calculated as a function of the number of undelivered orders. Even though the percentage is small, there is a difference between scenarios with different parameters.

The factors that affect the response variables in the second order interaction are DL*%I1, DL*TP, and %I1*TP. The only exception is the external customer cost, with a significant interaction between %I2*%I3, and re-delivery costs, with interactions between DL*%I1, DL*%I2, DL*TP, and %I1*%I2. DL*%I1.

The parameter interaction for each factor in each response variable is shown graphically in Appendix A. The horizontal reference line represents the overall mean for the data, and the straight lines representing the factors show the magnitudes of the effects. The greater the slope of the line, the greater the impact (positive or negative) that the factor has on the response variable.

Based on the parameter interaction graphs, we can confirm that the most influential parameters on the responses are %DL, %I1, and TP. Although %I2 and %I3 show significant variance for re-delivery costs, the graph shows that %I1 has a greater slope than the others. Therefore, we can conclude that parameters %DL, %I1, and TP significantly change most results, and consequently costs, when varied. These are relevant factors for consideration when developing simulation scenarios.

DoE is very useful for simulation studies, similar to the one conducted here, since we were dealing with many agents, while each one had several input parameters. This technique allowed us to save simulation time, in addition to pointing out the most significant parameters. Without DoE, we would have had to simulate 48 scenarios, varying all parameters. DoE reduces this by 75%, requiring that only 12 scenarios be simulated.

5. Discussion

Our results point to the first delivery attempt failure rate as being an input variable that impacts the simulation results, and consequently the logistical operational costs (see Appendix A). These data have not been researched much, although they are very important, and there are few studies that estimate delivery failure rates [66]. Most data are out of date, or do not converge to a common value. First delivery attempt failure rates in the literature range from 11 to 30% [3,4,67].

One explanation for this variation is differentiated services provided by some carriers that notify customers when deliveries were attempted and no one was at home. According to [66], this increases the chance of delivery during the second attempt. An order tracking option to inform customers of the delivery status and the ability to choose different delivery locations also contribute to increasing delivery success rates [4].

As a study on e-commerce and last-mile deliveries, [66] showed that e-shoppers in the UK increasingly use different places to receive goods, e.g., at work, click and collect at stores, pick-up points, and Delivery Lockers [68]. In São Paulo, Brazil, e-commerce consumers have also adopted new delivery options for receiving goods, as there are now more DLs available (+30% during the pandemic) [69]. According to [66], delivery success rates are higher when the delivery location is not set as the customer’s home.

The DoE aided us in verifying that DL use rate is another factor that influences logistical operational results. Various authors have been studying and highlighting the advantages of using DLs for some time [2,11,14,15]. In fact, DLs reduce unsuccessful attempts and can reduce re-delivery cost and help improve the reputation of e-commerce retailers [15].

According to [15], another benefit of implementing DLs is providing extra income for stores that host the DLs. Extra flows of people visiting stores with DLs can increase sales. According to [7], for every four DL users, one will make purchases when collecting their orders. Despite the numerous advantages, these systems could lead to customers making extra trips to pick up their orders using their own private vehicles if these systems are poorly planned. To minimize this, the authors of [38] stated that location is important for system success, since this can facilitate trips; i.e., customers can make trips to DLs while on their daily commutes [7,18,38,39,70].

In addition to the first attempt delivery failure rate and the DL implementation rate, Truck Stoppage Time was one input parameter that significantly affected the results. This can be explained as follows: the number of orders delivered by a truck per day is smaller when simulating scenarios with longer Truck Stoppage Time; thus, more trips are required to deliver the same number of orders as during scenarios simulating with smaller Truck Stoppage Time. The number of trips is used to calculate the Cost with Time, which explains why this parameter has a direct impact on the results. According to [34], a higher delivery density is crucial for achieving last-mile delivery efficiency.

The COVID-19 pandemic has brought about substantial changes to online consumption patterns. According to [18], the “higher-than-expected” growth in online sales has brought new challenges to urban logistics stakeholders. Studies such as the present one that seek to understand the main variables and parameters for e-commerce deliveries are ever more important now. It still cannot be said whether online sales volumes will return to pre-pandemic levels, or whether consumer behavior changes will be irreversible [71]. Some researchers have tried to understand this behavior [21]. What is known is that the pandemic accelerated e-commerce growth [19]. It forced companies to change their sales and delivery strategies, in order to provide consumers with more safety and less physical contact. This meant that they could rethink traditional delivery modes [37], and implement different initiatives to promote social distancing, e.g., DLs. In this uncertain scenario, the use of agents is very useful for capturing changes, increasing precision in decision making. In this sense, the framework presented in this study can also help freight carriers and online store operators to increase the accuracy of decision-making and evaluation scenarios.

Although this study was on a specific region in Belo Horizonte, the results can be generalized to most large Brazilian and Latin American cities, since deliveries are usually not left unsupervised on the street, given the risk of robbery. Since e-commerce is still developing, stores cannot offer different delivery options to their customers, e.g., a service window that is compatible with convenient times for customers. This means that, in most cases, there is a greater chance of re-deliveries, because delivery times for transport companies usually coincide with when customers are at work. Studying using DLs is useful for customers, since it offers them more flexibility and could reduce the number of re-deliveries for carriers.

6. Conclusions

This study sought to model, build, implement, and analyze the implementation of Delivery Lockers as a last-mile solution for e-commerce deliveries using Agent-Based Modeling and Simulation (ABM), and by using “what if” scenarios. This article contributes to scientific knowledge surrounding the use of ABM as a support tool for decision-making applied to urban logistics. The main theoretical achievement of this study is the framework presented, which can be generalized to other cities and regions, since the studied agents follow the same rules that were modeled here, and since the GIS region can be easily changed, along with the input parameters that support the model. The main scientific contribution is using ABM and DoE together to assess e-commerce deliveries, highlighting which parameters most influence the system, as presented in Appendix A.

The model is extensive, and was implemented in Anylogic^®. We sought to investigate a solution for e-commerce last-mile deliveries, in terms of external operating costs and profit. Given the complexity of the studied system, the model was effective, generating coherent solutions, which allowed us to represent different situations under different scenarios. The simulations were justified given the many variables, which need to be studied together and systematically, to evaluate their influence on each other. Given the large number of input parameters in the model, Design of Experiments was used to measure the degree of relevance for certain parameters and their impacts on the system.

The sensitivity analysis provided by the DoE performed here (Appendix A) allowed us to verify that the DL use rate, the first attempt delivery failure rate, and the Truck Stoppage Time most impact costs and profits in the modeled system. This technique allowed us to reduce the number of simulated scenarios, which is an important gain, given the complexity of the model and the relatively large computational time needed to run a new scenario. Therefore, different studies can use the results presented here to plan scenarios in their simulation projects.

This study contributes to different areas, showing how e-commerce deliveries are performed and what improvements can be made for sustainability. This research can be extended to other fields of knowledge, such as the study of last-mile delivery; consumers’ behavior, which could provide insights into the decision-making process of future customers of DLs; and especially urban logistics, to investigate the use of DLs combined with other city logistics measures such as cargo cycles, electric freight vehicles, and off-hour delivery. A study assessing the impact of combining these measures would make a positive contribution to the scientific community interested in last-mile delivery.

The higher-than-expected increase in e-commerce sales caused by the COVID-19 pandemic has resulted in more individual deliveries, further aggravating the negative impacts of urban freight. Future studies could assess how sales growth impacts factors such as CO₂ emissions, number of vehicles, and routes, in addition to the parameters studied here.

Data collection is of critical importance when conducting simulation studies. In this study, because it is a complex, multi-stakeholder model, we found that it was very difficult to gather information. Future studies that better explore this issue, especially in terms of congestion, loading and unloading times, and speed, would make a good contribution to the scientific community interested in last-mile deliveries. Finally, we emphasize that these limitations do not invalidate the gains achieved with this work, since it successfully achieved the goals of modeling the stakeholders of e-commerce urban freight transportation and evaluating the implementation of Delivery Lockers.