A Data-Intelligence-Driven Digital Twin Framework for Improving Sustainability in Logistics

Abstract

As supply chains evolve toward the adoption of the Industrial Internet of Things (IIoT), vast amounts of data are collected by different systems across the manufacturing, logistics and transportation value chain. John G Russell (Transport) is a UK-based company involved in multiple lines of business in the supply chain. As the company adopts the utilization of data intelligence as a way to collect, process and utilize data for insights, this presents an opportunity for applying artificial intelligence (AI) approaches such as reinforcement learning (RL), to identify trends, and offer recommendations for improving the sustainability and efficiency of its logistics. Preliminary results show that we can achieve up to a 20–30% reduction in carbon emissions from the fleet of a segment of the transport business lines of the Russell Group. This paper presents a holistic framework for achieving sustainable supply chains, reducing costs as well as achieving operational efficiency using a supply chain digital twin.

1. Introduction

A report by EY in [1] discussed how businesses in supply chain identify the need for managing sustainability but lack the technological innovation, and often, business case identification for achieving this cost effectively. In terms of the technology requirements, the concept of Supply Chain 4.0, as described in [2], constitutes emerging technologies such as the Internet of Things (IoT), cloud computing, big data analytics, machine learning, etc., applicable within procurement, warehousing, manufacturing, logistics and transportation, which provide tools capable of improving performance and resilience, and reducing costs, in supply chain management. A technology that combines the application of these emerging technologies is a digital twin (DT). Digital twins (DTs) can be seen as a replica of a physical asset in the digital space, with synced operations and data, in real time, suitable for simulations and applying analytics [3]. Business-case-wise, this work focuses on sustainable logistics through the application of a data-driven DT framework to support logistics sustainability management, which can be described as an integration of eco-social governance with supply chain processes, balancing profitability and sustainability for organizational growth [1,2]. We use our vast amount of data as a valuable asset [4] to further enhance our transport managers’ capacity to improve the business using digital tools [5]. In our case study, we focus on logistics, where costs and emissions can be reduced by optimizing operations.

One of the most prevalent challenges in logistics is empty running. The report in [6] defined empty running as “trips by a transport vehicle without any freight loaded” [6]. This work proposes a solution that leverages the distributed digital framework discussed in [7] to stream real-time location and carbon footprint data of the fleet and incoming job orders from the enterprise resource planning (ERP) system to offer recommendations for the optimized planning of pick-ups and drop-offs of orders to minimize empty runs, which constitutes a costs and emissions reduction. The recommendation system is based on a reinforcement learning with human feedback model and a traveling salesman problem algorithm for route optimization. The DT framework provides a solution architecture for handling time and resource constraints efficiently, using historical data and real-time data, along with data, edge, and cloud platforms.

1.1. Contributions

In this paper, we contribute the following toward achieving our goal of using an intelligent digital twin for improving sustainability in our transport business.

• Conceptualization and design of a digital twin framework leveraging data integration to create a realistic digital replication of our assets and operations.
• Use of the DT framework to integrate multiple data sources and train and test a reinforcement learning model with the goal of reducing empty running and optimizing job allocation.
• Showcase preliminary results on the impact of applying the model.

1.2. Related Work

The backbones of DTs are the enabling technologies based on Industry 4.0 concepts [8], a solid two-way automatic dataflow between the physical object and the digital object via a connector [9] and the use of this physical–digital fusion to extract knowledge [10]. Some works in the literature have explored the concept of a supply chain digital twin. For instance, the work in [11] described a supply chain digital twin as a simulation model of a supply chain capable of predicting dynamic behaviors to make immediate decisions (short to medium term). The authors of [11] discussed how the need for DTs in supply chains is important to overcome the limitations of ERPs in including real-time feedback and responses to dynamic supply chain environments when dealing with orders, processes, and assets. One of the solutions highlighted in the literature is the digital twin features provided by AnyLogistix supply chain software [12], which provides tools for leveraging ERP data for integration with IT infrastructure and business intelligence (BI), along with supply chain resilience analysis. In the systematic review in [2], the insights from analyzing the performance benefits of applying key emerging technologies to different supply chain companies included an improvement in productivity, demand forecasting, traceability, customer satisfaction, responsiveness, decision making and so much more. While [2] did not describe a case study for a supply chain DT with company data, the capability of a DT to utilize the Supply Chain 4.0 technologies to handle both commercial and sustainability benefits was evident. Our work showcases the potentials of DTs with some more context.

As a supply chain DT must handle big data to be responsive in real time, there exist challenges. The work in [13] identified some of the challenges when handling big data for transportation systems—from the modeling, processing, and analyzing to visualizing aspects. The authors highlighted some challenges of intersecting the Supply Chain 4.0 technologies of the IoT, cloud computing and big data to include real-time data analytics (lambda architectural issues), data sources and security and privacy. Challenges associated with traffic management are often solved by machine learning approaches, as discussed in [14]. Some of the traffic management challenges adopted in our work include the use of real-time geo-positioning as a data source for some of the algorithm workflows in the proposed DT.

In relation to routing optimizations, reinforcement learning has been explored in [15] for multi-modal transport in supply chains, focusing on optimizing the costs, including emissions, but this requires further refinement for optimal decision making. The proposed DT in this work considers the human feedback for such a solution. While the development of DTs is not fully standardized, hence the lack of an all-purpose technological platform that could perform all the activities peculiar to a business case study, adopting standards such as ISO-23247 can significantly help improve the benefits of applying DTs [7].

Researchers continue to explore strategies for achieving NET Zero by 2050, such as alternative energy sources like adopting synthetic fuels for decarbonizing transport [16], swarm intelligence for traffic optimization [17], and the application of AI approaches for optimizing power electronics [18]. Furthermore, Industry 5.0 presents opportunities for sustainability through adopting 5G for intelligent transport [19], the Internet of Things (IoT) as an enabler of transport sustainability [20], and fault diagnosis [21]. The study in [22] identified the challenges of verifying the confidence interval of optimization algorithms based on machine learning approaches. The work in [23] showcased the framework and implementation of a digital twin for collaborative logistics solutions to the emissions and resources wastage associated with truck logistics. While the simulation in [23] showed potential, the authors did not address the empty-running problem with an actual industry case study. Another effort by researchers in terms of the application of ML for transport optimization focused more on traffic optimization to minimize emissions [24,25]. Hence, we find that existing works do not apply to a real industry setting with live data from an actual logistics business showing quantifiable emissions and costs reduction benefits. In this project, we aim to showcase how utilization of these technologies can include human agents to confidently optimize logistics operations, leading to a contribution to cutting down CO₂ emissions.

1.3. Case Study

The case study is simply that of the pick-up and drop-off segment of the transport company in which assets are transported based on customer orders across the UK. A fleet consists of a truck and trailer. Figure 1 provides an overview of a sample daily collection of jobs at a given time and the real-time location of vehicles/drivers, depicting a non-optimized plan with no routes included.

The business requirements for this case study are to develop a solution that will track the fleet in real time and apply a model that can recommend optimized job allocations and routes to minimize the occurrence of empty runs and cut down emissions. Therefore, the objective of the DT framework is to achieve the following.

• Track emissions and empty runs in real time from journeys related to jobs.
• Apply an intelligent model that recommends optimized job plans and routes.
• Facilitate feedback mechanisms from human agents to inform and improve the model’s accuracy.

2. Materials and Methods

The methodology adopted for this work comprises (A) selecting a business case study, which is in the logistics and transport domain, (B) the application of a distributed DT framework that implements an end-to-end data pipeline based on Supply Chain 4.0 technologies, and (C) a recommendation and optimization system based on formulating the RL problem using the Markov decision process (MDP) and optimization using the traveling salesman problem (TSB). The methodology is categorized and described as follows.

2.1. Data Description

As described in the case study description above, the data used for the model experiments and simulations are based on real data from the operations of a transport business line within the company. However, to obtain the full context of actual operations data, integration from multiple data sources relevant to the proposed solution was performed. The following

Table 1. Data description.

No.	Name	Description	Type/Latency
1	Transport Management System	ERP for planning and assigning pick-up and drop-off goods in-trailer to drivers	ERP/Batch stream
2	Truck Telematics	On-vehicle sensors transmitting vehicle telematics such as location, trips, carbon emission, driving events, etc.	API/Realtime stream
3	Trailer Telematics	On-trailer sensors transmitting trailer telematics such as location, temperature, speed, weight, etc.	API/Realtime stream

describes these data in a summary, from which a JSON representation is used to create the features of the DT for training and inference.

2.2. Data Processing and Integration

To create a digital twin representing the physical assets and processes of the logistics operations, the data sources described above must be processed and integrated into the DT model. Processing the data involves extracting raw data and processing it to infer context to be fed into the DT model and intelligence-driven model. An example of this is identifying whether a trailer is empty or full. To achieve this, the weight in tonnes of the trailers over a certain period must be processed to identify the threshold that depicts an empty or full trailer. Figure 2 below provides a description of achieving this for a certain trailer type in which the data over a month were processed to identify this threshold.

In terms of integration, the DT model needs to integrate all the different data sources to provide a full picture of the assets relating to events. An example of this is achieving integration in the DT by turning data into information that translates business processes into features for the model. This integration helps answer the following questions.

• Which driver and truck are assigned to a certain job and what is its status at any given point in time?
• Where is the truck (assigned or unassigned to a trailer) at any given point in time?
• Where is the trailer (loaded or unloaded) at any given point in time?
• What are the nearest vehicles to a given location or trailer?
• What truck/driver based on all the information from 1–4 is best suited to be assigned an incoming job to optimally utilize resources and reduce costs and carbon emissions?

From the questions above, question 5 is ultimately the goal of our intelligent DT model as identifying this achieves our goal. Nonetheless, Figure 3 below provides an example of this integration within our DT interface, where trailers (blue markers) within a geozone (green highlight) are identified, with the closest vehicles identified as well. This closest vehicles feature enables us to build a proximity score of all the available trucks/drivers, taking into account their status, distance and availability to take on jobs.

With data coming from the different systems, and the integration as described above, the DT gathers information to feed into our model and ultimately allows our model to train and infer what actions best optimize the next step of a business process, with the most recent data.

2.3. Digital Twin Framework

Digital twins serve as a replica of a physical object in the digital space with a two-way automatic flow of data and information, unlike a digital model or shadow that simply has a one-sided manual flow [26]. Our work utilizes a DT with a two-way automatic flow using an end-to-end pipeline that leverages Supply Chain 4.0 technologies. A digital twin framework that utilizes cloud computing, the Internet of Things (IoT), real-time/soft-real-time data processing, and machine learning is necessary. The requirement to handle data storage, multiple data sources, ETL/ELT, API handling, and cloud and on-premises data systems constitutes the need for a distributed, modular approach with two-way feedback, which can perform efficiently in terms of time and resource constraints such as API call limits, latency, and response time. This work adopted a distributed DT framework that uses edge/fog computing from [3] to implement the solution. Section 3 describes the architectural framework adopted in more detail.

Tools

Our digital twin instance is based on the following suite of tools, enabling us to perform real-time data ingestion, pre-processing and insights discovery.

• Azure Digital Twin (ADT): A platform for creating digital models of real-world assets, processes and events [27]. Using the Digital Twin Definition Language (DTDL) [28], we created a JSON-based model depicting the virtual representation of our operations, syncing in real time with our data lakehouse.
• Databricks: An all-in-one data intelligence platform enabling data and AI/ML workloads [29]. The use of Apache Spark for real-time processing with the Delta Lake format allows a seamless integration of the ADT with our intelligence models powered by Mosaic AI [30].

2.4. Fleet Utilization Using Reinforcement Learning and Traveling Salesman Problem

The core of the solution is the intelligent approach of identifying the efficient utilization of resources within a logistics environment. This is where the RL recommendation framework for route and job assignment optimization comes in. Having defined the problem, we use the MDP to formulate the case study as an RL problem, as follows [15].

• State (S): current location of a vehicle and the status of each job (assigned or unassigned).
• Action (A): decisions relating to job assignment such as assigning a vehicle or leaving a job unassigned.
• Transition (T): the probability of transitioning to the next state after taking an action such as vehicle movement and job assignment statuses.
• Reward (R): the reward for each state action fair, such as positive rewards for efficiently completed jobs and negative rewards for late pick-ups/drop-offs and empty runs. The DT framework also incorporates feedback from the human transport planners as rewards.

Figure 4 below describes the RL algorithm and the transition function given by:

T(s′∣ s,a,h) = P(s_t+1 = s′ ∣ s_t = s, a_t = a, h_t)

where T(s′∣ s,a,h) gives the probability of transitioning to state s’ given the current state s, action a, and human input h, all factors at time t. To achieve the optimized policy, the agent will traverse and utilize the environment to discover the optimal policy. Having formulated the case study into an RL problem, the approach for the model is the application of RL to a TSP. The model was enhanced to achieve advantages better than the use of classical optimization techniques. With this design in the DT framework, we can tweak the constraints, change rewards, add information such as the expected time of arrival (ETA) in real time and integrate machine learning (ML) prediction techniques [31]. Solving the problem in our case study also requires an optimization approach to find the best routes while tackling the empty-running problem. For this, we use a TSP solution [31], within the logistics DT. Figure 5 depicts a typical day’s job for a single driver modeled as graph instances.

3. Proposed Solution

Figure 5 above depicts the journey, from postcode to postcode, for a given driver in a day, processing jobs. Optimizing a particular driver’s journey requires several considerations. Additionally, the reward system is based on the RL with human feedback approach. As such, the DT facilitates bi-directional feedback between the physical space and the DT. This requirement makes it necessary for the DT to be distributed to handle latency, computational capacity, and response times as factors such as the real-time fleet location, traffic, pick-up and drop-off turnaround times, new job requests, etc., are dynamic, hence requiring the model to constantly update with new data points. Figure 6 below describes the proposed DT leveraging RL with human feedback. The proposed DT solution leverages the IoT for fleet tracking and distributed DT modules with real-time data feeds. The system recommends optimized plans to transport planners, who accept or reject the proposal, from which the RL model uses as a reward (human feedback) to improve. With the human in the loop, transport planners constantly receive these recommendations and evaluate the proposals to either accept or reject them based on human intelligence and subject matter expertise beyond the data fed to the model. The continuous iteration is envisioned to provide a more accurate RL model, as shown in Figure 6.

Figure 7 zooms out to depict the overall DT architecture. The architecture is in three layers. The edge layer at the bottom represents the physical assets such as trucks and trailers from where the real-time data (state) are generated. The middle layer is the distribution where there are ERP systems, human planners and localized processing and inference-enabling data intelligence. Additionally, in terms of the latency efficiency, the distribution of nodes [3] based on depot improves the resource utilization. At the topmost layer, extensive model training with historical data is performed in the cloud [32].

Figure 8, below, describes the workflow of the data intelligence DT. An automated data pipeline plugs in all the data in batch and real time, from where the model is invoked for inference and a recommendation is sent to the human planners via an active alert dashboard. The human planners’ response is accepted back to the model and plan reassignment is performed if necessary.

Human in the Loop

The use of a human-in-the-loop approach enables the model to learn from other factors such as internal business rules that require human intelligence, which the data and trends about jobs and driver proximity do not take into account. This feedback is accepted through the training pipeline from the DT dashboards. As such, the human-in-the-loop aspect is fundamental to the digital twin framework as it serves as a recommendation system for humans.

4. Experiments and Preliminary Results

4.1. Exploratory Data Analysis

Utilizing the different data sources across the company, a preliminary model was implemented as proof of concept for the proposed solution. Firstly, an exploratory data analysis was performed to ascertain the trends of empty runs in the operations of the year 2023. Figure 9 and Figure 10 show these trends. This exploratory data analysis enabled us to identify two aspects in our study. Firstly, as can be seen in Figure 9, reducing the monthly empty runs by about 20–30% will make a significant reduction in the cost of operations by cutting down the resources wastage (driver time, fuel, etc.) and associated emissions of approximately 300 journeys in a month

Secondly, as in Figure 10 below, we identified the top 10 routes with the most empty runs in 2023. This shows us where to focus our optimization problem for the most significant impact.

4.2. Training and Inference

The analysis from Section 4.1 above also helped us identify the most culpable depots and routes in terms of the occurrence of empty runs, and we selected some of the fleet for our simulation. The example is from Figure 5 earlier. The driver’s current job assignment is taken as graph instances, considering all the possible solutions that optimize the journey. The DT is relevant in this, as it considers incoming job orders, traffic conditions and other factors at a given time to simulate these initial possible solutions. Figure 11, below, depicts the graph of possible solutions from our example for Driver_A in Figure 5.

The RL model formulated as an MDP is trained with the environment, specifying the state, action, transition and reward, as described in Section 2 of this paper.

Figure 12 shows the training iteration where the flattening of the learning curve suggests that the agent started with exploration and exploitation and later learned a good strategy.

For the same driver, we experimented with a day’s job planning and our model was able to identify empty run instances that could have been avoided. Figure 13 shows this solution modeled as a graph instance extending the initial empty run problem found in Figure 5. In this solution, it was found that another driver, Driver_B, could in fact have processed Driver_A’s job (IV1 to IV2), as Driver_B was already within the area in a time window that is acceptable and within a range that will not make other deliveries late.

4.3. Discussion: Preliminary Results

As we continue to improve our methodology with more experiments and improved models, the preliminary results obtained by comparing normal operations and the what-if scenarios informed by the model results show that we can reduce our carbon footprint by between 20 and 30%.

Our result is expected to be significantly improved as the model keeps improving using feedback from human agents. Figure 14 below shows an overview of a comparison for a certain day’s job where this was simulated. From the figure, we can see the total emissions for selected customers (un-optimized vs. optimized). Notably, the factors that have more rewards include the proximity score for suggesting trucks/drivers, which includes drivers’ previous driving behavior and suggesting optimal drivers for longer-distance trips.

5. Conclusions, Limitations and Future Work

5.1. Conclusions

This paper presents a distributed DT framework as a concept for implementing a logistics digital twin that uses real-time fleet telematics to track the assets and carbon footprints associated with jobs. It then uses reinforcement learning to optimize the job assignments and routing for pick-up and drop-off, which significantly reduces empty runs, thereby improving the sustainability of the transport business. The requirement of the solution to accommodate batch and real-time feeds of data requires low latency and computational capacity for handling multiple data sources. This justifies adopting a hybrid approach of using both on-premises and cloud resources. Despite our choice of premium cloud platforms, the same framework supports open-source technologies and can easily be replicated to support SMEs in subsidized cloud environments using credits or local set-ups.

5.2. Limitations

These preliminary experiments and results show the potential of the proposed DT solution; however, the model needs to be improved over time with human feedback and the inference-sharing mechanism. This results are limited to the most impactful routes to the emissions and further expansion of the framework will offer more robust implementation.

5.3. Future Work

Future work on this issue envisions the company using this solution to improve the model using the feedback of transport planners who manually consider all the factors before planning a job. Due to the increasing number of jobs processed daily, an intelligent tool capable of detecting solutions to empty run scenarios in real time is needed to significantly reduce costs and improve sustainability.