Toward Efficient Blockchain for the Internet of Vehicles with Hierarchical Blockchain Resource Scheduling

Abstract

With the development of advanced information and communication technology, the traditional centralized cloud architecture cannot satisfy the exploding demand for data exchange in Internet of Vehicle (IoV) systems. Moreover, the traditional centralized architecture of the vehicular network has the potential risk of a single point of failure and lacks autonomy since the system highly relies on a trusted third party (TTP) to provide identity management. Fortunately, the emergence of blockchain technology provides a potential direction to address these problems. However, there are still some problems existing in the construction of an efficient blockchain system in IoV systems, such as the dynamic network topology and limited resources. In this paper, we propose a hierarchical resource scheduling scheme for blockchain-enabled IoV systems that improves the performance of the blockchain-enabled IoV system by efficiently allocating computational resources. The superiority of the proposed method is fully demonstrated by comparing it with existing baseline methods.

1. Introduction

With the development of big data technology, cloud computing, and communication technology, more and more devices have been connected together to share information and exchange data. Most of these devices are connected in a distributed way to reduce the burden of centralized cloud servers and lower the communication latency [1].

Unfortunately, these devices are not always cooperating efficiently. In some cases, it is easier for a malicious node to attack the system or deny the benefit they got from the system and refuse to fulfill its obligations if there is no centralized control center [2]. In addition, it is hard to distinguish whether the data are authentic or have been tampered with by malicious nodes which increases the difficulty of the collaboration [3].

To address these problems, some researchers try to introduce blockchain into the traditional distributed systems. Blockchain technology is currently a very popular research topic in academia, and it has attracted the attention of researchers from various fields [4]. It provides new approaches to solve many traditional distributed system problems. Many researchers have focused their attention on the use of blockchain technology to solve the problem of data exchange and sharing between heterogeneous devices in the Internet of Vehicles (IoV) [5].

Since blockchain technology was proposed as the underlying technical framework of Bitcoin, its original design purpose was only to serve digital currency and finance. For high-frequency and dynamic networks like the Internet of Things (IoT), the current blockchain technology is quite inefficient to adapt to the dynamic system [6]. Especially when it is used in combination with the IoV systems to solve some traditional distributed problems in the vehicular networks, its limitations will be particularly obvious [7]. The dynamic features of the network topology and uneven resource distribution of the IoV have brought major challenges to the application of blockchain technology in IoV. Building a blockchain-enabled IoV system implies that the system needs to be able to process high-frequency transactions under dynamic network topology, unstable connectivity, and limited resources. To improve the efficiency of blockchain technology in IoV environments, it is necessary to design a particularly customized blockchain architecture for vehicular networks. It should be flexible enough to cope with the unstable workload coursed by the dynamic topology, and efficient enough to make the best use of limited resources in the vehicular networks [8].

Most of the existing studies try to improve blockchain-based IoV system performance from the blockchain parameter perspective, such as block size, block forming interval, and channel configuration [9,10,11]. While we focus on the resource utilization efficiency of the blockchain-based IoV system. Blockchain systems are usually computing-intensive architecture. The more resources are allocated to the system, the better it will perform. However, in a blockchain-based IoV system, the number of vehicles is always changing, and the data flow generated by the vehicles will also change with the density of vehicles and road conditions [12]. If we allocate the resources for the system based on the data flow during the peak period, there will be a huge waste of resources in idle time. Correspondingly, if we allocate resources based on the average value, we can reduce waste, but it will greatly affect the performance of the system during peak periods. Thus, it is necessary to improve the resource utilization efficiency of the blockchain system under different traffic conditions and provide a flexible network scaling mechanism to reduce system construction and operating costs and improve system performance.

In our previous work [13], we tried to explore the possibility of using CPU shares to control resource allocation and proposed a heuristic control method. It helps to adjust the CPU utilization priority of the blockchain peer containers according to the operating status of the peer containers collected by the monitor. However, the blockchain services are configured to be running on the RSUs without any isolation or resource distributing method. Thus, when the network traffic is high, the blockchain services will consume all the resources on the RSUs. Meanwhile, since the communication of the blockchain services is based on the pandemic protocol, it will also interfere with the other processes running on the RSUs. Thereby, in this paper, we propose a hierarchical resource scheduling scheme for blockchain-enabled IoV systems to isolate the blockchain services on the allocated resources so that the blockchain services cannot consume all the resources of the RSU as well as preventing the blockchain services and various other services running on the RSU from interfering with each other. Moreover, since the resources are hierarchically distributed, instead of allocating large amounts of resources to blockchain services from the beginning, the number of the allocated resources can be adjusted as needed. The main contributions of this paper are summarized as follows:

• We propose a hierarchical resource scheduling scheme for blockchain-enabled IoV systems. A three-layered resource management architecture, namely the blockchain service layer, infrastructure layer, and network layer, is designed to provide resource isolation and improve the efficiency and flexibility of the resource utilization of the system.
• An upgraded monitoring system is also designed to help implement the resource scheduling of the system by continuously monitoring and analyzing the running status of the system for the proposed scheme to allocate system resources more efficiently. The monitor in our previous work only monitors the status of the peer containers and since we have segmented the resources of the system, we upgrade the monitoring system to monitor the status of the other tasks and available resources on all the related infrastructure nodes.
• We take the influence of the coexisting tasks into account and further extend the heuristic control method of our previous work to a resource control algorithm to improve the system performance under the unstable workload in vehicular networks. The proposed resource control algorithm adjusts the priority and proportion of the resource utilization according to the system status obtained from the resource monitor, such as transaction workload or CPU utilization status. The adjustment is implemented by tuning the CPU utilization of each component of the blockchain system running on the infrastructure nodes such as roadside units (RSU) and base stations (BS).
• A scaling control algorithm is also proposed to make better use of the limited resources in the IoV system. It helps the system scale up as needed when the resources allocated to the blockchain services are running out. The algorithm will help the system apply for more resources locally on the infrastructure layer or remotely by activating the backup peers on the network layer according to the resource utilization of the system and the number of vehicles. The scaling control algorithm also helps arrange the roles of the nodes to make sure that the one with the most available resources always takes the most responsibility.
• We design and conduct extensional simulations on multiple distributed nodes with the Hyperledger Fabric platform and the caliper blockchain benchmark to evaluate the proposed hierarchical resource scheduling scheme for the blockchain-enabled IoV system.

The rest of the paper is organized as follows. Section 2 introduces the related work about blockchain and IoV. Section 3 presents the design of the proposed hierarchical resource scheduling scheme for blockchain-enabled IoV systems. Details of the proposed resource control algorithm and scaling control algorithm are also introduced in this section. The simulation design and experimental results are presented in Section 4. Finally, conclusions will be given in Section 5.

2. Related Work

Over the past decade, thanks to the great efforts of researchers, blockchain technology has made significant progress in many academic and industrial fields such as finance, smart grids, supply chains, digital identity, and the Internet of things [14]. As one of the typical examples of IoT applications, IoV is one of the current hotspots to integrate blockchain technology. Research in related fields mainly focuses on the application of blockchain technology to help vehicle networks in trust management, incentive mechanisms, vehicle positioning, and system performance improvement [15].

2.1. Trust Management

In terms of trust management, it has become one of the main directions in applying blockchain technology in many different research areas. It is usually used to help eliminate the dependence of a distributed system on a trusted third party (TTP), and thus enables information sharing among different entities in a trustless environment. Xia et al. [16] proposed a Bayesian-game-based electricity trading scheme in blockchain-enabled IoV systems. Blockchain is used to enable trustworthiness trading in a Vehicle-2-vehicle (V2V) manner. In [17], the authors designed a key management scheme with blockchain to eliminate the handover delay caused by the communication with the TTP. Luo et al. [18] proposed a bidirectional auction mechanism based on the Bayesian game and designed a new price adjustment strategy to improve social welfare and cost performance. The authors employed the blockchain to record energy transactions and protect the privacy of the users. Gao et al. [19] pointed out the advantages of the VANET system combining blockchain with SDN, and also designed an SDN-based blockchain trust management model for the VANET system to prevent malicious activities.

2.2. Incentive Mechanisms

Another main direction of combining blockchain and IoV is designing an incentive mechanism for the collaboration of heterogeneous devices and applications in the IoV system. Yin et al. [20] proposed a new blockchain-based incentive model to coordinate multi vehicles collaboration situations for both general and emergent tasks. A bidding mechanism and a novel time-window-based method are devised to further encourage the vehicles to participate. Zhou et al. [21] proposed a consortium blockchain framework for energy trading in the Internet of Electric Vehicles (IoEV). A contract theory-based incentive mechanism with various contract items is proposed to promote more vehicles to participate in the scheduling and trading process. In [22], the authors proposed a quality-driven incentive mechanism for information sharing in vehicular networks. It also considered both the on-chain and off-chain scenarios to maximize social welfare and minimize the cost.

2.3. Vehicle Positioning

Vehicle positioning is also one of the main directions of applying blockchain technology in the IoV environment. Song et al. [23] proposed a blockchain-enabled vehicular positioning framework with Cooperative Positioning. A deep neural network (DNN) algorithm based self-positioning correction scheme and a multi-intelligent vehicle positioning error sharing model are designed to improve the positioning accuracy and reduce positioning error. This system is built on a blockchain-enabled architecture to realize information selection and sharing between vehicles and to ensure security. In [24], the authors proposed a trusted cloaking area construction in positioning services. Blockchain technology and identity pseudonym are used to improve privacy protection. The construction also employed edge computing to lower the computing latency during the evaluation of the trust value. Li et al. [25] proposed a blockchain-based positioning error evolution sharing framework for IoV. The authors consider malicious nodes and utilize blockchain technology to store and share the evolution of positioning errors of the vehicles as well as protecting privacy and ensuring credibility.

2.4. System Performance Improvement

The performance of blockchain in different environments and resource configurations has also attracted the attention of many researchers. [26] studies the impact of mobility on the performance of the blockchain-enabled vehicular ad hoc network (VANET) from three key aspects: block confirmation probability in a rendezvous, rendezvous stability, and exchanged block amount in a rendezvous period. The numerical results indicated that the blockchain system performance was greatly affected by the mobility of the vehicles. Nguyen et al. [27] presented a comprehensive evaluation for the impact of network latency on Hyperledger Fabric, a well-known consortium blockchain platform. The results demonstrated that Hyperledger Fabric does not provide sufficiently efficient consistency to be deployed in a network environment with large delays. The authors in [28] investigated the performance of the Hyperledger Fabric under different situations. A pipelined execution method is proposed to improve the efficiency of validation/commit phases and a new type of peer called sparse peer is designed to reduce redundant work.

In summary, the emergence of blockchain technology has attracted great attention in the related fields. Nevertheless, the existing works either focus on how to utilize one or more characteristics of blockchain technology to solve the traditional pain points of the IoV system, or improve the performance of blockchain systems from the blockchain configuration perspective. Since the performance of the blockchain system highly relies on the computing resources allocated to the system, it is important to explore the possibility of improving the performance of the blockchain system in vehicular networks from the resource scheduling perspective.

3. Materials and Methods

The proposed hierarchical resource scheduling scheme for blockchain-enabled IoV systems will be discussed in detail in this section.

3.1. System Architecture

While many studies discuss the integration of blockchain in vehicular networks, the problem of how to ensure the efficiency of resource utilization in dynamic vehicular environments is still underexplored. Since blockchain systems are usually computing-intensive, the performance of the system can vary significantly according to the computing ability of the infrastructure and the computing load weight running on the infrastructure, including blockchain service itself and coexisting services [29]. Thereby, we develop a hierarchical resource scheduling scheme for the IoV system to ensure the resources allocated to the blockchain system can be adjusted as effectively. We also design a monitoring system to achieve the working condition of the system. To make the system more efficient when the workload changes in the system, a resource control algorithm is designed to adjust the priority and proportion of the resource utilization according to the workload. A scaling control algorithm is proposed to effectively adjust the total amount of resources allocated to the blockchain system so that the blockchain system does not need to occupy large amounts of resources all the time. The proposed scheme adjusts the resource allocation on three different layers, which are the blockchain service layer, infrastructure layer, and network layer to balance the workloads across the network and make the best use of the system resources.

Moreover, in vehicular networks, it is critical to ensure the security of every single node. Thus, in order to reduce the possibility of malicious nodes occurrence, all the vehicles should be identifiable [30]. Therefore, permissioned blockchain architecture is more suitable for vehicular networks due to its access control mechanism. There are two main blockchain transaction models in permissioned blockchain architecture: execute–order–validate (EOV), represented by Hyperledger Fabric, and order–execute (OE), represented by Quorum [31]. We are exploring the performance of the blockchain-based IoV system with multiple peers running on the RSUs, and the EOV model has higher performance on a large number of peers scenarios. Moreover, Hyperledger Fabric has a highly modular architecture that allows most of its components such as consensus mechanism and membership services to be pluggable. This characteristic of Fabric makes it more flexible to be customized in the target system. Hence, in our proposal, we adopt Hyperledger Fabric, a well-known EOV-based permissioned blockchain platform [32].

The proposed system architecture is shown in Figure 1. There are three main components in the system, which are clients, endorsement and commitment peers, and ordering services.

• Ordering services: The ordering services are made up of ordering peers provided by different organizations in the blockchain systems and can be distributed on different cloud servers. They are responsible for the ordering process of the EOV models like sorting and packing all transactions into contiguous blocks in chronological order and broadcasting the blocks to the commitment peers to be verified without examining the content of the transactions. The ordering peers stores complete blockchain data.
• Peers: The endorsement and commitment peers are deployed on infrastructures like base stations or RSUs in the vehicular network. Endorsement peers are responsible for the execution process of the EOV models like verifying and endorsing the transactions proposed by the vehicular blockchain applications from the vehicles. Commitment peers are responsible for the validation process of the EOV models like checking the conflicts in the blocks received from ordering services and adding the new received block into the ledger. Each infrastructure node in the network will set up a pre-configured backup node for sharing its spare resources to the network and help balance the workloads in the system.
• Clients: The vehicles and smart devices in the system only utilize the system as clients and are responsible for generating transactions with preinstalled vehicular blockchain applications. The vehicles here do not participate in the consensus process of the blockchain system, since the resources and connectivity of the vehicles are usually limited and unstable.

3.2. Resource Distribution and Layering Principles

The system resource distribution of the system is shown in Figure 2. The resources of the proposed system are divided into 3 different layers as follows to better support the system and improve the efficiency of the resource utilization of the system.

• Service layer: As we can see that every infrastructure node allocates a percentage of resources for the blockchain service in advance. These allocated resources can be regarded as the blockchain service layer. It is isolated from other services to prevent interference among services running on the infrastructure node since the blockchain system is usually built on a Peer-2-Peer (P2P) communication pattern and uses epidemic protocols to propagate blocks.
• Infrastructure layer: The rest of the resources on the infrastructure node constitute the infrastructure layer. When the resources of the blockchain service layer run out, the proposed scheme can apply for more resources from the local infrastructure node to improve the performance of the system.
• Network layer: The network layer is composed of all the resources in the network. When the resource utilization of an infrastructure node is at a low level, and another infrastructure node is running out of resources, the pre-configured backup peer can be activated to support the others in the network.

3.3. Operating Procedure of the Proposed Scheme

The priority of the resource utilization proceeds in the service–infrastructure–network order. As shown in Figure 3, after the system initialization, the proposed scheme will keep monitoring the operating status of the blockchain-enabled IoV system. When the current metrics of the system are performing lower than expected, which is a set of metrics measured from the system under sufficient resources, it will check the blockchain service layer. If the resources of the service layer did not run out, the proposed scheme will invoke the proposed resource control algorithm to increase the resource utilization of the Fabric peer container running on the service layer. If the resources of the service layer run out, the proposed scheme will check the resource utilization of the infrastructure layer for help.

If the resources of the local infrastructure node did not run out, the proposed scheme will invoke the proposed scaling control algorithm to apply for more resources for the blockchain services. On the contrary, if the infrastructure node is busy, the proposed scheme will check for idle nodes in the network. If there are idle nodes available in the network, the proposed scheme will invoke the proposed scaling control algorithm to activate the backup peer on the available node. Otherwise, the system will report overload.

3.4. Proposed Monitoring System and Control Algorithems

The designs of the proposed monitoring system and control algorithms are elaborated in detail in this section. The proposed scheme adjusts the system resource scheduling on three different layers according to the metrics collected by the monitoring system.

3.4.1. Proposed Monitoring System

Since we need to control the resource allocation according to the system performance, we design a monitoring system to obtain the performance metrics such as throughput, latency, transaction success ratio, and CPU utilization of the system by continuously monitoring and analyzing the running statistics of the system, including peer containers, coexisting tasks, and available resources on all the related infrastructure nodes. The monitor can either be running on an edge node or cloud server. It is connected to the other infrastructure nodes through the Docker 2375 port to monitor all the resource statistics and updates the values every 10 s. The update interval can be adjusted according to the workload circumstances and sensitivity requirements. In our calculation, the throughput is equal to the number of transactions recorded on the ledger divided by the elapsed time.

$$Throughput=\frac{Recordedtransactions}{Elapsedtime}$$

(1)

The latency is measured through the difference between the time that a transaction is proposed and the time that the transaction is committed in the system.

$$Latency=Committedtime-Proposedtime$$

(2)

The transaction success ratio is used to measure the transactions marked as valid by the committed peers and successfully applied on the user account value. This is necessary because invalid transactions are also included in throughput calculation. We need to know that how many of the transactions are valid. It is calculated as the number of successful transactions divided by the total number of transactions.

$$Successratio=\frac{Successfultransactions}{Alltransactions}$$

(3)

These parameters can be easily found in the log files of the docker containers. The performance metrics can be collected from the local ledger since all the ledgers are the same and only the system resource utilization data is collected from remote peers. That is to say, the monitor reads 1 number from the docker log of each peer container every 10 s. Thus, the overhead is very low, and can be ignored compared to other network traffic.

3.4.2. Proposed Resource Control Algorithm

Since most blockchain systems are computing-intensive, the performance of the system highly depends on the allocated resources. However, it is impractical to equip all the nodes with sufficient resources in the blockchain system. There will be a huge cost and waste of resources during the idle period of the IoV systems.

According to [28], the CPU resource allocated to the Fabric peer is the bottleneck of the system performance. The monitoring data we got also evidence the conclusion. As the throughput increases, the peer container always runs out of resources first. So we can improve the resource utilization efficiency of the peer nodes to get better performance.

The proposed resource control algorithm is shown in Algorithm 1. It can adjust the CPU utilization percentage by changing the CPU share of each component running in the blockchain service layer. The CPU share is the CPU utilization portion of a process or a task. It is 1024 by default, and allocated by the CPU scheduler. Increasing the CPU share of a container means increasing the priority and proportion of the resource utilization. The controller adjusts the resources allocated to the peers according to the metrics from the monitoring system and the current traffic conditions.

Algorithm 1 Resource control algorithm.

Input: Current number of vehicles $N_V$, peer CPU utilization $U_P$, and coexisting task CPU utilization $U_C$

Output: New CPU share of the peer container $S_N$

1: Import reference CPU utilization from Table 1. Reference CPU utilization table. Number of Vehicles Sending Rate (TXN/s) CPU Utilization (%) 1000 100 75.03 1200 120 80.83 1400 140 86.26 1600 160 92.65 1800 180 102.36 2000 200 113.25 2200 220 118.78 2400 240 122.51 2600 260 137.97 2800 280 141.35 3000 300 152.84 3200 320 157.92 3400 340 163.88 3600 360 171.27 3800 380 178.89

2: Get reference CPU utilization $U_R$ from reference CPU utilization table according to $N_V$

3: for each $U_P$ <$U_R$ do

4: if coexisting task = true, then

$$S_N=S_D\ast \left(\frac{1-U_C}{U_P}\right)$$

5: else

$$S_N=\frac{S_D}{U_P}$$

6: return $S_N$

In Algorithm 1, $S_N$ represents the new CPU share of the peer container, $S_D$ represents the default value of the CPU share, $U_P$ represents the current Fabric peer CPU utilization, and $U_C$ represents the coexisting task CPU utilization. The coexisting tasks here refer to blockchain-related tasks running on the service layer with the blockchain system services, such as services for the vehicular blockchain applications. When the proposed scheme calculates the available resources of the blockchain service layer and adjusts the resource allocation, it will take the coexisting tasks into account so that the peer containers will not preempt all the allocated resources.

We assume that all vehicles initiate the same number of transactions within a certain period of time. Thus, the density of the vehicles is proportional to the transaction sending rates of the caliper workload. For example, a vehicle in VANET sends a safety beacon message every 10 s, so when there are 1000 vehicles, the sending rate will be 100 transactions per second. As shown in Table 1, a baseline parameter set that records the best performance of the system with sufficient resources under different transaction arriving rates is tested in advance as the reference CPU utilization. Note that the CPU utilization is made in units of the percentage of a single CPU core, and the tested host has 8 CPU cores, so it can reach a maximum value of 800% theoretically.

When the monitor detects that the system is running under the reference performance, the proposed scheme will check if there are allocated resources left in the blockchain service layer. If there are spare computing resources, the resource controller will increase the CPU share of the peer container according to the equation shown below.

$$\begin{array}{cc}\hfill & S_N=S_D+S_D\ast \left(\frac{1-U_P-U_C}{U_P}\right)\hfill \\ \hfill & =S_D\ast \left(\frac{1-U_C}{U_P}\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & S_N=S_D+S_D\ast \left(\frac{1-U_P}{U_P}\right)\hfill \\ \hfill & =\frac{S_D}{U_P}\hfill \end{array}$$

(5)

Before adjusting the CPU share of the peer container, the controller will check for coexisting tasks first. If there are coexisting tasks running in the service layer, it will adjust the CPU share of the peer container according to Equation (4). Otherwise, the controller will adjust the CPU share of the peer container according to Equation (5). 1024 is the default value of the CPU share for all projects. The numerator on the right side of Equations (4) and (5) represents the resources left in the service layer. The fraction on the right side of the equation represents the ratio of unused CPU resources to the CPU resources being used by the Fabric peer. The value of the new CPU share of the peer container $S_N$ is inversely proportional to the current Fabric peer CPU utilization UP. The more resources are left, the higher $S_N$ will be. When the $U_P$ is big enough, 100% for example, the $S_N$ will remain near 1024 because if the allocated resources are all used by Fabric peer container, it is meaningless to increase its priority.

3.4.3. Proposed Scaling Control Algorithm

The resource control algorithm only works on the blockchain service layer. When the service layer runs out of resources, the proposed scheme will apply for more resources from the local infrastructure node or activate backup peers from other available nodes in the network through the proposed scaling control algorithm, as shown in Algorithm 2.

The CPU utilization of the system is used to determine whether the system is under high pressure. When the transaction sending rate is higher than 2000 where the CPU utilization of the system is 92.2% under limited resources, the performance of the system starts to drop fast. Thus, the threshold of the high pressure is set to 90% to prevent the system performance from a dramatic drop. The threshold can be adjusted flexibly according to the configuration of the system and the priority of the blockchain services. The lower the threshold, the earlier the blockchain system takes up more resources.

When the resource utilization of the service layer is running out, the system will invoke the proposed scaling control algorithm to acquire reinforcement for the blockchain services, as shown in Figure 4. If there are spare recourses at the local infrastructure node, the algorithm will allocate more CPU resources for the blockchain-related containers.

Algorithm 2 Scaling control algorithm.

Input: Peer CPU utilization $U_P$, coexisting task CPU utilization $U_C$, and total CPU resources allocated to the blockchain service layer $U_T$

Output: Selected scaling method

1: for situation under high pressure $U_P$ + $U_C$ >90% $U_T$ do

2:  if there are spare resources on the localhost then

3:   apply more resources for the blockchain service layer

4:  else if there are idle hosts available in the network

5:   activate the backup peer on the available host, for all new activated peer $U_P$ do

6:   Anchor node = min ($U_{P1}$, $U_{P2}$, $U_{P3}$), Update anchor node

7:   end for

8:  else

9:   report system overload

10: end for

11: return

During system initialization, every infrastructure node will set up a backup peer. If the local host is fully occupied, the proposal will check for idle hosts in the network layer and activate the backup peer on the available host as shown in Figure 5. After the backup peer was activated, the scaling control algorithm will compare the resources allocated to the peers and update the one with the most available resources as anchor node. The anchor node can be regarded as the communication junction in the network, and is responsible for the dissemination of the blocks and the ledger synchronization of the peers.

4. Results and Discussion

The performance evaluation of the proposal is presented in this section. We first introduce the simulation design and assumptions. Then we evaluate the experiment results in terms of the throughput, latency, and transaction success ratio.

4.1. Experimental Design

Figure 6 shows the system setup of the simulations. The simulations are conducted on 3 computer hosts representing 3 different components of the blockchain-enabled IoV network. The first computer is a desktop consisting of an Intel i3-8100H CPU, 8G RAM, and a GeForce GTX 1050Ti graphic card. It is configured as the ordering server (host1). Three ordering nodes are configured on the ordering server.

The second and third computers (host2 and host3) are laptops with Intel i5-10300H CPU, 16G RAM, and a GeForce GTX 1650Ti graphic card. They are configured as RSUs where the endorsement and committing peers are running. Each laptop has configured 2 peers on it. Each peer can be regarded as running on an independent RSU since we have isolated the resources allocated to the peers. We use Fabric2.2 as our simulation platform and LevelDB as our Fabric database. The block size is set to 100 transactions per block in all simulations. The batch timeout (blockchain formation interval) is 2 s.

To produce proper workloads, we employ Hyperledger Caliper to simulate the transaction traffic of the system network. Caliper is a blockchain performance benchmark tool that is proposed by Huawei to the Hyperledger project. It can be used to generate various kinds of blockchain transactions at different sending rates to the blockchain system under test. The variation of vehicle density is simulated by changing the sending rate of the caliper workload. The blockchain transactions used in our simulation are all simple transfer transactions that transfer value between two accounts.

The simulation topology is an area of 1000 × 1000 m with 3 horizontal roads and 3 vertical roads, forming 4 square blocks. Every block has an infrastructure node in it and all the vehicles on these roads can at least communicate with one of the infrastructure nodes directly.

The performances of the system for different vehicle densities under sufficient resources and under limited resources with even allocation are measured first as baselines in our simulation. Then we compare the system performance under limited resources with the proposal to the baselines to evaluate the impact of the proposed method. The sufficient resources scenario utilizes all the 8 cores of the laptop while the limited resources scenario isolates the blockchain services on a single core to simulate a limited resource situation.

Two different load scenarios are tested which are the situation with coexisting tasks and the situation without coexisting tasks to evaluate the performance of the proposed scheme. The simulations with coexisting tasks are used to evaluate the impact of the proposal when the system is running together with other tasks. The coexisting tasks here can be the vehicular blockchain app or other tasks coexisting with the blockchain services on the infrastructure nodes. The simulations for evaluating the proposed scaling control algorithm are also tested with coexisting tasks since the algorithm is designed to improve the system performance under high pressure.

4.2. Results of the Scenario without Coexisting Tasks

The latency performance of the system is shown in Figure 7. While we can see that the latency of the system under sufficient resources does not change much with the increasing number of vehicles, the latency metrics of the system under limited resources increase dramatically after the number of vehicles surpasses 2500. This indicates that the transactions cannot be processed in time and begin queuing for verification at the peer nodes under the given computing resources. This is because, when the ordering peer broadcast a newly packed block to a peer, it will trigger the validation process in a peer which invokes the validation system chaincode (VSCC) [33]. During this stage, all the transactions need to wait for validation. Therefore, if the peer does not have enough resources, the waiting time of the transactions will become longer which will result in higher latency. So, we raise the CPU share of the peer container with the proposed control method to increase the CPU utilization of the peer container. As we can see from Figure 7 that the latency of the proposal increases visibly slower than the one with even allocation. This shows that our method can effectively reduce the delay caused by the increase in the number of vehicles with limited resources.

The performance of the system delivery success ratio is shown in Figure 8. In contrast to the latency performance, we can see that the success ratio metrics of the limited resources drop significantly when the number of vehicles exceeds 2500. This is because of the multi-version concurrency control (MVCC) which is employed by many concurrency systems to control the consistency of the key-value pairs [34]. Since more transactions waiting for validation at the peers, the value of the key that a transaction trying to update may be different from the value of this key when this transaction is proposed which causes MVCC read conflict. As the waiting line of the transaction increases, the possibility of the MVCC read conflict will also increase, resulting in more transactions marked as invalid. As we can see from Figure 8, the success ratio of the proposal is always higher than the one with even allocation, which means that our method can effectively lessen the probability of the transactions being marked as invalid, due to the lack of computing resources coursed by the increasing number of vehicles.

The performance of the system throughput under a different number of vehicles is shown in Figure 9. As we can see, throughput performance does not change obviously as it is shown in the latency and success ratio result. This is because the throughput performance depends on the cooperation of all the components in the system. Increasing the CPU utilization priority of the peer node will also influence the other components of the system, and the total resource of the system does not change. However, our proposal still keeps a higher throughput than the one with even allocation.

4.3. Results of the Scenario with Coexisting Tasks

The coexisting tasks in this scenario are conducted with the stress-ng tool to produce pressure to the blockchain service layer. It is configured to produce a container in the service layer that consumes 50% of the CPU resources allocated to the isolated blockchain service.

The latency performance of the system with coexisting tasks is shown in Figure 10. We can see from the figure that the latency of the system under sufficient resources is not influenced badly by the increasing number of vehicles. And the latency metrics of the system under limited resources with coexisting tasks start increasing from the beginning of the simulation with the increasing number of vehicles. This is because with the coexisting tasks running together, the system needs to arrange the CPU schedule for the coexisting tasks anyhow which will result in the waiting of the blockchain services.

Since the system resources are not highly occupied in the early stage of the simulation, the difference between the performance with the proposal and with even allocation is not quite obvious. But when the number of vehicles surpasses 2200, increasing the priority of the Fabric peer container with the proposed algorithm became more effective. However, along with the increase of the vehicle number, it returns to a small difference between the two lines. This shows that the proposed resource control algorithm is not very reliable when the system resources are running out. It needs some other methods to further improve the performance of the blockchain-enabled IoV system.

The performance of the system success ratio with coexisting tasks is shown in Figure 11. The success ratio metrics of the system under sufficient resources are also influenced by the increasing number of vehicles. because although there are enough resources in the system, along with the increasing transaction rate, the probability of two transactions that are updating the same value also increases. So, when any one of them is verified and marked as valid first, the other one will be marked as invalid and need to be proposed again with the new version of the target key value. It will result in the failure of the proposed transaction thereby lowering the success ratio.

As we can see from Figure 11, the success ratios of the system under limited resources drop at nearly the same speed before the number of vehicles comes to 2200. Moreover, similar to the latency performance, the proposal performs prominent from 2200 and returns to the performance similar to that of the performance with even allocation at around 2800 when the system resources are running out. However, we can see from the figure that the performance metrics of the proposal are better than those of the situation with even allocation. This is because when the system runs out of resources, the one with a higher CPU share will be preemptive and take over the resources from the others.

The performance of the system throughput with coexisting tasks is shown in Figure 12. As we can see that the throughput performance of the system under sufficient resources is approximately equal to the transaction arriving rate which means the throughput is hardly influenced by the coexisting tasks. The other two are both influenced by the coexisting tasks and the limited resources. This is because the proposed resource control algorithm does not change the total amount of the resources allocated to the blockchain services, to which the throughput performance is highly related.

4.4. Results of the Scenario with the Backup Peer Activated

This scenario has tested the system performance under 4 different situations:

• Full configuration with even allocation. As a baseline, all the resources are allocated to the system directly from the beginning with even allocation which means all the tasks and processes will share the allocated resources evenly without any priority or hierarchical control.
• One backup peer activated with even allocation. The system is first running on the limited resources with even allocation as a baseline. When the resources run out, the system will activate 1 backup peer with even allocation.
• One backup peer activated with the proposal. The system is first running under the control of the proposed resource control algorithm, and when the system runs out of resources, it activates 1 backup peer with the proposed scaling control algorithm.
• Scaling locally with the proposal. The system is first running under the control of the proposed resource control algorithm, and when the system runs out of resources, it scales up locally with the proposed scaling control algorithm.

The first two simulations above are tested as the baselines. The last three situations have the same allocated resources through the whole simulations: limited resources (1 CPU core) at the beginning and doubled resources (2 CPU cores) after the backup peer activated or scaled up locally. And the first one with full configuration has doubled resources (2 CPU cores) from the beginning.

The latency performance of the system is shown in Figure 13. As we can see that the latency of the system with full configuration keeps staying low with a higher construction cost. And the latency of the system with 1 backup peer activated with even allocation increases dramatically along with the increasing number of vehicles. This is because, in a distributed system, one more peer does not simply mean more resources for the system. It also means that the system has one more agent to communicate and discuss with which will generate extra overhead for the system.

The latency of the system with one backup peer with the proposal and scale up locally both grow a little with the increasing number of vehicles before 1800 and then the proposed monitoring system detects that the system resources are running out, so one backup peer is activated with the proposal in the third simulation and scales up locally with the proposal in the fourth simulation.

We can see that the latency of the system with one backup peer with proposal begins to grow after the number of the vehicles surpasses 2200, but much slower than the one in the former one with even allocation which means the backup peer has shared the workloads of the original peers. The proposal updates the peer with the most available resources as the anchor node. And this measure helps to balance the system load and improves the resource utilization efficiency of the system.

As we can see from Figure 13, the latency performance of the system scaling up locally with the proposal is even lower than the one with full configuration when the number of vehicles is high. This shows that our methods can help the system scale effectively with low latency. However, the one scaling locally is better than the one with an activated backup peer because although they have the same resources, 1 more peer will still bring unnecessary overhead to the system. So only when the local host is running out of resources should the system activate the backup peer for reinforcement.

To further illustrate the superior of the proposal, the performance–cost ratio of the system latency is shown in Figure 14. Generally, the performance-cost ratio of the system is the performance metrics divided by the cost, but since the performance to be compared here is latency, we divide the inverse of the latency by the cost of resources.

Even though the system under full configuration has better performance at the beginning of the simulation result, it can be found from Figure 14 that its performance-cost ratio is much lower than those with the proposal. This means it will need more construction and operating cost to achieve similar performance with the system with the proposal.

The performance of the system success ratio is shown in Figure 15. The success ratio of the system in all simulations is similar to each other until the number of vehicles surpasses 1800. The success ratio of the system with full configuration and the one that scales up locally perform stable and only drop a little through the whole test. The success ratio of the system with one backup peer activated with even allocation begins to drop first when its allocated resources run out and one backup peer is activated after the number of vehicles exceeds 1800. Then the performance of the system with one backup peer activated with the proposal starts to drop slowly, along with the increasing number of vehicles. It starts to drop faster when the number of vehicles comes to 2800. This is because the proposal can help balance the load among the peers, but this method does not increase average resources for each peer. Thus, the resources of the peers under this situation will run out earlier than that in the scaling locally scenario.

The success ratio of the system scales up locally performs the best of the four. It is even a little bit better than the one with the full configuration at some point. This is because, in this situation, all the resources are directly allocated to the existing peers and do not introduce more peers into the system. Furthermore, the fewer nodes in a distributed system, the higher the efficiency of the system to reach a consensus will be. And the resource control algorithm also helps the success ratio of the system with proposal performs better than the one with full configuration.

To further illustrate the superior of the proposal, the performance–cost ratio of the system is shown in Figure 16. The performance-cost ratio is equal to the success ratio of the system divided by the cost of resources. We can see that the performance-cost ratio of the proposal is almost twice as much as the system with full configuration before the number of vehicles reaches 2000. Although the one with one backup peer has a similar performance–cost ratio to those with the proposal, it drops quickly as soon as the number of vehicles exceeds 2000.

The performance of the system throughput is shown in Figure 17. The throughput performances of the four simulations are similar most of the time during the simulation. The reason is similar to that of the former simulations. At the beginning of the simulations, the system is not under pressure, so the throughput of the system is close to the transaction sending rate. We can see that around 2000–2200, the throughput of the system under 1 backup peer with even allocation starts to drop which means the system here is running out of resources, but the backup peer is activated from here and the throughput returns similar to others. Nevertheless, as we can see that the throughputs of the scenarios with the proposal are still slightly higher than that under one backup peer activated with even allocation.

5. Conclusions and Future Work

In this paper, we proposed a hierarchical blockchain resource scheduling scheme for the IoV system to enhance resource utilization efficiency and improve the system performance of the blockchain-enabled IoV system under limited resources. Since the proposed scheme does not change the internal mechanism of the blockchain system, it is suitable for most of the blockchain-enabled IoV systems with digital value transfer or data exchange demands. For example, electronic toll collection systems (ETC) and distributed energy trading systems. The proposed scheme acquires the system operating status through the proposed monitoring system and adjusts the system resource scheduling with the proposed resource control algorithm and scaling control algorithm. The resource control algorithm increases the CPU share of the Fabric peer to improve the performance of the system according to the allocated resources and the number of vehicles. The scaling control algorithm helps the system to scale up locally or remotely according to the resource utilization of the system. We evaluated the performance of the proposed scheme on the Hyperledger Fabric platform. Extensional simulations are conducted to evaluate the proposed hierarchical blockchain resource scheduling scheme and the results fully demonstrate the superiority of the proposal.

Because blockchain is a relatively new technology, there is no proper tool to simulate and evaluate the performance of the blockchain system in an IoV simulator such as OMNeT++ [35]. We only simulate the changes of the network workloads with Hyperledger Caliper. In the future, we will try to combine Fabric and OMNeT++ so that we can produce the blockchain transactions with OMNeT++. It will be more realistic and we can evaluate the influence of mobility on the proposed scheme. Moreover, since the performance of the system always changes dramatically when the system scales up, multistep thresholds for scaling up and more elaborate resource division are also under consideration for making the performance of the system smoother and more flexible.