Minimizing Redundant Hash and Witness Operations in Merkle Hash Trees

Abstract

Reusing cached data is a widely adopted technique for improving network and system performance. Future Internet architectures such as Named Data Networking (NDN) leverage intermediate nodes—such as proxy servers and routers—to cache and deliver data, reducing latency and alleviating load on original data sources. However, a fundamental challenge of this approach is the lack of trust in intermediate nodes, as users cannot reliably identify and verify them. To address this issue, many systems adopt data-oriented verification rather than sender authentication, using Merkle Hash Trees (MHTs) to enable users to verify both the integrity and authenticity of received data. Despite its advantages, MHT-based authentication incurs significant redundancy: identical hash values are often recomputed, and witness data are repeatedly transmitted for each segment. These redundancies lead to increased computational and communication overhead, particularly in large-scale data publishing scenarios. This paper proposes a novel scheme to reduce such inefficiencies by enabling the reuse of previously verified node values, especially transmitted witnesses. The proposed scheme improves both computational and transmission efficiency by eliminating redundant computation arising from repeated calculation of identical node values. To achieve this, it stores and reuses received witness values. As a result, when verifying 2ⁿ segments (n > 8), the proposed method achieves more than an 80% reduction in total hash operations compared to the standard MHT. Moreover, our method preserves the security guarantees of the MHT while significantly optimizing its performance in terms of both computation and transmission costs.

1. Introduction

In many data services, users are often required to verify multiple pieces of received data. For instance, future Internet architectures, such as Named Data Networking (NDN), treat each segment of requested content as an individual network packet. NDN assumes that intermediate network devices like routers can temporarily cache content segments and directly provide them to users [1,2,3]. If a router receives a request packet for a segment it has already cached, the router responds with the cached segment instead of forwarding the request to the original data source. While this in-network caching improves efficiency and reduces latency, it also means that users may receive segments from unidentified and potentially untrusted intermediate nodes. Since the authenticity of such network devices cannot be guaranteed, NDN mandates that users verify the integrity and authenticity of every received segment [4,5,6,7,8]. Similarly, in blockchain systems like Bitcoin, a block includes multiple transactions or contracts, and users must verify each individual component upon receiving the block [9,10,11]. In general, when data are composed of multiple subcomponents, users must verify not only the integrity of each subcomponent but also whether it is a valid part of the original dataset.

Traditionally, data verification relies on validating digital signatures or message authentication codes [12,13]. While these mechanisms ensure data integrity and authenticate the origin, they are less effective in determining whether a specific segment is a legitimate part of a larger dataset. Moreover, verifying numerous segments individually can cause significant latency. As the quality of digital content continues to improve, for instance, through high-resolution video, immersive media, and richer interactive applications, the required dataset size necessarily increases. In parallel, global digital data volumes are rapidly expanding. Analysts forecast that the world will enter the so-called “Zettabyte Era”, with up to 175 ZB of data generated annually by 2025 [14]. Similarly, mobile network traffic is projected to rise from approximately 120 EB per month in 2024 to about 280 EB per month by 2030 [15]. These trends clearly indicate that higher content quality directly translates into larger data volumes and, consequently, a greater number of segments that must be transmitted and verified in the network. In the context of NDN, this raises significant challenges for efficient segment verification. If a user must verify the digital signature of each segment of high-quality content, it can cause serious service delays, undermining the practical deployment of NDN for large-scale and latency-sensitive applications. Therefore, there is a strong need for an efficient verification scheme that can preserve security guarantees while minimizing computational and communication overheads.

Merkle Hash Tree (MHT) was proposed to efficiently verify large datasets composed of multiple segments [16,17,18,19]. MHT builds a binary tree consisting of hierarchically computed hash values: the hash value of each segment is sequentially placed at a leaf node, and each parent node’s value is calculated by hashing the concatenation of its two child nodes’ values. This tree structure ensures that any change in a leaf node (i.e., a segment) affects the root node value. Various studies have been conducted to improve the efficiency of MHT implementations: [20] proposed an optimized circuit design for SHA3-256-based MHT verification. By integrating the MHT with the Goldwasser–Kalai–Rothblum (GKR) protocol introduced in [21], the witness processing overhead can be reduced to $O\left(logn\right)$. In [22], a technique was proposed that enhances witness verification by replacing traditional AND-based proofs with OR aggregation, thereby simplifying logical complexity. The authors in [23] introduced a deterministic authenticated data structure that combines the properties of binary search trees, heaps, and Merkle trees to support efficient and verifiable operations. These studies contribute to improving the performance of MHTs either through efficient hardware implementation or by enabling seamless integration with cryptographic protocols and authenticated data structures to support efficient and verifiable dataset operations.

To verify a segment using MHT, a user must recompute all node values along a path from the corresponding leaf node to the root node. This requires not only recursive hash computation but also access to sibling node values along the path. These sibling node values, provided by the publisher, are referred to as a witness. Thus, for partially overlapping paths across multiple segments, publishers and users perform redundant hash calculations and share redundant witnesses. This redundancy becomes more severe as the dataset grows and the MHT increases in size. With the increasing scale and volume of data, such inefficiencies could become a significant bottleneck. To address this, some studies have proposed using dynamic programming to store and reuse intermediate node values, thereby avoiding recalculation [24].

Contribution. This paper focuses on the fact that the witness generated by the original content publisher consists of a set of valid node values. Once a user verifies a segment, the corresponding witness is also validated. So, the user can obtain valid node values directly from the transmitted witnesses without recomputing them. Building on this observation, we propose an improved MHT-based data verification scheme that enhances computational efficiency by storing and reusing transmitted witnesses from previously verified segments. Since these witnesses have already been validated in earlier verifications, they can be safely reused without additional checks. By reusing valid witnesses, the proposed scheme achieves more than an 80% reduction in total hash operations compared to the standard MHT when verifying $2^n$ segments ($n>8$). In addition, it significantly reduces witness transmission overhead. Finally, we present the proposed scheme in the context of NDN to demonstrate its practical applicability.

Organization. The remainder of this paper is organized as follows: Section 2 reviews the standard MHT structure and a previously proposed optimization [24], followed by performance analysis. Section 3 presents the proposed method. Section 4 provides evaluation results. Finally, Section 5 concludes this paper.

2. MHT-Based Data Authentication Overview

This section describes an MHT-based data authentication scheme and its computational inefficiency. Also, it introduces the improved scheme reusing the computed node values presented in [24].

2.1. Background: Merkle Hash Tree

Assume a scenario in which a user receives segments of a dataset over a distributed network. The user must validate each received segment based on the following three criteria:

(1)

The integrity of the segment is preserved.

(2)

The publisher of the segment is authenticated.

(3)

The segment is a valid component of the dataset requested by the user.

The first two requirements can be satisfied using traditional authentication mechanisms such as digital signatures and message authentication codes. However, these schemes are insufficient to fulfill the third requirement, verifying that a given segment is genuinely a part of the original dataset.

MHT enables efficient validation of both the integrity of each segment and its rightful inclusion within the original dataset. Furthermore, MHT offers a more efficient alternative for authenticating the data publisher than applying digital signatures to each segment individually [12].

In NDN, when a publisher distributes content, the content is first divided into small segments transmitted independently. Each segment forms the payload of a distinct network packet. One of the key features of NDN is that the user may retrieve these segments not only from the original publisher but also from nearby network nodes (e.g., switches, routers, or base stations) that have temporarily cached them. The caching behavior depends on each node’s cache management policy. When a network node caches a segment, subsequent requests for that segment may be served directly from the cache without forwarding the request to the original publisher. However, this decentralization means that the recipient cannot determine whether the segment originated from a trusted source, as the receiver cannot even identify its sender. Thus, because NDN lacks end-to-end authentication, users must verify the integrity and authenticity of each received segment for themselves.

To facilitate this, NDN leverages MHT-based segment and content verification. Figure 1 illustrates an example where a publisher generates content consisting of $2^4$ segments. A binary hash tree with $2^4$ leaf nodes is constructed, and each segment is assigned to one leaf node in order. Each node $N_i$ has a sequence number $i$, where $0\le i<2^5$ and a node level $⌊{log}_{2}i⌋$. The value $V_i$ of node $N_i$ of the tree is computed as follows:

(1)

If $N_i$ is a leaf node, $V_i=H\left(S_k\right)$, where $S_k$ is a segment having a sequence number $k\in \left\{0,1,\cdots ,2^4-1\right\}$, and $i=k+2^4$.

(2)

Otherwise, $V_i=H\left(V_{2i}\right|\left|V_{2i+1}\right)$, where $V_{2i}$ and $V_{2i+1}$ are the values of left and right child nodes of $N_i$, respectively.

Once the root node value $V_1$ is calculated, the publisher signs it. Since $V_1$ is derived from all segments using a hash function, it represents the integrity of the entire dataset.

To verify a segment $S_k$, a user must be able to reconstruct the root node value $V_1$ using $S_k$ along with a set of intermediate node values. For example, suppose the user receives the first segment $S_0$. The user then computes $V_{16}$, $V_8$, $V_4$, $V_2$, and finally $V_1$ sequentially. While $V_{16}=H\left(S_0\right)$ can be computed directly, calculating lower-level node values (e.g., $V_8=H\left(V_{16}\right|V_{17})$) requires access to sibling node values such as $V_{17}$. This continues recursively up to the root node value $V_1$. The sibling node values, which cannot be computed locally, are collectively referred as the witness of the segment. For MHT with $2^n$ leaf nodes, let $\left\{w_i|i=1,2,\cdots ,n\right\}$ denote the witness of a segment, where $w_i$ corresponds to the sibling node value at level $i$ along the path from the leaf node to the root. Accordingly, the publisher transmits a packet that includes the following:

(1)

A segment $S_k$.

(2)

The witness $\left\{w_i|i=1,2,\cdots ,n\right\}$ corresponding to $S_k$.

(3)

The digital signature of the root node value $V_1$.

When a user receives the segment $S_k$, the segment verification process is as follows:

(1)

Initial segment verification: When the user receives the first packet including $S_0$, it computes a root node value $V_1^{\prime }$ using $S_0$ and its witness. The digital signature over $V_1$ is then verified using $V_1^{\prime }$. If valid, the user accepts $S_0$ as a correct segment and stores $V_1^{\prime }$ as the trusted root value.

(2)

Subsequent segment verification: For any other segment $S_k$ where $k>0$, the user computes $V_1^{″}$ using $S_k$ and its corresponding witness. The digital signature need not be verified again. Instead, the user simply checks whether $V_1^{″}=V_1^{\prime }$, where $V_1^{\prime }$ has been stored during the initial verification process. If the values match, $S_k$ is accepted as valid.

By avoiding repeated signature verifications and limiting computation to hash operations, MHT significantly reduces overhead compared to traditional data authentication schemes. Moreover, MHT not only authenticates each segment of the content but also verifies that each segment is a legitimate part of the content. Therefore, by verifying all received segments, users can authenticate the entire content.

2.2. Reusing Computed Valid Node Values of MHT

As discussed in the previous section, when verifying segments, a content verification scheme that integrates digital signature technology with MHT can be more efficient than conventional schemes that rely solely on digital signature technology. MHT requires a user to compute the root node value $V_1$ every time segment $S_k$ is received. To verify $S_k$, the user must compute all node values along the path from the corresponding leaf node to the root node $N_1$. Many node values are computed repeatedly during the validation of all segments. If MHT consists of $2^n$ leaf nodes, the root node value $V_1$ is computed $2^n$ times, and $V_2$ is computed $2^{n-1}$ times. In general, each node value $V_i$ is repeatedly computed $2^{n-⌊{log}_{2}i⌋}$ times. For example, in Figure 1, a user computes $V_8$ two times: one is to verify $V_{16}$, and the other is to verify $V_{17}.$ When verifying 16 segments, the user is required to compute $V_4$ four times, $V_2$ eight times, and $V_1$ sixteen times, respectively. Hence, when verifying large-size content consisting of many segments, the MHT-based scheme results in redundant computation due to repeated calculation of identical node values. Such redundant computation can still lead to inefficiencies in the verification process.

To address this problem, ref. [24] proposes a dynamic programming method that stores and reuses node values computed during the verification of earlier segments of content. For that, ref. [24] assumes that segments are received sequentially. If MHT has $2^n$ leaf nodes, the total number of nodes in the tree is $2·{(2}^n-1)$. In this case, the total number of hash value computations required for verifying all segments is ${\displaystyle {\sum }_{i=0}^n}{(2}^i·2^{n-i})=2^n·(n+1)$. This implies that a total of $2^n·(n-1)$ of the hash computations are redundant resulting from recalculations of the same intermediate node values.

If a segment is verified as valid, all computed node values along the node path from the corresponding leaf node to the root node $N_1$ can also be considered valid. For example, as shown in Figure 1, if the first segment is verified, it is also confirmed that the computed node values $V_{16}$, $V_8$, $V_4$, $V_2$, and $V_1$ are all valid. In [24], it is proposed that a user stores these valid node values except for the leaf node value. In Figure 1, after the first segment is verified, the user stores the computed $V_8$, $V_4$, $V_2$, and $V_1$. Hence, a user maintains $n$ stored node values when the MHT contains $2^n$ leaf nodes. While the standard MHT [16] requires the user to calculate $V_1$ whenever verifying each segment, ref. [24] proposed that a user use stored valid node values to avoid redundant node value calculations. For instance, in Figure 1, when a user verifies the third segment after the first segment has been verified, it suffices for the user to compute only $V_{18}$, $V_9$, and $V_4$. If the newly computed $V_4$ matches the previously stored $V_4$, then the yet-to-be-computed $V_2$ and $V_1$ can be regarded as being identical to the stored $V_2$ and $V_1$, thus eliminating the need for further computation of $V_2$ and $V_1$. Hence, if a user can store the valid node values obtained during the verification of each segment, it is not necessary to compute all node values along the full verification path for subsequent segments. Instead, when verifying a segment, the user only needs to identify an intermediate node on the full path corresponding to the segment that matches one of the nodes whose node values have been saved for the previous segment verification process. For example, in Figure 1, when verifying the third segment, the user does not need to consider the full path from $N_{18}$ to the root node $N_1$. Instead, the user sufficiently considers a subpath from $N_{18}$ to $N_4$ because the user has stored the valid node value of $N_4$. Hence, the user must only need to identify $N_4$ for verifying the third segment. If the computed node value of the matched node equals the corresponding saved node value, the remaining node values along the path—though not explicitly computed—can be considered to match their previously stored counterparts, thereby reducing redundant computations. Hence, the user can utilize a short subpath—from a leaf node corresponding to the segment to the matched node—to improve performance efficiency.

When utilizing the scheme proposed in [24], each internal node’s value is computed twice. Hence, if the MHT contains $2^n$ leaf nodes, the total number of internal node value computations is $2^n·(n+1)$.

3. Minimizing Redundant Hash and Witness Operations

The scheme in [24] reuses valid node values computed locally by the user. However, it discards the received witness after segment verification even though the witness itself consists of valid node values. Considering all segments, the entries of witnesses are also significantly redundant. Practically, this repeated redundant witness transmission results in significant inefficiency. This paper focuses on the observation that a witness essentially consists of valid node values and that the same witness element values are often transmitted repeatedly. Based on this insight, this paper proposes a scheme that stores and reuses witnesses to additionally improve MHT: by selecting a shorter subpath than those employed in [24] and by reducing witness transmission, thereby enhancing overall efficiency. As in [24], this paper assumes that segments are received sequentially.

3.1. Analysis of Redundant Witness Transmission

When verifying segments, some transmitted witnesses contain overlapping node values. For instance, in Figure 1, $V_9$ is redundantly transmitted two times, as it is included in the witnesses for verifying two segments corresponding to $V_{16}$ and $V_{17}$, respectively. Similarly, $V_5$ is transmitted four times when verifying four segments corresponding to $V_{16}$, $V_{17}$, $V_{18}$, and $V_{19}$. $V_3$ is transmitted eight times. As illustrated in Figure 1, the number of times a node value is transmitted as part of a witness varies depending on the location (i.e., depth) of the corresponding node in the MHT. This concept of a node’s depth is formally defined as follows:

Definition 1.

Node Level: Each node of MHT is assigned a unique sequence number in a top-down, left-to-right manner—from the root node to the leaf nodes. Suppose the MHT has $2^n$ leaf nodes. The root node is assigned sequence number $1$, and the last leaf node is assigned sequence number $2^{n+1}-1$. Let $N_i$ denote the node with sequence number $i$. The  node level of $N_i$, denoted by $L\left(i\right)$, is defined as $L(i)=⌊{log}_{2}i⌋$, indicating the depth of the $N_i$ within the binary tree. Let $L_k$ denote the node level $k$.

The number of times a node value is repeatedly transmitted as a witness depends on the node level $L\left(i\right)$ and the size of MHT. For example, the node $N_3$ has level $L\left(3\right)=1=L_1$. If the MHT contains $2^4$ leaf nodes, the value $V_3$ is transmitted eight times during all segment verifications. However, if the MHT has $2^{10}$ leaf nodes, $V_3$ is transmitted $512$ times. The repeated transmission count $\mathrm{t}\left(i\right)$ of a node value $V_i$ is defined as follows:

Property 1.

Repeated Transmission Count of a Witness Node Value: Assume the MHT has $2^n$ leaf nodes and let $i>1$. The number of times that the node value $V_i$ is transmitted as a witness during the verification of all $2^n$ segments is given by the following:

$${t\left(i\right)=2}^{n-L\left(i\right)}$$

where $L(i)=⌊{log}_{2}i⌋$ is the level of $N_i$.

Property 2.

Total Transmission Count of Witness Values at Level $\mathit{l}$: Assume the MHT has $2^n$ leaf nodes. Let $l$ denote a level in the MHT. There are $2^l$ nodes at level $l$, and each node at this level is included as a witness in $2^{n-l}$ packet transmissions. Hence, the total number of node value transmissions for all witness values at level $l$ is the following:

$$T\left(l\right)=2^l·2^{n-l}=2^n$$

Property 3.

Total Transmission Count of All Witness Values: Assume MHT has $2^n$ leaf nodes. The tree consists of $2^{n+1}$-1 nodes. The total number $T\left(n\right)$ of node value transmissions required as witnesses for verifying all $2^n$ segments is the following:

$$T_n={\displaystyle {\sum }_{i=1}^{2^{n+1}-1}}t\left(i\right)={\displaystyle {\sum }_{i=1}^{2^{n+1}-1}}{2}^{n-L\left(i\right)}={\displaystyle {\sum }_{l=0}^n}T\left(l\right)={\displaystyle {\sum }_{l=0}^n}{2}^l{·2}^{n-l}=(n+1)·2^n$$

Figure 2 illustrates the number of packets that include a witness value at level $L_k$. For example, the curve labeled $L_1$ represents the number of packets that contain $V_3$ for MHT with $2^n$ leaf nodes. As shown in Figure 2, if $n>13$, $V_2$ and $V_3$ are transmitted more than 5000 times, respectively.

3.2. Witness Reuse in MHT

A witness used to verify a segment consists of certain valid node values. Hence, when a user verifies a segment, the user effectively receives certain valid node values as part of the witness. Based on this observation, this paper proposes to reuse the transmitted, not computed by a user, valid node values. This section introduces a scheme that selectively stores valid node values contained in the received witness and then reuses them when verifying subsequent segments. As a result, the proposed scheme reduces the computation and transmission overheads associated with the MHT.

Definition 2.

The Verification Path of a Leaf Node: Let $\left(N_i,N_k\right)$ denote a path from a leaf node $N_i$ to an ancestor node $N_k$. The verification path $\left(N_i,N_x\right)$ of the leaf node $N_i$ is defined as the shortest such subpath where $V_x$ has been verified as valid.

In Figure 1, suppose a user receives the first segment assigned to $N_{16}$. At this point, the user has not yet verified any node values and therefore does not store any valid node values. As a result, the user must compute all node values along the full path, from $N_{16}$ to $N_1$. In this case, the verification path of $N_{16}$ is $\left(N_{16},N_1\right)$. According to the scheme presented in [24], if the user stores the computed node values, $V_8$, $V_4$, $V_2$, and $V_1$ after verifying the first segment, the verification path of a leaf node $N_{17}$ becomes $\left(N_{17},N_8\right)$, rather than $\left(N_{17},N_1\right)$.

Definition 3.

Level of the Verification Path of a Leaf Node: Assume MHT has $2^n$ leaf nodes. Let the given $k^{th}$ segment be assigned to a leaf node $N_i$. If the verification path of the $N_i$ is $\left(N_i,N_x\right)$, the level $L_p\left(n,k\right)$ of verification path  $\left(N_i,N_x\right)$ is defined as the node level of the $N_x$, that is, $L_p\left(n,k\right)=⌊{log}_{2}x⌋$.

For example, the level of verification path of the leaf node assigned to the first segment is always $0$. In Figure 1, if the user adopts the scheme in [24], the level $L_p(4,2)$ of the verification path $\left(N_{18},N_4\right)$ is $2=⌊{log}_{2}4⌋$.

Assume that a user receiving and verifying segments maintains a cache for transmitted witnesses. If segments are transmitted sequentially, the user can determine the verification path for each segment according to the following procedure: Let the size of MHT be 2ⁿ. The user maintains a set of internal caches $\left\{W_i|i=0,1,\cdots ,n\right\}$, for storing the verified root node value and the received witness. Specifically, $W_0$ stores the verified root node value, while $W_i$ (for $i>0$) stores a witness entry corresponding to an internal node $N_k$ at level $i$. Each cache entry is initialized to NULL.

(1)

Packet Reception: The user receives a packet containing a segment, its corresponding witness, and the digital signature. The segment has a sequence number $i$, where $0\le i\le 2^n-1$.

(2)

Determining the Verification Path Level: The user computes the level $L_p(n,i)$ of verification path of the leaf node corresponding to the received segment. If the packet contains the first segment, $L$ is $0$. Otherwise, the user searches for the deepest-level node on the verification path that matches the node corresponding to one of the stored values in $\left\{W_i|i=0,1,\cdots ,n\right\}$. The level $L_p\left(n,i\right)$ is set to the level of this matched node. The user can predict the level $L_p\left(n,i\right)$ prior to computing node values of the full verification path. To achieve this, the user refers to Algorithm 1, which defines how to determine $L_p\left(n,i\right)$ based on the sequence number $i$ of the segment.

(3)

If $L_p\left(n,i\right)=0$, the user computes $V_1$ to verify the digital signature. If the signature is valid, the segment is considered valid. The user then stores the $V_1$ in the cache $W_0$ and the transmitted witness values $\left\{w_i|i=1,\cdots ,n\right\}$into the cache $\left\{W_i|i=1,\cdots ,n\right\}$, placing each value in the entry corresponding to its node level.

(4)

If $L_p\left(n,i\right)>0$, the user selects a verification path $\left(N_i,N_k\right)$ whose level is $L_p\left(n,i\right)$. The user then computes the value $V_k$. If the computed value $V_k$ matches the one stored in $W_{L_p\left(n,i\right)=⌊{log}_{2}k⌋}$, the segment is considered valid. The user then stores the witness values $\left\{w_k|k=L_p\left(n,i\right)+1,\cdots ,n\right\}$, which were used in the computation of $V_k$, into the cache $\left\{W_i|i=L_p\left(n,i\right)+1,\cdots ,n\right\}$, indexed according to their corresponding levels.

Algorithm 1 ComputeLevel(n, sn): Pseudo-Code for Calculating the level of verification path for a given segment
Input	$n$: The height of MHT, where the MHT has $2^n$ leaf nodes $sn$: The sequence number of a segment $(0\le sn<2^n-1$)
Output	${Level=L}_p\left(n,sn\right)$: The level of a verification path $(0\le L_p\left(n,sn\right)\le n$)
01: if sn is 0 then          //the first segment 02: set$Level$$\leftarrow$ 0 03: else if sn mod 2 is 1 then     //odd-numbered segments 04: set$Level$$\leftarrow$ n 05: else                 //even-numbered segments 06: set bit_mask $\leftarrow$ 2 07: for Level, from n-1 down to 1 do 08:  if (sn bit-AND-operation bit_mask) is not 0 then 09:   break 10:  else 11:   set bit_mask $\leftarrow$ bit_mask << 1  //<< operation: a left bit shift 12:  end if 13: end for 14: end if 15: return$Level$

Algorithm 2 presents the pseudo-code of the above verification procedure for MHT with $2^n$ leaf nodes. For example, if $n=5$ and $sn=8$, the user has cached $\left\{V_3,V_5,V_9,V_{19},V_{38}\right\}$ in $\{W_1,W_2,\dots ,W_n\}$ after verifying $S_7$. $ComputeLevel(5,8)$ returns $L=2$ as the level of the verification path for $S_8$. Then, the shortest verification path is $\left(N_{40},N_5\right)$, as shown in line 13 of Algorithm 2. Then, the user computes $V_5$ using both $S_8$ and witness $\left\{w_{3=L+1},w_4,w_5\right\}$, corresponding to line 15 in Algorithm 2. If the computed $V_5$ equals the cached value in $W_2$, $S_8$ is verified as valid. Finally, the user caches $\left\{w_{3=L+1},w_4,w_5\right\}$ according to their level, as described in line 18 of Algorithm 2.

3.3. Witness Generation for Reusing Witness

If a user stores and reuses transmitted witnesses, the publisher of segments may no longer need to repeatedly transmit the same witness values. To apply this idea in practical scenarios, assume that network and device conditions are stable enough for the user to successfully receive all segments sequentially. Under this assumption, the publisher can generate witnesses as follows: Assume that MHT has $2^n$ leaf nodes and the published content is fragmented into $2^n$ segments. The segment has a sequence number $i$, where $0\le i\le 2^n-1$. Let $\left\{w_1,w_2,\cdots ,w_n\right\}$ denote the original witness of a segment.

(1)

For the First Segment: If $i=0$, the publisher transmits the full original witness as is.

(2)

For Even-Numbered Segments: If $i$ is even, the witness for the segment is empty, that is, the publisher does not transmit the witness for it.

(3)

For Odd-Numbered Segments (After the First): If $i$ is odd, using Algorithm 1, the publisher computes the level $L_p\left(n,i\right)$ of the verification path for the segment. Then, the publisher constructs a partial witness $\left\{w_k|k=L_p\left(n,i\right)+1,\cdots ,n-1,n\right\}$.

Algorithm 2 VerifySeg(n,sn,Ssn,ω,σ,Ω): Pseudo-Code for Verifying a Segment using ComputeLevel(n, sn)
Input	$n$: The height of MHT, where the MHT has $2^n$ leaf nodes $sn$$:\mathrm{The}\mathrm{sequence}\mathrm{number}\mathrm{of}\mathrm{a}\mathrm{segment}(0\le sn<2^n-1$) $\{\mathrm{segment}{S}_{sn}$ $,\mathrm{witness}\omega =\left\{w_1,w_2,\cdots ,w_n\right\}$$,\mathrm{signature}\sigma$}: A transmitted packet $\mathsf{\Omega }=\left\{W_0,W_1,\cdots ,W_n\right\}$: Caches for the root node value and witness
Output	$vResult$: Segment Verification Result
01: if $sn$ is 0 then                    //the first segment verification 02:  compute $V_1$ $\mathbf{using}\mathbf{both}{S}_0$ $\mathbf{and}\left\{w_1,w_2,\cdots ,w_n\right\}$ 03:  verify $\sigma$$\mathbf{using}\mathrm{the}\mathrm{computed}{V}_1$ 04:  if $\sigma$ is valid then 05:   set $W_0\leftarrow V_1$                    //caching the valid root node value 06:   store $\left\{w_1,w_2,\cdots ,w_n\right\}$ $\mathbf{in}\left\{W_1,W_2,\cdots ,W_n\right\}$ in order      //caching valid witness 07:   set $vResult\leftarrow Success$ 08:  else 09:   set $vResult\leftarrow Fail$ 10:  end if 11: else                         //the other segments verification 12:  set $L\leftarrow ComputeLevel(n,sn)$              //finding the shortest path 13:  select $\mathrm{the}\mathrm{shortest}\mathrm{verification}\mathrm{path}\left(N_{2^n+sn},N_{(2^n+sn)/⌊2^{n-L}⌋}\right)$ 14:  set $wlevel\leftarrow \mathrm{m}\mathrm{i}\mathrm{n}(L+1,n)$              //finding witness level 15:  compute $V_{(2^n+sn)/⌊2^{n-L}⌋}$$\mathbf{ using}{S}_{sn}$$\mathbf{ and}\left\{w_i|wlevel\le i\le n\right\}$ 16:  compare $W_L$$\mathbf{ with}{V}_{(2^n+sn)/⌊2^{n-L}⌋}$            //the node value verification 17:  if $W_L$$\mathrm{ equals}{V}_{(2^n+sn)/⌊2^{n-L}⌋}$ then 18:   store $\left\{w_i|wlevel\le i\le n\right\}$ $\mathbf{in}\left\{W_i|wlevel\le i\le n\right\}$ in order //caching witness 19:   set $vResult\leftarrow Success$ 20:  else 21:   set $vResult\leftarrow Fail$ 22:  end if 23: end if 24: return vResult

4. Evaluation

4.1. Security Evaluation

When verifying segments, a user must check two properties:

(1)

The integrity of each segment: The verifier ensures that the segment has not been tampered with during transmission. In MHT, this is achieved by verifying the hash value of the segment using the full verification path associated with the segment.

(2)

The correctness of each segment: This ensures that the segment is an authentic part of the original content requested by the user. In MHT, the original content is represented by the root node value. If the user correctly recomputes the root node value using the segment, the segment is considered an authentic part of the content.

This paper considers a potential attack scenario to forge a segment. Let the number of leaf nodes of the MHT be $2^n$, a dataset be $D=\left\{S_0,S_1,\cdots ,S_{2^n-1}\right\}$, and a local cache be $\mathsf{\Omega }=\left\{W_0,W_1,\cdots ,W_n\right\}$. We assume that a secure hash function $h$ is a collision-resistant (CRHF) and the second preimage-resistant, and it is optionally modeled as a random oracle (RO) for simpler proofs. The digital signature scheme is secure in the sense of existential unforgeability under chosen-message attacks (EUF-CMA). For segment $S_i$ ($i\ge 0$), assume that the user has already received and verified every segment $S_j$, where $0\le j<i$. Security goals are as follows:

(1)

G1 (Cache-safety): It does not open new forgery avenues to reuse cached (previously verified) witness nodes.

(2)

G2 (Integrity): A probabilistic polynomial-time adversary ($\mathsf{\alpha }$) cannot cause node values to accept a modified segment $S_i^{\prime }\ne S_i$ as valid.

(3)

G3 (Correctness): The $\mathsf{\alpha }$ cannot cause acceptance of a segment not in the dataset.

Property 4.

Subtree commitment and valid verification path. Let $N_k$ be a node of the MHT over $D$ with node value $V_k$. Under CRHF, the value  $V_k$ uniquely commits to all node values in the subtree rooted at $N_k$; i.e., for fixed $D$, there is a unique assignment of descendant values consistent with $V_k$. Moreover, for any leaf node $N_{2^n+i}$ that lies in the subtree of $N_k$ (equivalently, whose index satisfies $k·2^{n-l}\le 2^n+i\le \left(k+1\right)·2^{n-l}-1$, where $l=⌊{log}_{2}k⌋$), the path ($N_{2^n+i}$, $N_k$) is a valid verification path: if the sibling values along this path are provided, one can recompute $V_k$ from $S_i$.

Property 5.

Caching rule and stability of cached node values. Let $V_k$ be transmitted as a witness and $l=⌊{log}_{2}k⌋$ be the node level of $N_k$. Then, $N_{p\left(k\right)}=N_{⌊k/2⌋}$ is the parent node of $N_k$. The leftmost leaf node in the subtree rooted at $N_{⌊k/2⌋}$ is the node $N_{left\left(⌊k/2⌋\right)}=N_{{p\left(k\right)·2}^{n-l+1}}=N_{2^n+i_{\left(⌊k/2⌋\right)}}$, where $i_{\left(⌊k/2⌋\right)}=p\left(k\right)·2^{n-l+1}-2^n$. When the segment $S_{i_{\left(⌊k/2⌋\right)}}$ (assigned to $N_{left\left(⌊k/2⌋\right)}$) is verified, the verifier stores $V_k$ in the cache entry $W_l\in \mathsf{\Omega }$. Also, for the fixed dataset $D$ and under CRHF, the stored $V_k$ remains unchanged while verifying any other segment whose leaf node lies in the subtree of $N_k$.

Theorem 1.

Cache safety (for G1): Let $V_k$ be a cached node value validated in previous verifications. If VerifySeg($n$,$i$,$S_i$,$\omega$,$\sigma$,$\mathsf{\Omega }$) selects the shortest verification path ($N_{2^n+i}$, $N_k$), computes $V_k^{\prime }$, and finally returns Success, accepting $S_i$ is as secure as a full path verification to the root node $N_1$.

Proof. 

We prove the theorem by induction on the segment index $i$.

Base case ($i=0$): When $i=0$, VerifySeg verifies $S_0$ using the full path ($N_{2^n}$,$N_1$). Thus, acceptance of $S_0$ is identical to full-path verification, and the claim holds trivially.

Induction hypothesis: Assume that for all $j<i$, if VerifySeg accepts $S_j$, this acceptance is equivalent to full-path verification to $N_1$. Consequently, the caches $\mathsf{\Omega }=\left\{W_1,W_2,\cdots ,W_n\right\}$ contain only valid node values that were previously received as witnesses.

Induction step ($i>0$): Suppose, for contradiction, that VerifySeg returns Success for segment $S_i$, yet this acceptance is not as secure as full-path verification to $N_1$. This implies that the computed root node value $V_1^{\prime }$ differs from the valid stored root node value $V_1$. Under CRHF, the root node value $V_1$ uniquely commits to all descendant node values along the path ($N_{2^n+i}$,$N_1$). Thus, if $V_1^{\prime }(\ne V_1)$ is computed, there must exist at least one mismatched intermediate value on the path. Let the shortest verification path chosen by VerifySeg be ($N_{2^n+i}$,$N_k$). One of the mismatched nodes must then occur at the computed value $V_k^{\prime }$, i.e., $V_k^{\prime }\ne V_k$, where $V_k$ is a valid node value. Since VerifySeg returns Success, it must have confirmed that $V_k^{\prime }=W_{{log}_{2}k}$. Since $N_{2^n+i}$ is a leaf node of subtree rooted at $N_k$, it is a leaf node of subtree rooted at $N_{⌊k/2⌋}$. By property 5, the cache entry $W_{{log}_{2}k}$ was set to $V_k$ when the leftmost leaf node value of the subtree rooted at $N_{⌊k/2⌋}$ was first verified, and this cached value remains unchanged during the verification of all leaf nodes in that subtree. Hence, $W_{{log}_{2}k}=V_k$ when $S_i$ is verified. Therefore, the condition $V_k^{\prime }\ne V_k$ cannot hold, leading to contradiction.

Conclusion: Thus, for segment $S_i$, acceptance by VerifySeg is equivalent in security to full-path verification. By induction, the claim holds for all $0\le i<2^n$. □

Theorem 2.

Soundness Verification (for G2 and G3): An adversary $\alpha$ has only negligible advantages in violating goals, G2 and G3, under CRHF and EUF-CMA.

Proof. 

We consider two cases depending on the segment index $i$. The first case is $i=0$. For the first segment $S_0$, the adversary $\mathsf{\alpha }$ must either forge corresponding signature $\sigma$ or produce a different verification path leading to a forged root $V_1^{\prime }=V_1$ that still verifies under $\sigma$. Forging $\sigma$ would directly break EUF-CMA security of the signature scheme, which is assumed to be infeasible. Producing the other path requires constructing a collision in the hash function along the path from the leaf node to the root node, contradicting the CRHF assumption. Thus, $\mathsf{\alpha }$ cannot forge $S_0$ with non-negligible advantage.

The second case is for subsequent segments $S_i$, $i>0$. The acceptance requires the recomputed intermediate node value $V_k^{\prime }$ to match the cached valid node value $V_k$. The cache entries are witnesses that were previously verified against valid segments. To successfully forge a segment $S_i^{\prime }$, the adversary must construct a fake witness ${\omega }^{\prime }$ corresponding to the $S_i^{\prime }$ such that the recomputed $V_k^{\prime }=V_k$ without using the legitimate $S_i$. This reduces finding a second preimage under the hash function when computing $V_k^{\prime }$. Such a second preimage attack contradicts the secure hash function assumption.

In both cases, forging an acceptable segment would either break EUF-CMA (by forging $\sigma$) or break CRHF (by creating a collision or second preimage). Since both are assumed to be secure, the adversary $\mathsf{\alpha }$ has only a negligible advantage in violating G2–G3. □

4.2. Performance Evaluation

To implement the proposed scheme, if MHT has $2^n$ leaf nodes, a user needs to store at most $n$ witness values, one per tree level. This results in a storage overhead of $O\left(n\right)$, which is negligible compared to the dataset size $O\left(2^n\right)$. Therefore, the storage requirement grows only logarithmically with the number of segments. In the following subsection, we theoretically analyze the computation and transmission overheads.

4.2.1. Hash Computation Overhead

To evaluate the computation overhead of the proposed scheme, we analyzed the total number of hash computations required during segment verification process for $2^n$ segments of the content. In the standard MHT approach [16], each segment verification requires computing all hash values along the full path from the corresponding leaf node to the root node. Therefore, the total computation overhead is the following:

$$C_{\left[MHT\right]}=2^n·(n+1)$$

In the case of [24], only the hash values along a partial path—from the leaf node to the internal node having the highest node level whose node value has been computed, verified, and stored previously—must be calculated. Hence, ref. [24] reduces the number of hash computations needed for verifying all segments except the first one. Therefore, the total computation overhead is the following:

$$C_{[MHT-1]}=2n+2^n+\sum _{i=1}^{n-1}{2}^i·(n-i)$$

Similarly, our proposed scheme also computes hash values along a partial path. However, this path terminates at the internal node whose node value was previously received as a witness. In terms of path length, the partial path in our proposed scheme is typically 1 shorter than that in [24], except for the first segment. Therefore, the total computation overhead is as follows:

$$C_{MHT-2}=\left(2n-1\right)+2^n+\sum _{i=1}^{n-2}{2}^i·(n-i-1)$$

Figure 3 shows the total number of hash computations for various values of $n$, where the number of segments is $2^n$. As $n$ increases, the difference in computation overhead between the standard and the proposed methods becomes more pronounced. When $n>8$, the proposed method achieves a reduction of more than 80% in total hash operations compared to the standard MHT. In comparison with [24], the proposed method yields an average reduction of approximately 33%. This reduction in computational cost is particularly advantageous for resource-constrained environments, such as IoT networks or edge devices, where minimizing CPU workload is essential.

4.2.2. Witness Transmission Overhead

In the standard MHT, each segment requires a set of node values, referred to as a witness, to reconstruct the root node value. For a dataset consisting of $2^n$ segments, each witness contains $n$ node values. Therefore, the total witness transmission overhead is the following:

$$T_{\left[MHT\right]}=2^n·n$$

This means that many witness values are transmitted redundantly, resulting in significant communication overhead.

In the case of [24], a subtree of the full MHT is used to verify each segment. Although verification requires only the path within the subtree, the scheme still transmits the entire witness of the standard MHT for each segment. If [24] was modified to transmit only the partial witness corresponding to the subtree path, the transmission overhead could be reduced. Under this assumption, the total witness transmission overhead of [24] is given by the following:

$$T_{[MHT-1]}=2n+\sum _{i=1}^{n-1}{2}^i·(n-i)$$

In contrast, the proposed scheme enables users to store and reuse previously received witness values when verifying subsequent segments. As a result, redundant witness transmissions are eliminated: Each node value except for the root node value in the MHT is transmitted only once throughout the entire segment verification process. Therefore, the total witness transmission overhead of [24] is given by the following:

$$T_{[MHT-2]}=\left(2n-1\right)+\sum _{i=1}^{n-2}{2}^i·(n-i-1)$$

Figure 4 illustrates the total number of transmitted witness values for various values of $n$, where the number of segments is $2^n$. Here, T [16] represents the standard MHT, T [24] shows the result when [24] is improved as proposed in this paper, and T denotes the proposed scheme. As shown in Figure 4, when $n>10$, the proposed method achieves a reduction of more than 90% in transmission overhead compared to the standard MHT. If [24] adopts a similar partial witness approach presented in this paper, the proposed method still yields a 50% reduction in transmission overhead. These results demonstrate that the proposed scheme significantly reduces transmission overhead compared to both the standard MHT and [24], particularly as the number of segments increases.

5. Conclusions

MHT is widely used for efficiently verifying both individual segments of a dataset and the dataset as a whole. However, when applied to large-scale datasets, MHT-based verification still suffers from redundant hash computations and excessive witness transmissions, leading to performance inefficiencies. Previous studies have addressed the computation inefficiency for computing hash values by storing and reusing previously computed node values.

This paper proposed a method to improve both computational and transmission efficiency by storing and reusing received witness values. Since witnesses consist of valid node values, if a user caches them after verifying a segment and later reuses them to verify subsequent segments, the user can verify the subsequent segment via a subtree of the original MHT. In this case, the user only needs to compute the root node value of the subtree, thereby reducing the number of hash operations. Furthermore, since only a subset of the full witness is required for subtree verification, transmission overhead is also significantly reduced.

The proposed scheme enhances the practicality of MHT-based verification, especially in large-scale or latency-sensitive environments such as IoT and edge networks.

However, the proposed scheme and [24] assume that users receive every segment sequentially, without considering packet loss or network status. If a user receives the ${i+1}^{th}$ segment before receving the $i^{th}$ segment, the verification of the received segment must be delayed until the $i^{th}$ segment is obtained. Furthermore, neither scheme addresses the integrity of the local cache. For example, if the local cache is corrupted, segment verification will always fail regardless of the actual validity of the segment. Future work will focus on the scheme to enable verification of each segment independently of its sequence, as well as addressing local cache integrity issues. In addition, we plan to explore further optimizations using lightweight cryptographic techniques tailored for resource-constrained devices.