Formalization and Verification of PaxosStore from Process Algebra Perspective

Abstract

PaxosStore is a high-availability storage system developed to support the comprehensive business of WeChat. With the widespread application of WeChat, it is particularly important to verify the safety of PaxosStore. This work proposes a formal model for the storage system PaxosStore using the process algebra Communicating Sequential Processes (CSP) to clearly reflect the interactions of the components in PaxosStore. More importantly, we utilize the model checker Process Analysis Toolkit (PAT) to simulate and verify the constructed CSP model. We specifically verify the validity of six properties: deadlock-freeness, divergence-freeness, robustness, consistency, nontriviality and liveness. Through the verification results, we demonstrate that our formalization model successfully satisfies these properties, confirming the correctness and effectiveness of the framework in ensuring secure interactions among the PaxosStore storage system components.

1. Introduction

PaxosStore is a high-availability storage system developed to support the comprehensive business of WeChat [1]. The overall architecture of PaxosStore comprises three layers, including the programming model, the consensus layer, and the storage layer, which is illustrated in Figure 1. For the programming model, it provides a variety of data structures exposed to the application clients. The consensus layer implements the Paxos-based storage protocol [2,3]. In order to fulfill diverse performance requirements, the storage layer offers multiple storage engines implemented based on different storage models (Bitcask, LSM-tree, and Main/Delta Table).

Figure 1. The overall architecture of PaxosStore.

With the widespread application of WeChat, it is particularly important to verify the safety of PaxosStore. The theory and tools of formal methods have become increasingly mature and have gained widespread recognition and application in industry. Since formal methods can help software developers identify errors that are difficult to detect using other approaches, the use of formal verification techniques to enhance the safety and reliability of safety-critical systems has been incorporated into safety standards across various fields in recent years. However, to the best of our knowledge, existing verification efforts on PaxosStore primarily focus on the safety verification of the Paxos protocol in the consensus layer, with limited attention given to the safety verification of the overall storage system.

Goel et al. proved the safety of Paxos by leveraging three structural features in its specification: spatial regularity, temporal regularity, and hierarchical composition [4]. They demonstrated an automatically-inferred inductive invariant for Lamport’s Paxos algorithm. Wang et al. presented a formal colored Petri net model to automatically generate a suite of test cases for the Paxos distributed consensus protocol [5]. The paper [6] provided a formal specification and verification of Lamport’s Multi-Paxos algorithm for the distributed consensus. In their work, the formal specification was written in TLA+ [7], and the proof was written and checked using TLAPS (a proof system for TLA+). Delzanna et al. presented a formal model of Paxos in the executable specification language Promela, extended with a new type of guards, called counting guards [8]. They applied the model checker Spin [9] to automatically validate finite instances of the model and extract preconditions on the size of quorums used in the election phases of the protocol. Paper [10] provided experience feedback on the use of animation and debugging tools to build, improve, and verify a UML model of the Paxos consensus algorithm. From the above research, it is evident that existing verification efforts focus primarily on validating the safety of the PaxosStore’s consensus protocol, while lacking safety verification of the PaxosStore’s storage layer and the interaction between the consensus and storage layers.

Process Algebra is a class of formal methods used to describe and analyze the behaviors of individual processes and their interactions in concurrent systems. It provides a mathematical framework for concurrent computation, enabling the systematic analysis of synchronization, communication, mutual exclusion, and other behaviors between concurrent processes. Process algebra can be used to model the concurrency of a system, describing how multiple processes coordinate their work through synchronization and communication, and analyzing properties such as safety and liveness. It emphasizes modeling the interactions between processes rather than the internal state changes of individual processes. CSP (Communicating Sequential Processes) is a process algebra introduced by Tony Hoare [11], which emphasizes describing communication between processes through event synchronization. In CSP, processes are used to represent each participant in the system, and processes interact with each other through events. Events can be either synchronous (where two processes proceed simultaneously) or asynchronous (where one process does not depend on the actions of another process). In this work, we want to verify the safety of the consensus layer and the storage layer of PaxosStore, as well as the safety of the interaction process between the layers. Through the above analysis, it can be seen that CSP focuses on how processes in the system interact with each other, making it better suited for precisely describing the interaction behavior between processes. Therefore, this paper chooses the process algebra CSP to construct the formal model for PaxosStore.

PAT (Process Analysis Toolkit) [12,13] is a self-contained framework for supporting the composing, simulating, and reasoning of concurrent, real-time systems and other possible domains. It comes with user-friendly interfaces, a featured model editor, and an animated simulator. Most importantly, PAT implements various model-checking techniques catering to different properties such as deadlock-freeness, divergence-freeness, reachability, LTL properties with fairness assumptions, refinement checking, and probabilistic model checking. Therefore, this paper selects PAT as the supporting tool for model checking.

These observations motivate us to propose the formalization and verification of PaxosStore from a process algebra perspective. In this work, our main contributions are summarized as follows:

• We construct a formal model for the storage system PaxosStore based on the process algebra CSP, particularly concerning the PaxosStore’s consensus layer and storage layer, as well as the interactions between these layers;
• We implement the constructed model based on the model checking tool PAT, and verify the validity of six properties: deadlock-freeness, divergence-freeness, robustness, consistency, nontriviality and liveness;
• Through the verification results, we demonstrate that our formalization model successfully satisfies these properties, confirming the correctness and effectiveness of the framework in ensuring secure interactions among the storage system components.

The remainder of this paper is organized as follows. Section 2 introduces the relevant background knowledge. Section 3 presents the formal model of PaxosStore based on CSP. Section 4 implements the constructed model using PAT. Section 5 provides detailed descriptions of six properties along with their verification results. Section 6 reviews related work on process algebra and model checkers for storage systems. Section 7 concludes the paper and discusses potential directions for future work.

2. Background

In this section, we first provide a brief introduction to the Paxos procedure in the consensus layer and the Bitcask model in the storage layer in PaxosStore. Then, we introduce the relevant concepts of process algebra and model checking. After that, we introduce the process algebra CSP and the model checker PAT.

2.1. Paxos and Bitcask

In the consensus layer, the Paxos procedure includes the prepare phase and the accept phase. The prepare phase is for making a preliminary agreement, and the accept phase is for reaching the eventual consensus. The Paxos procedure in PaxosStore uses message passing, illustrated in Figure 2. A proposal $\rho$ is defined as $\rho =(n,v)$, where n is the proposal number and v is the proposal value. $NA$ is the number of acceptors. $N{R}_1$ and $N{R}_2$ represent the number of responses from acceptors in the prepare phase and accept phase, respectively. $NV$ is the number of $acceptedV\ne null$ returned from acceptors.

Figure 2. The Paxos procedure.

• Proposer chooses a proposal number n and sends it to all acceptors.
• Acceptor receives a proposal number n and compares n and its $ResN$ (the proposal number promised by the acceptor to accept). If n is greater than $ResN$, then n is assigned to $ResN$. Acceptor returns $acceptedN$ and $acceptedV$ that it has accepted.
• If $N{R}_1\ast 2$ is greater than $NA$, the proposer checks the value returned from the acceptor. If $NV$ is greater than 0, then the proposer selects $value={\rho }^{\prime }.acceptedV$ as the value where ${\rho }^{\prime }$ is the one with the highest proposal the number is among the accepted proposals and sends $(n,value)$ to all acceptors. Otherwise, the proposer chooses a value $value=v$ and sends $(n,value)$ to all acceptors.
• Acceptor receives a proposal $(n,value)$ from the proposer. If n is greater than or equal to $ResN$, then n is assigned to $ResN$ and $acceptedN$ while $value$ is assigned to $acceptedV$. Acceptor returns $ok$.
• If $N{R}_2\ast 2$ is greater than $NA$, $value$ is chosen.

For the storage layer, we focus on the storage model Bitcask. In Bitcask, there is only one active file for writing by the server. When this active file meets a size threshold, it will be closed and a new active file will be created. Once a file is closed, it is considered immutable and will never be opened for writing again. With each write, a new entry is appended to the active file. The format of each key–value entry includes crc, time stamp, the key size, the value size, the key, and the value. When an append completes, a hash table in the memory is updated. A hash table maps every key to a fixed-size structure, giving the file, the value size, the value, position, and the time stamp.

Figure 3 shows the process of reading a value. For reading a value, we first look up the key in the hash table and obtain the information of the value, which includes the ID of the file, the size of the value, the position of the value, and the time stamp. Then, we can read the value according to the information returned from the hash table.

Figure 3. Reading a value.

2.2. Process Algebra and Model Checking

Formal methods are techniques based on a rigorous mathematical foundation for describing, developing, and verifying computer hardware and software systems. Their mathematical foundation is built upon a formal logical system that integrates formal languages, semantics, and deductive reasoning. Process algebra is a collection of formal languages used for modeling and analyzing concurrent systems, providing a concrete approach to describing interactions, communication, and synchronization among multiple processes. The study of process algebra began in the 1980s, associated with Robin Milner’s pioneering work on CCS and Tony Hoare’s work on CSP. Since its inception, process algebra has proven to be a very important formal language for modeling and analyzing concurrent properties, including mobility, interactivity, safety, and more.

Model Checking is an automated verification technique commonly used to verify the properties of hardware or software systems, particularly in the verification of concurrent systems. Its main goal is to ensure that the system satisfies certain specified properties by systematically checking all possible states of the system. The basic process of model checking is shown in Figure 4. The verification process in this paper is shown as follows.

Figure 4. Basic process of the model checking approach.

• The characteristics and behaviors of the PaxosStore system are described based on CSP, and a system model is constructed.
• The properties to be verified, such as robustness, data consistency, and others, are abstracted from the requirement specifications and defined as formal specifications.
• The system model and formal specifications are implemented using the model checking tool PAT for automated verification. If the described properties are satisfied, the verification result is returned; if the properties are not satisfied, a counterexample is returned, and the error point is identified through a simulation of the system model.

Table 1 provides a brief description of the relevant concepts. For more terminology descriptions, please refer to the Appendix A.

Table 1. Descriptions of technical terms.

Terms	Descriptions
Formal Methods	Techniques based on mathematics and logical reasoning, used to describe, develop, and verify computer systems, ensuring the correctness, safety, and reliability of the systems.
Process Algebra	A formal framework for modeling and analyzing concurrent systems using algebraic operations.
Model Checking	An automated technique that systematically explores all possible states of a system to verify correctness properties.
CSP	A process algebra developed by Tony Hoare for modeling interactions between processes using message passing.
PAT	A model checking tool that supports visual simulation and validation of the CSP models.

2.3. CSP and PAT

CSP was proposed by Turing Award recipient Tony Hoare and was mainly designed to model and analyze the behavior of concurrent systems. CSP emphasizes communication between processes, modeling the system by defining the behavior and interactions of the processes. This modeling language has been widely used in various domains, which include the protocols, memory model, NDN-based IoV, SDN under multi-controller architectures and OpenStack swift, OSEK/VDX operating system and Aeolus-based file system, nondeterministic communication in the wireless sensor networks, and more. The syntax of a subset of the CSP language is given in Table 2.

Table 2. Notations and meanings of CSP.

Notations	Meanings
$\alpha P$	the alphabet of process P
$\alpha c$	the set of messages communicable on channel c
$a\to P$	a then P
$P\left|\right|Q$	P in parallel with Q
$P\sqcap Q$	P or Q (non-deterministic)
$P\left|\right||Q$	P interleave Q
$P;Q$	P (successfully) followed by Q
$P◃b▹Q$	P if b else Q
$c!e$	on channel c output value of e
$c?x$	on channel c input to x

• $a\to P$ describes an object which first engages in the event a and then behaves exactly as described by process P.
• If P and Q are processes with the same alphabet, then the notation $P\left|\right|Q$ denotes the process that behaves like the system composed of processes P and Q interacting in lock-step synchronization.
• If P and Q are processes, then the notation $P\sqcap Q$ denotes a process which behaves either like P or like Q.
• $P\left|\right||Q$ denotes that two processes run concurrently without barrier synchronization. Both P and Q may perform their local actions without synchronizing with each other.
• If P and Q are sequential processes with the same alphabet, their sequential composition $P;Q$ is a process which first behaves like P, but when P terminates successfully, $P;Q$ continues by behaving as Q. If P never terminates successfully, neither does $P;Q$.

The system analysis in PAT is facilitated through two primary methods: simulation and model checking. The visualized simulator enables users to interactively explore their models by selecting one enabled action at a time, allowing the computer to generate system traces randomly, or even constructing the complete state graph. On the other hand, the embedded model checkers in PAT leverage advanced model-checking techniques for systematic analysis. Users can specify assertions in various forms, and with a single click, PAT evaluates their validity. If an assertion is false, a counterexample is generated. Here, we list some definitions and assertions in PAT.

• $\#defineN0;$
  It defines a global constant N with the initial value 0.
• $varDcenter\left[N\right]=[0,1,2,...,N-1];$
  This statement defines an array named $Dcenter$. The size of the array is N.
• $channelc1;$
  This statement declares that c is the channel name and 1 is the buffer size. The channel buffer size must be greater than or equal to 0. Notice that a channel with buffer size 0 sends/receives messages synchronously.
• $\#assertP\left(\right)deadlockfree;$
  Given $P\left(\right)$ as a process, this assertion asks whether $P\left(\right)$ is deadlock-free or not.
• $\#assertP\left(\right)divergencefree;$
  Given $P\left(\right)$ as a process, this assertion asks whether $P\left(\right)$ is divergence-free or not.
• $\#assertP\left(\right)reachescond;$
  Given $P\left(\right)$ as a process, this assertion asks whether $P\left(\right)$ can reach a state at which some given condition is satisfied.
• $\#assertP\left(\right)\vDash F;$
  PAT supports the full set of LTL syntax. This assertion asks whether $P\left(\right)$ satisfies the LTL formula F, where the syntax of F is defined as the following rules:
  $F=e\left|prop\right|\left[\right]F|<>F|XF\left|F1UF2\right|F1RF2$
  where e is an event, $prop$ is a pre-defined proposition, $\left[\right]$ reads as “always”, $<>$ reads as “eventually”, X reads as “next”, U reads as “until”, and R reads as “release”.

3. Modeling PaxosStore

In this section, we propose the CSP model of PaxosStore. The formalization is proceeded based on the introduction to PaxosStore in Section 2. Figure 5 illustrates the interaction and communication framework among all processes in the PaxosStore model.

Figure 5. The framework of the PaxosStore model.

The entire system is modeled as the interleaving of two CSP processes, $Write$ and $Read$, where $Write$ represents the process of writing a value, and $Read$ represents the process of reading a value. Equation (1) provides the formal definition of the entire system.

$$PaxosStore{=}_{df}Write\left|\right||Read$$

(1)

For the writing process, there are three crucial processes running in parallel, which are abstracted as the processes $Proposer$, $Acceptor$, and $Storage$. Equation (2) describes the formal definition of the writing process.

$$Write{=}_{df}Proposer\left|\right|Acceptor\left|\right|Storage$$

(2)

Next, we provide the formal definitions of the three processes $Proposer$, $Acceptor$, and $Storage$, respectively.

3.1. Proposer

In the proposer operations, each proposer is assigned a unique ID, denoted as i, and corresponds to a process called $propose{r}_i$. Therefore, the $Proposer$ process can be formalized as in Equation (3). Here, I represents the set of all proposer IDs associated with the proposer tasks.

$$Proposer{=}_{df}{\left|\right||}_{i\in I}propose{r}_i$$

(3)

The $propose{r}_i$ process describes the details of proposer operations. We first define the channels and messages used in the $propose{r}_i$ process.

The channels in the $propose{r}_i$ process are as follows.

• $P_i{A}_j$: channel between $propose{r}_i$ and $accepto{r}_j$.
• $P_i{S}_k$: channel between $propose{r}_i$ and $storag{e}_k$.

The messages in the $propose{r}_i$ process are as follows, where # is just a separator.

• $acceptedInfo{=}_{df}ResN\#acceptedN\#acceptedV$
  $acceptedInfo$ is sent from $accepto{r}_j$ to $propose{r}_i$, which includes the proposal number promised by the acceptor to accept and for the proposal to be accepted.
• $proposal{=}_{df}n\#value$
  $proposal$ is sent from $propose{r}_i$ to $accepto{r}_j$, which includes the proposal number and the proposal value.

Figure 6 illustrates the message communication between the three processes $propose{r}_i$, $accepto{r}_j$, and $storag{e}_k$.

Figure 6. The message communication.

We formalize the process $propose{r}_i$ as in Equation (4). Equations (5) to (10) represent the formal definitions of subprocesses $P_1$ to $P_6$, respectively.

$$\begin{array}{cc}\hfill propose{r}_i{=}_{df}{P}_i{A}_j!n& \to (P_i{A}_j?acceptedInfo\to N{R}_1:=N{R}_1+1\to P_1)\hfill \\ \hfill & \sqcap (P_i{A}_j?error\to propose{r}_i)\hfill \end{array}$$

(4)

$$P_1{=}_{df}\left(P_2\right)◃N{R}_1\ast 2>NA▹\left(propose{r}_i\right)$$

(5)

$$P_2{=}_{df}(NV:=NV+1\to P_3)◃acceptedInfo.acceptedV\ne null▹\left(P_3\right)$$

(6)

$$P_3{=}_{df}(value:=select\left(acceptedInfo\right)\to P_4)◃NV>0▹(value:=v\to P_4)$$

(7)

$$\begin{array}{cc}\hfill P_4{=}_{df}{P}_i{A}_j!proposal& \to (P_i{A}_j?ok\to N{R}_2:=N{R}_2+1\to P_5)\hfill \\ \hfill & \sqcap (P_i{A}_j?error\to propose{r}_i)\hfill \end{array}$$

(8)

$$P_5{=}_{df}(choose\left(value\right)\to P_i{S}_k!value\to P_6)◃N{R}_2\ast 2>NA▹\left(propose{r}_i\right)$$

(9)

$$P_6{=}_{df}(P_i{S}_k?success\to propose{r}_i)\sqcap (P_i{S}_k?fail\to propose{r}_i)$$

(10)

In the process $propose{r}_i$, it first sends a proposal number n via channel $P_i{A}_j$ to the process $accepto{r}_j$. Then, $propose{r}_i$ either receives a message that the acceptor has already accepted, or receives an error message. If $propose{r}_i$ receives the message $acceptedInfo$, then the value of $N{R}_1$ increases by 1. If $N{R}_1\ast 2$ is greater than $NA$, then $propose{r}_i$ checks the message returned from the acceptor. If $acceptedV$ is not null, then the value of $NV$ increases by 1. If $NV$ is greater than 0, then $propose{r}_i$ selects a value $value=\rho .acceptedV$, as the value where $\rho$ is the one with the highest proposal number among the accepted proposals and sends $proposal=(n,value)$ to all acceptors. Otherwise, $propose{r}_i$ chooses a value $value=v$ and sends $proposal=(n,value)$ to all acceptors. After that, the process $propose{r}_i$ either receives a $ok$ message or receives an error message. If it receives a $ok$ message, then the value of $N{R}_2$ increases by 1. If $N{R}_2\ast 2$ is greater than $NA$, then the $value$ is chosen. $propose{r}_i$ sends $value$ to $storag{e}_k$ via channel $P_i{S}_k$. Then, $propose{r}_i$ either receives a $success$ message or receives a $fail$ message from the process $storag{e}_k$.

3.2. Acceptor

In the acceptor operations, each acceptor is assigned a unique ID, denoted j, which corresponds to a process called $accepto{r}_j$. Therefore, the $Acceptor$ process can be formalized as in Equation (11). Here, J represents the set of all acceptor IDs associated with the acceptor tasks.

$$Acceptor{=}_{df}{\left|\right||}_{j\in J}accepto{r}_j$$

(11)

The $accepto{r}_j$ process describes the details of the acceptor operations. We formalize the process $accepto{r}_j$ as in Equation (12). Equation (13) represents the formal definition of subprocess $A_1$.

$$\begin{array}{cc}\hfill accepto{r}_j& {=}_{ df}{P}_i{A}_j?n\to (Res{N}_j:=n\to P_i{A}_j!acceptedInfo\to A_1)\hfill \\ \hfill & \hspace{1em}\hspace{1em}◃n>Res{N}_j▹(P_i{A}_j!error\to accepto{r}_j)\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill A_1& {=}_{ df}{P}_i{A}_j?proposal\to (Res{N}_j:=proposal.n;\hfill \\ \hfill & \hspace{1em}\hspace{1em}acceptedN:=proposal.n;acceptedV:=proposal.value\hfill \\ \hfill & \hspace{1em}\hspace{1em}\to P_i{A}_j!ok\to accepto{r}_j)\hfill \\ \hfill & \hspace{1em}\hspace{1em}◃proposal.n>=Res{N}_j▹(P_i{A}_j!error\to accepto{r}_j)\hfill \end{array}$$

(13)

In the process $accepto{r}_j$, it receives a proposal number n from the proposer. If n is greater than the proposal number $Res{N}_j$ that the acceptor promises to accept, then n is assigned to $Res{N}_j$. After that, $accepto{r}_j$ sends $acceptedInfo$ to the proposer and receives a proposal from the proposer. If $proposal.n$ is greater than or equal to $Res{N}_j$, then the acceptor accepts this proposal and sends the message $ok$ to the proposer. Otherwise, it sends the message $error$ to the proposer.

3.3. Storage

In the storage operations, each storage is assigned a unique ID, denoted as k, and corresponds to a process called $storag{e}_k$. Therefore, the $Storage$ process can be formalized as Equation (14). Here, K represents the set of all storage IDs associated with the storage tasks in the writing process.

$$Storage{=}_{df}{\left|\right||}_{k\in K}storag{e}_k$$

(14)

The process $storag{e}_k$ describes the details of the storage operations in the writing process. We formalize the process $storag{e}_k$ as in Equation (15).

$$\begin{array}{cc}\hfill storag{e}_k{=}_{df}& P_i{S}_k?value\to d:=verifyDstate\to \hfill \\ \hfill & (store\left(value\right)\to updatehashtable\to P_i{S}_k!success\to storag{e}_k)\hfill \\ \hfill & ◃d=true▹(P_i{S}_k!fail\to storag{e}_k)\hfill \end{array}$$

(15)

For the process $storag{e}_k$, it first receives a value from a proposer. Then, it verifies whether these datacenters, which are used to store the value are available or not. If these datacenters are available, then the value is stored successfully. After that, $storag{e}_k$ updates the hash table and sends the $success$ message to $propose{r}_i$ via channel $P_i{S}_k$. Otherwise, it sends the $fail$ message to $propose{r}_i$.

3.4. Read

For the reading process, there are two crucial processes running in parallel, which are abstracted as the processes $Cread$, $StoreV$. Equation (16) describes the formal definition of the reading process.

$$Read{=}_{df}Cread\left|\right|StoreV$$

(16)

In the reading operations, each reader is assigned a unique ID, denoted as r, and corresponds to a process called $crea{d}_r$. Therefore, the $Cread$ process can be formalized as Equation (17). Here, R represents the set of all reader IDs associated with the reading tasks.

$$Cread{=}_{df}{\left|\right||}_{r\in R}crea{d}_r$$

(17)

The process $crea{d}_r$ describes the details of the reading operations. We first define the channels and messages used in the $crea{d}_r$ process.

The channels in the $crea{d}_r$ process are as follows.

• $C_r{S}_s$: channel between $crea{d}_r$ and $store{v}_s$.

The messages in the $crea{d}_r$ process are as follows, where # is just a separator.

• $readvalue$ is sent from $crea{d}_r$ to $store{v}_s$, indicating a request to initiate a data read.
• $valueInfo{=}_{df}fileid\#vsz\#vpos\#tstamp$
  $valueInfo$ is sent from $store{v}_s$ to $crea{d}_r$, which includes the ID of the file, the size of the value, the position of the value, and the time stamp.

We formalize the process $crea{d}_r$ as Equation (18).

$$crea{d}_r{=}_{df}{C}_r{S}_s!readvalue\to C_r{S}_s?valueInfo\to read\left(valueInfo\right)\to crea{d}_r$$

(18)

$crea{d}_r$ requests to read a value. Then, it receives the information of this value. After that, it can read this value according to this information.

The $StoreV$ process models the behaviors of the storage system in the reading operations. Each task is assigned a unique ID, denoted as s, and corresponds to a process called $store{v}_s$. Therefore, the $StoveV$ process can be formalized as in Equation (19). Here, S represents the set of all task IDs associated with the storage data node in the reading operations.

$$StoreV{=}_{df}{\left|\right||}_{s\in S}store{v}_s$$

(19)

The process $store{v}_s$ describes the details of the storage data ceter in the reading operations. We formalize the process $store{v}_s$ as in Equation (20).  

$$\begin{array}{cc}\hfill store{v}_s& {=}_{ df}{C}_r{S}_s?readvalue\to offset:=get\left(key\right)\to \hfill \\ \hfill & \hspace{1em}\hspace{1em}valueInfo:=read\left(offset\right)\to C_r{S}_s!valueInfo\to store{v}_s\hfill \end{array}$$

(20)

$store{v}_s$ receives a reading request from the client, then it looks up the offset of the key in the memory index (hash table). After that, it reads the key’s data structure based on the offset and sends it to the client.

3.5. Example of the Model

To better understand the execution process of the write model, we provide an example here. We only demonstrate the simplest communication scenario: the interaction among a proposer, an acceptor, and a storage node. Before the proposal begins, we make the following assumptions: the proposal number is 2, the proposal value is 3, the number of responses returned from the acceptors is 0, and the acceptor has not received any other proposals. According to the construction process of the write model, the first round of interactions among these three nodes is as follows.

$$\begin{array}{cc}\hfill propose{r}_1\left|\right|accepto{r}_1\left|\right|storag{e}_1=& P_1{A}_1!2\to Res{N}_1:=2\to P_1{A}_1!2.null.null\to \hfill \\ & N{R}_1:=1\to value:=3\to P_1{A}_1!2.3\to \hfill \\ & Res{N}_1:=2;acceptedN:=2;acceptedV:=3\to \hfill \\ & P_1{A}_1!ok\to N{R}_2:=1\to choose\left(3\right)\to \hfill \\ & P_1{S}_1!3\to d:=true\to store\left(3\right)\to \hfill \\ & updatehashtable\to P_1{S}_1!success\hfill \end{array}$$

We also provide an example to illustrate the execution process of the read model. We only demonstrate the simplest communication scenario: the interaction between a reading node and a storage node. Before the reading begins, we make the following assumptions: the key exists in the memory index with an offset of 4096. According to the construction process of the read model, the first round of the interaction between the two nodes is as follows.

$$\begin{array}{cc}\hfill crea{d}_1\left|\right|store{v}_1=& C_1{S}_1!readvalue\to offset:=4096\to valueInfo:=read\left(4096\right)\to \hfill \\ & C_1{S}_1!valueInfo\to read\left(valueInfo\right)\hfill \end{array}$$

4. Implementing the CSP Model in PAT

In this section, we implement the PaxosStore model based on the model checker PAT. We first provide some representative global definitions used in the process definitions. Then, we provide some process definitions as examples.

4.1. Global Definitions

For defining the processes, various parameters are configured for reading and writing values. Here, we provide some global constants and variables as examples.

In the model checker PAT, the process proposer(i) and process acceptor(j) communicate through message passing on channel c. The process proposer(i) and process storage(p) communicate with each other via channel d. The processes cread(r) and storev(s) communicate with each other via channel e. The constant N is the number of proposers and acceptors. n is the initial proposal number and v is the initial proposal value. responseC is the number of responses from acceptors, and its initial value is 0.

The array minN records the least proposal number that the acceptor is willing to accept. The arrays acceptedN and acceptedV store the proposal number and proposal value that the acceptor has accepted, respectively. The array proposeV holds the values that the proposer has proposed. The array chosenV stores the value that has been chosen. The array datacenterS represents the state of each datacenter, where 0 indicates that the datacenter is unavailable, and 1 indicates that the datacenter is available. Initially, the states of the three datacenters are set to 0, 1, and 1. The array writeV tracks whether the writer has successfully written the value: 0 indicates a failure, and 1 indicates success. Similarly, the array readV records whether the reader has successfully read the value. The last statement defines four communication messages.

4.2. Process Definitions

Here, we illustrate some representative process definitions as examples.

For the process acceptor(j), it first receives a proposal number from the proposer. If the proposal number received is greater than the least proposal number that acceptor j promises to accept, then the proposal number received is assigned to minN[j]. Acceptor j returns minN[j] and the proposal that it has accepted to the proposer. Otherwise, it returns minN[j]. For the first case, acceptor j prepares to receive a new proposal. When it receives a new proposal from the proposer, it compares the proposal number with minN[j]. If the proposal number is greater than or equal to minN[j], then it accepts this new proposal and returns the accepted result to the proposer.

For storage(p), it first receives the value chosen from the proposer. If at least one of the three datacenters is available, then the value is stored successfully. After that, it updates the hash table and sends the message success to the proposer. Otherwise, it sends the message fail to the proposer.

For the process cread(r), it first sends a request for reading a value to the process storev(s) via the channel e. Then, it will receive the information valueInfo about this value from the process storev(s). If at least one of the three datacenters is available, then it can read this value according to the information successfully.

For storev(s), it first receives a reading request. Then, it looks up the key in the hash table and returns the information of the value.

Figure 7 shows a simulation of the CSP model we constructed, highlighting the enabled events and event traces within the model.

Figure 7. A simulation for PaxosStore.

5. Verification

In this section, we verify the validity of six properties: deadlock-freeness, divergence-freeness, robustness, consistency, nontriviality and liveness. Through the verification results, we demonstrate that our formalization model successfully satisfies these properties, confirming the correctness and effectiveness of the framework in ensuring secure interactions among the PaxosStore storage system components.

5.1. The Properties

We first provide a description of deadlock-freeness.

• Property 1: Deadlock-freeness

In PaxosStore, we should avoid the situation in which some clients are waiting for the resources that have been occupied by other clients infinitely. In the model checker PAT, there is a primitive to describe this situation:

• Property 2: Divergence-freeness

A divergent system is usually undesirable. In the model checker PAT, there is a primitive to describe this situation:

• Property 3: Robustness

The primary objective of PaxosStore is to store the data reliably even in the presence of failures. In WeChat production, hardware failure and network outages are the two main sources of failure. Here, we focus on the datacenter failures. We verify that readers can read the value and writers can write the value successfully when some of the datacenters break down. Due to the limitations of the state space, we have only constructed three datacenters in the model, but this does not affect the verification of the model’s properties. We have tested that when the three datacenters have the states [1,1,1], [1,1,0], [1,0,1], [0,1,1], [1,0,0], [0,1,0], [0,0,1], where 0 means that the datacenter breaks down and 1 means that the datacenter is available, and property 3 is satisfied. This assertion about this property is described as below:

• Property 4: Consistency

PaxosStore is designed as a multi-homed system that runs actively on multiple datacenters around the clock, and meanwhile, employs the Paxos protocol for value consensus. The Paxos consensus algorithm must satisfy the consistency property, that is, only a single value may be chosen. This property can use two assertions to describe in PAT, which is shown below:

When only one of the two assertions is valid, the consistency property is satisfied, i.e., if consistency1 is valid and consistency2 is not valid, then the PaxosStore model satisfies the consistency property.

• Property 5: Nontriviality

In addition to the above properties, a Paxos algorithm used in PaxosStore also needs to satisfy the property nontriviality; that is, only a value that has been proposed may be chosen. This assertion is described as below:

• Property 6: Liveness

Liveness properties describe the infinite behavior of the model [14], ensuring that desirable events will eventually occur. In this model, every proposal and every file read request will be processed infinitely often, ensuring fair handling. This property can use two assertions to describe in PAT, which is shown below:

5.2. Evaluation Results

The verification results are shown in Figure 8. From Figure 8, we find that the properties deadlock-freeness, divergence-freeness, robustness, consistency, nontriviality, and liveness are all valid, indicating that the constructed model satisfies the PaxosStore requirements, and these properties confirm the correctness and effectiveness of the framework in ensuring secure interactions among the PaxosStore storage system components. We provide a detailed analysis and evaluation for each assertion.

Figure 8. Verification results for PaxosStore.

Table 3 provides a verification statistics table showing runtime, memory consumption, visited states, and total transitions for each assertion. From the statistical table, it can be seen that the verification process of the three assertions—deadlock-free, divergence-free, and consistency2—performs a depth-first search of all possible states of the system. These states do not lead to deadlock or divergence, nor do they satisfy consistency2. At the same time, it can also be seen that these three assertions are the most time and memory-consuming. Therefore, in the subsequent model extension work, a better abstraction of the model will be considered.

Table 3. Verification statistics.

Assertions	Time Used (s)	Memory Used (KB)	Visited States	Total Transitions
deadlockfree	178.1149229	291,660.256	4,190,208	21,620,158
divergencefree	484.5008373	334,991.168	4,190,208	21,620,158
robustness	0.08012	12,080.512	3954	3239
consistency1	0.003786	10,338.52	264	263
consistency2	128.7361892	74,894.504	4,190,208	21,620,158
nontriviality	0.0022756	10,404.032	272	271
liveness1	0.683174	58,302.288	31,630	88,199
liveness2	0.0122431	43,685.944	759	1752

We also provide the detailed verification results of the assertions. Figure 9 shows the detailed verification results of deadlock-freeness and divergence-freeness. Deadlock-freeness is valid, which means that the CSP model will not enter a deadlock state. Divergence-freeness is valid, which demonstrates that the system model is well defined. The verification assertions for the two properties are applicable to all storage systems, meaning that any storage system can adopt the verification method presented in this paper to validate the two properties.

Figure 9. Detailed verification results of deadlock-freeness and divergence-freeness.

Figure 10 shows the detailed verification results of robustness.

Figure 10. Detailed verification result of robustness.

From Figure 10, it can be seen that this assertion is valid. The result indicates that the system successfully stores data even when storage nodes experience failures. The verification method is applicable to all storage systems that use replication mechanisms to store data, including HDFS, Ceph, GFS, OpenStack swift, and others.

Figure 11 shows the detailed verification results of consistency. From Figure 11, it can be seen that the first assertion is valid, while the second assertion is invalid. The result indicates that only a single value may be chosen, in other words, only one value ultimately reaches consensus. The verification method can not only verify the consistency of the Paxos algorithm, but also be used for the consistency verification of other consensus algorithms, such as Raft, PBFT, HotStuff, and others.

Figure 11. Detailed verification result of consistency.

Figure 12 shows the detailed verification results of nontriviality. From Figure 12, it can be seen that this assertion is valid. The result indicates that only a value that has been proposed may be chosen. Similar to consistency verification assertions, this verification assertion can also be used for property verification of more consensus algorithms.

Figure 12. Detailed verification result of nontriviality.

Figure 13 shows the detailed verification results of liveness. From Figure 13, it can be seen that the two assertions are valid. The verification results show that every proposal and every read request will be processed infinitely often. The descriptions of the two verification assertions apply to the liveness verification of most storage systems that use consensus protocols.

Figure 13. Detailed verification result of liveness.

6. Related Work

In recent years, significant efforts have been made to explore the formal modeling and verification of storage systems. Existing process algebra systems provide a rigorous methodological foundation for modeling and analyzing the behavior and characteristics of concurrent systems. CSP [11], CCS [15], and ACP [16] have been introduced to model the communication among processes in reactive systems. Sun et al. proposed CSP# to specify concurrent systems by integrating CSP-like compositional operators with sequential programs that update shared variables [17]. Reed et al. extended CSP into timed CSP by incorporating a real-time model [18]. Since we focus more on the interaction behavior of components and the safety of the system, we choose CSP to construct the system model. Notably, CSP has been widely adopted across various domains. Chen et al. modeled and verified Constrained Application Protocol (CoAP) based on CSP [19]. Xu et al. used CSP to model and verify kafka messaging mechanism [20]. Fei et al. presented a formal model for AKA protocols using CSP [21]. Liu et al. presented a CSP model for AMQP [22]. Xiao et al. used CSP to model PSO and verified four properties based on PAT [23]. Chen et al. used CSP to formalize NDN-based IoV [24]. In their paper, they mainly focused on the data access mechanism. Huang et al. proposed a formal model of the OSEK OS at the code level based on CSP and verified three significant properties of the OSEK-based system [25]. The paper [26] presented a model for wireless sensor networks (WSNs) to be used for the formal verification of communication reliability in mesh networks based on CSP.

A variety of model checkers have been developed to support the simulation and verification of models. UPPAAL [27], based on the theory of timed automata [28], is designed for the modeling, validation, and verification of real-time systems. Chen et al. used the model checker UPPAAL to implement the CaIT model and verified six temporal properties of the model [29]. Fei et al. modeled and verified some fundamental properties of the NLSR protocol using the model checker UPPAAL [30]. UPPAAL is only suitable for time-related systems. Meseguer initially proposed the theory of rewriting logic [31], which later led to the introduction of the rewriting engine Maude by Meseguer and colleagues. Currently, Maude has been widely applied across various fields, including equivalence checking of quantum circuits based on dirac notation [32], raft log replication [33], and so on. Maude is complex to use, requires a strong theoretical background, and lacks a user-friendly graphical interface. SPIN is an efficient verification system for the models of distributed software systems [34], and it has been used to model and analyze the IoT protocols [35], predicate transition nets [36], and more. SPIN does not support the verification of temporal properties and cannot handle time-related issues. Rossi proposed a logic-based technique for verifying both security and correctness properties of multilevel service compositions [37]. With the aid of the NCSU Concurrency Workbench model checker, both non-interference and compliance can be verified. However, this verification technique lacks a user-friendly graphical interface. PAT [12,13] is a comprehensive framework for composing, simulating, and reasoning about concurrent and real-time systems, as well as other domains. It integrates multiple verification techniques, such as LTL verification, temporal verification, and probabilistic model checking. Additionally, it offers a graphical interface and a flexible plugin architecture. PAT has been widely used in verification tasks across various fields, such as smart contracts [38], autonomous systems [39], and more. Due to PAT’s versatility and user-friendly graphical interface, this paper selects PAT for model implementation and property verification.

7. Conclusions

This work presents a formal model of the PaxosStore storage system using CSP to clearly capture the interactions between its components. More importantly, we employ PAT to simulate and verify the constructed CSP model. Specifically, we verify the validity of six properties: deadlock-freeness, divergence-freeness, robustness, consistency, nontriviality, and liveness. The verification results demonstrate that our formalized model successfully satisfies these properties, confirming the correctness and effectiveness of the framework in ensuring secure interactions among the components of the PaxosStore storage system.

The current PaxosStore model’s scale in this paper has not led to a state explosion issue during the verification process. The storage layer in PaxosStore provides multiple storage models, including Bitcask, LSM-tree, and Main/Delta Table. In this work, the modeling and verification of the storage layer are primarily centered around the Bitcask model. Since the focus during model construction is on storage process communication and the correct storage and retrieval of data, rather than the differences in storage interfaces, the other two storage interfaces do not affect the property verification of the model.

Data integrity refers to the characteristic of data remaining accurate, complete, and unaltered during storage, transmission, and querying processes. In this paper, we have verified the robustness and data consistency within the storage system. In the future, we will incorporate the verification of query integrity to ensure that data are not tampered with.

To comprehensively model system behavior and conduct more analyses, the model will be expanded in the future. The potential expansion directions include the following:

• Incorporating the other two storage interfaces LSM-tree and Main/Delta Table into the model.
• Incorporating more analytical work, including read/write latency, the verification of query integrity to ensure that data are not tampered with.

As the model expands, the verification process may encounter the state explosion problem. We will consider using the following methods to address the potential state explosion problem.

• Theorem proving may provide a solution to the possible state space explosion problem. The commonly used tools for theorem proving include Coq [40] and Isabelle [41].
• Symbolic model checking [42] utilizes symbolic methods, such as Binary Decision Diagrams (BDDs) or SAT solvers [43], to efficiently represent and explore the system’s state space, thereby overcoming the state explosion problem. NuSMV [44] is a tool that supports both symbolic and explicit state space exploration. Some existing research works have provided valuable research ideas, especially those related to the property verification [45,46], process algebra [47,48], and state–space exploration [49].
• Abstraction techniques [50] are a method for simplifying system models, primarily used to mitigate the state explosion problem and enhance the scalability of formal verification. Some existing research works have also provided valuable ideas for future studies, especially those related to state–space exploration [51].