Design of Identical Strictly and Rearrangeably Nonblocking Folded Clos Networks with Equally Sized Square Crossbars

Abstract

Clos networks and their folded versions, fat trees, are widely adopted in interconnection network designs for data centers and supercomputers. There are two main types of Clos networks: strictly nonblocking Clos networks and rearrangeably nonblocking Clos networks. Strictly nonblocking Clos networks can connect an idle input to an idle output without interfering with existing connections. Rearrangeably nonblocking Clos networks can connect an idle input to an idle output with rearrangements of existing connections. Traditional strictly nonblocking Clos networks have two drawbacks. One drawback is the use of crossbars with different numbers of input and output ports, whereas the currently available switches are square crossbars with the same number of input and output ports. Another drawback is that every connection goes through a fixed number of stages, increasing the length of the communication path. A drawback of traditional fat trees is that the root stage uses differently sized crossbar switches than the other stages. To solve these problems, this paper proposes an Identical Strictly NonBlocking folded Clos (ISNBC) network that uses equally sized square crossbars for all switches. Correspondingly, this paper also proposes an Identical Rearrangeably NonBlocking folded Clos (IRNBC) network. Both ISNBC and IRNBC networks can have any number of stages, can use equally sized square crossbars with no unused switch ports, and can utilize shortcut connections to reduce communication path lengths. Moreover, both ISNBC and IRNBC networks have a lower switch crosspoint cost ratio relative to a single crossbar than their corresponding traditional Clos networks. Specifically, ISNBC networks use 46.43% to 87.71% crosspoints of traditional strictly nonblocking folded Clos networks, and IRNBC networks use 53.85% to 60.00% crosspoints of traditional rearrangeably nonblocking folded Clos networks.

1. Introduction

Clos networks [1] and fat trees [2] have been widely used in interconnection network designs for modern data centers and supercomputers such as Google Jupiter [3], Tianhe [4], TaihuLight [5], Frontera [6], Summit and Sierra [7], and the Multi-Plane Fat Tree for DeepSeek-V3 [8].

The traditional unidirectional nonblocking Clos network was originally designed for telecommunications. It is a type of multistage circuit-switching network that replaces a single large crossbar to reduce hardware costs in terms of crosspoints.

A three-stage traditional unidirectional Clos network topology [1] is parameterized by two integers (n and m, where n is the number of sources connecting to an ingress-stage crossbar switch or the number of destinations connecting to an egress-stage crossbar switch and m is the number of crossbar switches in the middle stage). The ingress stage has n crossbar switches, and the number of crossbar switches in the egress stage is also n. Therefore, the total number of sources is $n^2$, and the total number of destinations is also $n^2$. A switch in the ingress stage is an $n\times m$ (n inputs and m outputs) crossbar. A switch in the egress stage is an $m\times n$ crossbar. A switch in the middle stage is an $n\times n$ crossbar. There is exactly one connection between each ingress-stage switch and each middle-stage switch, and there is exactly one connection between each middle-stage switch and each egress-stage switch. $m\ge 2n-1$, indicates a strictly nonblocking network, meaning that the network can connect a free source to a free destination without interfering with existing connections. $m\ge n$ indicates a rearrangeably nonblocking network, meaning that the network can connect a free source to a free destination with rearrangements of existing connections.

Traditional unidirectional strictly nonblocking Clos networks have two drawbacks. One drawback is that all connections from sources to destinations pass through a fixed number of stages. The source is connected to the switch in the ingress stage, and the destination is connected to the switch in the egress stage. Let s be the number of stages of a Clos network. Then, the path length from the ingress stage to the egress stage is $s-1$. Counting the path from the source to the ingress-stage switch and the path from the egress-stage switch to the destination, the path length is $1+(s-1)+1=s+1$. For example, the path length in a three-stage unidirectional Clos network is always 4. It is not possible to exploit the locality of source–destination pairs. Another drawback is that it uses $n\times m$ and $m\times n$ crossbars with different numbers of input and output ports. However, nowadays, the available switches are square crossbars with the same number of input and output ports. If the same square crossbars are used for all switches, there will be many unused ports.

A fat tree is a folded version of a Clos network [9]. It merges the corresponding ingress and egress switches. Then, the merged stage is called a leaf stage, and the middle stage is called a root stage. A fat tree can utilize shortcut connections, that is, a connection does not have to go through all stages. For example, if the source and destination are connected to the same leaf switch, the connection does not need to go to the root stage. Thus, the path length is 2 instead of 4. A drawback of fat-tree networks is that the root stage uses differently sized crossbar switches than the other stages.

A k-ary n tree [10] is a kind of parametric fat tree where k is the arity or number of links of a switch that connect to the previous or next stage and n is the number of stages, that is, the switch radix is $2k$. A k-ary n-tree Clos network can be constructed with two back-to-back k-ary n-fly butterflies [11]. A 2-ary n-tree Clos network is also called a Beneš network [12]. A k-ary fat tree [13] is a bidirectional ($k/2$)-ary n-tree Clos network where k is even.

A packet can be routed to an arbitrary middle switch in a rearrangeably nonblocking Clos network or an arbitrary root switch in a rearrangeably nonblocking folded Clos network, then to its ultimate destination. This increases hardware cost and packet latency. Mirrored k-ary n-tree networks [14] and peer k-ary n-tree networks [15] focus on increasing network capacity and reducing hardware costs and packet latency.

A strictly nonblocking folded Clos network using same-sized crossbar switches was proposed in [16]. The proposed network has a two-stage structure and uses multiple links between a leaf switch and a root switch. It may have unused switch ports. The number of unused switch ports is reduced by adjusting the number of leaf switches, but the existence of unused switch ports increases the hardware cost. A flexible folded Clos network was proposed in [17]. To reduce the blocking probability, a second group of switches is added to the root stage. Ref. [18] extended the number of groups from two to a general number (S). All these networks have only two stages, making it difficult to scale the network.

The existing issues are summarized below. Traditional unidirectional strictly nonblocking Clos networks use switches with different numbers of input and output ports and route packets through a fixed number of stages. Fat trees, the folded version of Clos networks, use differently sized switches at the root and other stages. Recently proposed Clos networks have only two stages and introduce unused ports on the switches. This paper attempts to solve these problems with a low crosspoint ratio relative to a single crossbar.

The contributions of this paper are summarized as follows. An Identical Strictly NonBlocking folded Clos (ISNBC) network and an Identical Rearrangeably NonBlocking folded Clos (IRNBC) network are proposed. Both ISNBC and IRNBC networks can have any number of stages to increase the system’s scalability, can use equally sized square crossbars with no unused switch ports to accommodate currently available switches at low costs, and can utilize shortcut connections to reduce communication path lengths. Moreover, both ISNBC and IRNBC networks have a lower crosspoint ratio relative to a single crossbar than their corresponding traditional nonblocking Clos networks.

The rest of this paper is organized as follows. Section 2 reviews some related multistage interconnection networks. Section 3 proposes identical strictly and rearrangeably nonblocking folded Clos networks consisting of equally sized square crossbars. Section 4 evaluates the hardware cost from the crosspoint perspective and shows that the costs of the proposed networks are lower than those of their corresponding traditional networks. Finally, Section 5 concludes the paper and suggests some future research topics.

2. Related Works

There are many different types of multistage interconnection networks. This section reviews some related multistage interconnection networks.

2.1. Traditional Nonblocking Clos Networks

A traditional unidirectional strictly nonblocking Clos network [1] has odd switch stages. Consider a traditional unidirectional strictly nonblocking Clos network with three switch stages: The ingress stage has n switches, and each switch is an $n\times m$ (n inputs and m outputs) crossbar. The middle stage has m switches, and each switch is an $n\times n$ crossbar. The egress stage has n switches, and each switch is an $m\times n$ crossbar. The m outputs of an ingress switch are connected to m middle switches; each output is connected to a different middle switch. The n outputs of a middle switch are connected to n egress switches; each output is connected to a different egress switch, that is, there is exactly one connection between each ingress-stage switch and each middle-stage switch. There is exactly one connection between each middle-stage switch and each egress-stage switch. The total number of switches is $2n+m$, and the total number of compute nodes is $N=n^2$. It can be seen that crossbar switches with different numbers of input and output ports are used at the ingress and egress stages.

Assume that $N-1$ connections have been built, which means that an ingress switch has an idle input and an egress switch has an idle output. It is desirable to build a connection from the idle input to the idle output without interfering with existing connections. This is called strictly nonblocking. Suppose that the idle input is at switch i in the ingress stage and the idle output is at switch j in the egress stage, that is, $n-1$ connections are built into switch i and $n-1$ connections are built into switch j. In the worst case, these $2(n-1)$ connections use $2(n-1)$ switches in the middle stage. If there is another switch in the middle stage, a connection from the idle input to the idle output can be built through that switch without interfering with existing connections. Therefore, the condition required to achieve strictly nonblocking is $m\ge 2(n-1)+1=2n-1$.

An $x\times y$ (x inputs and y outputs) crossbar has $xy$ crosspoints. Let $m=2n-1$. Then, there are $nm\times n+n^2\times m+mn\times n=3n^{2}m=3n^2(2n-1)$ crosspoints in a traditional unidirectional strictly nonblocking Clos network. There are $N=n^2$ inputs and $N=n^2$ outputs. Using a single $N\times N$ crossbar requires $N^2=n^4$ crosspoints. For $n=6$, the Clos network requires $3n^2(2n-1)=3\times 6^2\times 11=1188$ crosspoints, which is less than the $N^2=n^4=6^4=1296$ crosspoints in the single-crossbar implementation.

A folded version of a traditional unidirectional strictly nonblocking Clos network can be constructed by combining ingress and egress switches to form leaf switches. The middle switches become root switches. A leaf switch is an $(n+m)\times (m+n)$ crossbar. A root switch is still an $n\times n$ crossbar. It can be seen that the differently sized crossbar switches are used at the leaf and root stages.

A Clos network is rearrangeably nonblocking if and only if $m\ge n$. In such a case, an input of an ingress switch can be connected to an output of an egress switch using a middle switch. A folded version of a traditional unidirectional rearrangeably nonblocking Clos network can be constructed by combining ingress and egress switches.

Traditional unidirectional strictly nonblocking Clos networks use crossbars with different numbers of input and output ports. The root stage of the folded version of a traditional unidirectional strictly and rearrangeably nonblocking Clos network uses differently sized crossbar switches than the other stages.

2.2. K-Ary N-Tree Clos Networks

A k-ary n-tree Clos network can be created with k-ary n-fly butterfly networks. A k-ary n-fly butterfly network [11] has $N=k^n$ compute nodes. It has n stages. Each stage has $N/k$ switches, and each switch has k input ports and k output ports ($2k$ is the radix of the switch). A k-ary 1-fly butterfly network has a $k\times k$ switch. There are k input ports and k output ports. Each of $N=k^1=k$ compute nodes is connected to an input port and an output port. A k-ary 2-fly butterfly network has two stages: stage 0 and stage 1. Each stage has k switches. $N=k^n=k^2$ compute nodes are connected to the input ports of switches in stage 0 and the output ports of switches in stage 1. Each output port of a switch in stage 0 is connected to an input port of a different switch in stage 1. A k-ary n-fly butterfly network with $k=3$ and $n=3$ is given in Figure A1 in Appendix A.

A butterfly network minimizes the network diameter and reduces the network cost. However, there is a lack of path diversity because there is only one path between the source node and the destination node. A butterfly network is a blocking network. In addition, it cannot exploit the locality of traffic because all packets must traverse the diameter of the network [9].

Multistage rearrangeably nonblocking Clos networks are also called k-ary n-tree Clos networks. A k-ary n-tree Clos network can be created by combining two k-ary n-fly butterfly networks [11] back to back, where the two back stages are fused [9]. There are $2n-1$ stages. A three-ary three-tree Clos network, built from three-ary three-fly butterfly networks, is given in Figure A2 in Appendix A.

When $k=2$, a k-ary n-tree Clos network is also called a Beneš network [12]. There are $2\log \left(N\right)-1$ stages, where N is the number of compute nodes. Each stage contains $N/2$ switches, and each switch is a $2\times 2$ crossbar. For example, a three-stage Beneš network has four compute nodes, a five-stage Beneš network has eight compute nodes, and a seven-stage Beneš network has sixteen compute nodes.

In a k-ary n-tree Clos network, a packet needs to be routed first to an arbitrary middle stage switch, then to its ultimate destination. It is a rearrangeably nonblocking network. The cost and latency of a k-ary n-tree Clos network is nearly double that of a k-ary n-fly butterfly network with equal node capacity [9].

Because in a unidirectional k-ary n-tree Clos network, the nodes on the input side and the nodes on the output side in the same row are the same compute nodes and the ports of a switch are unidirectional, the unidirectional k-ary n-tree Clos network can be folded, and two unidirectional switches can be combined to build a bidirectional switch. Furthermore, the bidirectional links are used to connect switch ports. This is called a k-ary n-tree folded Clos network or a k-ary n-tree fat tree [10]. It has n stages and $k^n$ compute nodes.

A k-ary n-tree fat-tree network can be thought of as a bidirectional k-ary n-fly butterfly network with compute nodes connected to stage 0. In contrast to a unidirectional k-ary n-tree Clos network, a k-ary n-tree fat-tree network can exploit traffic locality because a packet needs to be routed only to the nearest common ancestor (NCA) of the source and destination, then to its ultimate destination. This means that packets may no longer need to be routed to the root switch, reducing the path they take.

The switches in a unidirectional k-ary n-tree Clos network can be redesigned so that they have bidirectional ports. Bidirectional links are also used, and the number of compute nodes is doubled (the left nodes and right nodes are distinct compute nodes). Such a network has $2n-1$ stages and $2k^n$ compute nodes. All the switches in a bidirectional k-ary n-tree Clos network have the same radix, which is $2k$.

When $n=3$ and k is even, a bidirectional ($k/2$)-ary n-tree Clos network is also called a k-ary fat-tree network [13]. It has three fixed stages (layers). The root stage is called the core (spine) layer. There are k pods below the core layer. Each pod contains two layers: an aggregation layer in the middle stage and an edge layer in the leaf stage.

2.3. Mirrored and Peer K-Ary N-Tree Networks

To implement nonblocking routing, the network must provide a high level of path diversity so that a packet can be routed to an arbitrary middle-stage switch in a Clos network or an arbitrary root switch in a fat-tree network, then to its ultimate destination. This approximately doubles the number of switches and links, resulting in a high hardware cost and a high level of packet latency [9]. Both mirrored k-ary n-tree (MiKANT) networks [14] and peer k-ary n-tree networks [15] focus on increasing network capacity and reducing hardware costs and packet latency.

A k-ary n-tree fat-tree network has n stages. If the number 0 represents the leaf stage, then the root stage is numbered $n-1$. A MiKANT network [14] consists of two k-ary n-tree fat-tree networks joined back to back, where the switches in stage $n-2$ of a fat tree serve as root switches for the other fat tree, that is, the MiKANT network has $2n-2$ stages. Fat tree 0 and fat tree 1 are notations used to distinguish between two fat trees. Compared to k-ary n-tree fat tree network, MiKANT doubles the number of compute nodes. Compared to a bidirectional k-ary n-tree Clos network, MiKANT uses fewer switches with equal node capacity. If the source and destination nodes belong to the same fat tree, MiKANT behaves like a k-ary n-tree fat-tree network. MiKANT reduces the path length and path diversity when the source and destination nodes belong to different fat trees.

In a k-ary n-tree fat-tree network, the root switch has a radix of k, and the other switches have a radix of $2k$. The $2k$-radix switches can be used in the root stage, and k compute nodes can be connected to each root switch. Therefore, the root switches are the same as the leaf switches. This is called a peer k-ary n-tree or a peer fat tree [15] because there is no difference between roots and leaves. The peer fat tree reduces both hardware cost and average distance and, meanwhile, provides nonblocking routing functionality for half of the source–destination node pairs. Note that a peer k-ary n-tree network is not a bidirectional k-ary n-fly butterfly network. A bidirectional k-ary n-fly butterfly network connects compute nodes to either stage 0 or stage $n-1$. However, a peer k-ary n-tree network connects compute nodes to both stage 0 and stage $n-1$. Therefore, the number of compute nodes in a peer k-ary n-tree network is twice that in a bidirectional k-ary n-fly butterfly network. Actually, a bidirectional k-ary n-fly butterfly network is the same as a k-ary n-tree fat-tree network.

2.4. Twisted-and-Folded Clos Networks

A two-stage twisted-and-folded Clos network using equally sized square crossbar switches was proposed in [16]. In the proposed network, there are multiple links between a leaf switch and a root switch. It has k switches in the leaf stage and m switches in the root stage. The condition required to achieve strict nonblocking is $m\ge 2\lfloor (n-1)/v\rfloor +1$, where n is the number of ports connected to compute nodes at a leaf switch and v is the number of links between a leaf switch and a root switch. It has $N=nk$ compute nodes in total. It uses $r\times r$ square crossbars, where $r=max\{kv,n+mv\}$ and the number of leaf switches (k) is determined so that $kv$ is as close as possible to $n+mv$. This keeps the number of unused ports as low as possible. Then, if $kv=n+mv$, there is no unused port; otherwise, $|n+mv-kv|$ ports are unused on every switch in either the leaf stage (if $kv>n+mv$) or root stage (if $kv<n+mv$).

Let us take a look at the following examples proposed in [16]. For $v=2$ and $n=9$, $m=2\lfloor (n-1)/v\rfloor +1=2\lfloor (9-1)/2\rfloor +1=9$ and $n+mv=9+9\times 2=27$. There are two possible values of k, as shown below.

• Let $k=13$; then, $kv=13\times 2=26$, which is smaller than $n+mv=27$. The network uses $k+m=22$ switches, and each switch is an $r\times r=(n+mv)\times (n+mv)=27\times 27$ crossbar. There is one ($27-26$) unused port in each root switch. The number of compute nodes is $N=nk=9\times 13=117$.
• Let $k=14$; then, $kv=14\times 2=28$, which is larger than $n+mv=27$. The network uses $k+m=23$ switches, and each switch is an $r\times r=\left(kv\right)\times \left(kv\right)=28\times 28$ crossbar. There is one ($28-27$) unused port in each leaf switch. The number of compute nodes is $N=nk=9\times 14=126$.

A flexible twisted-and-folded Clos network was proposed in [17]. It eliminates the strict condition of $m\ge 2\lfloor (n-1)/v\rfloor +1$. To reduce the blocking probability, a second group of switches is added to the root stage. If it cannot establish a connection through the root switch in group 1, it will try to use the root switch in group 2 (two-step model).

The number of groups can be extended to a general number (S in an S-step model), as presented in [18]. To use it as a building block, a multiple-plane twisted-and-folded Clos network can be constructed [19]. A plane-selection layer consisting of $1\times M$ and $M\times 1$ crossbars is inserted between the terminals and the input–output layer, where M is the number of planes. This multiple-plane network uses crossbar switches of various sizes. The twisted-and-folded Clos networks introduced in [16,17,18,19] have only two stages with $nk$ compute nodes.

3. Proposed Identical Nonblocking Folded Clos Networks

This section first presents Unidirectional Strictly NonBlocking Clos (USNBC) networks and Unidirectional Rearrangeably NonBlocking Clos (URNBC) networks. Based on these unidirectional networks, the construction methods of identical strictly nonblocking folded Clos (ISNBC) networks and identical rearrangeably nonblocking folded Clos (IRNBC) networks, as listed in Table 1, are presented. Note that the number of stages in unidirectional networks is odd, e.g., $2s-1=3,5,7$ for $s=2,3,4$, where s is the number of stages for corresponding identical folded networks. Here, only the cases of $s=2$, 3, and 4 are shown, but for $s>4$, USNBC, URNBC, ISNBC, and IRNBC networks can be constructed similarly.

Table 1. Proposed nonblocking Clos networks.

Strictly or Rearrangeably	Network	Stage, Unfolded or Folded
Strictly nonblocking	USNBC	3-stage, 5-stage, and 7-stage Clos networks
	ISNBC	2-stage, 3-stage, and 4-stage folded Clos networks
Rearrangeably nonblocking	URNBC	3-stage, 5-stage, and 7-stage Clos networks
	IRNBC	2-stage, 3-stage, and 4-stage folded Clos networks

3.1. Proposed Identical Strictly NonBlocking Folded Clos (ISNBC) Networks

As mentioned before, there are two drawbacks of traditional strictly nonblocking Clos networks. One is the use of $n\times m$ and $m\times n$ crossbars with unequal numbers of input and output ports. Another is that all connections pass through a fixed number of stages. A drawback of traditional nonblocking folded Clos networks is that the root stage uses differently sized crossbar switches than the other stages. These problems can be solved by using an ISNBC network consisting of equally sized square crossbars. This subsection presents USNBC networks and the corresponding ISNBC networks.

3.1.1. Two-Stage ISNBC Networks

To construct a two-stage ISNBC network, a three-stage USNBC network is constructed first. In a three-stage USNBC network, let n be the number of inputs per switch in the ingress stage, m be the number of switches in the middle stage, and r be the number of switches in the ingress and egress stages. Let $m=2n$ to ensure the strictly nonblocking property. Let $r=n+m=3n$ to ensure that the ISNBC network uses equally sized square crossbars.

In the ingress stage, there are r switches, and each switch is an $n\times m$ crossbar (n inputs and m outputs); in the middle stage, there are m switches, and each switch is an $r\times r$ crossbar; and in the egress stage, there are r switches, and each switch is an $m\times n$ crossbar. Each output of a switch in the ingress stage is connected to an input of a different switch in the middle stage. Since there are r switches in the ingress stage, the number of inputs of a switch in the middle stage must be r—one input for an output of the r switches in the ingress stage. Each output of a switch in the middle stage is connected to an input of a different switch in the egress stage. Because the number of outputs of a switch in the middle stage is also r, each output is connected to an input of r switches in the egress stage. Then, the number of compute nodes is $N=nr=n\times 3n=3n^2$.

In summary, to construct a three-stage USNBC network, m and r are determined as shown in Formula (1), where N is the number of compute nodes.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill m& =2n\hfill \\ \hfill r& =n+m=3n\cdots \cdots (\text{3-stage} \mathrm{USNBC})\hfill \\ \hfill N& =nr=3n^2\hfill \end{array}\right.\end{array}$$

(1)

To construct a folded version of a three-stage USNBC network, the corresponding position switches in the ingress and egress stages are merged and expanded so that there are $n+m$ inputs and $m+n$ outputs in the combined switch. Then, these switches have the same number of inputs and outputs, which is $n+m$. The switch in the middle stage has r inputs and r outputs with $r=n+m$. Therefore, all switches in the folded Clos network use equally sized $(n+m)\times (m+n)=3n\times 3n$ crossbars.

Figure 1a shows a three-stage USNBC network with $n=2$, $m=2n=4$, and $r=n+m=6$. It has $N=nr=3n^2=12$ compute nodes. A two-stage ISNBC network, the folded version of the three-stage USNBC network with $n=2$, $m=2n=4$, and $r=n+m=6$, is shown in Figure 1b. It uses equally sized $(n+m)\times (m+n)=6\times 6$ square crossbars for all switches. It can utilize shortcut connections to reduce communication path lengths. For example, if the source and destination nodes are connected to the same leaf switch, the communication does not need to go through the root switch.

Figure 1. Proposed strictly nonblocking Clos networks ($m=2n$ and $r=n+m$). (a) A 3-stage USNBC network with $n=2$, $m=2n=4$, and $r=n+m=6$. (b) A 2-stage ISNBC network composed of equally sized $(n+m)\times (m+n)=6\times 6$ square crossbars.

The root switch in Figure 1b and the middle switch in Figure 1a are the same — both are the $r\times r$ crossbar switches. However, the leaf switch in Figure 1b is not simply a combination of the $n\times m$ ingress switch and the $m\times n$ egress switch in Figure 1a. A leaf switch is a square $(n+m)\times (m+n)$ crossbar.

Referring to Figure 2, for $n=2$ and $m=2n=4$, Figure 2a shows an $n\times m=2\times 4$ crossbar and an $m\times n=4\times 2$ crossbar. The number of crosspoints is $n\times m+m\times n=4n^2=16$. There are no paths from inputs $x_0$ and $x_1$ to outputs $y_4$ and $y_5$. Similarly, there are no paths from inputs $x_2$, $x_3$, $x_4$, and $x_5$ to outputs $y_0$, $y_1$, $y_2$, and $y_3$. Figure 2b shows an $(n+m)\times (m+n)=6\times 6$ crossbar. The number of crosspoints is $(n+m)\times (m+n)=9n^2=36$. Any input ($x_i$) can be routed to any output ($y_j$) for $i,j=0,1,\dots ,5$. A crosspoint can be implemented using two two-to-one multiplexers, as shown in Figure 2c.

Figure 2. Merging and expanding an $n\times m$ crossbar switch and an $m\times n$ crossbar switch to a big square $(n+m)\times (m+n)$ crossbar switch for $n=2$ and $m=2n=4$. (a) An $n\times m$ crossbar switch and an $m\times n$ crossbar switch, each with 8 crosspoints. (b) A square $(n+m)\times (m+n)$ crossbar switch with 36 crosspoints. The red line shows the path of $x_0\to y_1$. (c) Crosspoint states and implementation using two 2-to-1 multiplexers.

Figure 2b is a $6\times 6$ crossbar switch. The input port ($x_i$) and output port ($y_i$) form a bidirectional port ($p_i$, where $i=0,1,\cdots ,5$). An example of a shortcut is shown in Figure 3, where $p_0$ and $p_1$ connect to compute nodes 1 and 2, respectively. Then, node 1 can send packets to node 2 only through leaf switch 1, thereby reducing the communication path length.

Figure 3. A shortcut in an ISNBC network. When a packet is sent from a source (node 1) to a destination (node 2), it does not need to go through the root switch.

Figure 4 shows another three-stage USNBC network with $n=3$, $m=2n=6$, and $r=n+m=9$. There are $r=9$ ingress switches, $r=9$ egress switches, and $m=6$ middle switches. It has $N=nr=27$ compute nodes.

Figure 4. A 3-stage USNBC network with $n=3$, $m=2n=6$, and $r=n+m=9$.

A two-stage ISNBC network, the folded version of the three-stage USNBC network with $n=3$, $m=2n=6$, and $r=n+m=9$, is shown in Figure 5. It uses equally sized $3n\times 3n=9\times 9$ square crossbars for all switches. It can utilize shortcut connections to reduce communication path lengths.

Figure 5. A 2-stage ISNBC network with $n=3$, $m=2n=6$, and $r=n+m=9$ composed of equally sized $(n+m)\times (m+n)=9\times 9$ square crossbar switches.

Table 2 and Table 3 show the differences between this work and existing original work. Table 2 compares the proposed three-stage USNBC network with a traditional three-stage unidirectional strictly nonblocking Clos network. Table 3 compares the proposed two-stage ISNBC network with a traditional two-stage strictly nonblocking folded Clos network. The $[x\times y]\times z$ expression in the switch column indicates that there are z switches, and each switch is an $x\times y$ crossbar. N is the total number of compute nodes in the network. The ISNBC network uses equally sized square crossbar switches in the leaf and root stages.

Table 2. Comparison of three-stage unidirectional strictly nonblocking Clos networks.

	m	Ingress Switch	Middle Switch	Egress Switch	N
Traditional	$2n-1$	$[n\times (2n-1\left)\right]\times n$	$[n\times n]\times (2n-1)$	$\left[\right(2n-1)\times n]\times n$	$n^2$
USNBC	$2n$	$[n\times 2n]\times 3n$	$[3n\times 3n]\times 2n$	$[2n\times n]\times 3n$	$3n^2$

Table 3. Comparison of two-stage strictly nonblocking folded Clos networks.

	m	Leaf Switch	Root Switch	N
Traditional	$2n-1$	$\left[\right(3n-1)\times (3n-1\left)\right]\times n$	$[n\times n]\times (2n-1)$	$n^2$
ISNBC	$2n$	$[3n\times 3n]\times 3n$	$[3n\times 3n]\times 2n$	$3n^2$

3.1.2. Three-Stage ISNBC Networks

To construct a three-stage ISNBC network, a five-stage USNBC network is constructed first. By using three-stage USNBC networks as building blocks, a five-stage USNBC network can be constructed. For $m=2n$, a three-stage USNBC network has $N=(n+m)n=3n^2$ compute nodes. As a building block, the compute nodes are removed so that the three-stage USNBC network has $(n+m)n=3n^2$ inputs and $(n+m)n=3n^2$ outputs.

The building blocks can be thought of as virtually $3n^2\times 3n^2$ crossbars. m such building blocks are arranged in the middle stage. Then, in total, there are $3n^2\times m=6n^3$ inputs and $3n^2\times m=6n^3$ outputs in the middle stage. Correspondingly, the same number of outputs in the ingress stage and the same number of inputs in the egress stage can be arranged. Let r be the number of switches in the ingress and egress stages; then, $rm$ must be equal to $6n^3$. Therefore, $r=6n^3/m=3n^2\times m/m=3n^2=(n+m)n$.

In summary, to construct a five-stage USNBC network, given n as the number of inputs per switch in the ingress stage, there are $r=(n+m)n=3n^2$ switches in the ingress stage, and each switch is an $n\times m$ crossbar with $m=2n$. The middle stage has m building blocks, and each building block is a three-stage USNBC network with compute nodes removed. The egress stage has $r=(n+m)n=3n^2$ switches, and each switch is an $m\times n$ crossbar with $m=2n$, as shown in Formula (2). The linking method is similar to the three-stage USNBC network; each output of a switch in the ingress stage is connected to an input of a different building block in the middle stage. Each output of a building block in the middle stage is connected to an input of a different switch in the egress stage. Because $r=3n^2$, the number of compute nodes is $N=nr=3n^3$.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill m& =2n\hfill \\ \hfill r& =(n+m)n=3n^2\cdots \cdots (\text{5-stage} \mathrm{USNBC})\hfill \\ \hfill N& =nr=3n^3\hfill \end{array}\right.\end{array}$$

(2)

Figure 6 shows a five-stage USNBC network with $n=2$, $m=2n=4$, and $r=3n^2=12$. It has $N=nr=3n^3=24$ compute nodes. There are $m=2n=4$ building blocks in the middle stage, and each building block is a three-stage USNBC network with compute nodes removed. The detailed network of a building block is shown at the bottom of the figure. It can be seen from the figure how the switches are linked together.

Figure 6. A five-stage USNBC network with $n=2$, $m=2n=4$, and $r=3n^2=12$. There are $m=2n=4$ building blocks (3-stage USNBC, Figure 1a) in the middle stage.

A three-stage ISNBC network, the folded version of the five-stage USNBC network with $n=2$, $m=2n=4$, and $r=3n^2=12$, is shown in Figure 7. It uses equally sized $(n+m)\times (m+n)=6\times 6$ square crossbars for all switches. Four building blocks are depicted in the two switch columns on the right, and each building block is a two-stage ISNBC network with compute nodes removed, as shown in Figure 1b. It can utilize shortcut connections to reduce communication path lengths.

Figure 7. A three-stage ISNBC network with $n=2$ composed of equally sized $(n+m)\times (m+n)=6\times 6$ square crossbars (folded version of Figure 6).

3.1.3. Four-Stage ISNBC Networks

To construct a four-stage ISNBC network, a seven-stage USNBC network is constructed first. Similarly, by using five-stage USNBC networks as building blocks, a seven-stage USNBC network can be constructed. For $m=2n$, five-stage building block has $3n^3$ inputs and $3n^3$ outputs. m such building blocks are arranged in the middle stage. Then, in total, there are $3n^3\times m=6n^4$ inputs and $3n^3\times m=6n^4$ outputs in the middle stage. Correspondingly, $r=3n^3$ switches in the ingress stage and $r=3n^3$ switches in the egress stage are arranged.

In summary, to construct a seven-stage USNBC network, given n, which is the number of inputs per switch in the ingress stage, there are $r=3n^3$ switches in the ingress stage, and each switch is an $n\times m$ crossbar with $m=2n$. There are m building blocks in the middle stage, and each building block is a five-stage USNBC network with compute nodes removed. There are $r=3n^3$ switches in the egress stage, and each switch is an $m\times n$ crossbar with $m=2n$, as shown in Formula (3). The linking method is similar to that in the five-stage USNBC network.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill m& =2n\hfill \\ \hfill r& =(n+m)n^2=3n^3\cdots \cdots (\text{7-stage} \mathrm{USNBC})\hfill \\ \hfill N& =nr=3n^4\hfill \end{array}\right.\end{array}$$

(3)

Figure 8 shows a seven-stage USNBC network with $n=2$, $m=2n=4$, and $r=3n^3=24$. It has $N=nr=3n^4=48$ compute nodes. There are $m=2n=4$ building blocks in the middle stage, and each building block is a five-stage USNBC network with compute nodes removed. The detailed networks of building blocks are shown at the bottom of the figure.

Figure 8. A 7-stage USNBC network with $n=2$, $m=2n=4$, and $r=3n^3=24$. There are $m=2n=4$ building blocks (5-stage USNBC, Figure 6) in the middle stage.

A four-stage ISNBC network, the folded version of the seven-stage USNBC network with $n=2$, $m=2n=4$, and $r=3n^3=24$, is shown in Figure 9. It uses the equally sized $(n+m)\times (m+n)=6\times 6$ square crossbars for all switches. Four three-stage ISNBC networks are shown in the three switch columns on the right. It can utilize shortcut connections to reduce communication path lengths.

Figure 9. A four-stage ISNBC network with $n=2$ composed of equally sized $(n+m)\times (m+n)=6\times 6$ square crossbars (folded version of Figure 8).

Table 4 lists the numbers of compute nodes and switches of two-, three-, and four-stage ISNBC networks. Let s be the number of stages. Then, the number of compute nodes is $N=3n^s$, and the number of switches is $(2^{s+1}-3)n^{s-1}$. The derivation of the formula is given in Theorem A1 in Appendix B. The crossbar size is listed in the right column.

Table 4. The numbers of compute nodes and switches in ISNBC networks.

	Number of Compute Nodes			Number of Switches
n	2-Stage	3-Stage	4-Stage	2-Stage	3-Stage	4-Stage	Crossbar
2	12	24	48	10	52	232	$6\times 6$
3	27	81	243	15	117	783	$9\times 9$
4	48	192	768	20	208	1856	$12\times 12$
5	75	375	1875	25	325	3625	$15\times 15$
6	108	648	3888	30	468	6264	$18\times 18$
7	147	1029	7203	35	637	9947	$21\times 21$
8	192	1536	12,288	40	832	14,848	$24\times 24$
9	243	2187	19,683	45	1053	21,141	$27\times 27$
10	300	3000	30,000	50	1300	29,000	$30\times 30$

3.2. Proposed Identical Rearrangeably NonBlocking Folded Clos (IRNBC) Networks

This subsection presents a URNBC network and the corresponding IRNBC network composed of square crossbars of the same size.

3.2.1. Two-Stage IRNBC Networks

To construct a two-stage IRNBC network, a three-stage URNBC network is constructed first. In a three-stage URNBC network, let n be the number of inputs per switch in the ingress stage, m be the number of switches in the middle stage, and r be the number of switches in the ingress and egress stages. Let $m=n$ to ensure the rearrangeably nonblocking property. Let $r=n+m=2n$ to ensure that the IRNBC network uses equally sized square crossbars.

In the ingress stage, there are r switches, and each switch is an $n\times m$ crossbar (n inputs and m outputs); in the middle stage, there are m switches, and each switch is an $r\times r$ crossbar; and in the egress stage, there are r switches, and each switch is an $m\times n$ crossbar. Each output of a switch in the ingress stage is connected to an input of a different switch in the middle stage. Since there are r switches in the ingress stage, the number of inputs of a switch in the middle stage must be r — one input for an output of the r switches in the ingress stage. Each output of a switch in the middle stage is connected to an input of a different switch in the egress stage. Because the number of outputs of a switch in the middle stage is also r, each output is connected to an input of r switches in the egress stage. Then, the number of compute nodes is $N=nr=2n^2$. In summary, to construct a three-stage URNBC network, m and r are determined as shown in Formula (4).

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill m& =n\hfill \\ \hfill r& =n+m=2n\cdots \cdots (\text{3-stage} \mathrm{URNBC})\hfill \\ \hfill N& =nr=2n^2\hfill \end{array}\right.\end{array}$$

(4)

To construct a folded version of a three-stage URNBC network, the corresponding position switches in the ingress and egress stages are merged and expanded so that there are $n+m$ inputs and $m+n$ outputs in the combined switch. Then, these switches have the same number of inputs and outputs, which is $n+m$. The switch in the middle stage has r inputs and r outputs with $r=n+m$. Therefore, all switches in the folded Clos network use equally sized $(n+m)\times (m+n)=2n\times 2n$ crossbars.

Figure 10a shows a three-stage URNBC network with $n=2$, $m=n=2$, and $r=n+m=4$. It has $N=nr=8$ compute nodes. A two-stage IRNBC network, the folded version of the three-stage URNBC network with $n=2$, $m=n=2$, and $r=n+m=4$, is shown in Figure 10b. It uses equally sized $2n\times 2n=4\times 4$ square crossbars for all switches. It can utilize shortcut connections to reduce communication path lengths. For example, if the source and destination nodes are connected to the same leaf switch, the communication does not need to go through the root switch.

Figure 10. Proposed rearrangeably nonblocking Clos networks ($m=n$ and $r=n+m$). (a) A 3-stage URNBC network with $n=2$, $m=n=2$, and $r=n+m=4$. (b) A 2-stage IRNBC network composed of equally sized $(n+m)\times (m+n)=4\times 4$ square crossbars.

Figure 11 shows the blocking and rearrangements in a rearrangeably nonblocking Clos network. Referring to Figure 11a, connection $1\to 4$ cannot be built because source node 2 (connected to the same switch as node 1) and destination node 3 (connected to the same switch as node 4) use different middle switches for their connections ($2\to 5$ and $7\to 3$). Figure 11b shows the case of the folded version, where a bidirectional link consists of two oppositely oriented unidirectional links. Figure 11c–f show that the $1\to 4$ connection can be constructed after the rearrangement of existing connections.

Figure 11. Blocking and rearrangements in a rearrangeably nonblocking Clos network ($m=n=2$ and $r=n+m=4$). (a) Two connections ($2\to 5$ and $7\to 3$) were built. The $1\to 4$ connection cannot be built. (b) The case of the folded version of (a). Note that a bidirectional link consists of two oppositely oriented unidirectional links. (c) Two connections ($2\to 5$ and $7\to 3$) were built. The $1\to 4$ connection can also be built by the rearrangements of (a). (d) The case of the folded version of (c). (e) Two connections ($2\to 5$ and $7\to 3$) were built. The $1\to 4$ connection can also be built by the rearrangements of (a). (f) The case of the folded version of (e).

Figure 12a shows a three-stage URNBC network with $n=3$, $m=n=3$, and $r=n+m=6$. It has $N=nr=18$ compute nodes. A two-stage IRNBC network, the folded version of the three-stage URNBC network with $n=3$, $m=n=3$, and $r=n+m=6$, is shown in Figure 12b. It uses the equally sized $2n\times 2n=6\times 6$ square crossbars.

Figure 12. Proposed rearrangeably nonblocking Clos networks ($m=n$ and $r=n+m$). (a) A 3-stage URNBC network with $n=3$, $m=n=3$, and $r=n+m=6$. (b) A 2-stage IRNBC network composed of equally sized $2n\times 2n=6\times 6$ square crossbars.

Table 5 and Table 6 show the differences between this work and existing original work. Table 5 compares the proposed three-stage URNBC network with a traditional three-stage unidirectional rearrangeably nonblocking Clos network. Table 6 compares the proposed two-stage IRNBC network with a traditional two-stage rearrangeably nonblocking folded Clos network. The $[x\times y]\times z$ expression in the switch column indicates that there are z switches, and each switch is an $x\times y$ crossbar. N is the total number of compute nodes in the network. The IRNBC network uses equally sized square crossbar switches in the leaf and root stages.

Table 5. Comparison of 3-stage unidirectional rearrangeably nonblocking Clos networks.

	m	Ingress Switch	Middle Switch	Egress Switch	N
Traditional	n	$[n\times n]\times n$	$[n\times n]\times n$	$[n\times n]\times n$	$n^2$
URNBC	n	$[n\times n]\times 2n$	$[2n\times 2n]\times n$	$[n\times n]\times 2n$	$2n^2$

Table 6. Comparison of 2-stage rearrangeably nonblocking folded Clos networks.

	m	Leaf Switch	Root Switch	N
Traditional	n	$[2n\times 2n]\times n$	$[n\times n]\times n$	$n^2$
IRNBC	n	$[2n\times 2n]\times 2n$	$[2n\times 2n]\times n$	$2n^2$

3.2.2. Three-Stage IRNBC Networks

To construct a three-stage IRNBC network, a five-stage URNBC network is constructed first. By using three-stage URNBC networks as building blocks, a five-stage URNBC network can be constructed. For $m=n$, a three-stage URNBC network has $N=(n+m)n=2n^2$ compute nodes. As building blocks, the compute nodes are removed so that the three-stage URNBC network has $(n+m)n=2n^2$ inputs and $(n+m)n=2n^2$ outputs. m such building blocks are arranged in the middle stage. Then, in total, there are $2n^2\times m=2n^3$ inputs and $2n^2\times m=2n^3$ outputs in the middle stage. Correspondingly, the same number of outputs in the ingress stage and the same number of inputs in the egress stage can be arranged. Let r be the number of switches in the ingress and egress stages; then, $rm$ must be equal to $2n^3$. Therefore, $r=2n^3/m=2n^2\times m/m=2n^2=(n+m)n$.

In summary, to construct a five-stage URNBC network, given n, which is the number of inputs per switch in the ingress stage, there are $r=2n^2$ switches in the ingress stage, and each switch is an $n\times m$ crossbar with $m=n$. There are m building blocks in the middle stage, and each building block is a three-stage URNBC network with compute nodes removed. There are $r=2n^2$ switches in the egress stage, and each switch is an $m\times n$ crossbar with $m=n$, as shown in Formula (5). The linking method is similar to the three-stage URNBC network; each output of a switch in the ingress stage is connected to an input of a different building block in the middle stage. Each output of a building block in the middle stage is connected to an input of a different switch in the egress stage.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill m& =n\hfill \\ \hfill r& =(n+m)n=2n^2\cdots \cdots (\text{5-stage} \mathrm{URNBC})\hfill \\ \hfill N& =nr=2n^3\hfill \end{array}\right.\end{array}$$

(5)

Figure 13 shows a five-stage URNBC network with $n=2$, $m=n=2$, and $r=2n^2=8$. It has $N=nr=2n^3=16$ compute nodes. There are $m=n=2$ building blocks in the middle stage, and each building block is a three-stage URNBC network with compute nodes removed. The detailed network of a building block is shown at the bottom of the figure.

Figure 13. A 5-stage URNBC network with $n=2$, $m=n=2$, and $r=2n^2=8$. There are $m=n=2$ building blocks (3-stage URNBC, Figure 10a) in the middle stage.

A three-stage IRNBC network, the folded version of the five-stage URNBC network with $n=2$, $m=n=2$, and $r=2n^2=8$, is shown in Figure 14. It uses equally sized $2n\times 2n=4\times 4$ square crossbars for all switches. Two two-stage IRNBC networks (Figure 10b) are shown in the two switch columns on the right.

Figure 14. A 3-stage IRNBC network with $n=2$ composed of equally sized $2n\times 2n=4\times 4$ square crossbars (folded version of Figure 13).

3.2.3. Four-Stage IRNBC Networks

To construct a four-stage IRNBC network, a seven-stage URNBC network is constructed first. Similarly, by using five-stage URNBC networks as building blocks, a seven-stage URNBC network can be constructed. In summary, to construct a seven-stage URNBC network, m and r are determined as shown in Formula (6). The linking method is similar to the five-stage URNBC network.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill m& =n\hfill \\ \hfill r& =(n+m)n^2=2n^3\cdots \cdots (\text{7-stage} \mathrm{URNBC})\hfill \\ \hfill N& =nr=2n^4\hfill \end{array}\right.\end{array}$$

(6)

Figure 15 shows a seven-stage URNBC network with $n=2$, $m=n=2$, and $r=2n^3=16$. It has $N=nr=2n^4=32$ compute nodes. There are $m=n=2$ building blocks in the middle stage, and each building block is a five-stage URNBC network with compute nodes removed. The detailed networks of building blocks are shown at the bottom of the figure.

Figure 15. A 7-stage URNBC network with $n=2$, $m=n=2$, and $r=2n^3=16$. There are $m=n=2$ building blocks (5-stage URNBC, Figure 13) in the middle stage.

A four-stage IRNBC network, the folded version of the seven-stage URNBC network with $n=2$, $m=n=2$, and $r=2n^3=16$, is shown in Figure 16. It uses equally sized $2n\times 2n=4\times 4$ square crossbars for all switches. Two three-stage IRNBC networks (Figure 14) are shown in the three switch columns on the right.

Figure 16. A 4-stage IRNBC network with $n=2$ composed of equally sized $2n\times 2n=4\times 4$ square crossbars (folded version of Figure 15).

Table 7 lists the numbers of compute nodes and switches of two-, three-, and four-stage IRNBC networks. Let s be the number of stages. Then, the number of compute nodes is $N=2n^s$, and the number of switches is $(2s-1)n^{s-1}$. The derivation of the formula is given in Theorem A2 in Appendix B. The crossbar size is listed in the right column.

Table 7. The numbers of compute nodes and switches in the IRNBC networks.

	Number of Compute Nodes			Number of Switches
n	2-Stage	3-Stage	4-Stage	2-Stage	3-Stage	4-Stage	Crossbar
2	8	16	32	6	20	56	$4\times 4$
3	18	54	162	9	45	189	$6\times 6$
4	32	128	512	12	80	448	$8\times 8$
5	50	250	1250	15	125	875	$10\times 10$
6	72	432	2592	18	180	1512	$12\times 12$
7	98	686	4802	21	245	2401	$14\times 14$
8	128	1024	8192	24	320	3584	$16\times 16$
9	162	1458	13,122	27	405	5103	$18\times 18$
10	200	2000	20,000	30	500	7000	$20\times 20$
11	242	2662	29,282	33	605	9317	$22\times 22$
12	288	3456	41,472	36	720	12,096	$24\times 24$
13	338	4394	57,122	39	845	15,379	$26\times 26$
14	392	5488	76,832	42	980	19,208	$28\times 28$
15	450	6750	101,250	45	1125	23,625	$30\times 30$

4. Cost Evaluations

This section evaluates the hardware cost for USNBC, ISNBC, URNBC, and IRNBC networks from the perspective of switch crosspoints and compares them to the corresponding traditional Clos networks. Here, we evaluate the costs of the 24 networks listed in Table 8.

Table 8. Nonblocking Clos networks for cost evaluations.

Strictly or Rearrangeably	Unidirectional or Bidirectional	Network	Stage, Unfolded or Folded
Strictly nonblocking	Unidirectional	USNBC	3-, 5-, and 7-stage Clos networks
		Traditional	3-, 5-, and 7-stage Clos networks
	Bidirectional	ISNBC	2-, 3-, and 4-stage folded Clos networks
		Traditional	2-, 3-, and 4-stage folded Clos networks
Rearrangeably nonblocking	Unidirectional	URNBC	3-, 5-, and 7-stage Clos networks
		Traditional	3-, 5-, and 7-stage Clos networks
	Bidirectional	IRNBC	2-, 3-, and 4-stage folded Clos networks
		Traditional	2-, 3-, and 4-stage folded Clos networks

4.1. Cost Evaluations of Strictly Nonblocking Clos Networks

This subsection investigates the crosspoint ratios relative to a single crossbar for unidirectional strictly nonblocking Clos (USNBC) networks and identical strictly nonblocking folded Clos (ISNBC) networks. These ratios are compared to those of the corresponding traditional Clos networks.

4.1.1. Cost Evaluations of USNBC Networks

An $n\times m$ crossbar (n inputs and m outputs) has $nm$ crosspoints. In the proposed strictly nonblocking Clos networks, $m=2n$. Referring to Figure 1a, in a three-stage USNBC network, there are $r=n+m=3n$ switches in the ingress stage, and each switch is an $n\times m=n\times 2n$ crossbar. There are m switches in the middle stage, and each switch is an $r\times r=3n\times 3n$ crossbar. There are $r=m+n=3n$ switches in the egress stage, and each switch is an $m\times n=2n\times n$ crossbar. The number of total crosspoints is $nm\times r+r^2\times m+mn\times r=2n^2\times 3n+9n^2\times 2n+2n^2\times 3n=30n^3$. There are $N=nr=3n^2$ inputs and $N=3n^2$ outputs. A single $N\times N$ crossbar requires $N\times N=9n^4$ crosspoints. The crosspoint ratio of the three-stage USNBC network relative to a single crossbar is $30n^3/\left(9n^4\right)=10/\left(3n\right)$, which is less than 1 if $n\ge 4$. For example, when $n=4$, the three-stage USNBC network requires $30n^3=1920$ crosspoints, which is less than $9n^4=2304$ crosspoints in the single crossbar’s implementation. In contrast, a traditional strictly nonblocking Clos network requires $n\ge 6$, as mentioned in Section 2.

Referring to Figure 6, in the five-stage case, there are $2n^2\times 3n^2$ crosspoints in the ingress stage, there are $30n^3\times 2n$ crosspoints in the middle stage, and there are $2n^2\times 3n^2$ crosspoints in the egress stage, where $30n^3$ is the number of crosspoints in a three-stage USNBC network, as derived above. The total number of the crosspoints is $6n^4+60n^4+6n^4=72n^4$. There are $N=nr=3n^3$ inputs and $3n^3$ outputs. A single $N\times N$ crossbar requires $9n^6$ crosspoints. The crosspoint ratio of the five-stage USNBC network relative to a single crossbar is $72n^4/\left(9n^6\right)=24/\left(3n^2\right)$, which is less than 1 if $n\ge 3$.

Referring to Figure 8, in the seven-stage case, there are $2n^2\times 3n^3$ crosspoints in the ingress stage, there are $72n^4\times 2n$ crosspoints in the middle stage, and there are $2n^2\times 3n^3$ crosspoints in the egress stage, where $72n^4$ is the number of crosspoints in a five-stage USNBC network, as derived above. The total number of the crosspoints is $6n^5+144n^5+6n^5=156n^5$. There are $N=nr=3n^4$ inputs and $N=3n^4$ outputs. A single $N\times N$ crossbar requires $9n^8$ crosspoints. The crosspoint ratio of the seven-stage USNBC network relative to a single crossbar is $156n^5/\left(9n^8\right)=52/\left(3n^3\right)$, which is less than 1 if $n\ge 3$.

The number of crosspoints for a traditional unidirectional strictly nonblocking Clos network [1] is examined below. A three-stage traditional unidirectional strictly nonblocking Clos network has n switches in the ingress stage, $m=2n-1$ switches in the middle stage, and n switches in the egress stage. An ingress-stage switch is an $n\times m$ crossbar, a middle-stage switch is an $n\times n$ crossbar, and an egress-stage switch is an $m\times n$ crossbar. Then, the total number of crosspoints is $nm\times n+n^2\times m+mn\times n=3n^{2}m=3n^2(2n-1)$. The total number of compute nodes is $N=n^2$. A single $N\times N$ crossbar requires $n^4$ crosspoints. The crosspoint ratio of a three-stage traditional unidirectional strictly nonblocking Clos network relative to a single crossbar is $3n^2(2n-1)/n^4=3(2n-1)/n^2$. To guarantee that the ratio is less than 1, $n\ge 6$ is needed.

Consider the five-stage case. There are $nm\times n^2$ crosspoints in the ingress stage, there are $3n^2(2n-1)\times m$ crosspoints in the middle stage, and there are $mn\times n^2$ crosspoints in the egress stage, where $3n^2(2n-1)$ is the number of crosspoints in a three-stage traditional unidirectional strictly nonblocking Clos network, as derived above. The total number of crosspoints is $nm\times n^2+3n^2(2n-1)\times m+mn\times n^2=n^3(2n-1)+3n^2(2n-1)(2n-1)+n^3(2n-1)=n^2(8n-3)$ $(2n-1)$. The total number of compute nodes is $N=n^3$. A single $N\times N$ crossbar requires $n^6$ crosspoints. The crosspoint ratio of a five-stage traditional unidirectional strictly nonblocking Clos network relative to a single crossbar is $n^2(8n-3)(2n-1)/n^6=(8n-3)(2n-1)/n^4$. To guarantee that the ratio is less than 1, $n\ge 4$ is needed.

Consider the seven-stage case. There are $nm\times n^3$ crosspoints in the ingress stage, there are $n^2(8n-3)(2n-1)\times m$ crosspoints in the middle stage, and there are $mn\times n^3$ crosspoints in the egress stage, where $n^2(8n-3)(2n-1)$ is the number of crosspoints in a five-stage traditional unidirectional strictly nonblocking Clos network, as derived above. The total number of crosspoints is $nm\times n^3+n^2(8n-3)(2n-1)\times m+mn\times n^3=n^4(2n-1)+n^2(8n-3)(2n-1)(2n-1)+n^4(2n-1)=$ $n^2(18n^2-14n+3)(2n-1)$. The total number of compute nodes is $N=n^4$. A single $N\times N$ crossbar requires $n^8$ crosspoints. The crosspoint ratio of a seven-stage traditional unidirectional strictly nonblocking Clos network relative to a single crossbar is $n^2(18n^2-14n+3)(2n-1)/n^8=(18n^2-14n+3)(2n-1)/n^6$. To guarantee that the ratio is less than 1, $n\ge 4$ is needed.

Table 9 summarizes the crosspoint ratio relative to a single crossbar for traditional unidirectional strictly nonblocking Clos networks and USNBC networks.

Table 9. Crosspoint ratio for unidirectional strictly nonblocking Clos networks.

Network	3-Stage	5-Stage	7-Stage
Traditional	$3(2n-1)/n^2$	$(8n-3)(2n-1)/n^4$	$(18n^2-14n+3)(2n-1)/n^6$
USNBC	$10/\left(3n\right)$	$24/\left(3n^2\right)$	$52/\left(3n^3\right)$

Figure 17 plots the crosspoint ratio relative to a single crossbar for unidirectional strictly nonblocking Clos networks, showing that USNBC networks have a lower crosspoint cost than traditional strictly nonblocking Clos networks.

Figure 17. Crosspoint ratio relative to a single crossbar in unidirectional strictly nonblocking Clos networks.

4.1.2. Cost Evaluations of ISNBC Networks

The number of crosspoints for an ISNBC network that uses equally sized square crossbars of $(n+m)\times (m+n)=3n\times 3n$ for $m=2n$ is examined below. Referring to Figure 1b, an ISNBC network based on the three-stage USNBC network has two stages. There are $r=n+m=3n$ leaf switches and m root switches. The total number of switches is $r+m=5n$, and each switch is a square $(n+m)\times (m+n)=3n\times 3n$ crossbar. Then, the total number of crosspoints is $3n\times 3n\times 5n=45n^3$. The total number of compute nodes is $N=nr=3n^2$.

A single $N\times N$ crossbar requires $9n^4$ crosspoints. The crosspoint ratio of a two-stage ISNBC network relative to a single crossbar is $45n^3/\left(9n^4\right)=5/n$. To guarantee that the ratio is less than 1, $n\ge 6$ is needed.

Consider a three-stage ISNBC network. Referring to Figure 7, there are $r=(n+m)n=3n^2$ switches in the leaf stage; there are m building blocks, and each building block is a two-stage ISNBC network whose number of switches is $5n$, as derived above. The total number of switches is $3n^2+5n\times 2n=13n^2$, and each switch is a square $(n+m)\times (m+n)=3n\times 3n$ crossbar. Then, the total number of crosspoints is $3n\times 3n\times 13n^2=117n^4$. The total number of compute nodes is $N=nr=3n^3$.

A single $N\times N$ crossbar requires $9n^6$ crosspoints. The crosspoint ratio of a three-stage ISNBC network relative to a single crossbar is $117n^4/\left(9n^6\right)=13/n^2$. To guarantee that the ratio is less than 1, $n\ge 4$ is needed.

Consider a four-stage ISNBC network. Referring to Figure 9, there are $r=(n+m)n^2=3n^3$ switches in the leaf stage; there are m building blocks, and each building block is a three-stage ISNBC network whose number of switches is $13n^2$, as derived above. The total number of switches is $3n^3+13n^2\times 2n=29n^3$, and each switch is a square $(n+m)\times (m+n)=3n\times 3n$ crossbar. Then, the total number of crosspoints is $3n\times 3n\times 29n^3=261n^5$. The total number of compute nodes is $N=nr=3n^4$.

A single $N\times N$ crossbar requires $9n^8$ crosspoints. The crosspoint ratio of a four-stage ISNBC network relative to a single crossbar is $261n^5/\left(9n^8\right)=29/n^3$. To guarantee that the ratio is less than 1, $n\ge 4$ is needed.

Table 10 lists the crosspoints of ISNBC networks. The “Crossbar” column shows the number of crosspoints in a single crossbar. The “ISNBC” column shows the number of crosspoints in an ISNBC network. The number of crosspoints of the ISNBC network is better (smaller) than that of a single crossbar when $n\ge 6$ for the two-stage ISNBC network and $n\ge 4$ for the three- and four-stage ISNBC networks.

Table 10. The numbers of crosspoints in ISNBC networks.

	2-Stage			3-Stage			4-Stage
n	Node	Crossbar	ISNBC	Node	Crossbar	ISNBC	Node	Crossbar	ISNBC
2	12	144	360	24	576	1872	48	2304	8352
3	27	729	1215	81	6561	9477	243	59,049	63,423
4	48	2304	2880	192	36,864	29,952	768	589,824	267,264
5	75	5625	5625	375	140,625	73,125	1875	3,515,625	815,625
6	108	11,664	9720	648	419,904	151,632	3888	15,116,544	2,029,536
7	147	21,609	15,435	1029	1,058,841	280,917	7203	51,883,209	4,386,627
8	192	36,864	23,040	1536	2,359,296	479,232	12,288	150,994,944	8,552,448
9	243	59,049	32,805	2187	4,782,969	767,637	19,683	387,420,489	15,411,789
10	300	90,000	45,000	3000	9,000,000	1,170,000	30,000	900,000,000	26,100,000

The number of crosspoints for a traditional strictly nonblocking folded Clos network that uses crossbars of different sizes is examined below. A two-stage traditional strictly nonblocking folded Clos network has n switches in the leaf stage and $m=2n-1$ switches in the root stage. A leaf switch is an $(n+m)\times (m+n)=(3n-1)\times (3n-1)$ crossbar. A root switch is an $n\times n$ crossbar. Then, the total number of crosspoints is ${(3n-1)}^2\times n+n^2\times m=n(11n^2-7n+1)$. The total number of compute nodes is $N=n^2$.

A single $N\times N$ crossbar requires $n^4$ crosspoints. The crosspoint ratio of a two-stage traditional strictly nonblocking folded Clos network relative to a single crossbar is $n(11n^2-7n+1)/n^4=(11n^2-7n+1)/n^3$. To guarantee that the ratio is less than 1, $n\ge 11$ is needed.

Consider the three-stage case. There are $n^2$ switches in the leaf stage, and each switch is an $(n+m)\times (m+n)=(3n-1)\times (3n-1)$ crossbar. There are $m=2n-1$ building blocks, and each building block has $n(11n^2-7n+1)$ crosspoints, as derived above. Then, the total number of crosspoints is ${(3n-1)}^2\times n^2+n(11n^2-7n+1)\times (2n-1)=n(31n^3-31n^2+10n-1)$. The total number of compute nodes is $N=n^3$. A single $N\times N$ crossbar requires $n^6$ crosspoints. The crosspoint ratio of a three-stage traditional strictly nonblocking folded Clos network relative to a single crossbar is $n(31n^3-31n^2+10n-1)/n^6=(31n^3-31n^2+10n-1)/n^5$. To guarantee that the ratio is less than 1, $n\ge 6$ is needed.

Consider the four-stage case. There are $n^3$ switches in the leaf stage, and each switch is an $(n+m)\times (m+n)=(3n-1)\times (3n-1)$ crossbar. There are $m=2n-1$ building blocks, and each building block has $n(31n^3-31n^2+10n-1)$ crosspoints, as derived above. Then, the total number of crosspoints is ${(3n-1)}^2\times n^3+n(31n^3-31n^2+10n-1)\times m=n(71n^4-99n^3+52n^2-12n+1)$. The total number of compute nodes is $N=n^4$. A single $N\times N$ crossbar requires $n^8$ crosspoints. The crosspoint ratio of a four-stage traditional strictly nonblocking folded Clos network relative to a single crossbar is $n(71n^4-99n^3+52n^2-12n+1)/n^8=(71n^4-99n^3+52n^2-12n+1)/n^7$. To guarantee that the ratio is less than 1, $n\ge 4$ is needed.

Table 11 summarizes the crosspoint ratio relative to a single crossbar for traditional strictly nonblocking folded Clos networks and ISNBC networks. The general formula for calculating the ISNBC crosspoint ratio relative to a single crossbar is $(2^{s+1}-3)/n^{s-1}$, where s is the number of stages and the number of compute nodes is $N=3n^s$.

Table 11. Crosspoint ratio for strictly nonblocking folded Clos networks.

Network	2-Stage	3-Stage	4-Stage
Traditional	$(11n^2-7n+1)/n^3$	$(31n^3-31n^2+10n-1)/n^5$	$(71n^4-99n^3+52n^2-12n+1)/n^7$
ISNBC	$5/n$	$13/n^2$	$29/n^3$

Table 12 lists the crosspoint ratios relative to a single crossbar for strictly nonblocking folded Clos networks. The ratios are calculated based on the formulas in Table 11.

Table 12. Crosspoint ratio relative to a single crossbar in strictly nonblocking folded Clos networks.

	ISNBC			Traditional
n	2-Stage	3-Stage	4-Stage	2-Stage	3-Stage	4-Stage
2	2.500000	3.250000	3.625000	3.875000	4.468750	4.132812
3	1.666667	1.444444	1.074074	2.925926	2.415638	1.605396
4	1.250000	0.812500	0.453125	2.328125	1.491211	0.770569
5	1.000000	0.520000	0.232000	1.928000	1.007680	0.425485
6	0.833333	0.361111	0.134259	1.643519	0.725180	0.258748
7	0.714286	0.265306	0.084548	1.431487	0.546379	0.168757
8	0.625000	0.203125	0.056641	1.267578	0.426239	0.116044
9	0.555556	0.160494	0.039781	1.137174	0.341699	0.083163
10	0.500000	0.130000	0.029000	1.031000	0.279990	0.061608

Figure 18 plots the crosspoint ratio relative to a single crossbar for strictly nonblocking folded Clos networks, showing that ISNBC networks have a lower crosspoint cost than traditional strictly nonblocking folded Clos networks. Also note that ISNBC networks use equally sized square crossbars for all switches in the network.

Figure 18. Crosspoint ratio relative to a single crossbar in strictly nonblocking folded Clos networks.

Figure 19 plots the crosspoint ratio relative to a single crossbar for strictly nonblocking folded Clos networks, showing that ISNBC networks have a lower crosspoint cost than the strictly nonblocking folded Clos network proposed in [16]. The network proposed in [16] is labeled “Redesign” in the figure, only supports two stages, and uses multiple links ($v=2,3,\cdots$) between a leaf switch and a root switch. If the network is constructed using equally sized crossbar switches, there will be unused ports on the leaf switch, the root switch, or both the leaf and root switches.

Figure 19. Crosspoint ratio relative to a single crossbar in strictly nonblocking folded Clos networks.

4.2. Cost Evaluations of Rearrangeably Nonblocking Clos Networks

This subsection investigates the crosspoint ratios relative to a single crossbar for unidirectional rearrangeably nonblocking Clos (URNBC) networks and identical rearrangeably nonblocking folded Clos (IRNBC) networks. These ratios are compared to the corresponding traditional Clos networks. It is unfair to compare a rearrangeably nonblocking Clos network to a single crossbar, since a single crossbar is a strictly nonblocking network. The reason the ratios are presented here is to make it easier to see the difference between the proposed network and a traditional network.

4.2.1. Cost Evaluations of URNBC Networks

As described in the previous section, in URNBC networks, $m=n$ and $r=n+m=2n$. Referring to Figure 10a, in a three-stage URNBC network, there are $r=n+m=2n$ switches in the ingress stage, and each switch is an $n\times m$ crossbar; there are m switches in the middle stage, and each switch is an $r\times r$ crossbar; and there are $r=n+m=2n$ switches in the egress stage, and each switch is an $m\times n$ crossbar. The total number of crosspoints is $nm\times r+r^2\times m+mn\times r=2n^3+4n^3+2n^3=8n^3$. There are $N=nr=2n^2$ compute nodes. A single $N\times N$ crossbar requires $N\times N=4n^4$ crosspoints. The crosspoint ratio of a three-stage URNBC network relative to a single crossbar is $8n^3/\left(4n^4\right)=2/n$.

Referring to Figure 13, in a five-stage URNBC network, there are $r=(n+m)n=2n^2$ switches in the ingress stage, and each switch is an $n\times m$ crossbar; there are m three-stage URNBC networks, and each URNBC network has $8n^3$ crosspoints, as derived above; and there are $r=(n+m)n=2n^2$ switches in the egress stage, and each switch is an $m\times n$ crossbar. The total number of crosspoints is $nm\times 2n^2+8n^3\times m+mn\times 2n^2=2n^4+8n^4+2n^4=12n^4$. There are $N=nr=2n^3$ compute nodes. A single $N\times N$ crossbar requires $4n^6$ crosspoints. The crosspoint ratio of a five-stage URNBC network relative to a single crossbar is $12n^4/\left(4n^6\right)=3/n^2$.

Referring to Figure 15, in a seven-stage URNBC network, there are $r=(n+m)n^2=2n^3$ switches in the ingress stage, and each switch is an $n\times m$ crossbar; there are m five-stage URNBC networks, and each URNBC network has $12n^4$ crosspoints, as derived above; and there are $r=(n+m)n^2=2n^3$ switches in the egress stage, and each switch is an $m\times n$ crossbar. The total number of crosspoints is $nm\times 2n^3+12n^4\times m+mn\times 2n^3=2n^5+12n^5+2n^5=16n^5$. There are $N=nr=2n^4$ compute nodes. A single $N\times N$ crossbar requires $4n^8$ crosspoints. The crosspoint ratio of a seven-stage URNBC network relative to a single crossbar is $16n^5/\left(4n^8\right)=4/n^3$.

The number of crosspoints for a traditional unidirectional rearrangeably nonblocking Clos network is examined below. In a three-stage traditional unidirectional rearrangeably nonblocking Clos network, $m=n$ and $N=n^2$. The total number of crosspoints is $nm\times n+n^2\times m+mn\times n=n^3+n^3+n^3=3n^3$. A single $N\times N$ crossbar requires $n^4$ crosspoints. Therefore, the crosspoint ratio of a three-stage traditional unidirectional rearrangeably nonblocking Clos network relative to a single crossbar is $3n^3/n^4=3/n$.

In the five-stage case, $m=n$ and $N=n^3$. The total number of crosspoints is $nm\times n^2+3n^3\times m+mn\times n^2=n^4+3n^4+n^4=5n^4$, where $3n^3$ is the number of crosspoints in a three-stage traditional unidirectional rearrangeably nonblocking Clos network, as derived above. A single $N\times N$ crossbar requires $n^6$ crosspoints. Therefore, the crosspoint ratio of a five-stage traditional unidirectional rearrangeably nonblocking Clos network relative to a single crossbar is $5n^4/n^6=5/n^2$.

In the seven-stage case, $m=n$ and $N=n^4$. The total number of crosspoints is $nm\times n^3+5n^4\times m+mn\times n^3=n^5+5n^5+n^5=7n^5$, where $5n^4$ is the number of crosspoints in a five-stage traditional unidirectional rearrangeably nonblocking Clos network, as derived above. A single $N\times N$ crossbar requires $n^8$ crosspoints. Therefore, the crosspoint ratio of a seven-stage traditional unidirectional rearrangeably nonblocking Clos network relative to a single crossbar is $7n^5/n^8=7/n^3$.

Table 13 summarizes the crosspoint ratio relative to a single crossbar for traditional unidirectional rearrangeably nonblocking Clos networks and URNBC networks. The cost ratios for the proposed URNBC networks relative to traditional networks are $66.67\%$, $60.00\%$, and $57.14\%$ for three-stage, five-stage, and seven-stage networks, respectively.

Table 13. Crosspoint ratio for unidirectional rearrangeably nonblocking Clos networks.

Network	3-Stage	5-Stage	7-Stage
Traditional	$3/n$	$5/n^2$	$7/n^3$
URNBC	$2/n$	$3/n^2$	$4/n^3$
URNBC/Traditional	$66.67\%$	$60.00\%$	$57.14\%$

Figure 20 plots the crosspoint ratio relative to a single crossbar for unidirectional rearrangeably nonblocking Clos networks, showing that the URNBC networks have a lower crosspoint cost than traditional rearrangeably nonblocking Clos networks.

Figure 20. Crosspoint ratio relative to a single crossbar in unidirectional rearrangeably nonblocking Clos networks.

4.2.2. Cost Evaluations of IRNBC Networks

The number of crosspoints for IRNBC networks that use equally sized square crossbars of $(n+m)\times (m+n)=2n\times 2n$ for $m=n$ is examined below. Referring to Figure 10b, in a two-stage IRNBC network, there are $r=n+m=2n$ switches in the leaf stage and m switches in the root stage. The total number of crosspoints is $(n+m)\times (m+n)\times (2n+m)=12n^3$. There are $N=nr=2n^2$ compute nodes. A single $N\times N$ crossbar requires $N\times N=4n^4$ crosspoints. The crosspoint ratio of a two-stage IRNBC network relative to a single crossbar is $12n^3/\left(4n^4\right)=3/n$.

Referring to Figure 14, in a three-stage IRNBC network, the total number of crosspoints is $(n+m)\times (m+n)\times 2n^2+12n^3\times m=8n^4+12n^4=20n^4$, where $12n^3$ is the number of crosspoints in a two-stage IRNBC network, as derived above. There are $N=nr=2n^3$ compute nodes. A single $N\times N$ crossbar requires $4n^6$ crosspoints. The crosspoint ratio of a three-stage IRNBC network relative to a single crossbar is $20n^4/\left(4n^6\right)=5/n^2$.

Referring to Figure 16, in a four-stage IRNBC network, the total number of crosspoints is $(n+m)\times (m+n)\times 2n^3+20n^4\times m=8n^5+20n^5=28n^5$, where $20n^4$ is the number of crosspoints in a three-stage IRNBC network, as derived above. There are $N=nr=2n^4$ compute nodes. A single $N\times N$ crossbar requires $4n^8$ crosspoints. The crosspoint ratio of a four-stage IRNBC network relative to a single crossbar is $28n^5/\left(4n^8\right)=7/n^3$.

Table 14 lists the crosspoints of IRNBC networks. The “Crossbar” column shows the number of crosspoints in a single crossbar. The “IRNBC” column shows the number of crosspoints in an IRNBC network. The number of crosspoints of the IRNBC network is better (smaller) than that of a single crossbar when $n\ge 4$ for the two-stage IRNBC network, $n\ge 3$ for the three-stage IRNBC network, and $n\ge 2$ for the four-stage IRNBC networks.

Table 14. The number of crosspoints in IRNBC networks.

	2-Stage			3-Stage			4-Stage
n	Node	Crossbar	IRNBC	Node	Crossbar	IRNBC	Node	Crossbar	IRNBC
2	8	64	96	16	256	320	32	1024	896
3	18	324	324	54	2916	1620	162	26,244	6804
4	32	1024	768	128	16,384	5120	512	262,144	28,672
5	50	2500	1500	250	62,500	12,500	1250	1,562,500	87,500
6	72	5184	2592	432	186,624	25,920	2592	6,718,464	217,728
7	98	9604	4116	686	470,596	48,020	4802	23,059,204	470,596
8	128	16,384	6144	1024	1,048,576	81,920	8192	67,108,864	917,504
9	162	26,244	8748	1458	2,125,764	131,220	13,122	172,186,884	1,653,372
10	200	40,000	12,000	2000	4,000,000	200,000	20,000	400,000,000	2,800,000

The number of crosspoints for traditional rearrangeably nonblocking folded Clos networks with $m=n$ is examined below. In a two-stage traditional rearrangeably nonblocking folded Clos network, the total number of crosspoints is $(n+m)(m+n)\times n+n^2\times m=4n^3+n^3=5n^3$. There are $N=n^2$ compute nodes. A single $N\times N$ crossbar requires $N\times N=n^4$ crosspoints. The crosspoint ratio of a two-stage traditional rearrangeably nonblocking folded Clos network relative to a single crossbar is $5n^3/n^4=5/n$.

In the three-stage case, the total number of crosspoints is $(n+m)(m+n)\times n^2+5n^3\times m=4n^4+5n^4=9n^4$, where $5n^3$ is the number of crosspoints in a two-stage traditional rearrangeably nonblocking folded Clos network, as derived above. There are $N=n^3$ compute nodes. A single $N\times N$ crossbar requires $n^6$ crosspoints. The crosspoint ratio of a three-stage traditional rearrangeably nonblocking folded Clos network relative to a single crossbar is $9n^4/n^6=9/n^2$.

In the four-stage case, the total number of crosspoints is $(n+m)(m+n)\times n^3+9n^4\times m=4n^5+9n^5=13n^5$, where $9n^4$ is the number of crosspoints in a three-stage traditional rearrangeably nonblocking folded Clos network, as derived above. The crosspoint ratio of a four-stage traditional rearrangeably nonblocking folded Clos network relative to a single crossbar is $13n^5/n^8=13/n^3$.

Table 15 summarizes the crosspoint ratio relative to a single crossbar for traditional rearrangeably nonblocking folded Clos networks and IRNBC networks. The cost ratios for the proposed IRNBC networks relative to traditional networks are $60.00\%$, $55.56\%$, and $53.85\%$ for three-stage, five-stage, and seven-stage networks, respectively. The general formula for calculating the IRNBC crosspoint ratio relative to a single crossbar is $(2s-1)/n^{s-1}$, where s is the number of stages and the number of compute nodes is $N=2n^s$.

Table 15. Crosspoint ratio for rearrangeably nonblocking folded Clos networks.

Network	2-Stage	3-Stage	4-Stage
Traditional	$5/n$	$9/n^2$	$13/n^3$
IRNBC	$3/n$	$5/n^2$	$7/n^3$
IRNBC/Traditional	$60.00\%$	$55.56\%$	$53.85\%$

Table 16 lists the crosspoint ratios relative to a single crossbar for rearrangeably nonblocking folded Clos networks. The ratios are calculated based on the formulas in Table 15.

Table 16. Crosspoint ratio relative to a single crossbar in rearrangeably nonblocking folded Clos networks.

	IRNBC			Traditional
n	2-Stage	3-Stage	4-Stage	2-Stage	3-Stage	4-Stage
2	1.500000	1.250000	0.875000	2.500000	2.250000	1.625000
3	1.000000	0.555556	0.259259	1.666667	1.000000	0.481481
4	0.750000	0.312500	0.109375	1.250000	0.562500	0.203125
5	0.600000	0.200000	0.056000	1.000000	0.360000	0.104000
6	0.500000	0.138889	0.032407	0.833333	0.250000	0.060185
7	0.428571	0.102041	0.020408	0.714286	0.183673	0.037901
8	0.375000	0.078125	0.013672	0.625000	0.140625	0.025391
9	0.333333	0.061728	0.009602	0.555556	0.111111	0.017833
10	0.300000	0.050000	0.007000	0.500000	0.090000	0.013000

Figure 21 plots the crosspoint ratio relative to a single crossbar for rearrangeably nonblocking folded Clos networks, showing that IRNBC networks have a lower crosspoint cost than traditional rearrangeably nonblocking folded Clos networks.

Figure 21. Crosspoint ratio relative to a single crossbar in rearrangeably nonblocking folded Clos networks.

From the above discussion in this section, it can be seen that the crosspoint ratios of ISNBC and IRNBC networks are lower than those of their corresponding traditional folded Clos networks. The crosspoint ratio of IRNBC networks is lower than that of ISNBC networks because IRNBC networks are rearrangeably nonblocking folded Clos networks that require rearrangements of existing connections to make new connections and ISNBC networks are strictly nonblocking folded Clos networks that do not require rearrangements of existing connections to make new connections.

The proposed ISNBC and IRNBC networks are summarized in Table 17, where the “Node” column shows the number of compute nodes, the “Crossbar” column shows the number of crosspoints in a single crossbar, the “Crosspoint” column shows the number of crosspoints in the proposed network, and the “Switch” column shows the number of switches in the proposed network. Both networks use square crossbar switches.

Table 17. Summary of the proposed ISNBC and IRNBC networks, where n is the number of compute nodes connected to a leaf switch and s is the number of stages.

Network	Node	Crossbar	Crosspoint	Switch	Switch Size
ISNBC	$3n^s$	$9n^{2s}$	$9(2^{s+1}-3)n^{s+1}$	$(2^{s+1}-3)n^{s-1}$	$3n\times 3n$
IRNBC	$2n^s$	$4n^{2s}$	$4(2s-1)n^{s+1}$	$(2s-1)n^{s-1}$	$2n\times 2n$

Figure 22 compares the crosspoint ratios of ISNBC and IRNBC networks for different numbers of compute nodes. Based on this figure, the number of stages can be selected for a given number of compute nodes such that the system has a low crosspoint ratio.

Figure 22. Crosspoint ratios versus the numbers of compute nodes in the proposed networks.

A limitation of the proposed identical nonblocking folded Clos networks is that in ISNBC networks, $3n\times 3n$ square crossbar switches must be used, whereas in IRNBC networks, $2n\times 2n$ square crossbar switches must be used, where n is the number of compute nodes connected to a leaf switch. Therefore, the proposed networks cannot use arbitrary $n\times n$ square crossbar switches, where $n\in \mathbb{N}$, without any unused switch ports. For example, by using practical $16\times 16$ crossbar switches, an IRNBC network can be constructed with no unused switch ports ($16=2n$ with $n=8$). By constructing an ISNBC network with $16\times 16$ crossbar switches, one port on each switch is left unused ($16=3n+1$ with $n=5$). Again, if an ISNBC network is constructed using $15\times 15$ crossbar switches, there will be no unused switch ports.

5. Conclusions

Nowadays, available switches are square crossbars with the same number of input and output ports. Traditional unidirectional strictly nonblocking Clos networks use switches with different numbers of input and output ports and route packets through a fixed number of stages. Fat trees, the folded version of Clos networks, use differently sized switches at the root and other stages. Recently proposed Clos networks have only two stages and introduce unused ports on the switches.

To address these issues, this paper proposed two new folded Clos variants: an identical strictly nonblocking Clos (ISNBC) network and an identical rearrangeably nonblocking Clos (IRNBC) network. These designs use equally sized square crossbar switches across all stages, eliminate unused switch ports, increase system scalability by accommodating any number of stages, and reduce communication path lengths by supporting shortcut connections. Both ISNBC and IRNBC networks have lower switch crosspoint costs compared to their traditional counterparts. Specifically, ISNBC networks use 46.43% to 87.71% of the crosspoints of traditional strictly nonblocking folded Clos networks, and IRNBC networks use 53.85% to 60.00% of the crosspoints of traditional rearrangeably nonblocking folded Clos networks.

The limitation is that ISNBC networks require the use of $3n\times 3n$ square crossbar switches, and IRNBC networks require the use of $2n\times 2n$ square crossbar switches, where n is the number of compute nodes connected to a leaf switch.

Future work should develop load-balancing adaptive routing algorithms and fault-tolerant routing algorithms for the proposed identical strictly and rearrangeably nonblocking folded Clos networks and evaluate performance through simulations.