Sparse Power-Law Network Model for Reliable Statistical Predictions Based on Sampled Data

Abstract

A projective network model is a model that enables predictions to be made based on a subsample of the network data, with the predictions remaining unchanged if a larger sample is taken into consideration. An exchangeable model is a model that does not depend on the order in which nodes are sampled. Despite a large variety of non-equilibrium (growing) and equilibrium (static) sparse complex network models that are widely used in network science, how to reconcile sparseness (constant average degree) with the desired statistical properties of projectivity and exchangeability is currently an outstanding scientific problem. Here we propose a network process with hidden variables which is projective and can generate sparse power-law networks. Despite the model not being exchangeable, it can be closely related to exchangeable uncorrelated networks as indicated by its information theory characterization and its network entropy. The use of the proposed network process as a null model is here tested on real data, indicating that the model offers a promising avenue for statistical network modelling.

1. Introduction

Network science [1,2,3,4] is one of the most rapidly advancing scientific fields of investigation. The success of this field is deeply rooted in its interdisciplinarity. In fact, network science characterizes the underlying structure and dynamics of complex systems ranging from on-line social networks to molecular networks and the brain. Additionally, the theoretical tools and techniques used by network science are coming from different disciplines including statistical mechanics, statistics, machine learning and computer science.

In the last twenty years significant attention has been addressed to modelling framework of complex networks. Since most real networks, from the Internet to molecular networks, are sparse, i.e., they have an average degree that does not depend on the network size, statistical mechanics models focus on modelling sparse networks. These statistical mechanics models can be divided between non-equilibrium growing network models [5,6,7,8,9,10,11,12,13] such as the famous Barabási–Albert model [5] and equilibrium models such as maximum entropy network ensembles [14,15,16,17,18,19] including Exponential Random Networks [16,17,20,21,22] and block models [23,24]. The non-equilibrium growing network models have the power to explain the fundamental mechanisms giving rise to emergent properties such as scale-free distributions [5,6,7,8], degree correlations [6], communities [9,10,11] and network geometry [11,12,13]. On the contrary, maximum network ensembles constitute the least biased models satisfying a given set of constraints. These models are not explanatory but constitute the ideal null hypothesis to which real networks can be compared.

Recently the need to formulate reliable statistical models is receiving significant attention [25]. A reliable statistical model will include projectivity and exchangeability [26,27,28,29,30]. The projectivity of the statistical network model guarantees that the conclusions reached by considering a subsample of the data are consistent with the ones that can be drawn starting from a larger sample of the data. The exchangeability of the nodes implies that the probability of a network does not depend on the specific labels of the nodes. However, how to reconcile these statistical requirements with the sparseness of the networks, i.e., a average degree that is independent of the network size, constitutes a major impasse of network modelling. For instance it has been shown that random uncorrelated networks are only projective if the average degree $\langle k\rangle$ of the network increases linearly with the network size N, i.e., if the network is maximally dense and $\langle k\rangle =O\left(N\right)$ [27,30,31].

In physical terms, the desired projective and exchangeable network process mimicking the subsequent sampling of an increasing portion of the network is a modelling framework that goes beyond the traditional statistical mechanics division between equilibrium and non-equilibrium network modelling approaches. This observation reinforces the belief that actually combining these two properties might be not an easy task.

Already several works have addressed this problem [32,33,34,35,36,37,38], using different approaches such as relaxing the condition $\langle k\rangle =O\left(N\right)$ but always characterizing models with average degree diverging with the network size N, considering edge exchangeable models or alternatively using an embedding space as a basic mechanism to combine sparsity with projectivity and exchangeability [31,39].

Here we propose a network process describing a network evolution mimicking the sampling of a network by subsequently expanding the nodes set. Each node is assigned an hidden variable from a hidden variable distribution. This distribution is the key quantity determining the properties of the network process. If the hidden variable is power-law distributed and the network is sufficiently sparse, the degree distribution displays a power-law tail with the same power-law exponent as the hidden variable distribution.

This model is a projective network process but it is not exchangeable. Nevertheless, this non-equilibrium network model can be directly related to an equilibrium uncorrelated network ensemble in the sparse regime. In fact, by permuting the order in which nodes are sampled it is possible to calculate the probability that two nodes are connected given their corresponding hidden variables. This connection probability is equal to the connection probability in an uncorrelated exchangeable network ensemble in which the hidden variable of each node is identified with half of its expected degree. The “proximity” between the network process and the uncorrelated network ensemble is here quantified by using information theory tools and comparing the entropy of the two models. In particular, we use the entropy of the two network models [14,15,16,17,40] to evaluate the difference in the information content of the two models, finding that the two models have small relative entropy difference.

Finally we study how well the proposed model can be used as a null model for real power-law network datasets. To this end we identify the hidden variable of each node with half of its observed degree and we run the model by adding the nodes in the network according to a random permutation of the nodes’ labels. The degree distribution of the real dataset and the degree distribution of the simulation results are in good agreement when starting from power-law networks, and the agreement remains good if the network is grown by only considering a subsample of the nodes of the real data. We also compare the correlations of the real dataset with the correlations of the simulation results to show that the simulations are able to generate only weak correlations of the degrees. Therefore a more refined model should be formulated to capture this additional network property.

The paper is structured as described in the following. In Section 2 we introduce the definition of the desired statistical properties of network models: projectivity and exchangeability. In Section 3 we discuss major examples of sparse network models (the Barabási–Albert model and the uncorrelated network ensembles) and characterize them with respect to the properties of projectivity and exchangeability. In Section 4 we present an account of the difficulties in combining projectivity and exchangeability with the sparseness of networks and we give a brief review of the approaches investigated in the recent literature on the subject. In Section 5 we present a network process mimicking a network sampling process. We characterize its structural and dynamical properties relating this non-equilibrum model to equilibrium uncorrelated network ensembles, and we characterize its statistical properties. In Section 6 we show the possible use of the proposed network process as a null model for modelling real power-law network datasets. Finally in Section 7 we give the conclusions.

2. Statistical Terms

Projectivity and exchangeability are two very basic and very natural statistical requirements for reliable statistical network models. In physical terms, projectivity is directly related to the principle of locality, while exchangeability is related to symmetry. In this section, we first discuss projectivity and exchangebility to make clear that they really are “must-haves” in any statistically useful network model, while in the next two sections we will comment on difficulties in combining them both in models of sparse networks, i.e., having average degree independent of the network size N [41]. While projectivity and exchangeability are desired properties of statistically reliable network models, the relevance and of these requirements for any realistic network model is a subject of scientific debate (see for instance contribution of Karthik Bharath in the discussion of the F. Caron and E. Fox paper [33]). In fact it is often observed that most real networks can hardly be exchangeable. Indeed, in a vast majority of real networks nodes are labelled with labels related to some rich metadata and a random permutation of the nodes labels would result in a different network whose probability to be produced by the same stochastic process that produces the real network is certainly not expected to be equal to the probability with which it generates the real network.

In order to investigate the properties of reliable statistical models we consider a network process mimicking the subsequent sampling a network by expanding the set of sampled nodes and detecting all the interactions among this set of nodes.

To this end we consider a set of networks ${\left\{G_t\right\}}_{t=1,2,\dots }$ with $G_t=(V_t,E_t)$ and increasing network size $N_t=\left|V_t\right|=t$. The sequence of networks defines a network process, i.e., $G_t=(V_t,E_t)$ is an induced subgraph of the network $G_{t^{\prime }}=(V_{t^{\prime }},E_{t^{\prime }})$ for all $t<t^{\prime }$ with node set $V_t\subset V_{t^{\prime }}$ if $t<t^{\prime }$. We label the nodes in order of their appearance in the network such that

$$\begin{array}{c}\hfill V_t=\{1,2,\dots ,t\}.\end{array}$$

(1)

and assign a probability $P\left(G_t\right)$ to each network $G_t$.

2.1. Projectivity

Given the set of networks ${\left\{G_t\right\}}_{t=1,2,\dots }$projectivity implies that the statistical properties of the network $G_t$ are directly related to the statistical properties of the network $G_{t^{\prime }}$ with $t^{\prime }>t$ by a proper marginalization of the probability of the network $G_{t^{\prime }}$ over its subgraph $G_t$.

By definition [26,27], a projective network model is a model that attributes a given probability $P\left(G_t\right)$ to each network $G_t$ of the sequence, such that

$$P\left({\pi }_{t^{\prime },t}\left(G_{t^{\prime }}\right)\right)=P\left(G_t\right),$$

where the projective map ${\pi }_{t^{\prime },t}$ maps networks $G_{t^{\prime }}$ of a larger size $t^{\prime }>t$ to their subgraph $G_t$ of a smaller size t.

In other words this means that one can first generate a larger graph $G_{t^{\prime }}$ using the model, then reduce its size to t by throwing out some $t^{\prime }-t$ nodes according to the projective map specification, and the probability with which the resulting graph $G_t$ is generated using this two-step procedure will be the same as if graph $G_t$ was generated by the model directly.

2.2. Exchangeability

Exchangeability implies that the order in which two nodes are observed or labelled is not important. Specifically, a network model is exchangeable if, by definition [29,30], the probability $P\left(G_t\right)$ of a network $G_t=(V_t,E_t)$ is independent on the nodes labels, i.e.,

$$\begin{array}{c}\hfill P\left(G_t\right)=P({\tilde{G}}_t)\end{array}$$

(2)

where ${\tilde{G}}_t$ is any network isomorphic to the network $G_t$, i.e., it is any network obtained from the network $G_t$ by permuting the nodes labels ${\left\{i\right\}}_{i=1,2,\dots ,N}$ according to the permutation $\mathbf{\sigma }$. If a network model is exchangeable it follows that the marginal the probability $p_{ij}$ of the generic link between node i and node j is unchanged if the node labels are permuted, implying that they are sampled in a different order, i.e.,

$$\begin{array}{c}\hfill p_{ij}=p_{\sigma \left(i\right),\sigma \left(j\right)}.\end{array}$$

(3)

Therefore exchangeability enforces the symmetry of the model with respect to the group of graph isomorphisms.

3. Characterization of Relevant Sparse Network Models from the Statistical Perspective

In this section we investigate major examples of non-equilibrium (growing) network models and equilibrium (static) network models widely used to model sparse complex networks. In particular we discuss the Barabási–Albert model [5] and the uncorrelated network ensembles from the statistical perspective. This discussion will reveal that neither of these two very popular frameworks for modelling sparse complex networks display both projectivity and exchangeability, indicating the difficulties in combining these properties with the sparseness of the networks.

3.1. Barabási–Albert Model

The Barabási–Albert model begins with an initial finite network and at each time t a new node enters in the network and is connected to the network by establishing m new links. Each of these links connect the new node to a node i with degree $k_i$ chosen with probability

$$\begin{array}{c}\hfill {\tilde{\Pi }}_i=\frac{k_i}{{\sum }_{i^{\prime }}{k}_{i^{\prime }}}.\end{array}$$

(4)

This probability enforces preferential attachment, i.e., allows nodes with higher degree to more rapidly acquire new links.

The Barabási–Albert model describes a model that is projective, because as the network grows the network $G_t$ obtained at time t is an induced subgraph of the network $G_{t^{\prime }}$ obtained at a later time $t^{\prime }>t$. However the Barabási–Albert model is not exchangeable. The fact that the network is not exchangeable is revealed for instance by the expression for the average number of links $k_i(t,t_i)$ of a node i arrived in the network at time $t_i$,

$$\begin{array}{c}\hfill k_i(t,t_i)=m{\left(\frac{t}{t_i}\right)}^{1/2}.\end{array}$$

(5)

This expression explicitly indicates that the older nodes are statistically different from the younger nodes, and their degree is much larger than that of younger nodes. Additionally it is possible to observe that the model is not exchangeable because the order of the addition of the nodes, i.e., their time of arrival in the network, is the key property that determines the connection probability [42], i.e.,

$$\begin{array}{c}\hfill p_{ij}\simeq \frac{m}{2}\frac{1}{\sqrt{t_i{t}_j}}.\end{array}$$

(6)

Nevertheless we observe the interesting property that for this model the connection probability $p_{ij}$ between node i and node j can be also expressed as

$$\begin{array}{c}\hfill p_{ij}\simeq \frac{k_i(t,t_i)k_j(t,t_j)}{{\sum }_{i^{\prime }}{k}_{i^{\prime }}(t,t_{i^{\prime }})},\end{array}$$

(7)

indicating that actually, although the network process has different statistical properties than the uncorrelated network with the same degree distribution, the expected degree correlations are weak. The relation between the Barabási–Albert (BA) model and the uncorrelated network ensemble with the same degree distribution is investigated in detail using information theoretic tools in Ref. [43].

3.2. Uncorrelated Network Ensembles

The Barabási–Albert model is projective but not exchangeable. On the contrary the widely used uncorrelated network ensembles are exchangeable models but they are not projective in the sparse regime. In order to show this let us consider an uncorrelated network model in which each node i has an expected degree ${\theta }_i$, where the expected degrees of the nodes are consistent with a structural cutoff, i.e.,

$$\begin{array}{c}\hfill {\theta }_i\le \sqrt{\langle \theta \rangle N}.\end{array}$$

(8)

In this case the probability $p_{ij}$ of a link between node i and node j is given by

$$\begin{array}{c}\hfill p_{ij}=\frac{{\theta }_i{\theta }_j}{〈\theta 〉N},\end{array}$$

(9)

and therefore it only depends on the expected degrees ${\theta }_i$ and ${\theta }_j$ of the nodes i and j and not on the order in which node i and node j have been sampled. The model is therefore exchangeable as long as we consider the simultaneous permutation of the node labels and the expected degrees of the nodes. However if we consider a large sample of the network with $N^{\prime }>N$ nodes, we see that the model is projective if and only if it is also dense, with the number of links scaling as $L=O\left(N^2\right)$. In fact if we assume that in the larger sample the expected degrees of nodes i and j are given by ${\theta }_i^{\prime }$ and ${\theta }_j^{\prime }$, the probability that node i and node j are connected in the larger network models including $N^{\prime }$ nodes is

$$\begin{array}{c}\hfill p_{ij}^{\prime }=\frac{{\theta }_i^{\prime }{\theta }_j^{\prime }}{〈{\theta }^{\prime }〉N^{\prime }}\end{array}$$

(10)

If we impose projectivity, i.e.,

$$\begin{array}{c}\hfill p_{ij}=p_{ij}^{\prime }\end{array}$$

(11)

for $i,j\le N$, and we assume that the number of nodes $N^{\prime }>N$ can be written as

$$\begin{array}{c}\hfill N^{\prime }=zN,\end{array}$$

(12)

it is easy to see that we should also have

$$\begin{array}{cc}\hfill {\theta }_i^{\prime }& =z{\theta }_i,\hfill \\ \hfill \langle {\theta }_i^{\prime }\rangle & =z\langle \theta \rangle .\hfill \end{array}$$

(13)

Therefore to guarantee projectivity the expected degree of each node should grow linearly with the network size, resulting in a dense network with the total number of links L scaling with the network size N as $L=O\left(N^2\right)$. This implies that the random network $\mathbb{G}(N,p)$ with p independent of N is an exchangeable model whereas the Poisson random network $\mathbb{G}(N,p)$ with $p=\frac{z}{N}$ and z independent of N is not exchangeable. In fact one cannot throw out $N^{\prime }-N$ nodes from a network of size $N^{\prime }$ produced by $\mathbb{G}(N^{\prime },z/N^{\prime })$, and hope that the resulting network will have the same probability as in $\mathbb{G}(N,z/N)$, simply because the links in the $\mathbb{G}(N^{\prime },z/N^{\prime })$ and $\mathbb{G}(N,z/N)$ ensembles exist with different probabilities $z/N^{\prime }$ and $z/N$ that depend on the graph size N. Alternatively, if one attempts to formulate $\mathbb{G}(N,z/N)$ as a growing model, then since the edge existence probability depends on N, the addition of a new node affects the probability of existence of edges in the existing network. Since this probability is a decreasing function of N ($z/N$), upon the addition of a new node all the existing edges must be removed with some probability ($1/N$). In other words, in such a growing model new node additions must necessarily affect the existing network structure.

4. Impasse with Sparsity

Surprisingly, combining projectivity and exchangeability with the additional constraint of sparsity, i.e., the requirement that the average degree of the sampled networks is independent of the network size, has been a major impasse. If we exclude spatially embedded networks [31], to the best of our knowledge there exists no model of sparse networks that would be both projective and exchangeable at the same time. This situation is in stark contrast with the case of dense graphs. Dense graphs are known to have well-defined thermodynamic limits known as graphons, and any graphon-based network model is both exchangeable and projective [30].

The thermodynamic limits of sparse graphs are at present quite poorly understood, which appears to be one of the reasons behind the mentioned impasse. Several attempts have been made to understand the limits of sparse graphs, including, for example, sparse $L^p$ graphons [32], which are not projective, or stretched graphons a.k.a. graphexes [33,34,35]. In the latter case, graphs are sparse, exchangeable and projective, but with two major caveats:

(1)

the average degree cannot be constant, it must diverge with N (but possibly slower than linearly),

(2)

exchangeability is completely redefined: it is not with respect to node labels $1,\dots ,N$, but with respect to artificial labels which are positive real numbers.

Another class of attempts suggests to completely give up on the node label exchangeability requirement, and to consider edge exchangeability instead, e.g., using variations of Pitman–Yor processes [36,37,38]. It remains unclear at present whether these developments imply that too many network models that were found to be quite useful in practice and that do use node labels $1,\dots ,N$, are statistically hopeless. It seems more likely that further research is needed to understand and resolve this projectivity vs. exchangeability impasse in sparse network models.

Proposed Solution of the Impasse Based on Network Geometry

In [31] it was shown that a generic network model is projective if the probability of edge existence, i.e., the connection probability, does not depend on the network size N. In fact if the connection probability does depend on N, then, the addition of new nodes to the existing network in the growing formulation of the model necessarily affects the existing network structure and the network cannot be projective.

In order to formulate network models in which the connection probability does not depend on the network size N, embedding networks in space can turn out to be very useful. In fact spatially embedded networks can combine projectivity with a constant average degree [31] as their spatial embedding ensures projectivity when the connection probability is local and nodes connect typically to nodes that are spatially close. For instance if the nodes are uniformly distributed in ${\mathbb{R}}^2$ and each node connects only to the nodes with a constant radius $r_0$, by sampling the network by progressively expanding the spatial region of interest we can build a projective model with constant average degree. This is clearly a realistic scenario in most real networks as it unlikely that a local event in a spatial network causes a global change in the network. For instance in the Internet, the appearance of a new customer of a local Internet provider in Bolivia cannot lead to immediate severance of customers by a local Internet provider in Bhutan.

It turns out that models that are not explicitly constructed from spatial embeddings can also be analysed using geometrical arguments, hence shedding light on their statistical properties. In this vein, it was recently shown that the hypersoft configuration model, which defines maximum-entropy random graphs with a given degree distribution, is sparse and either exchangeable or projective [39]. Both sparsity and exchangeability definitions are traditional in the model, i.e., the average degree is constant and exchangeability is with respect to labels $1,\dots ,N$, so that the only caveats are in “either-or” and also in that this “either-or” is achieved only for specific degree distributions (power law with exponent $\gamma =3$ in [39]).

In the exchangeable equilibrium formulation of the model, nodes are points sprinkled at random onto an interval $A_N$ of an N-dependent length $L_N$, where $L_N$ is a growing function of N, according to a non-uniform point density (if this point density is exponential, then the resulting degree distribution is a power law), and then all pairs of points/nodes i and j, $j>i=1,\dots ,N$, at sprinkled coordinates $x_i$ and $x_j$ are connected by an edge with the entropy-maximizing Fermi–Dirac connection probability

$$\begin{array}{c}\hfill p(x_i,x_j)=\frac{1}{e^{x_i+x_j}+1}\end{array}$$

(14)

that does not depend on the network size N.

In the projective growing formulation of the same model, the interval $A_N$ grows with N, its length growing according to $L_n$, new node $N+1$ appears in the interval increment $A_{N+1}\backslash A_N$ of length $L_{N+1}-L_N$, and then connects to existing nodes with the same connection probability as in the exchangeable formulation.

The difficulty of combining projectivity and exchangeability is evident in this example: in the exchangeable formulation, node labels i are random and uncorrelated with their coordinates $x_i$, while in the projective formulation, nodes are labelled in the increasing order of their coordinates: $i<j\iff x_i<x_j$. If nodes are labelled this way, then the projective map ${\pi }_{N^{\prime },N}$ trivially throws out nodes with labels $N+1,\dots ,N^{\prime }$, and the resulting graph satisfies the projectivity requirement since the connection probability does not depend on N, and since the remaining N nodes lie in $A_N$. If the node labels are random, however, as they are in the exchangeable formulation, then it remains unclear if even an asymptotically correct projective map can be constructed.

5. Statistical Mechanics Model with Hidden Variables

Our goal is here to reconcile sparseness with a reliable statistical modelling framework without assuming the existence of an embedding geometrical space. In this endeavour we will define a projective network process yielding a sequence of networks growing by the subsequent addition of nodes and links. To each node i we associate a hidden variable ${\theta }_i$ that is a proxy for the degree that the node will acquire in the model. The statistical properties of the network model when we average over all the possible sequences determining the subsequent addition of the links obey scaling laws and reduce to the uncorrelated network model of any size N in the sparse regime.

Although this model does not ultimately reconcile sparseness with both exchageability and projectivity, we will see in Section 6 that it provides a very reliable null model for power-law networks also if only a subsample of the original network is considered.

5.1. The Model

The model can be interpreted as a weighted growing network model where we allow multiedges. In the model every node i is assigned a hidden variable ${\theta }_i$ from a hidden variable distribution $\rho \left(\theta \right)$.

Starting at $t=1$ from a single isolated node, at each time $t>1$ a new node i is added to the network and draws ${\kappa }_i$ links to the existing nodes of the network, where ${\kappa }_i$ is chosen according to the Poisson distribution with average ${\theta }_i$, i.e.,

$$\begin{array}{c}\hfill \widehat{P}\left({\kappa }_i\right|{\theta }_i)=\frac{1}{{\kappa }_i!}{\theta }_i^{{\kappa }_i}{e}^{-{\theta }_i}.\end{array}$$

(15)

Each new link is attached to a node j already present in the network with probability

$$\begin{array}{c}\hfill {\Pi }_j=\frac{{\theta }_j}{{\sum }_{r=1}^{t-1}{\theta }_r}.\end{array}$$

(16)

Note that not all the new links will yield new connections because the nodes i and j might be already connected. Additionally note that this model does not implement preferential attachment as the linking probability is only dependent on the externally attributed hidden variable ${\theta }_i$ and not to the dynamically acquired degree $k_i$. Whenever a new link connects node i to an already connected node j the multi-edge between node i and node j is reinforced, i.e., the weight of the links between node i and node j increases by one.

Here and in the following we will indicate by $\mathbf{a}$ the adjacency matrix of the network, with $t_i$ the time at which node i has been added to the network, with $k_i$ the node degree and with $s_i$ the node strength, i.e., the sum of the weights of the links incident to node i.

5.2. The Strength of a Node and Its Dependence on the Hidden Variable $\theta$

The hidden variable ${\theta }_i$ modulates the temporal evolution of the strength of the node i. In fact in the mean-field approach [1,5,44], since at each time an average of $\langle \theta \rangle$ links are added and reinforced, the average strength $s_i\left(t\right|t_i,{\theta }_i,{\kappa }_i)$ of node i given the time $t_i$ of its arrival in the network, its hidden variable ${\theta }_i$ and its initial strength ${\kappa }_i$ obeys the equation

$$\begin{array}{c}\hfill \frac{d{s}_i}{dt}=\langle \theta \rangle \frac{{\theta }_i}{\langle \theta \rangle t}=\frac{{\theta }_i}{t}\end{array}$$

(17)

with initial condition $s_i\left(t_i\right|t_i,{\theta }_i,{\kappa }_i)={\kappa }_i$. The solution of this equation is

$$\begin{array}{c}\hfill s_i\left(t\right|{\theta }_i,{\kappa }_i)={\theta }_{i}ln\left(\frac{t}{t_i}\right)+{\kappa }_i.\end{array}$$

(18)

Therefore in this model the strength depends both on the time of arrival of the node in the network and on its hidden variable. If we average the strength over the nodes with the same hidden variable however, we see that the average strength ${\tilde{s}}_i\left({\theta }_i\right)$ of nodes with hidden variable ${\theta }_i$ is given in the large network limit $t\gg 1$ by

$$\begin{array}{c}\hfill {\tilde{s}}_i\left({\theta }_i\right)=2{\theta }_i.\end{array}$$

(19)

In fact we have

$$\begin{array}{ccc}\hfill 〈{\kappa }_i|{\theta }_i〉& =& {\theta }_i\hfill \\ \hfill {\tilde{s}}_i\left({\theta }_i\right)& =& \frac{1}{t}{\int }_1^t{\theta }_{i}ln\left(\frac{t}{t_i}\right)d{t}_i+〈{\kappa }_i|{\theta }_i〉\hfill \\ & =& 2{\theta }_i+O\left(\frac{lnt}{t}\right).\hfill \end{array}$$

(20)

This implies that if we attribute to a node a hidden variable ${\theta }_i$ and we consider a set of models in which the time of arrival of node i is taken randomly, the strength of node i is (on average over the different network models) determined only by its hidden variable.

5.3. Strength Distribution

The strength distribution of the model is a convolution of exponentials. To find the strength distribution we use the master equation approach [44] under the assumption that the hidden variable distribution has a well defined average value $\langle \theta \rangle$. To this end we write the equation for $N_{\theta }^t\left(s\right)$, the average number of nodes with hidden variable $\theta$ that have strength $s\ge 0$ at time t, as

$$\begin{array}{c}\hfill \frac{N_{\theta }^t\left(s\right)}{dt}=\langle \theta \rangle \Pi \left(\theta \right)N_{\theta }^t(s-1)[1-\delta (s,0)]-\langle \theta \rangle \Pi \left(\theta \right)N_{\theta }^t\left(s\right)+\rho \left(\theta \right)\widehat{P}(\kappa =s|\theta ),\end{array}$$

(21)

where $\delta (x,y)$ indicates the Kronecker delta and where we denote by $\Pi \left(\theta \right)$ the probability that a node with hidden variable $\theta$ is attached to the new node arrived in the network at time t by one of its connections, i.e.,

$$\begin{array}{c}\hfill \Pi \left(\theta \right)=\frac{\theta }{{\sum }_{{\theta }^{\prime }}{\theta }^{\prime }{\sum }_s{N}_{{\theta }^{\prime }}^t\left(s\right)}\simeq \frac{\theta }{\langle \theta \rangle t}.\end{array}$$

(22)

Given the continuous growth of the network asymptotically in time, for $t\gg 1$ it is possible to assume that

$$\begin{array}{c}\hfill N_{\theta }^t\left(s\right)\simeq t{P}_{\theta }\left(s\right),\end{array}$$

(23)

where $P_{\theta }\left(s\right)$ is the probability that a random node has strength s and hidden variable $\theta$.

By inserting this asymptotic expression in the master Equation (21) and solving for $P_{\theta }\left(s\right)$ we get

$$\begin{array}{c}\hfill P_{\theta }\left(s\right)=\rho \left(\theta \right)\sum _{\kappa =0}^s\widehat{P}\left(\kappa \right|\theta )\frac{1}{1+\theta }{\left(\frac{\theta }{1+\theta }\right)}^{s-\kappa }.\end{array}$$

(24)

Therefore given the value of the hidden variable $\theta$ and the initial number of links $\kappa$ the strength distribution is exponential. The overall strength distribution $P\left(s\right)$ of the model determining the probability that a random node has strength s is given by the integral of $P_{\theta }\left(s\right)$ over all possible value of the hidden variable $\theta$, i.e.,

$$\begin{array}{c}\hfill P\left(s\right)=\int d\theta \rho \left(\theta \right)\sum _{\kappa =0}^s\widehat{P}\left(\kappa \right|\theta )\frac{1}{1+\theta }{\left(\frac{\theta }{1+\theta }\right)}^{s-\kappa }.\end{array}$$

(25)

This result reveals that the strength distribution can be different from the distribution of hidden variables. For instance if all the hidden variables are the same, the strength distribution will still allow for fluctuations of the strengths. However for power-law hidden variable distributions

$$\begin{array}{c}\hfill \rho \left(\theta \right)\simeq C{\theta }^{-\gamma }\end{array}$$

(26)

the strength distribution has a power-law tail with the same exponent $\gamma$

$$\begin{array}{c}\hfill P\left(s\right)\simeq \widehat{C}{s}^{-\gamma }\end{array}$$

(27)

for $s\gg 1$. In fact, by inserting the explicit expression of $\widehat{P}\left(\kappa \right|\theta )$ and of $\rho \left(\theta \right)$ in Equation (25) we get

$$\begin{array}{c}\hfill P\left(s\right)=C\int d\theta \frac{{\theta }^{-\gamma }}{1+\theta }\sum _{\kappa =0}^s\frac{1}{\kappa !}\frac{{\theta }^s}{{(\theta +1)}^{s-k}}{e}^{-\theta }.\end{array}$$

(28)

For $s\gg 1$ we can approximate the sum over $\kappa$ with the infinite sum getting

$$\begin{array}{c}\hfill P\left(s\right)=C\int d\theta \frac{{\theta }^{-\gamma }}{1+\theta }{\left(\frac{\theta }{\theta +1}\right)}^s{e}^{-1}\simeq \widehat{C}{s}^{-\gamma }\end{array}$$

(29)

where the last expression is valid if $s\gg 1$. Therefore, although in general it is not true that the hidden variable distribution is the same as the strength distribution, in the case of power-law distributed hidden variables the strength distribution displays a power-law tail with the same exponent. Note that this is valid for power-law exponents in the range $\gamma \in (2,3]$ but also in the range $\gamma \in (1,2]$. Therefore in this case the hidden variables can be used to directly tune the strength distribution.

5.4. Connection Probability

In this section we derive the expression for the connection probability between any two nodes. Let us consider the probability $P(a_{ij}=1|{\theta }_i,{\theta }_j,{\kappa }_j,t_j>t_i)$ that node i is connected to node j, i.e., $a_{ij}=1$ given the hidden variables of node i and node j, their time of arrival with $t_j>t_i$ and the initial strength ${\kappa }_j$ of node j. This probability is one minus the probability that all of the initial links of node j do not connect to node i, i.e.,

$$\begin{array}{c}\hfill P(a_{ij}=1|{\theta }_i,{\theta }_j{\kappa }_j,t_i<t_j)=1-{\left(1-\frac{{\theta }_i}{{\sum }_r{\theta }_r}\right)}^{{\kappa }_j}.\end{array}$$

(30)

If we now average over the probability $\widehat{P}\left({\kappa }_j\right|{\theta }_j)$ we get the closed form expression

$$\begin{array}{c}\hfill P(a_{ij}=1|{\theta }_i,{\theta }_j,t_j,t_i<t_j)=\sum _{{\kappa }_j}P\left({\kappa }_j\right)\left[1-{\left(1-\frac{{\theta }_i}{{\sum }_{r=1}^j{\theta }_r}\right)}^{{\kappa }_j}\right]=\left[1-exp\left(-\frac{{\theta }_i{\theta }_j}{\langle \theta \rangle t_j}\right)\right],\end{array}$$

(31)

where we have assumed that the average of the hidden variables $〈\theta 〉$ is well defined. Therefore we have found that the connection probability between two nodes depends both on the hidden variables and on their time of arrival in the network. It follows that the model is not expected to be exchangeable, as this would require a connection probability independent of the time of arrival of the two nodes. However the fact that this connection probability does not only depend on the time of arrival of the nodes in the network (or the order in which they are sampled) can be a useful characteristic of a reliable statistical model.

5.5. Degree Distribution in the Sparse Regime

Here we derive the degree distribution of the model in the sparse regime, when we can assume that $p_{ij}\ll 1$. We will show that in this regime, each node has a Poisson degree distribution with an expected average degree ${\overline{k}}_i$ depending both on the value of its hidden variable and on the time of its arrival in the network.

The probability $P\left(k_i\right|{\theta }_i,t_i)$ that a node i arrived in the network at time $t_i$ and, having hidden variable ${\theta }_i$, has degree $k_i$ can be calculated starting from the connection probabilities $p_{ij}$ given by Equation (31). Let us indicate with ${\mathbf{a}}_i=\left\{a_{ij}\right|j\in \{1,2,\dots ,N\}\}$ the elements of the adjacency matrix in the i-th row indicating the connections of node i. Since node i is connected with each node j with probability $p_{ij}$ given by Equation (31), the probability $\mathcal{P}\left({\mathbf{a}}_i\right)$ is given by

$$\begin{array}{c}\hfill \mathcal{P}\left({\mathbf{a}}_i\right)=\prod _{j=1}^N\left[p_{ij}{a}_{ij}+(1-p_{ij})(1-a_{ij})\right].\end{array}$$

(32)

Using this result we can express the probability $P\left(k_i\right|{\theta }_i,t_i)$ that node i has degree $k_i$ as

$$\begin{array}{ccc}\hfill P\left(k_i\right|{\theta }_i,t_i)& =& \sum _{{\mathbf{a}}_i}{\mathcal{P}}_i\left({\mathbf{a}}_i\right)\delta \left(k_i,\sum _{j=1}^N{a}_{ij}\right)=\sum _{{\mathbf{a}}_i}\mathcal{P}\left({\mathbf{a}}_i\right)\int \frac{d\omega }{\sqrt{2\pi }}{e}^{-i\omega (k_i-{\sum }_{j=1}^N{a}_{ij})}\hfill \end{array}$$

(33)

where we have used the integral representation of the Kronecker delta $\delta (x,y)$. By performing the sum over all the elements of ${\mathbf{a}}_i$ we get

$$\begin{array}{ccc}\hfill P\left(k_i\right|{\theta }_i,t_i)& =& \int \frac{d\omega }{\sqrt{2\pi }}{e}^{-i\omega k_i}\prod _{j=1}^N\left[1-p_{ij}(1-e^{-i\omega })\right]=\int \frac{d\omega }{\sqrt{2\pi }}{e}^{F\left(\omega \right)}\hfill \end{array}$$

(34)

where

$$\begin{array}{cc}\hfill F\left(\omega \right)& =-i\omega k_i+\sum _{j=1}^{N}ln\left[1-p_{ij}(1-e^{-i\omega })\right].\end{array}$$

(35)

For $p_{ij}\ll 1$ we can approximate $F\left(\omega \right)$ with

$$\begin{array}{ccc}\hfill F\left(\omega \right)& =& i\omega k_i-\sum _{j=1}^N{p}_{ij}(1-e^{-i\omega })=i\omega k_i-{\overline{k}}_i(1-e^{-i\omega })\hfill \end{array}$$

(36)

where ${\overline{k}}_i$ is the expected degree of node i given by

$$\begin{array}{c}\hfill {\overline{k}}_i=\sum _{j=1}^N{p}_{ij}.\end{array}$$

(37)

Note here that since the connection probability $p_{ij}$ depends both on the hidden variables of the nodes i and j and on their arrival time in the network, it follows that also the expected degree ${\overline{k}}_i$ of node i will be both a function of the node’s hidden variable and its time of arrival in the network. Using Equations (34) and (36) we can derive the explicit expression for $P\left(k_i\right|{\theta }_i,t_i)$. In fact we have

$$\begin{array}{ccc}\hfill P\left(k_i\right|{\theta }_i,t_i)& \simeq & \int \frac{d\omega }{\sqrt{2\pi }}{e}^{i\omega k_i-{\overline{k}}_i(1-e^{-i\omega })}=\sum _{h=0}^{\infty }\frac{1}{h!}{\overline{k}}_i^h{e}^{-{\overline{k}}_i}\int \frac{d\omega }{\sqrt{2\pi }}{e}^{i\omega (k_i-h)},\hfill \end{array}$$

(38)

and by identifying the last integral with the Kronecker delta $\delta \left(h,k_i\right)$ we get the Poisson distribution

$$\begin{array}{c}\hfill P\left(k_i\right|{\theta }_i,t_i)=\frac{{\overline{k}}_i^{k_i}}{k_i!}{e}^{-{\overline{k}}_i}.\end{array}$$

(39)

Therefore the probability that node i, which arrived in the network at time $t_i$ with hidden variable ${\theta }_i$, has degree $k_i$ is given by the Poisson distribution with average ${\overline{k}}_i$ given by Equation (37). It follows that the degree distribution $P\left(k\right)$ of the network at time t is given by

$$\begin{array}{c}\hfill P\left(k\right)=\int d\theta \rho \left(\theta \right)\frac{1}{t}\sum _{t^{\prime }=1}^{t}P\left(k\right|\theta ,t^{\prime }).\end{array}$$

(40)

Note that for sufficiently sparse networks where each two connected nodes are typically connected by a link of weight one, the degree of a node can be identified with its strength

$$\begin{array}{c}\hfill k_i\simeq s_i.\end{array}$$

(41)

It follows that in this case the degree distribution can be approximated by the strength distribution and we have that if the hidden variables are power-law distributed with power-law $\gamma$ (as described in Equation (26)) then also the degree distribution has a power-law tail with the same exponent $\gamma$, i.e.,

$$\begin{array}{c}\hfill P\left(k\right)=\tilde{C}{k}^{-\gamma }\end{array}$$

(42)

for $k\gg 1$.

5.6. Random Permutation of the Node Sequence

Here we investigate whether the described network process can be related to the generation of uncorrelated networks. In this way we aim at reconciling the non-equilibrium growing nature of the network model, displaying projectivity, with the properties of exchangeable but not projective uncorrelated network models.

We observe that this expression depends both on the hidden variable and on the time of arrival of the nodes i and j in the network. However if we consider several realizations of the model in which the times of arrival of node i and node j are random, but the hidden variables are preserved, we observe that the probability that node i and node j are connected satisfies

$$\begin{array}{cc}\hfill P\left(a_{ij}=1|{\theta }_i,{\theta }_j,t=N\right)& =\frac{1}{N^2}{\int }_1^{N}d{t}_i{\int }_1^{N}d{t}_j{\int }_1^{N}d\tau \delta (\tau ,min(t_i,t_j))\left[1-exp\left(-\frac{{\theta }_i{\theta }_j}{\langle \theta \rangle \tau }\right)\right]\hfill \\ & =\frac{2}{N^2}{\int }_1^{N}d\tau \tau \left[1-exp\left(-\frac{{\theta }_i{\theta }_j}{\langle \theta \rangle \tau }\right)\right]=2\frac{{\theta }_i{\theta }_j}{\langle \theta \rangle N}+o\left(\frac{{\theta }_i{\theta }_j}{\langle \theta \rangle N}\right).\hfill \end{array}$$

(43)

Therefore if the network is sufficiently sparse, i.e.,

$$\begin{array}{c}\hfill \frac{{\theta }_i{\theta }_j}{〈\theta 〉N}\ll 1,\end{array}$$

(44)

we have that the expected degree $k_i\left({\theta }_i\right)$ of a random node i of hidden variable ${\theta }_i$ is given by

$$\begin{array}{c}\hfill {\tilde{k}}_i\left({\theta }_i\right)=2{\theta }_i,\end{array}$$

(45)

and the probability that a node with hidden variable ${\theta }_i$ is connected with a node with hidden variable ${\theta }_j$ independently of their time of arrival in the network, is given by the uncorrelated network marginal corresponding to the number of nodes in the sample, i.e.,

$$\begin{array}{c}\hfill {\tilde{p}}_{ij}=P\left(a_{ij}=1|{\theta }_i,{\theta }_j,t=N\right)=\frac{{\tilde{k}}_i\left({\theta }_i\right){\tilde{k}}_j\left({\theta }_j\right)}{\langle \tilde{k}\left(\theta \right)\rangle N}.\end{array}$$

(46)

Note that in this case if the sample increases in size and includes $N^{\prime }>N$ nodes, the probability that node i and node j are connected will satisfy

$$\begin{array}{c}\hfill {\tilde{p}}_{ij}^{\prime }=P\left(a_{ij}=1|{\theta }_i,{\theta }_j,t=N^{\prime }\right)=\frac{{\tilde{k}}_i\left({\theta }_i\right){\tilde{k}}_j\left({\theta }_j\right)}{\langle \tilde{k}\left(\theta \right)\rangle N^{\prime }}.\end{array}$$

(47)

In this case the network process induces a probability ${\tilde{p}}_{ij}$ that depends on the network size N and at the same time enforces the sparseness of the network. In fact the expected degrees $\left\{{\tilde{k}}_i\right\}$ of the nodes are only determined the the hidden variable and are independent on the network size.

5.7. Entropy of the Network Model

In order to compare our model with hidden variable distribution $\rho \left(\theta \right)$ to an uncorrelated network ensemble in which the expected degrees are ${\tilde{k}}_i=2{\theta }_i$, in this section we use information theory tools. Specifically we will compare the entropy of the two ensembles. The entropy of a network model or of a network ensemble [14,15,16,17,40] is a fundamental tool to evaluate the information content in the network model. It indicates the logarithm of the typical number of networks generated by the ensemble and as such evaluates the complexity of the model and can be used in inference problems [40]. Since for our network model the connection probability $p_{ij}$ of any two pair of nodes is i and j is given by Equation (31), the entropy of the model is given by

$$\begin{array}{c}\hfill S=-\sum _{i<j}\left[p_{ij}ln{p}_{ij}+(1-p_{ij})ln(1-p_{ij})\right].\end{array}$$

(48)

where in the sparse regime we can approximate $p_{ij}$ with $t_j>t_i$ as

$$\begin{array}{c}\hfill p_{ij}\simeq \frac{{\theta }_i{\theta }_j}{\langle \theta \rangle t_j}.\end{array}$$

(49)

Similarly for the uncorrelated network ensemble with connection probability ${\tilde{p}}_{ij}$ the entropy is given by

$$\begin{array}{c}\hfill \tilde{S}=-\sum _{i<j}\left[{\tilde{p}}_{ij}ln{\tilde{p}}_{ij}+(1-{\tilde{p}}_{ij})ln(1-{\tilde{p}}_{ij})\right].\end{array}$$

(50)

In order to compare these two entropies we use the explicit expression for the connection probability ${\tilde{p}}_{ij}$ when we put ${\tilde{k}}_i\left({\theta }_i\right)=2{\theta }_i$ which reads

$$\begin{array}{c}\hfill {\tilde{p}}_{ij}=2\frac{{\theta }_i{\theta }_j}{\langle \theta \rangle N}.\end{array}$$

(51)

By performing a straightforward calculation we find that S is given, up to the linear terms in N, by

$$\begin{array}{c}\hfill S=\langle \theta \rangle ln(N!)-2N\langle \theta ln\theta \rangle +N\langle \theta \rangle ln\langle \theta \rangle +\langle \theta \rangle N\end{array}$$

(52)

and that the entropy S of our model is smaller than the entropy of the uncorrelated network ensemble. In fact, S differs from $\tilde{S}$ only by

$$\begin{array}{c}\hfill \Delta S=S-\tilde{S}\simeq \langle \theta \rangle ln\left(\frac{N!2^N}{N^N}\right)\simeq \langle \theta \rangle Nln\left(\frac{2}{e}\right).\end{array}$$

(53)

The entropy difference $\Delta S$ quantifies the information loss when the proposed network process is approximated with its corresponding uncorrelated network model. We observe here that the uncorrelated network model is obtained when the causal construction of the original network model is disregarded and the only retained information is the probability ${\tilde{p}}_{ij}$ that two nodes of hidden variables ${\theta }_i$ and ${\theta }_j$ are connected regardless of their time of arrival in the network. Therefore $\Delta S$ captures the loss of information when the causal nature of the original model is disregarded. Interestingly in the large network limit $N\gg 1$, $|\Delta S|$ is low when compared to S revealing the proximity between the two models. Additionally $\Delta S$ is only dependent on $〈\theta 〉$ indicating that the information loss from one model to the other is independent of the particular distribution of the hidden variables $\rho \left(\theta \right)$ as long as $〈\theta 〉$ is kept constant.

6. Statistical Testing of the Model

In order to study the utility of the proposed model as a null model for sampled data we consider three power-law networks: the arxiv hep-ph (high energy physics phenomenology) citation network [45,46], the Berkeley–Stanford web network [47] and the Notre Dame web network [48] of network sizes $N=$ 34,546, $N=$ 685,230, $N=$ 325,000 respectively. All data are freely available on the Stanford Network Analysis Project webpage. To each node of the network we assign a different label $i\in 1,2,\dots ,N$ according to a random permutation of the indices from 1 up to N. We then assign to each node i of the network a hidden variable

$$\begin{array}{c}\hfill {\theta }_i=\frac{1}{2}{k}_i,\end{array}$$

(54)

where $k_i$ is the observed degree of node i in the dataset. Given our random node labelling and the hidden variables ${\left\{{\theta }_i\right\}}_{i=1,2,\dots ,N}$ we have generated a random network according to the proposed network process. Interestingly the proposed model preserves to a large extent the degree distribution (see comparison of the real degree distribution with the one generated by the model in Figure 1). Additionally these results are quite stable if we consider a model generated only by adding a subsample of randomly chosen nodes, showing that the model preserves the degree distribution under random sub-sampling of the nodes (see Figure 1).

The generated model however is to be considered mostly as uncorrelated. In fact if we compare the degree correlations of the real datasets with the degree correlations of the network generated by the model we observe that the model deviates from the real data and displays very weak/marginal degree correlations (see Figure 2). In fact from the results obtained for the three studied network datasets it seems that the model is able to better reproduce weakly assortative behaviour than strongly disassortative behaviour. In future, modifications of the proposed model could be envisaged to capture also the degree correlations of real datasets.

7. Conclusions

In conclusion, we have given a wide overview of the desirability of the projectivity and exchangeability properties in good statistical models and we have emphasized the difficulty in combining these properties with the sparseness of the network. While this problem is a widely discussed subject in statistics of networks and graph theory, here we have proposed a model that provides a trade-off solution. Our model describes a network process in which nodes and links are subsequently added according to a probability dependent on some hidden variables associated to the nodes. As long as the hidden variables are power-law distributed this model generates a scale-free network with the same exponent. This model is projective but not exchangeable. However, the expected probability that two nodes are connected when one considers a random permutation of the sequence in which nodes are added to the network reduces to the expression valid for the marginal of an uncorrelated exchangeable network with the same expected degrees (given by the double of the hidden variables) provided the network is sufficiently sparse. Finally, we tested this model as a statistical null model for scale-free sparse real networks, showing that it can reproduce the degree distribution (but not degree correlations) also if a partial subset of the data is considered.