Hidden Drivers in Ecological Networks: Detecting Possible Latent Forcing from Time-Series Data

Abstract

Ecological interaction networks are often influenced by unobserved upstream forcing, which can make an observed link between two variables appear causal even when it primarily reflects mediation within a deeper causal chain. This challenges the interpretation of trophic control, the identification of dominant ecological drivers, and the design of effective monitoring strategies. Here, we introduce detecting pairwise effect (DPE), a data-driven method based on empirical dynamic modeling and multivariable state-space reconstruction, to assess whether an observed directional link is more consistent with a direct ecological driver or with a mediated effect shaped by latent upstream forcing. Using benchmark dynamical system motifs, we show that DPE can identify conditions under which a strong upstream driver acting on the putative cause may distort pairwise causal interpretation. We further demonstrate its ecological relevance using real-world time series, where DPE reveals diagnostic signals consistent with hidden trophic forcing in a plankton food-chain dataset and possible latent external forcing in soil-temperature depth dynamics. Overall, DPE provides a diagnostic signal that is consistent with the presence of latent upstream influence, thereby improving the interpretability of inferred ecological causal relationships.

1. Introduction

Ecological interaction networks are shaped by coupled biotic and abiotic processes acting across space and time. In practice, however, ecological monitoring is inevitably incomplete: key environmental drivers, missing taxa, or unrecorded management actions can act as unobserved upstream forcing, leaving only partial time-series observations of the system. This creates a recurrent inferential challenge: an apparent causal relationship between two measured variables may be interpreted as a direct ecological driver, even when it largely reflects mediation within a deeper causal chain [1]. Such ambiguity can limit our ability to interpret trophic control, identify dominant ecological drivers, and design effective monitoring strategies.

To interrogate dynamic interactions from time series, ecologists have increasingly adopted causality inference tools [2]. These include Granger causality [3,4,5], dynamical Bayesian inference [6,7,8,9,10,11], convergent cross mapping (CCM) and related empirical dynamic modeling (EDM) approaches [12,13,14,15,16,17,18], as well as neural network-based methods [19]. These methods have proven valuable for detecting directional dependencies and information flow among observed variables. Nevertheless, real-world ecosystems are inherently complex [20,21,22,23,24], typically comprising large networks of interacting components, and many relevant drivers may remain unknown, unmeasured, or difficult to observe. As a result, existing approaches—largely focused on relationships among measured variables—often provide limited guidance on a critical ecological question: when an observable directional link X → Y is detected, does X represent a direct driver of Y, or is the link primarily a mediated effect shaped by unobserved upstream forcing? If the latter holds, interventions or monitoring centered on X may target a mediator rather than the more proximal leverage point, and mechanistic interpretation of the inferred network can be misleading.

Addressing this gap requires diagnostic tools that can extract evidence of latent upstream structure using only the available time series. To our knowledge, systematically detecting latent causation solely from observable time series in dynamical ecological systems has not yet been fully explored [25,26,27]. In this paper, we propose a data-driven method termed detecting pairwise effect (DPE) to infer latent upstream forcing from two interconnected time series X and Y when an observable directional relationship X → Y has been established. The central idea of DPE is to detect a characteristic signature—termed the pairwise effect—that indicates the observed link X → Y is unlikely to be directly fundamental, but instead embeds a deeper latent chain Z → X → Y, in which X primarily acts as a mediator rather than the root cause of Y.

We validate DPE in simulated benchmark ecosystem motifs with controlled causal structures and then demonstrate its ecological relevance using real-world time series networks. Under appropriate conditions, DPE provides additional insight into whether an inferred link is more consistent with a direct driver or a mediated relationship shaped by latent forcing. By providing an interpretable screening step prior to targeted data collection, DPE complements existing causality inference methods and helps improve the interpretability of causal relationships in complex ecological networks.

The contribution of this study is threefold. First, we propose DPE as a diagnostic framework for assessing whether an observed directional link is more consistent with a direct driver or with a mediated effect shaped by latent upstream forcing. Second, we provide a mechanistic interpretation of how the pairwise effect can arise in systems with hidden upstream structures. Third, we evaluate the method using benchmark dynamical systems and representative ecological and environmental time-series datasets.

2. Materials and Methods

2.1. Detecting Pairwise Effect

Convergent cross mapping (CCM) is a widely used method for causal inference in ecological systems [12]. Based on Takens’ embedding theorem [28], CCM reconstructs shadow manifolds from time series data and exploits nearest-neighbor relationships in the reconstructed state-space to assess causality via predictive skill $\rho$ [29]. The key idea is that if X causally influences Y (i.e., X → Y), then information about X is embedded in the time series of Y [12]. Consequently, the effect variable can be used to predict the cause. For a pair of time series X and Y, if Y exhibits high predictive skill in reconstructing X, that is, the predicted series $\widehat{X}\mid M_Y$ closely matches the observed X, this indicates causality in the direction X → Y.

In the original CCM formulation, only the time series of Y is used to reconstruct the shadow manifold $M_Y$. In addition, it has been shown that a time series can accurately predict itself, yielding a high self-prediction skill $\rho (Y,\widehat{Y}|M_Y)$ that typically approaches 1 [30] (see Section 2.3 for details). A similar property holds for $\rho (X,\widehat{X}|M_X)$.

In this study, we propose a data-driven method termed detecting pairwise effect (DPE), which extends CCM by introducing multivariable embedding [29,31,32] into the reconstruction process. Specifically, when reconstructing the manifold of Y, we incorporate not only Y itself but also information from X, resulting in a multivariate shadow manifold $M_{Y,X}$. When predicting Y based on $M_{Y,X}$, the inclusion of X may act as an additional perturbation, reducing the purity of the reconstructed manifold. As a result, the predictive skill typically decreases, i.e.,

$$\rho \left(Y,\widehat{Y}|M_{Y,X}\right)<\rho \left(Y,\widehat{Y}|M_Y\right)$$

In a simple dynamical system involving two variables X and Y, if there exists a unidirectional causal relationship X → Y, then Y, as the effect, contains information about X, whereas X, as the cause, contains limited information about Y. Consequently, under multivariable embedding, we expect:

$$\rho \left(X,\widehat{X}|M_{X,Y}\right)>\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$$

However, real-world dynamical systems are typically more complex and often involve additional latent variables beyond X and Y, which we collectively denote as Z. In the DPE framework, these variables may be unobserved or entirely unknown. If the underlying causal structure includes a chain Z → X → Y, then the reconstruction of X from Y can be substantially affected by the upstream influence Z → X. As the strength of this causal link increases, the predictive skill $\rho (X,\widehat{X}\mid M_{X,Y})$ may decrease significantly. We hypothesize that when the influence of Z → X exceeds a certain threshold, the inequality may reverse, leading to:

$$\rho \left(X,\widehat{X}|M_{X,Y}\right)<\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$$

We refer to this counterintuitive phenomenon as the pairwise effect.

In summary, the central idea of DPE is to compare the predictive skills $\rho \left(X,\widehat{X}|M_{X,Y}\right)$ and $\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$. Under an approximately unidirectional causal relationship X → Y, the occurrence of $\rho \left(X,\widehat{X}|M_{X,Y}\right)<\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$ indicates the presence of the pairwise effect. This, in turn, suggests the existence of a deeper causal structure (e.g., Z → X → Y), in which X functions primarily as a mediator rather than the more proximal causal driver of Y.

2.2. Normal Reconstruction

In CCM theory, given a pair of time series X and Y, if Y has a high forecasting skill for predicting X, this indicates a causal relationship in the direction of X → Y. In the original CCM framework, the time series of Y is used to reconstruct the shadow manifold $M_Y$. The parameter of embedding dimension can be computationally determined by the simplex projection method [31,33]. Given an embedding dimension $E$, the reconstructed manifold $M_Y$ consists of a series of $E$-dimensional state vectors ${\mathbf{Y}}_t$, where each state vector is defined as:

$${\mathbf{Y}}_t=\left(y_t,y_{t-1},\dots ,y_{t-\left(E-1\right)}\right)$$

(1)

The time delay $\tau$ is fixed to 1 in this study for all experiments unless otherwise specified. Thus, the reconstructed shadow manifold is written as

$$M_Y=\left\{{\mathbf{Y}}_t\right\}$$

The manifold $M_Y$ can be used to predict the historical data of X. Specifically, the prediction of $x_t$ at time $t$ based $M_Y$ is denoted as $\widehat{x_t}|M_Y$. To estimate $x_t$ from $M_Y$, we first identify the $E+1$ nearest neighbors of the target point ${\mathbf{Y}}_t$, corresponding to time indices $t_1,\dots ,t_{E+1}$. Let $d_i$ denote the Euclidean distance between target point and its $i$-th nearest neighbor, and let

$$d_{\mathrm{m}\mathrm{i}\mathrm{n}}=\underset{j}{\min } d_j$$

(2)

be the minimum of these distances. Euclidean distance is used to identify nearest neighbors in all embeddings. The weight assigned to the $i$-th neighbor is defined as:

$$W_{\left(i\right)}={\displaystyle \frac{exp(-d_i/d_{min})}{\sum _{j=1}^{E+1}exp(-d_j/d_{min})}}$$

(3)

Accordingly, the prediction of $x_t$ based on $M_Y$ is given by:

$$\widehat{x_t}|M_y=\sum _{i=1}^{E+1}{W}_i{x}_{t\left(i\right)}$$

(4)

The predictive skill of CCM, denoted as $\rho (X,\widehat{X}|M_y)$, can be quantified by the Pearson correlation coefficient between real $X$ and predicted $\widehat{X}$. If the causal direction is X → Y, the $\rho (X,\widehat{X}|M_y)$ will generally increase and eventually converge to a numerical value as the library length $L$ increases, although there may be small fluctuations in the numerical value. In this study, we use the value obtained at the largest library length $L$ as the predictive skill. By contrast, under unidirectional causality, $Y$ cannot be reconstructed from the time series of $X$ with comparable accuracy, and therefore:

$$\rho \left(Y,\widehat{Y}|M_X\right)<\rho \left(X,\widehat{X}|M_Y\right).$$

2.3. Computing Process

As indicated by previous research [30], the time series of Y can also be used to predict itself, typically yielding a very high predictive skill, denoted by $\rho (Y,\widehat{Y}|M_Y)$. Similarly, the time series of $X$ can be used to predict itself, with predictive skill denoted by $\rho (X,\widehat{X}|M_X)$. In the DPE method, multivariable embedding [31] is introduced into the reconstruction of the shadow manifold. Specifically, when reconstructing the manifold of Y, we incorporate information from $\mathrm{X}$ into the state vector. The simplest multivariable state vector is defined as:

$${(\mathbf{Y}, X)}_t=\left(y_t,y_{t-1},\dots ,y_{t-\left(E-1\right)},x_t\right)$$

(5)

The corresponding multivariate shadow manifold is denoted by:

$$M_{Y, X}=\left\{\right({\mathbf{Y}, X)}_t\}$$

More generally, the form of multivariable embedding is not unique. For example, $\left(x_t{,y}_t,y_{t-1},\dots ,y_{t-\left(E-1\right)}\right)$ and $\left(y_t,y_{t-1},\dots ,y_{t-\left(E-1\right)},x_t,x_{t-1}\right)$ are also valid alternatives. In this study, we use the form given in Equation (5) for consistency and clarity. This is chosen as the simplest extension of the univariate embedding, incorporating only the contemporaneous value of the additional variable to avoid unnecessary increases in dimensionality and potential overfitting.

Prior to embedding, all time series are normalized to zero mean and unit variance. This ensures that variables with different scales contribute comparably to the Euclidean distance calculation and prevents any single variable from dominating the nearest-neighbor search and weight assignment.

Using $M_{Y,X}$, the predictive skill for reconstructing $Y$ is quantified by the Pearson correlation coefficient between the observed series $Y$ and the predicted series $\widehat{Y}\mid M_{Y,X}$, denoted by $\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$. In this paper, the predictive skill $\rho$ is defined as the Pearson correlation coefficient between the observed series and the predicted series obtained via cross mapping. Predictions are generated using a leave-one-out strategy, where the target point is excluded from the library when identifying nearest neighbors, ensuring that the prediction is out-of-sample. The predictive skill is evaluated as a function of library size $L$, and the value at the largest $L$ is used for comparison.

Because the introduced variable $x_t$ does not directly improve self-reconstruction of $Y$ in the pairwise setting and may act as an additional perturbation, we typically observe

$$\rho \left(Y,\widehat{Y}|M_{Y,X}\right)<\rho \left(Y,\widehat{Y}|M_Y\right)$$

Similarly, when reconstructing the manifold of X with the aid of Y, we define:

$$({\mathbf{X}, Y)}_t=\left(x_t,x_{t-1},\dots ,x_{t-\left(E-1\right)},y_t\right)$$

with the corresponding multivariate shadow manifold:

$$M_{X, Y}=\left\{\right({\mathbf{X}, Y)}_t\}$$

The associated predictive skill is denoted by $\rho \left(X,\widehat{X}|M_{X,Y}\right)$.

For a dynamical system with two observed variables X and Y, if there is a unidirectional causal relationship X → Y, then Y contains information about X, whereas X does not contain comparable information about Y. Consequently, under multivariable embedding, we expect:

$$\rho \left(X,\widehat{X}|M_{X,Y}\right)>\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$$

However, real dynamical systems are usually more complex and often involve additional latent variables beyond X and Y. In this paper, such latent variables are collectively denoted by Z. If the true causal structure contains a latent upstream chain Z → X → Y, then the ability to reconstruct X from information contained in Y may be substantially affected by the hidden driver Z. In particular, as the causal strength of Z → X increases, the predictive skill $\rho \left(X,\widehat{X}|M_{X,Y}\right)$ may decrease markedly.

Importantly, the presence of Z does not invalidate the conventional CCM inference for the observed pair X → Y. That is, standard CCM can still correctly indicate the directional asymmetry:

$$\rho \left(Y,\widehat{Y}|M_x\right)<\rho \left(X,\widehat{X}|M_y\right)$$

Nevertheless, when the causal effect of Z → X is sufficiently strong, the multivariable embedding comparison may yield:

$$\rho \left(X,\widehat{X}|M_{X,Y}\right)<\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$$

which appears counterintuitive under the simple pairwise interpretation. In this paper, we refer to this phenomenon as the pairwise effect.

In addition, if the latent variable Z also exerts a strong influence on Y (i.e., Z → Y), then $\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$ may also be substantially affected, in which case the pairwise effect may weaken or even disappear. Therefore, when analyzing a pair of observed variables, the presence of the pairwise effect suggests that unobserved factors in the system exert a strong influence on X while having limited influence on Y.

For convenience, we define the DPE contrast as

$${\mathit{\Delta }}_{DPE}=\rho \left(X,\widehat{X}|M_{X,Y}\right)-\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$$

Under ordinary pairwise causality X → Y, we expect ${\mathit{\Delta }}_{DPE}>0$. When a strong latent upstream driver affects X, ${\mathit{\Delta }}_{DPE}$ will become negative ${\mathit{\Delta }}_{DPE}<0$, indicating the occurrence of the pairwise effect.

2.4. Mechanistic Interpretation

To better understand the mechanism underlying the pairwise effect, we provide a conceptual and analytical explanation of how an upstream latent driver alters the asymmetry of predictive skill in multivariable embeddings.

Unlike conventional machine learning frameworks, CCM does not employ an explicit train/test split or rolling validation. Instead, predictive performance is evaluated within the library-based framework, where predictions are generated using nearest neighbors selected from a library of size L. By progressively increasing L, the convergence of predictive skill reflects the strength of the underlying causal signal.

In CCM, the ability to reconstruct a variable from another relies on the extent to which information about the former is redundantly encoded in the latter through the underlying dynamical system. When a causal relationship X → Y exists, the dynamics of Y are partially driven by X, and therefore the reconstructed manifold $M_Y$ contains information about X. This redundancy enables accurate recovery of X from Y-based embeddings.

However, consider a system in which an additional latent variable Z influences X but has little or no direct effect on Y, forming a causal chain Z → X → Y. In this case, the variability of X can be decomposed into two components: (i) the portion transmitted to $Y$ through the link X → Y, and (ii) the portion induced by Z that is not fully propagated to Y. As the strength of Z → X increases, a larger fraction of the variation in X originates from Z, while only a limited part of this variation is reflected in Y. Consequently, the redundancy of information about X in the reconstructed manifold $M_Y$ is reduced.

Under multivariable embedding, when constructing $M_{X,Y}$ and $M_{Y,X}$, this asymmetry becomes more pronounced. The reconstruction of X from $M_{X,Y}$ relies on the information about X that can be inferred from Y. When the influence of Z is strong, the component of X driven by Z is not sufficiently encoded in Y, leading to a degradation in predictive skill $\rho (X,\widehat{X}\mid M_{X,Y})$. In contrast, the reconstruction of Y from $M_{Y,X}$ is less affected, since Y remains primarily driven by X, and the additional variability introduced by Z does not directly disrupt the mapping from X to Y. As a result, the inequality $\rho \left(X,\widehat{X}|M_{X,Y}\right)>\rho \left(Y,\widehat{Y}|M_{Y,X}\right)$ may be weakened and eventually reversed as the strength of Z → X, giving rise to the pairwise effect.

To further illustrate this mechanism, consider a simplified linear dynamical system:

$$\begin{gathered} X_t=a{X}_{t-1}+b{Z}_{t-1}+{\epsilon }_t, \\ {Y}_t=c{Y}_{t-1}+d{X}_{t-1}+{\eta }_t, \end{gathered}$$

where Z is an exogenous latent driver, and ${\epsilon }_t,{\eta }_t$ are noise terms. In this system, X is influenced by both its own past and the upstream variable Z, while Y is driven by X.

When $b$ is small, most of the variability in X is inherited from its own dynamics and is partially transmitted to Y through the coefficient $d$. In this case, Y retains substantial information about X, and the predictive skill $\rho (X,\widehat{X}\mid M_{X,Y})$ remains high.

As $b$ increases, a larger portion of the variability in X is driven by Z. If the influence of Z is not proportionally transmitted to Y, the component of X that originates from Z becomes increasingly unpredictable from Y. This reduces the effective information overlap between X and Y, leading to a decline in $\rho (X,\widehat{X}\mid M_{X,Y})$. In contrast, the prediction of Y from $M_{Y,X}$ remains relatively stable, as Y continues to depend directly on X.

Therefore, when the contribution of Z to the variance of X exceeds that of the transmitted component through X → Y, the inequality between predictive skills may reverse, yielding

$$\rho (X,\widehat{X}\mid M_{X,Y})<\rho (Y,\widehat{Y}\mid M_{Y,X}).$$

This provides a mechanistic explanation for the emergence of the pairwise effect in systems with latent upstream drivers.

2.5. Uncertainty Quantification

To quantify uncertainty in the DPE contrast, we define

$${\mathit{\Delta }}_{DPE}=\rho (X,\widehat{X}\mid M_{X,Y})-\rho (Y,\widehat{Y}\mid M_{Y,X}).$$

For each simulation and empirical analysis, uncertainty is assessed using a block bootstrap procedure to preserve temporal dependence in the observed series. Specifically, the original time series are resampled using contiguous blocks of length $b$, and each bootstrap replicate is used to recompute the predictive skills $\rho (X,\widehat{X}\mid M_{X,Y})$, $\rho (Y,\widehat{Y}\mid M_{Y,X})$, and the resulting ${\mathit{\Delta }}_{DPE}$. We use $B=1000$ bootstrap replicates in all analyses. The 95% confidence interval of ${\mathit{\Delta }}_{DPE}$ is estimated using the percentile bootstrap. A pairwise effect is considered statistically supported when the upper bound of the 95% confidence interval is below zero; if the interval overlaps zero, the result is interpreted as inconclusive rather than as a binary decision. A one-sided bootstrap p-value is computed as the proportion of bootstrap replicates with ${\mathit{\Delta }}_{DPE}\ge 0$.

3. Results

3.1. Detect Latent Causation in Benchmark System

We first use the following simulated benchmark system of three interacting variables to validate our DPE method, as shown in Equation (6). Different values of the coupling parameters ${\beta }_{xy}$, ${\beta }_{xz}$, ${\beta }_{yx}$, ${\beta }_{yz}$, ${\beta }_{zx}$, and ${\beta }_{zy}$ can lead to various interacting modes and dynamic systems. Although this dynamic system includes the time series of X, Y and Z, only X and Y are involved in the causality detection computation. Z is considered unobservable and therefore excluded from the inference process, limited to result validation.

$$\left\{\begin{array}{c}{x}_t=x_{t-1}\left(3.60-3.6{0x}_{t-1}-{\beta }_{xy}{y}_{t-1}-{\beta }_{xz}{z}_{t-1}\right)\\ y_t=y_{t-1}\left(3.72-3.72y_{t-1}-{\beta }_{yx}{x}_{t-1}-{\beta }_{yz}{z}_{t-1}\right)\\ z_t=z_{t-1}\left(3.68-3.68z_{t-1}-{\beta }_{zx}{x}_{t-1}-{\beta }_{zy}{y}_{t-1}\right)\end{array}\right.$$

(6)

We formulated seven different causal network structures (Figure 1) based on Equation (6), encompassing various basic causal scenarios among three variables. Among these scenarios, a unidirectional strong causal relation X → Y (${\beta }_{yx}$ = 0.35) is fixed. The reconstruction parameters in manifold reconstruction for this benchmark system are E = 3, τ = 1 and L = 1500, aligning with common practices in empirical dynamic modeling analysis [34].

The experimental results are shown in

Table 1. Experimental results for scenarios A–G.

Scenario	$\mathit{\rho }\left(\mathit{X},\widehat{\mathit{X}}|{\mathit{M}}_{\mathit{x}+\mathit{y}}\right)$	$\mathit{\rho }\left(\mathit{Y},\widehat{\mathit{Y}}|{\mathit{M}}_{\mathit{y}+\mathit{x}}\right)$	${\mathit{\Delta }}_{\mathit{D}\mathit{P}\mathit{E}}$	95% CI $\mathbf{of} {\mathit{\Delta }}_{\mathit{D}\mathit{P}\mathit{E}}$	p-Value	Pairwise Effect
A	0.999909	0.999725	>0	[0.00011, 0.00025]	<0.001	-
B	0.999255	0.999511	<0	[−0.00034, −0.00018]	<0.001	★
C	0.999863	0.999743	>0	[0.00034, 0.00018]	<0.001	-
D	0.999423	0.999097	>0	[0.00024, 0.00041]	0.002	-
E	0.999226	0.999489	<0	[−0.00035, −0.00019]	<0.001	★
F	0.999842	0.999158	>0	[0.00057, 0.00080]	<0.001	-
G	0.999889	0.999752	>0	[0.00007, 0.00021]	0.001	-

“★” indicates the presence of pairwise effect and “-” indicates the absence of pairwise effect.

. It can be seen that the DPE method successfully detects pairwise-effect signals consistent with latent external influence (i.e., Z) based on pairwise effect, using only the data from X and Y. Noticeably, the pairwise effect is observed when the latent external variable Z significantly influences X (as observed in scenarios B and E in Figure 1) in the X → Y direction. Specifically, in scenarios where the external factor Z has no impact on either X or Y, $\rho \left(X,\widehat{X}|M_{x+y}\right)$ is higher (as observed in scenario A), indicating the absence of the pairwise effect. Conversely, in cases where the external variable Z strongly influences X while having weak impact on Y, $\rho \left(Y,\widehat{Y}|M_{y+x}\right)$ is higher, leading to the presence of pairwise effect (as observed in scenarios B and E). However, when Z had strong effects on both X and Y, the pairwise effect is absent (as observed in scenario D). Moreover, when the influence of Z on X was weak, regardless of Z’s impact on Y, the pairwise effect was always absent (as observed in scenarios C, F, and G).

Here, the DPE interpretation is based on the sign of ${\mathit{\Delta }}_{DPE}$ together with its bootstrap confidence interval. A pairwise effect is considered statistically supported only when the 95% confidence interval of ${\Delta }_{DPE}$ lies entirely below zero; otherwise, the result is treated as no supported pairwise effect or as inconclusive. We validated our proposed hypothesis in the aforementioned benchmark system, where the pairwise effect only appears in specific causal modes. To further investigate the application of DPE in complex scenarios, we conducted a sensitivity analysis on the strength of causal links.

Firstly, we studied the causal chain Z → X → Y, as depicted in scenarios B and C. We maintained ${\beta }_{yx}$ fixed at 0.35 and incrementally increased the value of ${\beta }_{xz}$, reflecting the causal strength in the Z → X direction, to observe the emergence of the pairwise effect. The experimental results are presented in Figure 2A. In the figure, when the value of the blue curve surpasses that of the red curve, it indicates the presence of pairwise effect. As observed from the figure, as the causal strength ${\beta }_{xz}$ in the Z → X direction increases, the red curve representing $\rho \left(X,\widehat{X}|M_{x+y}\right)$ demonstrates a significant downward trend. This decline implies a reduction in the information on X contained within Y. This reduction occurs due to the growing causal strength of Z → X, which may disrupt the information prediction in X → Y direction. Moreover, the blue curve representing $\left(Y,\widehat{Y}|M_{y+x}\right)$ remains basically stable. This stability is attributed to the fact that X, as the cause of Y, inherently contained minimal information about Y, which is the consequence of X. Thus, the prediction process remains less affected by the Z → X causal chain link. To conclude, when ${\beta }_{xz}$ is relatively low, the pairwise effect is not observed. However, as ${\beta }_{xz}$ gradually increases and surpasses a certain threshold (in this case, 0.025 in Figure 2A), the pairwise effect becomes apparent. The experimental findings aligned consistently with the theoretical analysis.

To quantify uncertainty in the DPE contrast, we further estimated 95% bootstrap confidence intervals for ${\Delta }_{DPE}$ in each benchmark scenario (Table 1). The results show that scenarios B and E yield negative ${\Delta }_{DPE}$ values, and in both cases the corresponding 95% confidence intervals lie entirely below zero, providing statistical support for the presence of the pairwise effect. By contrast, in scenarios A, C, D, F, and G, ${\Delta }_{DPE}$ remains positive and the corresponding confidence intervals remain above zero, indicating no supported pairwise effect under the current decision criterion. These results strengthen the interpretation that the pairwise effect emerges only under specific latent-causal configurations, rather than as a generic consequence of multivariable embedding. Therefore, the benchmark results support the proposed hypothesis that the pairwise effect is selectively induced when the latent upstream variable exerts a sufficiently strong influence on X while having limited influence on Y.

Secondly, we studied the causal network of Z → X → Y and Z → Y as represented by scenarios D and E, with fixed ${\beta }_{yx}$ at 0.35 and ${\beta }_{xz}$ at 0.20. We systematically adjusted the value of ${\beta }_{yz}$, reflecting the causal strength in the Z → Y direction. This continuous adjustment allowed us to observe whether the pairwise effect emerged in the results (Figure 2B). It is evident that when ${\beta }_{yz}$ was relatively low, indicating a weak causal strength in the Z → Y direction, the pairwise effect became evident. However, as ${\beta }_{yz}$ progressively increased, the pairwise effect progressively disappeared. In a comprehensive perspective, the blue curve, representing $\left(Y,\widehat{Y}|M_{y+x}\right)$, displayed a descending trend, while the red curve, representing $\rho \left(X,\widehat{X}|M_{x+y}\right)$, exhibited an ascending trend. This transition of the pairwise effect from presence to absence underscores that when Z exerts a strong causal influence on both X and Y simultaneously, acting as a confounder in the causal link X → Y, the pairwise effect cannot be observed.

To evaluate the robustness of DPE under realistic data conditions, we conducted a simple stress-test analysis using two representative benchmark scenarios. Scenario A represents a case without latent upstream forcing and is therefore expected not to exhibit the pairwise effect, whereas Scenario B represents a case with a latent upstream chain Z → X → Y and is expected to exhibit the pairwise effect.

For each scenario, we generated simulated time series using the same benchmark system described in Equation (6). We then examined two factors that commonly affect real-world time series analysis: observation noise and time-series length.

For the noise experiment, Gaussian observation noise was added to X and Y:

$$X_t^{obs}=X_t+\sigma {ϵ}_t,Y_t^{obs}=Y_t+\sigma {\eta }_t$$

(7)

where ${ϵ}_t$ and ${\eta }_t$ are independent standard normal variables and $\sigma$ denotes the noise level. We considered $\sigma$ = 0, 0.01, 0.05, and 0.10. For the sample-length experiment, the DPE analysis was repeated using truncated series with library lengths L = 1500, 1000, 500, and 250.

For each setting, we computed the DPE contrast ${\mathit{\Delta }}_{DPE}$, its 95% bootstrap confidence interval, and the corresponding decision outcome. A pairwise effect was considered statistically supported when the 95% confidence interval of ${\mathit{\Delta }}_{DPE}$ lay entirely below zero.

As shown in

Table 2. Robustness to observation noise.

Scenario	Noise Level σ	${\mathit{\Delta }}_{\mathit{D}\mathit{P}\mathit{E}}$	$95\% \mathbf{CI} \mathbf{of} {\mathit{\Delta }}_{\mathit{D}\mathit{P}\mathit{E}}$	Pairwise Effect Interpretation
A	0.00	0.00018	[0.00011, 0.00025]	no supported
A	0.01	0.00015	[0.00007, 0.00023]	no supported
A	0.05	0.00008	[−0.00001, 0.00017]	no supported
A	0.10	0.00003	[−0.00010, 0.00015]	inconclusive
B	0.00	−0.00026	[−0.00034, −0.00018]	supported
B	0.01	−0.00022	[−0.00031, −0.00013]	supported
B	0.05	−0.00011	[−0.00031, −0.00013]	supported
B	0.10	−0.00005	[−0.00017, 0.00006]	inconclusive

, DPE remained informative under low to moderate observation noise. In Scenario A, which does not contain latent upstream forcing, ${\Delta }_{DPE}$ remained positive across all noise settings, indicating no supported pairwise effect. However, as the noise level increased, the absolute magnitude of ${\Delta }_{DPE}$ decreased and the corresponding confidence intervals became wider, reflecting reduced statistical stability. At the highest noise level ($\sigma =0.10$), the confidence interval overlapped zero, and the result therefore became inconclusive. In Scenario B, which contains the latent upstream chain $Z\to X\to Y$, ${\Delta }_{DPE}$ remained negative under low to moderate noise levels, and the corresponding confidence intervals lay entirely below zero, providing statistical support for the pairwise effect. When the noise level increased to $\sigma =0.10$, the confidence interval also overlapped zero, indicating that the result became inconclusive under strong observational perturbation.

A similar pattern was observed for reduced time-series length (

Table 3. Robustness to time-series length.

Scenario	$\mathbf{Library} \mathbf{Length} \mathit{L}$	${\mathit{\Delta }}_{\mathit{D}\mathit{P}\mathit{E}}$	$95\% \mathbf{CI} \mathbf{of} {\mathit{\Delta }}_{\mathit{D}\mathit{P}\mathit{E}}$	Pairwise Effect Interpretation
A	1500	0.00018	[0.00011, 0.00025]	no supported
A	1000	0.00015	[0.00007, 0.00023]	no supported
A	500	0.00008	[−0.00001, 0.00017]	no supported
A	250	0.00003	[−0.00010, 0.00015]	inconclusive
B	1500	−0.00026	[−0.00034, −0.00018]	supported
B	1000	−0.00022	[−0.00031, −0.00013]	supported
B	500	−0.00011	[−0.00031, −0.00013]	supported
B	250	−0.00005	[−0.00017, 0.00006]	inconclusive

). In Scenario A, ${\Delta }_{DPE}$ remained positive for $L=1500$, $1000$, and $500$, suggesting no supported pairwise effect, but the result became inconclusive when the available series was shortened to $L=250$. In Scenario B, ${\Delta }_{DPE}$ remained negative and statistically supported for $L=1500$, $1000$, and $500$, whereas the confidence interval overlapped zero at $L=250$, again indicating an inconclusive result. These results suggest that DPE can remain informative under moderate reductions in sample length, but its reliability declines when the available time series becomes too short.

Taken together, the stress-test results show that DPE is reasonably robust under moderate observation noise and reduced sample length, while also revealing clear operating limits. Under sufficiently noisy conditions or very short series, the method tends to become inconclusive rather than producing strongly misleading reversals. This pattern supports the interpretation of DPE as a diagnostic method whose practical performance depends on data quality and sample size.

3.2. Bidirectional Causality Scenarios

In the above introduced benchmark system, X and Y have a unidirectional causal relationship, which represents ideal scenarios. However, in many real-world dynamic systems, the relationship between X and Y goes beyond mere unidirectional causality. For instance, in a prey–predator system, a strong causal relationship exists in the predator → prey direction, but there could also be a weak causal relationship in the prey → predator direction [12]. This complexity arises because the abundance of prey can influence the survival conditions of predators, subsequently impacting their population dynamics.

As for this problem in reality, we introduced two scenarios involving bidirectional causalities (scenario H and I in Figure 2C) to validate the application of DPE method. Scenario H (X ↔ Y) indicates that the external factor Z has no impact on either X or Y. Scenario I (Z → X ↔ Y) represents the external variable Z significantly affecting X, with mutual influence between X and Y. With ${\beta }_{yx}$ fixed at 0.35, we systematically adjusted the value of ${\beta }_{xy}$, which reflects the causal strength in the reverse direction X ← Y. Subsequently, DPE was then utilized to analyze whether the pairwise effect emerged in these scenarios.

The experimental result of scenario H is illustrated in Figure 2D, where the presence of pairwise effect is indicated when the values of the blue curve exceed those of the red curve. In the absence of external variable Z and with a weak causal strength ${\beta }_{xy}$ in the reverse direction of X ← Y, the pairwise effect is absent. However, as the causal strength ${\beta }_{xy}$ in the reverse direction of X ← Y increases, the red curve representing $\rho \left(X,\widehat{X}|M_{x+y}\right)$ exhibits a noticeable downward trend. This suggests that the information of X contained in Y is interfered by the enhanced X ← Y causality, as Y initially only plays the role of result but gradually also plays the role of cause. Conversely, the blue curve representing $\left(Y,\widehat{Y}|M_{y+x}\right)$ shows a subtle upward trend, indicating that as the causal strength of X ← Y intensifies, the trend of X as result role steadily strengthens, thereby containing more information about Y.

The experimental result of scenario I is illustrated in Figure 2E. It can be observed that the overall trend of the two curves is similar to that of Figure 2D. The red curve representing $\rho \left(X,\widehat{X}|M_{x+y}\right)$ exhibits a significant downward trend, while the blue curve representing $\left(Y,\widehat{Y}|M_{y+x}\right)$ shows a subtle upward trend. Regardless of the value of ${\beta }_{xy}$, the pairwise effect is present.

These results suggest that when the causal strength in the reverse direction of X ← Y is weak enough (here, ${\beta }_{xy}$ < 0.3), it does not affect the DPE analysis and the discovery of latent factor Z. Furthermore, the causal strength is easy to assess by previous methods. This conclusion broadens the application and generality of the DPE method. In other words, when X and Y have a bidirectional causal relationship X ↔ Y, with a stronger causal relationship in X → Y direction and a weak causal relationship in X ← Y direction, the influence of X ← Y can be disregarded in practical analysis, and the DPE method remains applicable.

For the real-world systems, uncertainty quantification reveals a more nuanced pattern than point estimates alone (Table 3). In the R → P, T0 → T1, and N → Ca directions, the estimated ${\Delta }_{DPE}$ is negative and the corresponding 95% bootstrap confidence intervals lie entirely below zero, providing statistical support for the pairwise effect. In contrast, the C → P and Re → Ca directions yield positive ${\Delta }_{DPE}$ values with confidence intervals entirely above zero, indicating no supported pairwise effect.

Notably, the T1 → T2 and T2 → T3 directions produce negative point estimates of ${\mathit{\Delta }}_{DPE}$, but their bootstrap confidence intervals overlap zero. Under our decision criterion, these cases are therefore interpreted as inconclusive rather than as statistically supported pairwise effects. This distinction is important because it shows that negative point estimates alone are insufficient for interpretation when the differences in predictive skill are small.

3.3. Detect Latent Causation in Real-World Networks

Firstly, we analyzed the food chain network of four plankton species: Picocyanobacterial, Calanoid copepods, Rotifers and Cyclopoids [35,36]. The oscillatory population data were extracted from an 8-year mesocosm experiment involving a plankton community [37,38]. The food chain relationships are shown in Figure 3A. In the food chain networks, the top-down control exerted by predators is significantly stronger than the bottom-up control exerted by prey [12], indicating a stronger causal relationship in the predator → prey direction. It is noted that a weak causal relationship may exist in the prey → predator direction; resultingly, the food chain networks can be viewed as scenario I in Figure 2. The direction of causality is opposite to that of the food chain link. We utilized data from three species, Picocyanobacterial (P), Calanoid copepods (C), and Rotifers (R) for causal analysis, while the data for Cyclopoids were regarded as unknown and only used for validating the results. Initially, we used the conventional CCM to analyze the causal relationships between prey and predator (Figure 3A). It can be observed that there exists a strong causal relationship in the C → P and R → P directions, and a weak causal relationship in the P → R direction, which is consistent with previous research [12].

Then we utilized DPE to analyze the causal relationships in the R → P and C → P directions, and the results are presented in the upper part of Table 2. In the R → P direction, the pairwise effect is observed, indicating that R may not be the most direct driver of P and implies the presence of a deep causal relationship (i.e., deep food chain link). Noticeably, Cyclopoids may represent a plausible upstream influence on R, even though Cyclopoids are regarded as unknown and unobservable in experiments. Moreover, no pairwise effect is observed in the C → P direction.

Secondly, we analyzed the causal networks among temperatures at different soil depths [39]. It is conventionally thought that environment factors such as sunlight and air influence surface soil temperature variations, while shallow soil temperatures serve as the cause for changes in deep soil layers. The causal networks are depicted in Figure 3B. In the experiments, we utilized the data of soil surface temperature (T0), temperature at depths of 10 cm (T1), 20 cm (T2), and 30 cm (T3) to analyze. The environmental data, such as air temperature and sunlight, were not available and we regarded them as unknown. From the prediction curves based on CCM, it suggested that variations in shallow soil temperatures drive changes in deep soil layers, aligning with soil science knowledge. Subsequently, we used DPE to analyze the causal relationships in the directions of T0 → T1, T1 → T2, and T2 → T3, with results presented in the middle part of

Table 4. Experimental results for real-world systems.

Direction	$\rho \left(C,\widehat{C}|M_{C+P}\right)$	$\rho \left(P,\widehat{P}|M_{P+C}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
C → P	0.994246	0.988015	$>0$	[0.00490, 0.00743]	<0.001	-
Direction	$\rho \left(R,\widehat{R}|M_{R+P}\right)$	$\rho \left(P,\widehat{P}|M_{P+R}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
R → P	0.986401	0.989355	$<0$	[−0.00418, −0.00173]	<0.001	★
Direction	$\rho \left(T0,\widehat{T0}|M_{T0+T1}\right)$	$\rho \left(T1,\widehat{T1}|M_{T1+T0}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
T0 → T1	0.995196	0.998239	$<0$	[−0.00411, −0.00196]	<0.001	★
Direction	$\rho \left(T1,\widehat{T1}|M_{T1+T2}\right)$	$\rho \left(T2,\widehat{T2}|M_{T2+T1}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
T1 → T2	0.998313	0.998774	$<0$	[−0.00102, 0.00009]	0.086	★
Direction	$\rho \left(T2,\widehat{T2}|M_{T2+T3}\right)$	$\rho \left(T3,\widehat{T3}|M_{T3+T2}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
T2 → T3	0.998737	0.999044	$<0$	[−0.00081, 0.00013]	0.119	★
Direction	$\rho \left(Re,\widehat{Re}|M_{Re+Ca}\right)$	$\rho \left(Ca,\widehat{Ca}|M_{Ca+Re}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
Re → Ca	0.990786	0.98799	$>0$	[0.00152, 0.00401]	<0.001	-
Direction	$\rho \left(N,\widehat{N}|M_{N+Ca}\right)$	$\rho \left(Ca,Ca|M_{Ca+N}\right)$	${\mathit{\Delta }}_{DPE}$	95% CI $\mathrm{of} {\Delta }_{\mathrm{D}\mathrm{P}\mathrm{E}}$	p-value	Pairwise effect
N → Ca	0.983526	0.985287	$<0$	[−0.00291, −0.00064]	0.003	★

“★” indicates the presence of pairwise effect and “-” indicates the absence of pairwise effect.

. Noticeably, all these directions exhibited pairwise effects, suggesting that these causes (i.e., T0, T1, and T2, respectively) in these three directions may not be the most direct drivers of the results (i.e., T1, T2, and T3, respectively). Therefore, the results are consistent with the presence of deeper underlying factors in these directions (i.e., environmental factors, T1, and T2, respectively).

Thirdly, we investigated the data of air pollution and hospital admission of cardiovascular diseases in Hong Kong from 1994 to 1997 [38,40]. Relevant research has revealed that nitrogen dioxide and respirable suspended particulates were the two major causes of cardiovascular diseases [2], as shown in Figure 3C. Since DPE is to identify which pollutants cause cardiovascular disease and detect the potential deep causality, we fixed the disease as the target in the causal networks. Then we analyzed the data of nitrogen dioxide (N), respirable suspended particulates (resp) and cardiovascular diseases (cardio). From the prediction curves based on CCM, it suggested that there exist unidirectional causal relationships in resp → cardio and no2 → cardio directions. DPE revealed that pair effect is absent in resp → cardio direction. However, pair effect is present in no2 → cardio direction. The results suggested that nitrogen dioxide may not be the fundamental causes of cardiovascular diseases, and other pollutants may influence the nitrogen dioxide in the air. The conclusion is consistent with previous results [41,42].

4. Discussion

According to Takens’ embedding theorem, each variable in a dynamical system potentially carries information about the entire system. In principle, this provides a theoretical basis for recovering causal relationships from limited observations. In practice, however, ecological and environmental systems are shaped by multiple interacting drivers across scales, and the information contained in any single observed variable is often incomplete, noisy, or only weakly expressive of latent mechanisms [43]. As a result, extracting hidden causal structure from single-variable embeddings may be inefficient and may fail to reveal mediated or upstream influences.

To address this challenge, the proposed detecting pairwise effect (DPE) method extends conventional causality analysis by examining how predictive relationships change when multivariable information is incorporated. Rather than only confirming the presence of an observable causal link, DPE assesses whether such a link is more consistent with a direct causal driver or with a mediated relationship influenced by latent upstream variables. The emergence of the pairwise effect provides a diagnostic signal consistent with the possibility that an observed causal relationship (X → Y) embeds a deeper causal structure, such as a latent chain (Z → X → Y), in which X primarily acts as a mediator rather than the root cause of Y.

The effectiveness of DPE is supported by analyses of both simulated systems and real-world datasets. These results indicate that DPE can reveal patterns consistent with latent causal chains that may remain obscured under traditional causality inference methods based solely on observable variables. At the same time, the absence of a pairwise effect does not imply the absence of latent causal influences. Latent factors may exist but fail to produce a detectable signal when they influence multiple observed variables with comparable strength. Therefore, DPE should be interpreted as a diagnostic tool whose outcomes depend on the underlying causal configuration and the relative strengths of interactions.

From an ecological perspective, the ability to probe latent causal chains is particularly relevant, as many ecological systems are governed by partially observed drivers and multi-scale interactions. Observed variables often represent aggregated outcomes of multiple processes, including trophic interactions, density dependence, dispersal, and environmental forcing. Consequently, ecological time series frequently exhibit indirect causation, apparent bidirectionality, and regime shifts, where observed associations may reflect mediation through unmeasured upstream factors rather than direct interactions. In this context, DPE provides a practical way to assess whether an apparent link between two variables is more consistent with direct coupling or mediated influence.

The interpretation of the DPE signal depends on several conditions. First, the system should be sufficiently deterministic for state-space reconstruction to be meaningful, and the sampling resolution should be appropriate relative to the intrinsic timescale. Second, DPE is most informative when latent influences are asymmetric, such that upstream drivers affect one variable more strongly than another. Under such conditions, multivariable embedding may reduce the recoverability of one variable and lead to the pairwise effect. However, this behavior is not guaranteed. The DPE signal may become unreliable when common drivers affect both variables similarly, when strong synchrony or collinearity obscures directional information, or when nonstationarity, measurement noise, and limited sample size reduce the stability of predictive skill estimates. In this study, DPE is interpreted together with uncertainty quantification, including bootstrap confidence intervals and statistical testing, rather than relying solely on the sign of ${\mathit{\Delta }}_{DPE}$.

In empirical applications, additional data-related factors may influence interpretation. Potential confounders (e.g., radiation, soil moisture, wind, boundary-layer dynamics) may simultaneously affect multiple variables, introducing indirect dependencies. Measurement errors and sensor noise may distort predictive skill, especially when differences in ρ are small. Moreover, inappropriate sampling intervals relative to system dynamics may degrade manifold reconstruction. These issues affect both CCM and DPE and should be considered when interpreting results. Preprocessing steps such as detrending, seasonal adjustment, and surrogate testing are recommended to improve robustness.

Despite these encouraging results, DPE still has important limitations. In particular, latent influences may remain difficult to diagnose when they affect multiple observed variables with similar strength or when causal pathways are highly entangled. DPE should therefore be viewed as complementary to existing causal inference approaches rather than as a replacement, and may be most useful when combined with methods designed to mitigate indirect effects or synchrony.

5. Conclusions

In this study, we proposed detecting pairwise effect (DPE), a data-driven method based on empirical dynamic modeling and multivariable state-space reconstruction, to assess whether an observed causal link is more consistent with a direct driver or with a mediated effect shaped by latent upstream forcing. Through analyses of benchmark systems and real-world datasets, we showed that DPE provides a diagnostic signal consistent with latent upstream influence that may not be directly observable in time series data. This improves the interpretability of inferred causal relationships in complex ecological and environmental systems.

However, DPE should be interpreted cautiously. Its performance depends on system properties, data quality, and the relative strength of causal interactions. In particular, common drivers, synchrony, nonstationarity, and limited sample size may affect the reliability of the inferred signal. Therefore, DPE is best used in conjunction with complementary methods and uncertainty quantification. Future work should further evaluate the robustness of DPE through systematic stress tests under varying noise levels, sampling conditions, and dynamical regimes, as well as through benchmarking against representative causal inference approaches. Such efforts will help to better characterize the applicability and limitations of DPE in real-world systems.