The Probabilistic Foundations of Surveillance Failure: From False Alerts to Structural Bias

Abstract

Forensic statisticians have long debated whether searching large DNA databases undermines the evidential value of matches. Modern surveillance faces an exponentially harder problem: screening populations across thousands of attributes using threshold rules. Intuition suggests that requiring many coincidental matches should make false alerts astronomically unlikely. This intuition fails. Consider a system monitoring 1000 attributes, each with a 0.5 percent innocent match rate. Matching 15 pre-specified attributes has probability ${10}^{-35}$, 1 in 30 decillion, effectively impossible. But operational systems may flag anyone matching any 15 of the 1000. In a city of one million innocents, this produces about 226 false alerts. A seemingly impossible event becomes guaranteed. This is a mathematical consequence of high-dimensional screening, not implementation failure. We identify fundamental probabilistic limits on screening reliability. Systems undergo sharp transitions from reliable to unreliable with small data scale increases, a fragility worsened by data growth and correlations. As data accumulate and correlation collapses effective dimensionality, systems enter regimes where alerts lose evidential value even when individual coincidences remain vanishingly rare. This framework reframes the DNA database controversy as a regime shift. Unequal surveillance exposures magnify failure, making “structural bias” mathematically inevitable. Beyond a critical scale, failure cannot be prevented through threshold adjustment or algorithmic refinement.

1. Introduction

1.1. From Database Search to Threshold Screening

For over two decades, forensic statisticians have debated a fundamental question: does searching a DNA database of a million profiles weaken the evidential value of a match? Stockmarr [1] argued that database searching increases coincidental match probability, while Balding [2] maintained that likelihood ratios preserve evidential weight regardless of search method. This controversy (concerning a single database with exact profile matching) remains unresolved [3,4].

Modern surveillance systems face an exponentially harder problem. Rather than searching one database for exact matches, they monitor populations across thousands of attributes (locations visited, transactions made, and communications sent) and flag individuals whose patterns match sufficiently many criteria using a threshold rule rather than exact matching. The intuition that “requiring many coincidences makes false matches astronomically unlikely” fails catastrophically in this regime.

Consider a concrete example that reveals the nature of this failure. A surveillance system monitors $k=1000$ binary indicators, each with innocent match probability $p=0.005$ (half a percent chance), and flags individuals matching at least $m=15$ attributes. At first glance, this threshold seems extraordinarily conservative. Requiring an exact match across 15 specific predetermined attributes would have probability ${\left(0.005\right)}^{15}\approx 3\times {10}^{-35}$, 1 in 30 decillion. Such exact-match odds suggest that false alerts should be astronomically impossible, even across populations of billions.

But operational systems do not require exact matches across 15 predetermined attributes. Instead, they may flag individuals matching at least 15 of any 1000 attributes. This seemingly subtle distinction changes everything. With $\lambda =kp=1000\times 0.005=5$ expected chance matches per innocent person and threshold $m=15$ (three times the expected value), direct calculation gives a per-person false alert probability of

$$q:=Pr(\mathrm{Poisson}\left(5\right)\ge 15)\approx 2.26\times {10}^{-4}=0.0226\%.$$

In a city of $n={10}^6$ innocents, this produces approximately $nq\approx 226$ false alerts, and the probability of at least one false alert is

$$Pr\left(\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{false}\mathrm{alert}\right)\approx 1-e^{-226}\approx 1-7\times {10}^{-99}.$$

Thus the probability of no false alerts is on the order of ${10}^{-99}$, making the system, for all practical and mathematical purposes, certain to flag at least one innocent person.

The “1 in 30 decillion” exact-match calculation is mathematically correct but operationally irrelevant. The threshold rule “at least m of k” transforms the per-person probability from vanishingly small (${10}^{-35}$) to operationally significant (${10}^{-4}$), and population size amplifies this individual-level vulnerability into system-level certainty of failure. This is not a failure of implementation or algorithm design; it is a mathematical inevitability arising from the combinatorics of threshold detection in high-dimensional attribute spaces.

This contrast between the DNA setting and modern multi-attribute surveillance also clarifies why the long-standing DNA database controversy has persisted. Classical DNA searches operate in an extremely sparse false-match regime, where random matches are so rare that enlarging the database does not meaningfully dilute the evidential value of a hit.

Multi-attribute threshold screening, however, operates in a dense false-match regime where accumulated coincidences quickly overwhelm the signal. These are mathematically distinct operating regimes. Recognizing this distinction resolves the apparent conflict in the DNA literature and motivates the broader probabilistic limits developed in this paper.

1.2. Motivation and Background

Modern surveillance and screening systems collect vast amounts of data about individuals and attempt to identify rare targets of interest, such as potential terrorists, criminals, or fraudsters. These systems compare observed attributes—biometric features, transaction patterns, travel histories, and communication metadata—against profiles or watchlists, flagging individuals whose patterns match suspiciously well [5,6,7].

The fundamental challenge is the curse of dimensionality in coincidence detection [8]: as monitored attributes grow, innocent individuals increasingly match multiple criteria by chance. When populations are large (millions to billions), even extremely rare coincidences become statistically certain, generating overwhelming false alerts. This parallels challenges in cybersecurity [9,10,11,12], where even highly accurate intrusion detectors are swamped by false alarms at scale, and in open-set recognition [13], where classifiers must reliably reject vast numbers of unfamiliar inputs.

1.3. Related Work and Connections

Our analysis builds on classical probability theory (Poisson approximation, Chernoff bounds, and concentration inequalities) applied to modern surveillance contexts, situating this work within the framework of large-scale inference [14]. The system-level bound $Pr\left(\mathrm{false}\mathrm{alert}\right)\approx 1-{(1-q)}^n$ corresponds to family-wise error rate (FWER) control in multiple testing [15,16,17], though our large-deviation analysis shows when such control becomes infeasible.

The base-rate analysis resolves the Stockmarr–Balding debate [1,2] in forensic DNA statistics by identifying distinct asymptotic regimes. Connections to anomaly detection [9,12], open-set recognition [13], and algorithmic fairness [18,19,20,21,22] appear throughout: false-positive explosion is a fundamental challenge across all screening domains.

1.4. Main Contributions

This paper develops a rigorous probabilistic framework for analyzing such screening systems. Using classical tools (Poisson approximations, Chernoff bounds, concentration inequalities, and Bayesian inference), we characterize fundamental limits on system reliability. Our analysis reveals that the single-database DNA controversy and multi-attribute surveillance failure are manifestations of the same mathematical phenomenon, differing only in which asymptotic regime they occupy.

Our main contributions include the following:

• Sharp critical population bounds: We derive two-sided bounds on both per-person and system-level false alert probabilities, showing that the critical population grows as $n_{\mathrm{crit}}\asymp \sqrt{\lambda }exp\left(\lambda D(c\parallel 1)\right)$ with explicit $\sqrt{\lambda }$ corrections (Theorem 1). The rate function $D(c\parallel 1)=clogc-c+1$ governs the exponential scaling.
• Finite system lifetimes: Under exponential data growth $k\left(t\right)=k_0{\gamma }^t$, systems fail at time $T^{*}\approx {\displaystyle \frac{1}{log\gamma }}log(m/k_{0}p)$ when $\lambda \left(t\right)$ reaches threshold m, with population corrections secondary (Theorem 2).
• Unified Bayesian–frequentist view: Both regimes exhibit the same exponential scaling $exp\left(\lambda D\right(c\parallel 1\left)\right)$, but Bayesian actionability requires $n\lesssim {\displaystyle \frac{1-\alpha }{\alpha }}rs\sqrt{\lambda }{e}^{\lambda D}$ (for desired PPV $\ge \alpha$) while frequentist reliability allows $n\lesssim \sqrt{\lambda }{e}^{\lambda D}$ (Section 4.1). Posterior probabilities collapse when $nq\gg rs$, resolving the Stockmarr–Balding controversy as a regime transition.
• Structural bias through group dominance: Differential surveillance exposure creates exponential outcome disparities that cannot be eliminated through threshold adjustments (Section 3.3), formalizing “structural bias” arguments from the algorithmic fairness literature.
• Effective dimensionality under correlation: Spatiotemporal dependencies reduce effective attribute counts to $k_{\mathrm{eff}}\approx A/\left(2\pi {\xi }^2\right)$ (spatial) or $k_{\mathrm{eff}}\approx k/\left(2\tau \right)$ (temporal), connecting surveillance analysis to multiple-testing corrections in genomics and neuroimaging (Section 3.4; technical details in Appendix B).

All main results are proved using elementary probability, with full technical details collected in Appendix A. Appendix C presents illustrative examples based on public datasets (UCI Adult Census and Chicago Crime) that show how the phenomena described here arise with real data.

Notation. 

The symbols above are defined precisely in later sections: $D(c\parallel 1)$ is the Poisson rate function (Lemma 1); $c>1$ is the threshold multiplier with $m=\lceil c\lambda \rceil$ (Section 2.2); $\gamma$ and $k_0$ parameterize exponential data growth (Section 3.2); r, s, and $\alpha$ are Bayesian parameters (Section 4.1); and $k_{\mathrm{eff}}$, $\xi$, and $\tau$ quantify correlation effects (Section 3.4). A complete notation summary appears in the symbol table.

2. Materials and Methods

2.1. The Basic Screening Model

Consider a population of n individuals, from each of whom we collect k types of data (e.g., locations visited, financial transactions, etc.). For each type of data, we also have lists of suspicious activity, for example, locations of interest where crimes have occurred. In general, we observe k binary attributes (indicators) for each individual, where each attribute has a small probability $p\ll 1$ of matching some profile of interest purely by chance.

For example, a simple model for the match probability p on the j-th data type might be found by calculating the probability that a person’s list (size t) overlaps the suspicious list (size s) on a domain of size V possible distinct items (e.g., V geo-location cells). With uniform sampling without replacement from the domain, the probability of an overlap comes from the zero-intersection hypergeometric probability:

$$p=Pr(\mathrm{overlap}\ge 1)=1-{\displaystyle \frac{\left(\genfrac{}{}{0pt}{}{V-t}{s}\right)}{\left(\genfrac{}{}{0pt}{}{V}{s}\right)}}=1-\prod _{\ell =0}^{s-1}\left(1-{\displaystyle \frac{t}{V-\ell }}\right),\mathrm{for}s\le V-t,$$

with $p=1$ when $s>V-t$ (guaranteed overlap).

More generally, the match probability p may be derived from any domain-appropriate model; the hypergeometric calculation above serves merely as one concrete example. Our analysis holds for any fixed value of $p\ll 1$, regardless of how it is obtained. With this understanding, let $X_i$ denote the number of matching attributes for individual i. Under the null hypothesis (individual i is innocent) and the modeling assumptions detailed below, we have

$$X_i\sim \mathrm{Binomial}(k,p).$$

For large k and small p, we write $\lambda =kp$ for the expected number of chance matches; this parameter will govern all subsequent results.

The screening system flags individual i as suspicious if they match some unsolved crime on m attributes with $X_i\ge m$ for an integer threshold m. Throughout, we consider thresholds of the form $m=\lceil c\lambda \rceil$ for constant $c>1$, representing detection criteria set above the expected number of chance matches; rounding affects only constant factors. The per-person false alert probability is

$$q=Pr(X_i\ge m\mid \mathrm{innocent}).$$

Since we screen n individuals, the system-level false alert probability, that is, the probability that at least one innocent person is flagged, is

$$Pr\left(\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{false}\mathrm{alert}\right)=1-{(1-q)}^n.$$

For small q, this is commonly approximated by

$$Pr\left(\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{false}\mathrm{alert}\right)\approx 1-e^{-nq}.$$

The system becomes unreliable when this probability approaches 1 (that is, when $nq\gtrsim 1$), in which case the system is near certain to flag innocent individuals as suspects.

2.2. Modeling Assumptions and Regime of Validity

Throughout this paper, we operate under the following assumptions:

• Large-k, small-p regime: We assume $k\gg 1$ (many attributes) and $p\ll 1$ (rare individual matches), with mean $\lambda =kp$ held moderate and fixed as $k\to \infty$. This is the natural Poisson regime for rare event detection, where the binomial distribution converges to Poisson$\left(\lambda \right)$ with total variation error bounded by $2{\sum }_{i=1}^k{p}_i^2$ (see Section 3.1).
• Independence across individuals: The match indicators for different individuals are statistically independent (Remark 2). In practice, common-mode events (e.g., large gatherings or seasonal shifts) may introduce positive dependence, which would only increase false alert rates beyond our bounds.
• Homogeneous match probability (baseline model): For analytical tractability, we initially assume that all attributes have the same match probability p. Section 3.3 extends to heterogeneous populations with different exposure rates $p_g$ across demographic groups. Remark 1 addresses heterogeneity across attributes within individuals.
• Binary attributes: Each attribute either matches ($M=1$) or does not match ($M=0$) a profile of interest. This models presence/absence at locations, yes/no transaction patterns, or binarized continuous features.
• Fixed threshold screening: The system flags individuals with m or more matches, where m is predetermined. We primarily analyze $m=c\lambda$ for constant $c>1$, representing thresholds scaled to exceed the expected number of chance matches under the null hypothesis.
• Null hypothesis analysis: We analyze false alerts under the assumption that all n individuals are innocent (the null hypothesis). Detection of actual targets requires separate treatment via signal detection theory and receiver operating characteristic (ROC) analysis [23], which is not addressed here. Our results characterize the false positive rate; in practice, system designers must balance this against detection power for true targets.

These assumptions define the scope of our analysis. The core results (Section 3.1 and Section 3.2) establish fundamental limits under these idealized conditions. Later sections extend the framework: Section 3.3 analyzes heterogeneous populations with differential exposure, and Section 3.4 summarizes how spatiotemporal correlation reduces effective dimensionality (with technical details in Appendix B).

The large-k, small-p regime is particularly natural for modern surveillance systems, where technological advances enable collection of thousands of attributes per person (GPS pings, transaction records, and communication events). In our model, each attribute contributes a binary match indicator (match or no match) of whether it aligns with a profile of interest, and each such match remains rare. For instance, with $k=1000$ location cells and $p=0.005$ (half-percent chance of presence at any flagged location), we have $\lambda =5$ expected false matches, a moderate value that falls squarely within our analytical framework.

3. Results

3.1. Fundamental Probabilistic Limits

3.1.1. The Poisson Approximation

When k is large, p is small, and $\lambda =kp$ is fixed, the $\mathrm{Binomial}(k,p)$ distribution can be approximated by $\mathrm{Poisson}\left(\lambda \right)$ [24]. Specifically, if $X\sim \mathrm{Binomial}(k,p)$, then

$$Pr(X=j)\approx e^{-\lambda }{\displaystyle \frac{{\lambda }^j}{j!}},\lambda =kp.$$

Moreover, by Le Cam’s inequality [25],

$$d_{\mathrm{TV}}\left(\mathrm{Binomial}(k,p),\mathrm{Poisson}\left(\lambda \right)\right)\le 2\sum _{i=1}^k{p}_i^2.$$

For the homogeneous case with $p_i=p$ for all i, this gives $d_{\mathrm{TV}}\le 2k{p}^2=2\lambda p$. When p is small and $\lambda =kp$ is moderate, this bound is $O\left(p\right)$, making the Poisson approximation numerically safe for practical applications. Throughout, we assume independence across individuals; common-mode shocks would require extensions via positively associated random variables (see Remark 2).

Under this approximation, the per-person false alert probability (tail) is

$$q=Pr(X\ge m)\approx \sum _{j=m}^{\infty }{e}^{-\lambda }{\displaystyle \frac{{\lambda }^j}{j!}}=1-\sum _{j=0}^{m-1}{e}^{-\lambda }{\displaystyle \frac{{\lambda }^j}{j!}}.$$

Remark 1

(Heterogeneous Attribute Probabilities). For analytical tractability we assume a homogeneous match probability p across attributes, but this assumption can be relaxed. Let the attribute-level match probabilities be $p_1,\dots ,p_k$, and define

$$X=\sum _{i=1}^k\mathrm{Bernoulli}\left(p_i\right),\lambda =\sum _{i=1}^k{p}_i.$$

By Le Cam’s theorem [25], the total variation distance between the distribution of X and a Poisson distribution with mean λ satisfies

$$d_{\mathrm{TV}}\left(X,\mathrm{Poisson}\left(\lambda \right)\right)\le 2\sum _{i=1}^k{p}_i^2.$$

Thus the Poisson approximation remains accurate whenever all $p_i$ lie in the small-probability regime, even if the $p_i$ differ substantially from one another. In this heterogeneous setting, the distribution of X, and therefore all results in this section, depends on the collection $\left\{p_i\right\}$ only through the aggregate rate λ, up to an approximation error of order ${\sum }_i{p}_i^2$.

Group-level heterogeneity (e.g., differing $p_g$ across demographic groups) is treated separately in Section 3.3.

Remark 2

(Independence Across Individuals). Since we screen n individuals independently, the probability that at least one innocent person is flagged is

$$Pr\left(\mathit{\text{at least one false alert}}\right)=1-{(1-q)}^n.$$

In real deployments, common-mode events may introduce positive dependence among individuals. Positive association means $Pr(X_i\ge x_i\mathit{\text{for all}}i)\ge {\prod }_{i}Pr(X_i\ge x_i)$: variables are more likely to be jointly large than under independence. This increases the system-level false alert probability beyond our bounds.

Our analysis assumes independence. Extensions to positively associated arrays via Chen–Stein’s or Stein’s method [26] are possible but omitted here. The key implication is that positive dependence increases tail probabilities, so our independence-based bounds underestimate true false alert rates when correlation is present. Thus, the independence-based bounds are conservative.

3.1.2. Tail Bounds for the Poisson Distribution

To understand how q depends on m and $\lambda$, we use Chernoff bounds and related concentration inequalities [27,28] for Poisson random variables.

Lemma 1

(Poisson Upper Tail). Let $Y\sim \mathrm{Poisson}\left(\lambda \right)$ and suppose $m>\lambda$. Then

$$Pr(Y\ge m)\le {\left({\displaystyle \frac{e\lambda }{m}}\right)}^m{e}^{-\lambda }=exp\left(-\lambda D\left({\displaystyle \frac{m}{\lambda }}\parallel 1\right)\right),$$

(1)

where $D(\alpha \parallel 1)=\alpha log\alpha -\alpha +1$ is the Poisson rate function.

Remark 3

(Rate Function Properties). The function $D(\alpha \parallel 1)$ is also called the Cramér rate function or Cramér transform of the Poisson distribution with unit mean. This differs from the binary Kullback–Leibler divergence $D_{\mathrm{KL}}(p\parallel q)=plog(p/q)+(1-p)log((1-p)/(1-q))$ used for Bernoulli random variables.

The large-deviation rate is asymptotically exact for $m/\lambda >1$: the tail probability satisfies $q\asymp {\lambda }^{-1/2}{e}^{-\lambda D(m/\lambda \parallel 1)}$, with factorial discreteness contributing only the ${\lambda }^{-1/2}$ subexponential prefactor. We write $D(c\parallel 1)$ or simply D for brevity when the argument is clear.

Proof. 

For any $t>0$,

$$\begin{array}{cc}\hfill Pr(Y\ge m)& =Pr\left(e^{tY}\ge e^{tm}\right)\le {\displaystyle \frac{\mathbb{E}\left[e^{tY}\right]}{e^{tm}}}=exp\left(\lambda (e^t-1)-tm\right).\hfill \end{array}$$

Optimizing by setting ${\displaystyle \frac{d}{dt}}[\lambda (e^t-1)-tm]=0$ gives $e^{t^{*}}=m/\lambda$, which yields the stated bound. □

3.1.3. Critical Population Size

We now establish the fundamental scale of population size for reliable screening when the decision threshold scales as $m=c\lambda$ with $c>1$.

Theorem 1

(Critical Population Scale). Consider a screening system with k binary attributes, per-attribute match probability p, threshold $m=c\lambda$ for some $c>1$, population size n, and $\lambda =kp$. Let $D(c\parallel 1)=clogc-c+1$ (the Poisson rate function from Lemma 1). Then the per-person false alert probability q satisfies the following asymptotic bounds:

$${\displaystyle \frac{1}{\sqrt{2\pi c\lambda }}}exp\left(-\lambda D(c\parallel 1)-\frac{1}{12c\lambda }\right)\le q\le exp\left(-\lambda D(c\parallel 1)\right),$$

(2)

which hold for $\lambda \ge 1$ and $c>1$ with the stated constants; for smaller λ the same exponential form holds with different absolute constants. The system-level false alert probability obeys

$$1-exp\left(-{\displaystyle \frac{n}{\sqrt{2\pi c\lambda }}}{e}^{-\lambda D(c\parallel 1)-\frac{1}{12c\lambda }}\right)\le Pr\left(\mathit{false}\mathit{alert}\right)\le 1-exp\left(-{\displaystyle \frac{n{e}^{-\lambda D(c\parallel 1)}}{1-e^{-\lambda D(c\parallel 1)}}}\right).$$

(3)

(For $e^{-\lambda D}\le 1/2$, the upper bound denominator is within a factor of 2 in the exponent, providing tight control.) In particular, the system becomes unreliable once

$$n\gtrsim \sqrt{\lambda }exp\left(\lambda D(c\parallel 1)\right)(\mathit{up}\mathit{to}\mathit{constant}\mathit{factors}),$$

and the critical population scale is

$$n_{\mathrm{crit}}\asymp exp\left(\lambda D(c\parallel 1)\right)=exp\left(kpD(c\parallel 1)\right),$$

(4)

ignoring subexponential $\sqrt{\lambda }$ corrections. Here we use the notation $f\asymp g$ to denote equality up to multiplicative constants independent of the main asymptotic parameters (λ, k, and n).

Proof sketch. 

The upper bound follows directly from Lemma 1 with $m=c\lambda$, giving $q\le exp(-\lambda D(c\parallel 1\left)\right)$. The lower bound uses Robbins’ refinement of Stirling’s formula, which shows that $Pr(Y=m)$ has the same exponential rate $D(c\parallel 1)$ up to a subexponential $\sqrt{\lambda }$ factor. Because $q=Pr(Y\ge m)\ge Pr(Y=m)$, these bounds yield matching exponential rates for q.

The system-level inequalities (3) follow from $e^{-nx/(1-x)}\le {(1-x)}^n\le e^{-nx}$ for $x\in (0,1)$. The critical population scale arises when $nq\sim 1$, giving $n_{\mathrm{crit}}\approx \sqrt{\lambda }{e}^{\lambda D(c\parallel 1)}$, and hence, up to subexponential factors, $n_{\mathrm{crit}}\asymp e^{\lambda D(c\parallel 1)}$, as stated. Full derivations appear in Appendix A. □

3.1.4. Sharp Threshold Behavior

The relationship between population size and system reliability exhibits an increasingly sharp threshold as the number of attributes grows.

Proposition 1

(Threshold Sharpness). Consider a screening system with $\lambda =kp$, threshold $m=c\lambda$ for fixed $c>1$, and population scaled as $n=\sqrt{\lambda }·e^{\alpha \lambda D(c\parallel 1)}$ for $\alpha >0$. Then the system-level false alert probability satisfies

$$\underset{\lambda \to \infty }{lim}Pr\left(\mathit{false}\mathit{alert}\right)=\left\{\begin{array}{cc}0\hfill & \mathit{if}\alpha <1,\hfill \\ 1\hfill & \mathit{if}\alpha >1.\hfill \end{array}\right.$$

(5)

The transition occurs in a window $\Delta \alpha \asymp 1/\left(\lambda D\right(c\parallel 1\left)\right)\to 0$, becoming arbitrarily sharp as λ increases; this follows from standard large-deviation theory for sums of i.i.d. rare events [29]. (Note: α here denotes the population scaling exponent, distinct from the posterior probability threshold α in Section 4.1.)

Proof sketch. 

From Theorem 1, $logq=-\lambda D(c\parallel 1)+O(log\lambda )$. With $n=\sqrt{\lambda }{e}^{\alpha \lambda D(c\parallel 1)}$, we obtain

$$log\left(nq\right)=(\alpha -1)\lambda D(c\parallel 1)+O(log\lambda ).$$

For $\alpha <1$, $nq\to 0$ exponentially; for $\alpha >1$, $nq\to \infty$ exponentially. The transition width $\Delta \alpha =O(1/(\lambda D\left)\right)\to 0$. Full details appear in Appendix A.2. □

Remark 4.

This sharp threshold implies no gradual degradation: systems operating near the critical line $logn=\lambda D(c\parallel 1)$ transition from reliable to unreliable essentially instantaneously as λ increases (Figure 1). Combined with exponential data growth $\lambda \left(t\right)={\lambda }_0{\gamma }^t$ with ${\lambda }_0=k_{0}p$, every system inevitably crosses this threshold at a finite time

$$T^{*}\sim {\displaystyle \frac{log(m/{\lambda }_0)}{log\gamma }}={\displaystyle \frac{1}{log\gamma }}log\left({\displaystyle \frac{m}{k_{0}p}}\right)$$

(Theorem 2), making failure predictable rather than merely possible.

3.1.5. Numerical Validation

Monte Carlo simulations (5000 runs per data point) validate the Poisson approximation and critical population predictions as k increases. Figure 1a compares simulated false alert rates (orange points) to theoretical predictions, showing a mean absolute error of 0.0015 across the critical transition region. The sharp threshold behavior predicted by Proposition 1 is clearly evident in the simulation data.

The accuracy of the Poisson approximation across varying $(k,p)$ combinations follows from Le Cam’s inequality (Section 3.1.1). In the homogeneous case $p_i=p$ used in this section, Le Cam’s bound simplifies to

$$d_{\mathrm{TV}}\le 2k{p}^2=2\lambda p,$$

where $\lambda =kp$.

Table 1. Poisson approximation error: theoretical bound versus actual total variation distance for representative $(k,p)$ combinations.

k	p	$\mathit{\lambda }=\mathit{kp}$	Le Cam Bound	Actual ${\mathit{d}}_{\mathbf{TV}}$
50	0.10	5.0	1.000	0.026
100	0.05	5.0	0.500	0.013
500	0.01	5.0	0.100	0.0025
1000	0.005	5.0	0.050	0.0012
5000	0.001	5.0	0.010	0.0002
100	0.10	10.0	2.000	0.026
1000	0.01	10.0	0.200	0.0025

compares this theoretical bound against the actual total variation distance between Binomial$(k,p)$ and Poisson$\left(\lambda \right)$ distributions across parameter combinations spanning the operating regime of interest.

The Le Cam bound is conservative by a factor of 17–40 in these regimes; the actual approximation error is substantially smaller than the bound suggests. For the primary parameters used throughout this paper ($k=1000$, $p=0.005$, $\lambda =5$), the actual total variation distance is approximately $0.001$, indicating that the Poisson approximation introduces negligible error relative to the phase transition effects we characterize. Even in less favorable regimes ($k=50$, $p=0.1$), the approximation error remains below $0.03$, confirming that the Poisson model is appropriate across the full range of parameters considered.

Additional numerical illustrations across a broader range of parameters, as well as examples based on real-world datasets, are given in Appendix C.

Key Takeaway. 

The per-person false alert probability decays as $q\sim e^{-\lambda D(c\parallel 1)}$, and the system becomes unreliable once the population size exceeds $n_{\mathrm{crit}}\sim \sqrt{\lambda }{e}^{\lambda D(c\parallel 1)}$. This behavior produces a sharp phase transition: systems move abruptly from reliable to unreliable operation, and the sharpness of the transition increases monotonically with $\lambda$.

3.2. Temporal Dynamics: Finite System Lifetimes

Real surveillance systems accumulate data over time. As more attributes are collected, the false alert problem intensifies until the system becomes unreliable. We now quantify how long a screening system can operate before this inevitable failure occurs. As Figure 1c demonstrates, systems with exponential data growth exhibit sharp temporal transitions from reliable to unreliable operation at predictable times.

Remark 5

(Change of Threshold Regime). In the preceding analysis (Section 3.1), we studied thresholds of the form $m=c\lambda$ with $c>1$ constant, where both m and λ scale together. In this section, we adopt a different perspective: we fix the threshold m at deployment time while allowing $\lambda \left(t\right)=k\left(t\right)·p$ to grow over time as data accumulates. This models realistic system operation, where detection thresholds are predetermined policy parameters that remain constant even as surveillance capabilities expand. The critical behavior occurs when the growing $\lambda \left(t\right)$ crosses the fixed threshold m, at which point false alerts become inevitable.

3.2.1. Exponential Data Growth

Modern data collection exhibits exponential growth [30,31]. We model this as follows:

$$k\left(t\right)=k_0{\gamma }^t,$$

(6)

where $k_0$ is the initial number of monitored attributes at time $t=0$ and $\gamma >1$ is the growth factor per time unit (e.g., annual growth rate).

Since each attribute has probability p of matching by chance for an innocent individual, the expected number of false matches grows exponentially:

$$\lambda \left(t\right)=k\left(t\right)·p=k_{0}p·{\gamma }^t.$$

(7)

3.2.2. Critical Time Derivation

The system flags an individual when they have m or more matching attributes. At time t, the per-person false alert probability is

$$q\left(t\right)=Pr\left(\mathrm{Poisson}\right(\lambda \left(t\right))\ge m).$$

With population size n, the probability of at least one false alert is approximately

$$Pr\left(\mathrm{system}\mathrm{false}\mathrm{alert}\mathrm{at}\mathrm{time}t\right)\approx 1-e^{-n·q\left(t\right)}.$$

The system becomes unreliable when $n·q\left(t\right)\gtrsim 1$.

Theorem 2

(System Lifetime Under Exponential Growth). Consider a screening system with exponential data growth $k\left(t\right)=k_0{\gamma }^t$ (where $\gamma >1$), per-attribute match probability p, fixed threshold m, and population size n.

For populations satisfying $n\gtrsim \sqrt{m}exp(m/2)$ (such that system failure occurs near $\lambda \left(T^{*}\right)\approx m$ rather than deep in the Poisson tail where $\lambda \ll m$), the system becomes unreliable at time

$$T^{*}\approx {\displaystyle \frac{1}{log\gamma }}log\left({\displaystyle \frac{m}{k_{0}p}}\right),$$

(8)

where the critical time is characterized by $\lambda \left(T^{*}\right)\approx m$.

The population size n introduces a correction of order ${\displaystyle \frac{1}{log\gamma }\sqrt{\frac{logn}{m}}}$ to the time to failure (equivalently, a shift of order $\sqrt{2mlogn}$ in λ), up to logarithmic factors in m and n. This is typically weak relative to the effects of γ and m.

Proof sketch. 

The system fails when the expected number of false alerts reaches order one: $n·q\left(T^{*}\right)\sim 1$. For $\lambda <m$, the Poisson tail probability $q\left(\lambda \right)=Pr\left(\mathrm{Poisson}\right(\lambda )\ge m)$ is exponentially suppressed by the factor $exp(-\lambda D(m/\lambda \parallel 1\left)\right)$. As $\lambda$ increases from well below m toward m, the rate function $D(m/\lambda \parallel 1)$ decreases to zero, causing $q\left(\lambda \right)$ to increase sharply from nearly zero to order one. When $\lambda >m$, the tail probability increases rapidly toward 1. The transition occurs sharply near $\lambda =m$ due to concentration of the Poisson distribution.

For $\lambda$ near m, we use the normal approximation Poisson($\lambda$) $\approx N(\lambda ,\lambda )$, which yields $q\left(\lambda \right)=Pr(X\ge m)\approx 1-\mathrm{\Phi }((m-\lambda )/\sqrt{\lambda })$. For $\lambda \lesssim m$, the Gaussian tail asymptotics (Mills’ ratio: $1-\mathrm{\Phi }\left(x\right)\sim \varphi \left(x\right)/x$ as $x\to \infty$) give the more precise form

$$q\left(\lambda \right)\sim {\displaystyle \frac{\sqrt{\lambda }}{(m-\lambda )\sqrt{2\pi }}}{e}^{-{(m-\lambda )}^2/\left(2\lambda \right)}(\lambda <m).$$

Setting $nq\left(T^{*}\right)\sim 1$ gives $\lambda \left(T^{*}\right)\approx m-\sqrt{2mlogn}$, which is a small correction to $\lambda \left(T^{*}\right)=m$ for typical parameters. Translating through $\lambda \left(t\right)=k_{0}p{\gamma }^t$ yields a time shift of order ${\displaystyle \frac{1}{log\gamma }}\sqrt{{\displaystyle \frac{logn}{m}}}$. Substituting into $\lambda \left(t\right)=k_{0}p{\gamma }^t$ yields (8) as the leading-order term. □

3.2.3. Key Insights

Equation (8) reveals several fundamental properties:

• Finite lifetime is inevitable: For any fixed threshold m and exponential growth $\gamma >1$, we have $T^{*}<\infty$.
  Even if one attempts to preserve reliability by scaling the threshold with the data ($m\propto \lambda \left(t\right)$), doing so would require the monitored population to grow as $n_{\mathrm{crit}}\left(t\right)\sim exp\left(C{\gamma }^t\right)$ for some $C>0$—a doubly exponential rate that far exceeds any realistic population dynamics.
  Because real surveillance systems use fixed thresholds and monitor approximately fixed populations, temporal failure becomes inevitable rather than merely possible. This mathematical structure parallels the “double birthday paradox” in our earlier work [32], though it arises here in a distinct screening context.
• Logarithmic dependence on threshold: Doubling the threshold m adds only $(log2)/log\gamma$ time units. For annual doubling ($\gamma =2$), this is exactly 1 year. Even increasing m tenfold extends lifetime by only ${log}_2\left(10\right)\approx 3.3$ years.
• Inverse dependence on growth rate: The factor $1/log\gamma$ means that faster data growth dramatically reduces system lifetime. Increasing $\gamma$ from 1.5 to 2 (from 50% to 100% annual growth) roughly halves the operational lifetime.
• Weak population dependence: While larger populations cause slightly earlier failure, this effect is logarithmic in n and secondary to the exponential effects of $\gamma$ and m. The system lifetime is primarily determined by data growth dynamics, not population size.

The temporal phase transition in Figure 1c illustrates these dynamics: a system with 50% annual growth ($\gamma =1.5$) operates reliably for approximately 4 years before crossing the critical threshold, after which false alerts become statistically inevitable.

3.2.4. Quantitative Examples

Example 1

(Border Security System). Consider a border security system with initial attributes $k_0=100$, annual growth rate $\gamma =1.5$ (50%), match probability $p=0.01$, and threshold $m=5$. The critical time is

$$T^{*}={\displaystyle \frac{log(5/(100\times 0.01\left)\right)}{log1.5}}={\displaystyle \frac{log5}{log1.5}}\approx 4.0\mathit{\text{years}}.$$

The lifetime sensitivity reveals fundamental constraints. Increasing m from 3 to 10,

$$\begin{array}{cc}\hfill T^{*}(m=3)& ={\displaystyle \frac{log(3/(100\times 0.01\left)\right)}{log1.5}}={\displaystyle \frac{log3}{log1.5}}\approx 2.7\mathit{\text{years}},\hfill \\ \hfill T^{*}(m=10)& ={\displaystyle \frac{log(10/(100\times 0.01\left)\right)}{log1.5}}={\displaystyle \frac{log10}{log1.5}}\approx 5.7\mathit{\text{years}}.\hfill \end{array}$$

Increasing the threshold from 3 to 10 extends lifetime from 2.7 to only 5.7 years (logarithmic benefit). In contrast, doubling γ from 1.5 to 2.0,

$$T^{*}(m=5,\gamma =2.0)={\displaystyle \frac{log5}{log2}}\approx 2.3\mathit{\text{years}},$$

cuts lifetime nearly in half (from 4.0 to 2.3 years). Growth rate dominates system longevity; threshold adjustments provide minimal protection.

3.2.5. Practical Implications

• Predictable obsolescence: Every system has a calculable expiration date $T^{*}\sim {\displaystyle \frac{1}{log\gamma }}log(m/k_{0}p)$. Monitoring $\lambda \left(t\right)/m$ provides early warning.
• Growth rate dominates: Lifetime scales as $1/log\gamma$; doubling $\gamma$ halves operational lifetime. Threshold adjustments provide only logarithmic benefit.

Remark 6

(Non-Exponential Data Growth). The analysis above focuses on exponential growth, but alternative data growth models lead to qualitatively different system lifetimes. For simplicity, we identify the onset of system unreliability with the heuristic condition $\lambda \left(t\right)=pk\left(t\right)\approx m$, which corresponds to the point at which false alert probabilities increase sharply. We assume $m>p{k}_0$ so that this regime is reached, if at all, at some $T^{*}>0$.

• Polynomial growth $k\left(t\right)=k_0+\gamma t^{\alpha }$ with $\alpha \ge 1$: Solving $p(k_0+\gamma T^{*\alpha })\approx m$ gives

  $$T^{*}\approx {\left({\displaystyle \frac{m/p-k_0}{\gamma }}\right)}^{1/\alpha },$$
  so $T^{*}\propto m^{1/\alpha }$ grows linearly when $\alpha =1$ and sublinearly when $\alpha >1$. Larger values of α correspond to faster growth of $k\left(t\right)$, leading to shorter system lifetimes. Moreover, because $1/\alpha$ decreases with α, increases in m yield progressively smaller gains in $T^{*}$, further reducing threshold protection. Nonetheless, polynomial growth still provides significantly more protection than exponential growth, where $T^{*}$ increases only logarithmically with m.
• Logarithmic growth $k\left(t\right)=k_0+\beta log(1+\gamma t)$: Here

  $$T^{*}\approx {\displaystyle \frac{exp\left((m/p-k_0)/\beta \right)-1}{\gamma }},$$
  so the time required for $\lambda \left(t\right)$ to reach m grows exponentially in the threshold m. Because $k\left(t\right)$ itself grows only logarithmically in t, this time can be extremely large, and practical data collection may saturate before a critical regime is ever reached.
• Bounded growth $k\left(t\right)\to k_{max}$: If the maximal attainable rate ${\lambda }_{max}=p{k}_{max}$ satisfies ${\lambda }_{max}<m$, the system never enters the high-false-alert regime and remains below the critical level for all time.

Exponential growth is the critical case because it eventually exceeds any subexponential threshold-adjustment strategy: if $\lambda \left(t\right)$ grows exponentially while $m\left(t\right)$ increases subexponentially, then $\lambda \left(t\right)>m\left(t\right)$ for all sufficiently large t. Empirical studies suggest that historical data collection capacity has grown approximately exponentially [30], making this the scenario of greatest relevance for policy analysis.

Key Takeaway. 

Under exponential data growth, every screening system has a finite, calculable lifetime $T^{*}\approx {\displaystyle \frac{1}{log\gamma }}log(m/k_{0}p)$ determined by when $\lambda \left(t\right)$ reaches the threshold m. Threshold adjustments provide only logarithmic protection against exponential growth; the growth rate $\gamma$ dominates system longevity, making temporal failure inevitable rather than merely possible.

3.3. Heterogeneous Population Structure

Not all individuals have equal representation in surveillance databases. Socioeconomic, geographic, and demographic factors lead to differential exposure rates [5,33].

3.3.1. Multi-Group Model

Partition the population into G groups such that group g has $n_g$ individuals, with ${\sum }_{g=1}^G{n}_g=n$, and each individual in group g has a per-attribute match probability $p_g$. Let ${\lambda }_g=k{p}_g$ and define the per-group false alert probability:

$$q_g=Pr(\mathrm{Poisson}\left({\lambda }_g\right)\ge m).$$

Proposition 2

(Heterogeneous System Risk). The system-level false alert probability is

$$Pr\left(\mathit{false}\mathit{alert}\right)=1-\prod _{g=1}^G{(1-q_g)}^{n_g}\approx 1-exp\left(-\sum _{g=1}^G{n}_g{q}_g\right).$$

(9)

Proof. 

The event “individual i in group g generates a false alert” is independent across individuals. The probability that no one in group g alerts is ${(1-q_g)}^{n_g}$. Since groups are disjoint, the probability that no one alerts is the product over groups. The approximation follows from ${(1-x)}^n\approx e^{-nx}$ for small x. □

3.3.2. Dominance and Disparity

Theorem 3

(Group Dominance Effect: Algebraic Decomposition). Let $g^{*}={arg\; max}_g\left\{n_g{q}_g\right\}$ be the group contributing the most to system risk. Then,

$$Pr\left(\mathit{false}\mathit{alert}\right)\approx 1-e^{-n_{g^{*}}{q}_{g^{*}}}+O\left(\left(\sum _{g\ne g^{*}}{n}_g{q}_g\right)e^{-n_{g^{*}}{q}_{g^{*}}}\right).$$

(10)

If $n_{g^{*}}{q}_{g^{*}}\gg {\sum }_{g\ne g^{*}}{n}_g{q}_g$, then group $g^{*}$ dominates system behavior. This is an algebraic decomposition showing how system-level risk concentrates in the highest-exposure group.

Proof. 

From Proposition 2,

$$1-exp\left(-\sum _{g=1}^G{n}_g{q}_g\right)=1-exp(-n_{g^{*}}{q}_{g^{*}})·exp\left(-\sum _{g\ne g^{*}}{n}_g{q}_g\right).$$

Using $e^{-x}\approx 1-x$ for small x,

$$\begin{array}{cc}\hfill & \approx 1-e^{-n_{g^{*}}{q}_{g^{*}}}\left(1-\sum _{g\ne g^{*}}{n}_g{q}_g\right)\hfill \\ \hfill & =1-e^{-n_{g^{*}}{q}_{g^{*}}}+e^{-n_{g^{*}}{q}_{g^{*}}}\sum _{g\ne g^{*}}{n}_g{q}_g.\hfill \end{array}$$

The main term is $1-e^{-n_{g^{*}}{q}_{g^{*}}}$, representing the contribution from group $g^{*}$. The correction term is of order $\left({\sum }_{g\ne g^{*}}{n}_g{q}_g\right)e^{-n_{g^{*}}{q}_{g^{*}}}$ and becomes negligible whenever $n_{g^{*}}{q}_{g^{*}}\gg {\sum }_{g\ne g^{*}}{n}_g{q}_g$. □

3.3.3. Numerical Example

Example 2

(Geographic Disparities). Consider a city with two neighborhoods:

• Neighborhood A (low surveillance): $n_A=\mathrm{100,000}$, $p_A=0.005$
• Neighborhood B (high surveillance): $n_B=\mathrm{100,000}$, $p_B=0.02$

With $k=100$ attributes and threshold $m=3$,

$$\begin{array}{cc}\hfill {\lambda }_A& =100·0.005=0.5,q_A=\sum _{j=3}^{\infty }{e}^{-0.5}{\displaystyle \frac{0.5^j}{j!}}\approx 0.014,\hfill \\ \hfill {\lambda }_B& =100·0.02=2,q_B=\sum _{j=3}^{\infty }{e}^{-2}{\displaystyle \frac{2^j}{j!}}\approx 0.323.\hfill \end{array}$$

The expected number of false alerts is

$$\begin{array}{cc}\hfill n_A{q}_A& \approx \mathrm{100,000}·0.014=1400,\hfill \\ \hfill n_B{q}_B& \approx \mathrm{100,000}·0.323=\mathrm{32,300}.\hfill \end{array}$$

Despite equal population sizes, Neighborhood B experiences approximately 23 times more false alerts. This exponential disparity, visualized in Figure 1d, demonstrates how differential exposure rates create fundamentally unequal outcomes that cannot be remedied through threshold adjustments alone. Moreover, by Theorem 3, since $n_B{q}_B\gg n_A{q}_A$, the system-level false alert probability is dominated by Neighborhood B:

$$Pr\left(\mathit{system}\mathit{false}\mathit{alert}\right)\approx 1-e^{-\mathrm{32,300}}\approx 1.$$

The system is essentially guaranteed to produce false alerts, driven almost entirely by the heavily surveilled neighborhood. This illustrates how heterogeneous exposure not only creates disparate individual-level burdens but also determines aggregate system reliability.

In the UCI Adult Census data, increasing the threshold produces the predicted divergence in group contributions: at $m=8$, a 2% subgroup generates 17% of false alerts, while the lowest-exposure group contributes none. See Appendix C.

3.3.4. Exposure Amplification Through Poisson Tails

Proposition 3

(Exposure Amplification Through Poisson Tails). Consider two groups with equal population sizes ($n_1=n_2=n/2$) but different per-attribute match probabilities $p_1<p_2$. Let ${\lambda }_i=k{p}_i$ and suppose both groups are screened with a common threshold m. Then, the following hold:

• If ${\lambda }_1<m\le {\lambda }_2$, the disparity in expected false alerts is exponential:

  $${\displaystyle \frac{n_2{q}_2}{n_1{q}_1}}={\displaystyle \frac{q_2}{q_1}}\ge exp(c·m)$$

  (11)
  for some constant $c>0$ depending on ${\lambda }_1$ for all sufficiently large m. The disparity grows super-exponentially because $q_1$ decays exponentially while $q_2$ remains bounded away from zero.
• The fraction of false alerts attributable to Group 2 approaches 1 as the exposure gap grows:

  $${\displaystyle \frac{n_2{q}_2}{n_1{q}_1+n_2{q}_2}}\to 1\mathit{\text{as}}{\lambda }_2-m\mathit{\text{increases}}.$$
• For fixed threshold m and exposure ratio $\alpha =p_2/p_1>1$, as k increases, Group 2 enters the critical regime (${\lambda }_2\approx m$) before Group 1, creating a temporal window of maximum disparity.

Proof sketch. 

(1) For ${\lambda }_2\ge m$, we have $q_2\ge Pr(\mathrm{Poisson}\left(m\right)\ge m)\ge 1/2-O\left(m^{-1/2}\right)$ by normal approximation, while $q_1\le e^{-{\lambda }_{1}D(m/{\lambda }_1\parallel 1)}$ is exponentially small (Lemma 1). The ratio grows super-exponentially. (2) Since $q_2/q_1\to \infty$, we have $q_2/(q_1+q_2)\to 1$. (3) Group 2 reaches ${\lambda }_2=m$ at $k=m/p_2<m/p_1$, so disparity is maximal for $k\in (m/p_2,m/p_1)$. Full details appear in Appendix A.3. □

Remark 7

(Within-Group Individual Heterogeneity). Even within a demographic group g, individuals may experience different exposure levels due to occupation, neighborhood, behavioral patterns, or daily routines, leading to person-specific match probabilities $p_g^{\left(i\right)}$. A convenient model treats $p_g^{\left(i\right)}$ as random with mean ${\overline{p}}_g$, for example $p_g^{\left(i\right)}\sim \mathrm{Beta}({\alpha }_g,{\beta }_g)$ with $\mathbb{E}\left[p_g^{\left(i\right)}\right]={\overline{p}}_g$. This induces a mixture of Poisson distributions for match counts whose variance exceeds the Poisson variance of the homogeneous case (classical overdispersion).

The qualitative phenomena we describe persist under within-group heterogeneity. As long as the group-level means ${\overline{p}}_g$ differ, the higher-exposure group reaches false alert thresholds earlier on average. Moreover, within-group heterogeneity may actually increase disparity: overdispersion inflates upper-tail probabilities, making heterogeneous groups more likely to produce false alerts than homogeneous groups with the same mean exposure. A full mixture analysis is beyond our present scope, but the structural exposure–disparity mechanism is robust to this generalization.

Key Takeaway. 

Differential surveillance exposure creates exponential disparities in false alert rates through Poisson tail behavior. Because this mechanism arises from data collection intensity rather than classifier design, neither threshold adjustment nor standard algorithmic fairness interventions can eliminate it. The disparity persists under within-group heterogeneity and may even intensify due to overdispersion. Achieving genuine outcome parity requires equalizing surveillance intensity itself.

3.4. Effective Dimensionality Under Correlation

Real surveillance data exhibit spatial and temporal dependencies that fundamentally alter tail probabilities. While correlation does not change the expected number of matches ($\mathbb{E}\left[{\sum }_i{X}_i\right]=kp$ regardless of dependence), it induces overdispersion that inflates tail probabilities beyond the independent Poisson approximation.

The key insight is that positive correlation reduces the effective degrees of freedom. The standard design effect methodology parameterizes this through an effective sample size $k_{\mathrm{eff}}$ satisfying $\mathrm{Var}\left(Y\right)=k_{\mathrm{eff}}·p(1-p)$. For spatial correlation with exponential decay over area A with correlation length $\xi$,

$$k_{\mathrm{eff}}\approx {\displaystyle \frac{A}{2\pi {\xi }^2}}.$$

(12)

For temporal correlation with correlation time $\tau$,

$$k_{\mathrm{eff}}\approx {\displaystyle \frac{k}{2\tau }}.$$

(13)

This modifies the critical population scale. The exponent $\lambda D(c\parallel 1)$ is reduced by the factor $k_{\mathrm{eff}}/k$, yielding

$$n_{\mathrm{crit}}^{\mathrm{corr}}\lesssim \sqrt{\lambda }exp\left({\displaystyle \frac{k_{\mathrm{eff}}}{k}}·\lambda D(c\parallel 1)\right).$$

(14)

The practical impact is severe. City-scale location monitoring with $k=\mathrm{10,000}$ cells and correlation length $\xi =500$ m yields $k_{\mathrm{eff}}\approx 64$, reducing critical populations by over two orders of magnitude. Year-long daily observations ($k=365$) with $\tau =30$ days yields $k_{\mathrm{eff}}\approx 6$, making reliable surveillance of even small groups challenging.

Remark 8

(Scope: Heuristic Approximations). This analysis uses heuristic approximations rather than rigorous large-deviation theory. The effective degrees of freedom approach captures the dominant effect (variance inflation) but does not constitute a formal theorem. The qualitative conclusion that positive correlation accelerates false alert saturation is robust and well established in the literature on design effects, effective sample size, and correlation-adjusted multiple testing. Full technical details, derivations, and worked examples appear in Appendix B.

4. Discussion

4.1. Bayesian Posterior Reliability and the Base-Rate Trap

The preceding analysis focused on frequentist system reliability: the probability that at least one innocent individual is flagged. However, practitioners ultimately need the posterior probability that a flagged person is truly a target. This Bayesian perspective reveals an even more stringent constraint: in the sparse-target regime (where the expected number of true targets r is small relative to population size), posterior reliability degrades once $nq$ becomes comparable to $rs$ and collapses when $nq\gg rs$. In this regime, flags become epistemically meaningless well before the frequentist transition at $nq\sim 1$.

4.1.1. The Bayesian Framework and Positive Predictive Value

We introduce standard notation from diagnostic testing and forensic statistics [2]:

$$\begin{array}{cc}\hfill \pi & \equiv Pr\left(\mathrm{target}\right)(\mathrm{base}\mathrm{rate}),\hfill \\ \hfill s& \equiv Pr(\mathrm{flag}\mid \mathrm{target})\left(\mathrm{sensitivity}\right),\hfill \\ \hfill q& \equiv Pr(\mathrm{flag}\mid \mathrm{innocent})(\mathrm{false}\mathrm{positive}\mathrm{rate}).\hfill \end{array}$$

By Bayes’ rule, the positive predictive value (PPV) is

$$Pr(\mathrm{target}\mid \mathrm{flag})={\displaystyle \frac{s\pi }{s\pi +q(1-\pi )}}.$$

(15)

In a screened population of size n with r true targets ($\pi =r/n$), the expected number of flagged individuals is

$$\mathbb{E}[\#\mathrm{flags}]\approx rs+nq.$$

Thus, the expected false discovery rate (FDR) is

$$\mathrm{FDR}\approx {\displaystyle \frac{nq}{rs+nq}},\mathrm{PPV}=1-\mathrm{FDR}\approx {\displaystyle \frac{rs}{rs+nq}},$$

(16)

matching (15) exactly.

Remark 9

(Sensitivity Dependence on Threshold). The analysis above treats sensitivity s as constant, but in practice $s=Pr(\mathit{\text{flag}}\mid \mathit{\text{target}})$ typically decreases as the threshold m increases: stricter criteria miss a larger fraction of true targets. A simple parametric model capturing this tradeoff is

$$s\left(m\right)=s_{max}exp\left(-\beta {(m-{\lambda }_{\mathrm{signal}})}_{+}\right),$$

(17)

where $s_{max}$ is sensitivity at low thresholds, ${\lambda }_{\mathrm{signal}}>\lambda =kp$ is the expected number of matching attributes for a true target, $\beta >0$ controls the decay rate, and ${\left(x\right)}_{+}=max(x,0)$. The exponential form reflects the common observation that match-score distributions exhibit approximately exponential tails, though any monotone decreasing form would yield the same qualitative conclusions.

Substituting (17) into the PPV formula (16) reveals competing effects as m increases:

• False positives decrease: For innocent individuals, $q\left(m\right)=Pr(Y\ge m)$ with $Y\sim \mathrm{Poisson}\left(\lambda \right)$ and $\lambda =kp$, and Lemma 1 shows that $q\left(m\right)$ decays very rapidly once $m>\lambda$ (indeed, faster than any fixed-rate exponential in m).
• True positives decrease: $s\left(m\right)$ remains near $s_{max}$ for $m\le {\lambda }_{\mathrm{signal}}$ but then decays exponentially for $m>{\lambda }_{\mathrm{signal}}$.

These effects create an intermediate regime in which PPV may plateau or improve more slowly than the constant-s analysis suggests, particularly once m exceeds ${\lambda }_{\mathrm{signal}}$ and losses in sensitivity offset some of the gains from reduced false positives. Under the model (17), however, Lemma 1 implies that $q\left(m\right)$ eventually decays much faster than $s\left(m\right)$, so $q\left(m\right)/s\left(m\right)\to 0$ and therefore $\mathrm{PPV}\left(m\right)\to 1$ as $m\to \infty$. Any plateau or dip in PPV can therefore occur only over this intermediate range of thresholds, not asymptotically.

Because sensitivity is bounded above by $s_{max}$, the constant-s model used earlier provides an upper bound on achievable PPV for any given threshold: $\mathrm{PPV}\left(m\right)$ is increasing in s, and $s\left(m\right)\le s_{max}$ for all m. Optimal threshold selection ultimately requires specifying both the innocent and target match distributions—equivalently, full ROC curve analysis [23]. Since the signal distribution is application-dependent, we do not pursue this direction here. Of course, driving m to extremely large values also drives the overall flag rate toward zero, so the $\mathrm{PPV}\left(m\right)\to 1$ limit is primarily of theoretical interest; in practice, the operationally relevant regime is the intermediate range of thresholds where nontrivial detection rates are maintained.

4.1.2. Posterior Reliability and Bayesian Critical Scales

In the large-deviation setting of Theorem 1, with threshold $m=c\lambda$ and $c>1$, the false positive rate satisfies

$$q\asymp \kappa \left(\lambda \right)e^{-\lambda D(c\parallel 1)},\kappa \left(\lambda \right)=O\left({\lambda }^{-1/2}\right),$$

up to subexponential prefactors. Combining this with (16) yields an explicit condition for maintaining actionable posterior probabilities.

Remark 10

(Sparse-Target Regime). The following proposition applies to the “needle in a haystack” setting, where r (the expected number of true targets) is fixed or grows slowly while population n increases. If instead $r\propto n$ (constant prevalence), then the ratio $r/n$ is fixed, and the condition for actionable PPV reduces to a bound on $q/s$ relative to $\alpha /(1-\alpha )$, independent of n. The sparse-target regime is the most challenging case for screening systems and is therefore the primary focus of this analysis.

Proposition 4

(Bayesian Critical Population for Actionable PPV). Fix a desired posterior level $\alpha \in (0,1)$ (e.g., $\alpha =0.9$) and sensitivity $s\in (0,1]$. Let r denote the expected number of true targets in the population (so the base rate is $\pi =r/n$). In the sparse-target regime where r is fixed or grows sublinearly with n, the condition

$$Pr(\mathit{\text{target}}\mid \mathit{\text{flag}})\ge \alpha$$

is satisfied whenever

$$n\le {\displaystyle \frac{(1-\alpha )rs}{\alpha q}}.$$

(18)

Under the large-deviation scaling $q\asymp \kappa \left(\lambda \right)e^{-\lambda D(c\parallel 1)}$, the Bayesian critical population size satisfies

$$n_{\mathrm{crit}}^{\mathrm{Bayes}}(\alpha ,s)\asymp {\displaystyle \frac{(1-\alpha )rs}{\alpha }}\sqrt{\lambda }exp\left(\lambda D(c\parallel 1)\right),$$

(19)

where ≍ hides subexponential factors absorbed into $\kappa \left(\lambda \right)$.

Proof. 

From (16), $\mathrm{PPV}\ge \alpha$ iff $rs\ge \alpha (rs+nq)$. Rearranging gives $rs(1-\alpha )\ge \alpha nq$, which yields (18). Substituting the large-deviation scaling for q proves (19). □

When sensitivity varies with the threshold, as in Remark 9, the bound (19) should be interpreted as an upper bound on achievable Bayesian reliability, since $s\left(m\right)\le s_{max}$ for all m.

Remark 11

(The Bayesian Trap). Frequentist reliability deteriorates once $nq\sim 1$, when the system is likely to produce at least one false alert. Bayesian actionability demands the stronger condition $nq\ll rs$. When $rs>1$, there is an intermediate regime where false alerts occur frequently but individual flags retain some evidential value. When $rs<1$ (extremely sparse targets), posterior reliability collapses before the system becomes statistically unreliable. In all cases, if $nq\gg rs$, posterior probabilities decay toward zero even when individual false positives remain rare.

4.1.3. Likelihood Ratios and Classical Fallacies

Analysts often cite tiny false positive rates q or large likelihood ratios

$$L={\displaystyle \frac{s}{q}},$$

and mistakenly infer that $Pr(\mathrm{target}\mid \mathrm{flag})$ is therefore large. This is the classical prosecutor’s fallacy [2]. Bayes’ rule shows that

$${\displaystyle \frac{Pr(\mathrm{target}\mid \mathrm{flag})}{Pr(\mathrm{innocent}\mid \mathrm{flag})}}={\displaystyle \frac{s}{q}}·{\displaystyle \frac{\pi }{1-\pi }}=L\times \mathrm{prior}\mathrm{odds}.$$

(20)

When the base rate $\pi =r/n$ is small, the prior odds can overwhelm any fixed likelihood ratio. Even extremely rare false positives ($q\ll 1$) do not guarantee high PPV. When $\pi \ll q$, most flagged individuals remain innocent despite individually low false positive rates.

4.1.4. Resolving the DNA Database Controversy

Forensic statisticians have long debated the evidential value of DNA “cold hits.” The authors of [1] argued that searching databases weakens evidence by inflating coincidental match probabilities; refs. [2,34] countered that likelihood ratios preserve evidential weight. Our framework resolves this apparent contradiction by identifying the relevant asymptotic regime.

Using (20), the posterior odds after a database match are

$${\displaystyle \frac{Pr(\mathrm{guilty}\mid \mathrm{match})}{Pr(\mathrm{innocent}\mid \mathrm{match})}}={\displaystyle \frac{s}{q}}·{\displaystyle \frac{r/n}{1-r/n}}.$$

The DNA regime. Standard STR genotype profiling yields match probabilities on the order of $q\sim {10}^{-12}$ or smaller. Even with databases of size $n\sim {10}^6$, the product $nq\sim {10}^{-6}\ll 1$ remains far below the critical scale where coincidental matches become probable.

In this extreme regime, likelihood ratios dominate the posterior odds. The prior odds $r/n$ may be small (perhaps ${10}^{-6}$ if we have one suspect among a million), but the likelihood ratio $s/q\sim {10}^{12}$ is so enormous that posterior probabilities remain overwhelmingly high. This validates Balding’s argument: database size does not meaningfully dilute evidential weight when $nq\ll 1$.

The surveillance regime. Multi-attribute surveillance systems operate in a fundamentally different regime. With $\lambda =5$ and threshold multiplier $c=3$ (i.e., $m=15$), we have $q\sim {10}^{-4}$ (Section 3.1). For populations of $n\sim {10}^6$, the product $nq\sim 100\gg 1$ places the system far above the critical scale.

In this regime, prior odds collapse faster than likelihood ratios can compensate. Even if the likelihood ratio $s/q$ is substantial, the prior odds $r/n$ are so unfavorable that posterior probabilities remain low. Stockmarr’s caution applies: match evidence loses evidential weight as search populations grow.

Resolution. The critical scale $n_{\mathrm{crit}}\sim exp\left(\lambda D\right)$ separates these regimes. Stockmarr and Balding are both correct in their respective contexts: DNA forensics operates where $nq\ll 1$ (likelihood-driven), while multi-attribute surveillance operates where $nq\gg 1$ (base-rate-dominated). The apparent contradiction dissolves once we recognize this regime distinction.

4.1.5. Key Takeaways

• Bayesian scaling mirrors frequentist scaling: Bayesian actionability inherits the exponential factor $e^{\lambda D(c\parallel 1)}$ but includes the additional multiplicative factor $(1-\alpha )rs/\alpha$; see (19).
• Posterior collapse can precede frequentist failure: In the sparse-target regime, posterior reliability collapses once $nq$ approaches $rs$, often well before the frequentist transition at $nq\sim 1$.
• Exponential data growth overwhelms adaptation: Reducing q by a factor of $\beta$ requires increasing k by only $log(1/\beta )/\left(pD\right)$, while real-world data growth is exponential: $k\left(t\right)=k_0{\gamma }^t$ (Section 3.2). Thus posterior collapse is temporally inevitable.
• Epistemic saturation: As n grows, base rates $\pi =r/n$ shrink. Even rare false positives become dominated by prior odds, causing $Pr(\mathrm{target}\mid \mathrm{flag})$ to decay toward zero.
• Resolution of the DNA debate: Stockmarr and Balding are correct in different regimes: Balding for $nq\ll 1$ (DNA) and Stockmarr for $nq\gtrsim 1$ (large-scale attribute screening).

Remark 12

(Connections to Classical Statistical Fallacies). This section unifies the base-rate fallacy, the prosecutor’s fallacy, false discovery rate control [15], and the PPV problem in medical screening [35]. Conceptually, these phenomena are identical: all reflect Bayes’ rule under low prevalence and imperfect specificity. The widely cited argument that “most published research findings are false” [36] is the same FDR/PPV problem in another domain.

4.2. Fairness Implications

The Group Dominance Effect (Theorem 3) has important implications for fairness in surveillance systems. When different population groups experience differential surveillance exposure, small differences in exposure rates create exponential disparities in outcomes. Proposition 3 shows that exposure ratios of 2–4 times generate false alert disparities exceeding 20 times near critical thresholds. This exponential amplification, which arises from Poisson tail behavior, means that even modest differences in surveillance intensity produce severe outcome inequalities.

Crucially, these disparities cannot be eliminated through threshold adjustment. Group-specific thresholds merely encode the underlying exposure inequality in a different form. Equalizing outcomes requires equalizing data collection intensity at the source, not algorithmic tuning. Moreover, since the high-exposure group drives system-level false alerts, aggregate reliability metrics obscure concentrated burdens on specific subpopulations, making demographic disaggregation essential for understanding actual system performance.

Remark 13

(Structural vs. Algorithmic Bias). Proposition 3 demonstrates that disparate outcomes arise from the probabilistic structure of screening systems, independent of algorithmic design choices. When different groups experience differential surveillance exposure rates ($p_1\ne p_2$), this mathematically guarantees unequal false positive rates ($q_1\ne q_2$) under any common threshold m, creating disproportionate false alert burdens through Poisson tail behavior.

The exponential amplification in part (1) is particularly striking: when both groups are screened using the same attribute set and threshold, small differences in exposure translate to exponential differences in false alert rates (Figure 1d).

The effect manifests temporally as different groups reach critical false alert rates at different times: Group 2 with exposure rate $p_2=0.020$ fails at $k_2^{*}=350$ attributes, while Group 1 with $p_1=0.005$ remains reliable until $k_1^{*}=1400$. While this fourfold difference in system lifetime simply reflects the fourfold difference in exposure rates (a linear relationship), the amplification becomes exponential when comparing simultaneous false alert rates: at any intermediate $k\in (350,1400)$, Group 2 experiences exponentially more false alerts than Group 1.

Connection to algorithmic fairness. These findings relate directly to classical impossibility theorems in the algorithmic fairness literature. Kleinberg et al. [37] and Chouldechova [19] show that equalizing false positive rates, false negative rates, and calibration is impossible when base rates differ. Our analysis identifies a complementary, and more fundamental, mechanism: surveillance exposure itself creates different effective base rates (${\lambda }_g=k{p}_g$), guaranteeing unequal false alert burdens even before any classifier is applied.

Standard fairness interventions operate at the classifier level: equalized odds [18], demographic parity, and calibration attempt to constrain predictions. None can correct the structural disparity we identify, because it arises from data collection intensity ($p_g$), not from how a classifier processes the collected data. Group-specific thresholds $m_g$ might equalize $q_1$ and $q_2$, but only by encoding exposure inequality directly. Achieving parity requires assigning a higher threshold to the more surveilled group.

This perspective connects to “fairness through awareness” [38] and formalizes “structural bias” arguments from the critical algorithm studies literature [22,33,39]. Disparities can be intrinsic to systems built on heterogeneous data collection, rather than artifacts of biased algorithms or training data.

Remark 14

(Policy Implications). These results suggest that surveillance system audits should perform the following:

1. 

Measure exposure rates ($p_g$) across demographic and geographic groups, not just aggregate false alert rates.

2. 

Recognize that system reliability is bounded by the worst-performing group (Theorem 3), making demographic disaggregation essential.

3. 

Account for temporal dynamics: groups with higher exposure fail first as data accumulates, creating windows of maximum disparity.

4. 

Acknowledge that threshold adjustments cannot eliminate disparities arising from differential exposure; only equalizing $p_g$ across groups can achieve fairness.

4.3. Limitations and Future Directions

This analysis operates under several simplifying assumptions that define its scope and suggest natural directions for future work.

Independence across individuals. Our core results (Theorems 1 and 2) assume statistical independence of match counts across individuals (Remark 2). Common-mode events (mass gatherings, natural disasters, viral social media content, and coordinated activities) introduce positive dependence that would increase false alert rates beyond our bounds. Positive dependence inflates upper-tail probabilities and therefore worsens system reliability relative to the independent case. Extensions via positively associated random variables or Chen–Stein methods [26] could quantify these effects, but our independence-based analysis provides a lower bound on false alert rates.

Binary attributes. We model attributes as binary indicators (match/no-match). Continuous features, count data, or multi-level categorical attributes would require different distributional assumptions and tail bounds. The qualitative insights about combinatorial explosion in high-dimensional spaces should persist, but quantitative thresholds would differ. Extensions to Gaussian or sub-Gaussian attributes would preserve the essential exponential tail behavior underlying our critical-scale results.

Fixed thresholds. Our analysis assumes detection thresholds m are fixed at deployment time. Adaptive systems that adjust thresholds based on observed alert rates or estimated base rates could potentially extend operational lifetimes. However, Theorem 2 suggests fundamental limits: under exponential data growth, even optimally adaptive thresholds would need to grow exponentially to maintain reliability, eventually exceeding meaningful detection capabilities.

Constant sensitivity. The Bayesian analysis (Section 4.1) initially treats sensitivity $s=Pr(\mathrm{flag}\mid \mathrm{target})$ as constant. Remark 9 introduces a simple threshold-dependent model, but a complete analysis would require specifying both the target match distribution and the full signal distribution of true targets and performing ROC optimization [23]. This is inherently application-dependent and beyond our current scope.

Heuristic correlation treatment. The effective degrees of freedom approach (Section 3.4; Appendix B) captures variance inflation but does not constitute rigorous large-deviation analysis. Formal treatment would require specifying mixing conditions or dependency graph structures [26,29]. Our heuristic provides qualitative guidance rather than formal guarantees.

Lack of empirical validation. We have not validated our predictions against operational surveillance data, which is typically proprietary, classified, or subject to confidentiality restrictions. Instead, we use proxy datasets for illustrative rather than validating analyses (Appendix C); these provide qualitative checks but do not constitute validation in the intended deployment environment.

Static population structure. We assume a fixed population composition with stable group sizes $n_g$ and exposure rates $p_g$. Dynamic populations with entry, exit, demographic shifts, and changing surveillance intensity would require stochastic population models. The temporal analysis (Section 3.2) addresses data growth but not population dynamics.

Formal versus heuristic results. Theorems 1, 2, and 3, along with Propositions 1 and 3, are formal results with complete proofs. The effective dimensionality analysis (Appendix B) and sensitivity threshold model (Remark 9) are heuristic approximations.

Broader applicability. While surveillance systems motivated this analysis, the mathematical framework applies to any domain where threshold rules screen large collections across many low-probability binary indicators. The critical population bounds (Theorem 1), temporal saturation dynamics (Theorem 2), and group-level disparity amplification (Theorem 3) characterize generic properties of high-dimensional threshold detection, independent of the specific application. Natural extensions include network intrusion detection, manufacturing quality control, financial fraud screening, medical diagnostic panels, and environmental monitoring systems. The binary indicator assumption could be relaxed to accommodate hybrid frameworks combining discrete and continuous variables [40], though the essential combinatorial explosion in threshold-based screening would persist. We developed the theory through the surveillance lens because it offered the clearest exposition of the societal stakes, but the probabilistic limits derived here constrain any system that aggregates rare coincidences across high-dimensional attribute spaces.

Despite these limitations, the core mathematical structure (exponential scaling of critical populations, finite system lifetimes under data growth, and structural amplification of exposure disparities) should prove robust across modeling variations. Modeling refinements would shift numerical thresholds but not the qualitative scaling laws, which arise from intrinsic high-dimensional coincidence phenomena.

5. Conclusions

We have established sharp probabilistic limits, under standard rare event and independence (or effective independence) assumptions, for large-scale screening systems. The critical population size beyond which false alerts become inevitable scales as

$$n_{\mathrm{crit}}\sim \sqrt{\lambda }exp\left(\lambda D(c\parallel 1)\right),$$

governed by large-deviation rate functions that cannot be circumvented through algorithmic refinement within these models. When data volume grows exponentially, $k\left(t\right)=k_0{\gamma }^t$, any screening system has a calculable operational lifetime

$$T^{*}\approx {\displaystyle \frac{logm-log\left(k_{0}p\right)}{log\gamma }},$$

typically measured in years rather than decades. Threshold adjustments provide only logarithmic protection against exponential growth, making temporal failure inevitable rather than merely possible.

The Bayesian analysis reveals an even more stringent constraint: posterior probabilities collapse when $nq\gg rs$, rendering flags epistemically meaningless well before frequentist reliability fails. This asymptotic framing clarifies the long-standing DNA database controversy: Stockmarr’s caution and Balding’s confidence apply in distinct regimes, separated by the critical scale $n\sim exp\left(\lambda D\right)$.

Differential surveillance exposure creates exponential outcome disparities whenever common thresholds are applied across groups with heterogeneous match probabilities $p_g$. The mathematics guarantees disproportionate false alert burdens through Poisson tail behavior. This is not an artifact of algorithmic bias but a structural inevitability arising from heterogeneous data collection.

These results impose non-negotiable constraints on system design. Operational surveillance systems require (1) explicit expiration dates calculated from data growth rates; (2) demographic exposure audits measuring $p_g$ across population groups; (3) capacity constraints limiting investigable flags; and (4) recognition that system-wide reliability is dominated by the subpopulation with the highest effective match probability. The mathematics is unforgiving: more data does not guarantee better decisions, and exponential growth ensures finite operational windows.

The policy implication is direct: screening systems operating near or beyond their critical parameters will generate false alerts regardless of implementation quality. Designers must either accept these mathematical limits or fundamentally restructure surveillance architectures to avoid accumulating correlated data over time. Threshold refinement and algorithmic optimization cannot solve problems rooted in probabilistic inevitability.