Sensor Fusion with Asynchronous Decentralized Processing for 3D Target Tracking with a Wireless Camera Network

Abstract

We present a method to acquire 3D position measurements for decentralized target tracking with an asynchronous camera network. Cameras with known poses have fields of view with overlapping projections on the ground and 3D volumes above a reference ground plane. The purpose is to track targets in 3D space without constraining motion to a reference ground plane. Cameras exchange line-of-sight vectors and respective time tags asynchronously. From stereoscopy, we obtain the fused 3D measurement at the local frame capture instant. We use local decentralized Kalman information filtering and particle filtering for target state estimation to test our approach with only local estimation. Monte Carlo simulation includes communication losses due to frame processing delays. We measure performance with the average root mean square error of 3D position estimates projected on the image planes of the cameras. We then compare only local estimation to exchanging additional asynchronous communications using the Batch Asynchronous Filter and the Sequential Asynchronous Particle Filter for further fusion of information pairs’ estimates and fused 3D position measurements, respectively. Similar performance occurs in spite of the additional communication load relative to our local estimation approach, which exchanges just line-of-sight vectors.

1. Introduction

Tracking multiple targets in a distributed-processing wireless-communication network (WCN) of cameras encompasses several areas such as surveillance, computer vision, threat detection [1], inertial navigation aided by additional sensors, and autonomous vehicles. In the tracking process investigated here, we assume fixed cameras whose poses are known and with computational processing resources (network nodes). The cameras estimate the location of a moving object of interest, here called a target, over time [2] in three-dimensional (3D) space. Network topology affects the distributed processing that aims at greater flexibility and robustness [3], and target tracking is expected to become less likely subject to failure. Each network node implements detection algorithms and state estimators, where state estimate refers to the position and velocity of the estimated target. Detected targets are represented by the location of their centroids on the image plane, and the corresponding measurements are the line-of-sight (LoS) vectors. Target 3D position arises from intersecting LoS vectors measured by neighboring nodes covering the 3D space the target traverses. Hence, the target 3D motion is not restricted to a reference ground plane, and targets can move freely in the 3D space covered by the field of view of each camera in the network. The LoS measurements are usually noisy or erroneous owing to the limitations of computer vision algorithms [4]. With a network of cameras, it is possible to improve target state estimation via fusion of information from different cameras observing the same target.

Two approaches for information exchange among cameras are found in the literature: measurement information exchange [3,5,6] and state estimate information exchange [7,8,9,10,11,12,13,14,15,16,17,18]. The former computes the fusion of exchanged measurements and subsequently proceeds to target state estimation. Concerning the target state estimate exchange, the i-th camera fuses the local state estimate with the same target’s state estimate received from neighboring cameras.

Each camera node has a local clock representing the timeline for each of its pre-programmed actions. The network nodes share the same inter-frame interval, though frame capture instants are asynchronous [8]. Moreover, for each captured frame, there is a processing delay when detecting the targets and computing the measurements. Here, this measurement is the 3D target position in world Cartesian coordinates from intersecting local and received LoS vectors of cameras with known poses followed by the unscented transform (UT) [19]. Processing delay and asynchronous capture are both integer multiples of the inter-frame interval and dealt with by means of temporal alignment of the 3D position measurements with the local frame capture instant and measurement fusion by averaging the aligned 3D position measurements. Then, we proceed with the local estimation of either the target information pair with Kalman filtering in information form or the target state estimate with particle filtering.

We compare the above with the Batch Asynchronous Filter (BAF) [15] and the Sequential Asynchronous Particle Filter (SA-PF) [17], both Bayesian decentralized filters, in combination with our fused 3D position measurement approach. BAF and SA-PF originally exchange information other than just asynchronous line-of-sight vectors to targets, respectively, asynchronous information pairs’ estimates, and fused 3D position measurements.

BAF exchanges asynchronous information pair estimates among neighboring cameras from local Kalman information filters, aligns them temporally at the frame capture instant for Kullback–Leibler Average (KLA) fusion [20], and proceeds to target information pair update followed by time propagation to the next local capture instant.

SA-PF [17] asynchronously exchanges fused 3D position measurements among neighboring cameras, then propagates each local particle state estimate to the reception instant of each measurement in the local reception sequence, iteratively updates each local particle weight with the likelihood of each received measurement, updates the local state estimate, and propagates to the next local capture instant.

1.1. Related Work

Target tracking in a distributed asynchronous camera network can either use a sequential method [13] or a batch method [6]. In sequential methods, a network node computes the target state not only when local measurements are obtained but also when the same node receives information from neighboring camera nodes. The nodes do not propagate the estimates received from neighboring nodes to alignment with the local frame capture instant prior to update; the reception instants are used as the sending camera frame capture instants and the probability density for target state estimation; namely, the particle weights are updated sequentially [15,17]. The SA-PF filter disregards the processing delays.

In batch processing methods, the update with data information received asynchronously from other nodes occurs after the received data are propagated to the local frame capture instant for the purpose of time alignment [6,15]. BAF [15] considers the asynchronous frame capturing among cameras and the processing delay in the estimation and fusion phases at each camera. In the estimation phase, a camera obtains the 3D position measurement assuming the target motion develops on a ground reference plane and estimates the information pair at the frame capture instant. Cameras exchange local target state information, and all information received during a time window period is stored in a buffer. At the end of the time window, a camera enters the fusion phase and propagates the received target information pairs of other cameras either backward or forward to temporal alignment with the local frame capture instant [21].

Previous work mostly related to target tracking with a camera network used information exchange of two-dimensional target states restricted to move on a reference ground plane [2,3,7,8,10,13,14,15,17,21]. However, much less attention has been given to the often-occurring scenario in which targets move in 3D space while taking into account network asynchrony and delays in image processing. For example, in [22], multi-view histograms were employed to characterize targets in 3D space using color and texture as the visual features; the implementation was based on particle filtering. In [23], a deep network model was trained with an RGB-D dataset, and then the 3D structure of the target was reconstructed from the corresponding RGB image to predict depth information. In [24], images with a detected hand and head were used with each camera transmitting to a central processor using a low-power communication network. The central processor performed 3D pose reconstruction combining the data in a probabilistic manner. In [16], state estimation with complete maximum likelihood information transfer produced 3D cartesian estimates of target position and velocity from azimuth and elevation provided by multiple cameras. Ref. [25] addressed the competing objectives of monitoring 3D indoor structured areas against tracking multiple 3D points with a heterogeneous visual sensor network composed of both fixed and Pan-Tilt-Zoom (PTZ) controllable cameras. The measurement was the projection of a distinct 3D point representing the target onto an image plane wherein all cameras followed the pinhole model with unit focal length. The authors applied decentralized EKF information filtering, neither handling camera network asynchrony nor the issue that in real image sequences a target occupies a volume; thus, cameras aiming at a target from different directions see quite distinct portions of the target surface and different 3D points.

None of the above methods considered frame processing delays and network asynchrony among camera nodes.

Reference [26] applied DAT2TF to an asynchronous network of 2D radars that provided local Kalman filter state estimates to a central fusion node performing global estimation of the 3D target state. While focusing on the fusion of state estimates, a linearized approach is highlighted to determine the Cramer–Rao estimation error uncertainty bound when converting radar measurements from polar to Cartesian coordinates, and the authors claim the result is equivalent to decentralized filtering.

1.2. Our Proposal

To track multiple actual targets undergoing unrestricted motion in a real image sequence with a network of calibrated cameras whose poses are known, we propose the decentralized approach wherein our focus is on the fusion of 3D position measurements combined with only local estimation, without involvement of a central node. For that, we need to invert the projection function from 3D space to the image plane with stereoscopy and the known camera poses and calibration data:

Step 1: Within a time window about the local frame capture instant and among neighboring cameras, transmit and receive asynchronously LoS vectors represented in world coordinate system {W} to a target centroid in the image plane and the corresponding time stamps among neighboring cameras within a time window about the frame capture instant [15];

Step 2: Use the unscented transform (UT) and stereovision principles to acquire 3D position measurements and covariances from the received LoS vectors when they intersect to within a 3D spatial distance threshold;

Step 3: Propagate forward or backward in time the 3D position measurements and respective covariances via interpolation until temporal alignment with the local frame capture instant;

Step 4: Fuse locally the 3D position measurements and covariances at the frame capture instant by means of averaging, use them to update the local estimate, and propagate to the next frame capture instant.

We compare our approach to SA-PF and BAF wherein additional communication is exchanged other than just LoS vectors to measure 3D positions. After the aforementioned time window in Step 1, each camera starts a fusion window after the time window to receive and process the additional communications. In SA-PF, a camera transmits and receives fused 3D position measurements and respective covariances at distinct instants in the fusion window. Differently, BAF receives asynchronous information pairs’ estimates at distinct instants in the fusion window.

Both BAF and SA-PF are compared to our proposal with the average root mean square error in the image planes of the cameras tracking the various targets in the scene. The comparison also aims at investigating whether there is a relevant change in performance when a camera fails to receive communications due to ongoing frame processing.

1.3. Contributions

Our innovative approach directly measures the 3D centroid position of an actual target with stereoscopy as different, calibrated cameras with known poses view distinct portions of the surface of the same target. Moreover, we explicitly dispense with the fusion node to add robustness to target tracking via decentralized processing in a camera network with partial connectivity. Our focus is on the fusion of 3D position measurements and respective covariances computed with LoS vectors received asynchronously from neighboring cameras. We also consider the image processing delay of each node when obtaining the local measurements.

We derive LoS vectors from the optical center of each camera to the various target centroids in the respective image plane with the corresponding error covariances. Then, we use stereoscopy combined with an unscented transform (UT) that employs the minimum distance search between pairs of LoS vectors to compute the corresponding 3D positions and error covariance in the world reference frame. One LoS vector is from the local camera and the other LoS vector is received asynchronously from any of the neighboring cameras tracking the same target. We handle asynchronous LoS vectors and the corresponding error covariances, all represented in the world coordinate frame, with time propagation from the received instant into alignment with the camera local capture instant. A set of 3D position measurements and covariances arise with representation in the world coordinate frame at the capture instant of the camera. Then, we fuse the measurements and covariances by means of averaging and proceed to local estimation.

We emphasize that our contribution is not in filtering, though we modified both BAF and SA-PF to add a fusion stage to test our fused 3D position measurement approach and local estimation against a further fusion step among the neighboring cameras, and thus investigate the tradeoff between performance and required communications. We find that local estimation provides similar performance with a smaller communication workload.

Moreover, in the real scenario evaluated here, namely the PETS2009 sequence [27], subjects enter and leave the fields of view of cameras the camera network, resulting in targets that merge and separate as they suddenly change motion direction in 3D space. Our approach handles such occurrences gracefully. According to our research, no previous investigation has focused on decentralized target tracking using LoS vectors exchanged asynchronously among neighboring camera nodes with frame processing delay to obtain position measurements in 3D space from stereoscopy principles. In

Table 1. Analytical comparison.

Ref.	Acronym	Characteristics				Fusion Method
		Processing Delays	Aligns Received Data to Local Capture Instant	Data Exchanged among Camera Nodes	Position Representation
[16]	GHF	no	no	State estimates	Polar coordinates	No fusion
[25]	Decentralized EKF	no	no	State estimates	Cartesian coordinates	Sequential
[26]	DAT2TF	no	no	State estimates	Polar coordinates	Sequential
[15]	BAF-LLM	yes	yes	State estimates	Cartesian coordinates	Batch
[17]	SA-PF	no	no	Measurements	Cartesian coordinates	Sequential
This paper [15]	BAF-LLM—local estimation 1	yes	yes 3	(1) LoS vectors to measure 3D positions	Cartesian coordinates	Batch
	BAF-LLM—with further fusion	yes	yes	(2) LoS vectors to measure 3D positions and information pairs’ estimates	Cartesian coordinates	Batch
This paper [17]	PF—local estimation 2	no	yes 3	(1) LoS vectors to measure 3D positions	Cartesian coordinates	Batch
	SA-PF—with further fusion	no	no	(2) LoS vectors to measure 3D positions and fused 3D position measurements	Cartesian coordinates	Sequential

¹ Kalman Filter in information form. ² Particle Filter. ³ 3D position measurements from asynchronously received LoS vectors undergo interpolation to local capture instant for the purpose of time alignment. See Section 2.1.6.

, we address an analytical comparison of methods found in the literature.

1.4. Paper Structure

The paper is organized as follows. Section 2 formulates how to measure 3D position considering the processing delay, network asynchrony, and tracking without constraining target motion to a reference ground plane. Section 3 describes the algorithms to obtain 3D measurements and the modifications to the Bayesian BAF and SA-PF filter algorithms to include an additional fusion window for comparison with just local estimation. Section 4 presents simulation results and discusses performance, and Section 5 concludes the paper.

2. Problem Formulation

Some background is required initially:

-

Variable k instants or time steps within a time window: they are integer multiples of the camera inter-frame interval and represent the capture instants of target centroids, and the instants of LoS vectors are received from neighboring cameras during the time window about the frame capture instant $k_c$ of a local camera. They are stored in the i-th camera node;

-

Frame capture instants $k_c$: the moment when a frame is captured and defines a time window;

-

Here we assume that the communication is ideal among cameras, i.e., there is no communication delays.

Unlike the fully connected network in [15,17], we investigate a partially connected network. Consider a WCN of cameras $C^i$, $i=\left\{1,2,\dots ,N_s\right\}$, where $N_s$ denotes the maximum number of sensors. This network is based on decentralized processing, meaning that each $C^i$ camera node fuses its information with the information received from the set ${\mathcal{N}}^{j,i}$ of camera nodes $C^j,j\ne i$ considered neighbors; i.e., ${\mathcal{N}}^{j,i}$ is the set of cameras with which the i-th camera communicates to track targets in 3D space. The network topology is fixed and independent of the tracked target. The term decentralized indicates that each camera in the network has a communication module, with detection and tracking algorithms independent from the rest of the cameras. There is no fusion center involved. Given that we aim at 3D tracking, the state of the t-th target at the i-th camera is given by ${\mathit{x}}_{t,k_c,3D}^i={\left[x_{t,k_c}^i,{\dot{x}}_{t,k_c}^i,y_{t,k_c}^i,{\dot{y}}_{t,k_c}^i,z_{t,k_c}^i,{\dot{z}}_{t,k_c}^i\right]}^T$, where $\left[x_{t,k_c}^i,y_{t,k_c}^i,z_{t,k_c}^i\right]{ }^T$ represents the position, and $\left[{\dot{x}}_{t,k_c}^i,{\dot{y}}_{t,k_c}^i,{\dot{z}}_{t,k_c}^i\right]{ }^T$ is the velocity at the frame capture instant $k_c$.

The first phase to model this problem involves obtaining the t-th target position measurements ${\mathit{z}}_{t,k_c,3D}^i$ in 3D space for the targets at the i-th camera node from intersecting LoSs measured asynchronously in neighboring camera nodes. Subsequently, at each camera node, position and velocity estimation are initialized for each target. The local estimation algorithms are the Bayesian filters BAF and particle filtering (PF) without engaging in the fusion step though [15,17]. They compute the probability density function of the target pdf $p({\mathit{x}}_{t,k_c,3D}^{i,i}|{\mathit{z}}_{t,1:k_c,3D}^i)$ based on the known predicted posterior pdf $p({\widehat{\mathit{x}}}_{t,k_c,3D}^i|{\mathit{z}}_{t,1:k_c-1,3D}^i)$ corresponding to the previous capture instant $k_c-1$ and propagated to capture instant $k_c$, the state transition pdf $p({\mathit{x}}_{t,k_c,3D}^{i,i}|{\widehat{\mathit{x}}}_{t,k_c-1,3D}^i)$, and the measurement likelihood pdf $p({\mathit{z}}_{t,k_c,3D}^i|{\mathit{x}}_{t,k_c,3D}^{i,i})$. The second superscript in i,i refers to the local state estimate processed at the i-th camera not using information received from a neighboring camera.

We consider the frame processing delay incurred to obtain ${\mathit{z}}_{t,k_c,3D}^i$ and the asynchrony of neighboring camera capture instants. To this end, we borrow from [15] the concept of a time window ${\mathit{K}}_k^i$ about the local camera capture instant $k_c$, wherein LoS vectors are received from neighboring cameras. We define ${\tau }_{k_c}^i$ as the frame processing delay to obtain the measurement ${\mathit{z}}_{t,k_c,3D}^i$ for the i-th camera, and ${\alpha }^{ij}$ is the relative offset between the capture instants of neighboring cameras $C^i$ and $C^j$.

The problems arising with false positives and false negatives in target detection were considered previously [28] and are addressed here as well. Scene complexity (quantity and speed of the targets) [29], construction parameters of the cameras, and accuracy in their calibration [30,31,32] are factors that also hinder target detection. Thus, if the detection algorithm fails to confirm the presence of the same target within the limits of a minimum capture count up to a maximum count, then the target is discarded. A new identifier number (Id) assignment is created, and a new initial target centroid measurement in the image plane, expressed in pixels, is generated when detecting a novel target. We assume the detection algorithm does not incur in a frame processing delay [33] to obtain a target centroid measurement in the image plane, and the delay is solely from computing the position measurement ${\mathit{z}}_{t,k_c,3D}^i$ in 3D space with the received LoS vectors transmitted by neighboring camera nodes.

As stated earlier, the cameras will fuse information from targets detected in their FoVs with information received from the targets of neighboring cameras belonging to ${\mathcal{N}}^{j,i}$. Recall the LoS vector from a single pinhole camera model to a target does not convey depth information for tracking in 3D space [32]. Then, subject to network asynchrony, j-th cameras in the i-th camera neighborhood set ${\mathcal{N}}^{j,i}$ transmit LoS vectors to distinct targets, each target being detected by at least two cameras with overlapping FoVs [30]. We use stereoscopy to measure depth along a LoS vector and determine 3D position as targets move freely. Stereoscopy must have views to the same target from at least two different directions. The LoS vector from the camera’s optical center to the target centroid in the image plane is represented in the global Cartesian coordinate system by means of camera calibration and pose [31] data. In a real scenario, the same target may have very distinct portions of its surface viewed from very different directions by cameras located far apart. In such a case, stereoscopy can produce a biased 3D position measurement located somewhere inside the target surface. A distance tolerance $d_{min}\le \delta$ is established to detect LoS proximity and ascertain that the measured 3D position is consistent with the pair of LoSs involved.

We evaluate three approaches to information exchange:

• First: We propose to exchange among neighboring cameras LoS vector measurements ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k_c+{\tau }_{k_c}^i,i}^w$ to the t-th target from the i-th camera at instant $k_c+$ ${\tau }_{k_c}^i$ within the time window ${\mathit{K}}_k^i$ period, and the data are represented in the world coordinate system {W}. The measured ${\mathit{z}}_{t,k,2D}^i$ (referring to the target centroid in the image plane in pixel coordinates) is transformed into ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ with the camera construction parameters as described in Section 2.1. The fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$ results from averaging intersecting pairs of vector constraints ${\overrightarrow{\mathit{r}}}_{t,k,i}^w, {\overrightarrow{\mathit{r}}}_{t,k,j}^w$ derived from LoS vectors of neighboring j-th cameras pointing at the same t-th target according to a LoS-vector 3D proximity determination algorithm. Local target state estimation uses the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$ to update and predict the state estimate in the next frame capture instant $k_c+1$.
• Second: For comparison, exchange among cameras $C^i$ and $C^j$, where $j \in {\mathcal{N}}^{j,i}$, the 3D fused measurement ${\mathit{z}}_{t,k_c,3D}^i$ for each target, transmitted and instantaneously received at instants within the fusion window with configurable p size after the ${\mathit{K}}_k^i$ time window. This is the used in a sequential approach in SA-PF.
• Third: Again, for comparison, exchange among cameras $C^i$ and $C^j$, where $j \in {\mathcal{N}}^{j,i}, j\ne i$, information pairs the estimates for each target, transmitted and instantaneously received at instants within the fusion window with configurable p size after the ${\mathit{K}}_k^i$ time window. This is used in the batch approach in BAF.

The first approach extends the concept of stereoscopy to neighboring calibrated cameras with known poses and requires a 3D proximity metric among locally detected LoS vectors in the i-th camera and LoS information received from cameras belonging to ${\mathcal{N}}^{j,i}$. Note that internal temporary storage of LoS vectors ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ at each instant within the time window ${\mathit{K}}_k^i$ period is required because the LoS vectors ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ at $C^i$ must be aligned temporally, through time stamps, with the vectors ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w$ received from $C^j$. In [34], the concept of “virtual synchronization” was used along a “dynamic” memory to store the captured data. Based on this concept, sending and receiving information from cameras are concurrent activities at instants $k$ within the time window period.

The information vector ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^i,i}^w$ exchanged between cameras conveys the LoS vector to the t-th target at the time instant $k_c+{\tau }_{k_c}^i$, the time stamp referring to the $k_c+{\tau }_{k_c}^i$ instant and the optical center ${\mathit{O}}_{c,i}^w$ of camera $C^i$ expressed in world coordinates {W}: ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^i,i}^w=\{k_c+{\tau }_{k_c}^i;{\mathit{O}}_{c,i}^w;{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k_c+{\tau }_{k_c},i}^w\}$; the information vector stored for each of the $k$ instants within the time window ${\mathit{K}}_k^i$ period is defined as ${\mathit{i}}_{t,k,i}^w=\{k;{\mathit{O}}_{c,i}^w;{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w\}$. We adopt capital I for the exchanged information vector and small i for the stored information vector.

2.1. Obtaining ${\mathit{z}}_{t,k_c,3D}^i$

To obtain measurements in 3D space, the following steps are required, from capturing the target centroid in the image plane ${\mathit{z}}_{t,k,2D}^i$ in pixels to measuring the centroid ${\mathit{z}}_{t,k_c,3D}^i$ in 3D space in meters.

2.1.1. Target Centroid Detection in the Image Plane in Pixels and Identification Number (Id) Assignment

For each target detected by the i-th camera, the target centroid in the image plane is represented in the image by a red dot, as shown in Figure 1. All targets are assigned an identification number (Id).

Concerning the detection algorithm, accurate data association for maintaining the assignment of an identifier to a target is a challenging problem. The “global nearest neighbor” algorithm [35], which in turn implements the Munkres algorithm [36], was used at each camera for this purpose. However, as shown in Figure 1, an Id assigned to a target by one of the cameras will not necessarily match the Id assigned to the same target by neighboring cameras. For example, in Figure 1, for the same target observed by Cameras 1 and 2, Camera 1 assigned Id = 1 while Camera 2 assigned Id = 3. This issue is addressed in Section 2.3.

Centroid pixels in Figure 1 should be represented in the corresponding camera {C} coordinate system, and the cameras’ calibration will correct the distortions caused by camera construction parameters.

2.1.2. Camera Calibration

All cameras follow the pinhole CCD model [32]. Using intrinsic or construction parameters of a camera, it is possible to correct its distortions and calibrate it so that the distorted pixel of the detected centroid of the t-th target, represented by point ${\mathit{P}}_d$ in Figure 2, corresponds to the corrected position in the image plane represented by point ${\mathit{P}}_u$. The transformation $(x_d,y_d)\to \left(x_u,y_u\right)$ follows [37].

After correcting point ${\mathit{P}}_d$ to point ${\mathit{P}}_u$ in the camera coordinate system {C}, we represent two main points in the world coordinate system {W} for each camera: the optical center point ${\mathit{O}}_{c,i}$ and point ${\mathit{P}}_u$ that give rise to the LoS vector from the i-th camera to the t-th target.

2.1.3. The Optical Center ${\mathit{O}}_{c,i}$ and Point ${\mathit{P}}_u$ Represented in the World Coordinate System {W}: A Preamble for Obtaining the LoS Vector

Given that the extrinsic parameters of camera $C^i$ represent its rotation ${\mathit{R}}^i$ and translation ${\overrightarrow{\mathit{t}}}^i$, the representation of the optical center ${\mathit{O}}_{c,i}$ defined as vector ${\tilde{\mathit{C}}}^i={\mathit{O}}_{c,i}$ [32], represented in camera coordinate system {C}, in world coordinate system {W} is expressed as follows:

$$\left\{C\right\}\to \left\{W\right\}\to {\mathit{O}}_{c,i}^w={}_c{}^w\overrightarrow{\mathit{t}}{}^i=-{}_c{}^w\mathit{R}{}^{i\prime }{\tilde{\mathit{C}}}^i$$

(1)

Point ${\mathit{P}}_u$ is located in image plane, as shown in Figure 2. We must first represent the pixels of ${\mathit{P}}_u$ in coordinates of the camera system {C}:

$$\left\{pixels\right\}\to \left\{C\right\}\to \left(x_u=f\frac{x_c}{z_c},y_u=f\frac{y_c}{z_c},z_c=f\right)\to \left(x_u=x_c, y_u=y_c,f\right)$$

(2)

Denoting the LoS vector ${\mathit{X}}_{t,c}^i=\left[x_c=x_u,y_c=y_u,f\right]$ as the coordinates obtained from point ${\mathit{P}}_u$ for the t-th target, now represented in coordinates of the camera system {C}, we can represent it in a vector base with origin at the camera optical center ${\mathit{O}}_{c,i}$ and parallel to the world coordinate system {W} as follows:

$$\left\{C\right\}\to \left\{W\right\}\to {\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w={}_c{}^w\mathit{R}{}^{i\prime }{\mathit{X}}_{t,c}^i$$

(3)

The rotation matrix ${}_c{}^w\mathit{R}{}^i$ is expressed as follows:

$${}_c{}^w\mathit{R}{}^i\left({\theta }_z,{\theta }_y,{\theta }_x\right)=\left[\begin{array}{ccc}\cos {\theta }_y\cos {\theta }_z& \cos {\theta }_z\sin {\theta }_x\sin {\theta }_y-\cos {\theta }_x\sin {\theta }_y& \sin {\theta }_x\sin {\theta }_z+\cos {\theta }_x\cos {\theta }_z\sin {\theta }_y\\ \cos {\theta }_y\sin {\theta }_z& \sin {\theta }_x\sin {\theta }_y\sin {\theta }_z+\cos {\theta }_x\cos {\theta }_z& \cos {\theta }_x\sin {\theta }_y\sin {\theta }_z-\cos {\theta }_z\sin {\theta }_x\\ -\sin {\theta }_y& \cos {\theta }_y\sin {\theta }_x& \cos {\theta }_x\cos {\theta }_y\end{array}\right]$$

(4)

This is the matrix that performs the rotation from system {W} to system {C} for each camera; its rotation order in camera axes is $zyx$ and angles ${\theta }_z,{\theta }_y,{\theta }_x$ rotate from {C} to {W} [31], then it is necessary to apply the transpose operation $(\prime )$ in Equations (1) and (3).

2.1.4. LoS Vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ and Parametric Equation of the Line

Note in Figure 3 that the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ is simply the point ${\mathit{X}}_{t,c}^i$ multiplied by the transposed rotation matrix of the i-th camera according to Equation (3).

It should be noted that ${}_c{}^w\overrightarrow{\mathit{t}}{}^i$ is a constant vector because it refers to the fixed position of the optical center of the i-th camera. The LoS vector, in turn, depends on the position of the target, referring to point ${\mathit{X}}_{t,c}^i$, in the image plane expressed in camera coordinates system {C}. Writing the constraint vector ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ in the form of a parametric equation [38] yields the following expression:

$${\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right)={}_c{}^w\overrightarrow{\mathit{t}}{}^i+\mathsf{\lambda }{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w, \lambda \in ℝ, \lambda >1$$

(5)

Thus, the vector constraint ${\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right)$ changes when extending the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ with parameter $\lambda$. Figure 4 shows the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ and points related to vectors constraint ${\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right)$ and optical center position ${}_c{}^w\overrightarrow{\mathit{t}}{}^i$ to Target 1 on Camera 1. The last two vectors have their origins coinciding with the origin $\left\{0;0;0\right\}$ of the world coordinate system {W}.

2.1.5. Proximity Metric between Vector Constraints Originating from LoS to a Target

Assume camera $C^i$ receives the information vector ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^j,j}^w$ from one of the transmitting cameras $C^j$. Suppose also that the observed scene contains only one target, and two synchronized cameras—$C^i$ and $C^j$ with ${\alpha }^{ij}=0$—observe it with overlapping FoVs at the same time of capture instant in both cameras; moreover, there is no image processing delay, i.e., ${\tau }_{k_c}^i={\tau }_{k_c}^j=0$. In these circumstances, we adopt for simplicity here subscript k to mean the capture instant $k_c$. The vector constraints ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ and ${\overrightarrow{\mathit{r}}}_{t,k,j}^w$, respectively, are computed from the information vectors of cameras $C^i$ and $C^j$, respectively, and can be written as follows:

$$\begin{array}{c}{\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right)={}_c{}^w\overrightarrow{\mathit{t}}{}^i+\lambda {\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w, \mathsf{\lambda } \in ℝ, \lambda >1\\ {\overrightarrow{\mathit{r}}}_{t,k,j}^w\left(\beta \right)={}_c{}^w\overrightarrow{\mathit{t}}{}^j+\beta {\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w, \beta \in ℝ, \beta >1\end{array}$$

(6)

Then, target triangulation must find ${\lambda }^{*},{\beta }^{*}=\underset{\lambda ,\beta }{\min }d\left({\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right),{\overrightarrow{\mathit{r}}}_{t,k,j}^w\left(\beta \right)\right)=\underset{\lambda ,\beta }{\min } d\left({}_c{}^w\overrightarrow{\mathit{t}}{}^i+\lambda {\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w ,{}_c{}^w\overrightarrow{\mathit{t}}{}^j+\beta {\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w\right)$ that minimize the distance between vector constraints ${\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right)$ and ${\overrightarrow{\mathit{r}}}_{t,k,j}^w\left(\beta \right)$. The concept of distance between reversed straight lines is applied [38,39], and the solution is obtained by calculating the partial derivative of the distance with respect to $\lambda \mathrm{and} \beta$.

$$\{\begin{array}{c}\frac{\partial d\left({\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right),{\overrightarrow{\mathit{r}}}_{t,k,j}^w\left(\beta \right)\right)}{\partial \lambda }=0\\ \frac{\partial d\left({\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right),{\overrightarrow{\mathit{r}}}_{t,k,j}^w\left(\beta \right)\right)}{\partial \beta }=0\end{array}.$$

(7)

Solving (7) produces the optimized nonlinear parameters expressed as follows:

$$\{\begin{array}{c}{\lambda }^{*}= \frac{\left({\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w·{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{\mathit{t},k,j}^w\right)\left[\left({}_c{}^w\overrightarrow{\mathit{t}}{}^i-{}_c{}^w\overrightarrow{\mathit{t}}{}^j\right)·{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w\right]-{\overrightarrow{||\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w{||}^2\left[\left({}_c{}^w\overrightarrow{\mathit{t}}{}^i-{}_c{}^w\overrightarrow{\mathit{t}}{}^j\right)·{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w\right]}{{\overrightarrow{||\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w{||}^2{\overrightarrow{||\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w{||}^2-{\left({\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w·{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w\right)}^2}\\ {\beta }^{*}=\frac{\left({}_c{}^w\overrightarrow{\mathit{t}}{}^i-{}_c{}^w\overrightarrow{\mathit{t}}{}^j\right)·{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w+{\lambda }^{*}\left({\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w·{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w\right)}{{\overrightarrow{||\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w{||}^2}\end{array}$$

(8)

The operator $·$ represents the inner product between the vectors, and $||.||$ represents the vector 2-norm operator. Appendix A shows how to find the optimized parameters ${\lambda }^{*} \mathrm{and} {\beta }^{*}$.

Once obtained, these parameters are substituted in (6), and the minimum distance occurs where the 3D position is in ${\overrightarrow{\mathit{r}}}_{t,k,i}^w\left({\lambda }^{*}\right)$. A tolerance $d_{min}\le \delta =0.3$ m was chosen to indicate whether the pair of LoS vectors point at the same target. Figure 5 shows LoS vectors ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ and ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,j}^w$ pointing at the same target.

The above description concerns information vectors exchanged synchronously. Next, we analyze asynchronous information exchange considering the processing time required for obtaining position measurements in 3D space.

2.1.6. Asynchronous Exchange of Information Vectors

Figure 6 shows the top and front views of the t-th target centroid trajectory during time window ${\mathit{K}}_k^i$ in one of the cameras, assuming $k_c=k_3$ to be the frame capture instant. Both the optical center ${\mathit{O}}_{c,i}^w$ and the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ that define the vector constraint ${\overrightarrow{\mathit{r}}}_{t,k,i}^w\left(\lambda \right)$ are observed in Figure 6. The frame processing delay of the detection algorithm is assumed negligible relative to acquiring the 3D position measurements of targets’ centroids instants $k$ within the time window ${\mathit{K}}_k^i$.

An example of a target centroid trajectory in the image plane expressed in world coordinates {W} is depicted in Figure 7.

Following an idea reported in [34], the information vectors ${\mathit{i}}_{t,k,i}^w$ are stored for each k-th instant within the time window ${\mathit{K}}_k^i$, as can be observed in Buffer 1 of the i-th camera in Figure 8. They are subsequently replaced by information vectors of the next time window ${\mathit{K}}_{k+1}^i$. Note that to each detected t-th target a buffer is allocated to store its information vectors.

Information vectors are sent and received by the i-th camera within the time window about the capture instant $k_c$, expressed as ${\mathit{K}}_k^i=\left[k_c-{\alpha }_1,k_c+{\alpha }_2\right]$, $\left({\alpha }_1>0 \mathrm{and} {\alpha }_2>{\tau }_{k_c}^i\right)$. The information vectors ${\mathit{i}}_{t,k,i}^w$ of the t-th target in the i-th camera (Buffer 1) are aligned according to their time stamps with respect to time stamps in the information vectors of various targets received from the j-th neighboring cameras (Buffer 2). After the temporal alignment, $C^i$ will contain local information vectors (Buffer 1) and received information vectors from the j-th neighboring cameras (Buffer 2), as depicted in Figure 9.

After temporal synchronization of the information vectors, the proximity metric algorithm is initialized to determine the distance between constraint pairs ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ and ${\overrightarrow{\mathit{r}}}_{t,k,j}^w$. The 3D position measurement in 3D space ${\mathit{z}}_{t,k,3D}^i$ results from local constraints ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ and constraints ${\overrightarrow{\mathit{r}}}_{t,k,j}^w$ received from neighboring cameras within each time window ${\mathit{K}}_k^i$ that are found to point to the same t-th target at the same instant k when distance between the constraint pair is less than tolerance $\delta$. A polynomial interpolates the 3D position measurements of the same target ${\mathit{z}}_{t,k,3D}^i$, propagating these 3D measurements backward or forward in time within the time window ${\mathit{K}}_k^i$ until alignment with the i-th camera captures instant $k_c$, and fuses these 3D measurements by averaging them, thus producing the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$. An internal variable $N_c$ defined for each t-th target counts the number of cameras tracking the same target for the purpose of averaging. Then follows local estimation via update with measurement ${\mathit{z}}_{t,k_c,3D}^i$ and time propagation to the next capture instant, either with BAF or SA-PF, without engaging in any further fusion phase and hence reducing the communication load.

For comparison, the fusion window takes place at the end of each time window ${\mathit{K}}_k^i$ at time $\left(k_c+{\alpha }_2\right)$, either using information pairs’ estimates in BAF or fused 3D position measurements in SA-PF received asynchronously from neighboring cameras. As described in [15], each camera node knows the maximum relative offset ${\alpha }^{max}=\underset{i,j}{\max }\left\{{\alpha }^{ij}\right\}$ and the maximum and minimum processing delays ${\tau }^{max}=\underset{i,k_c}{\max }\{{\tau }_{k_c}^i\}$ e ${\tau }^{min}=\underset{i,k_c}{\min }\{{\tau }_{k_c}^i\}$. To avoid the next capture instant $k_c$ coinciding with the fusion phase of the previous time window, a separation $T={\alpha }_1+{\alpha }_2$ between consecutive capture instants must exist where ${\alpha }_1={\alpha }^{max}-{\tau }^{min}$ and ${\alpha }_2={\alpha }^{max}+{\tau }^{max}$ [15]. Figure 10 shows the timeline that describes how the actions elapse within time window ${\mathit{K}}_k^i$. The algorithm that obtains the 3D position measurements takes into account the information vectors ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^i,j}^w$ exchanged at the time window within the period $[k_c-{\alpha }_1,k_c+{\tau }_{k_c}^i]$. The estimation phase takes place at $k_c+{\tau }_{k_c}^i$ using the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$.

An important point addressed next is how to model the uncertainties inherent to the whole process described above and thus find the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$.

2.1.7. Obtaining the 3D Position Measurement at Capture Instant $k_c$

In the initialization of the proximity metric algorithm to determine the distance between constraint pairs ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ and ${\overrightarrow{\mathit{r}}}_{t,k,j}^w$, we model the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w$ with the “Unscented Transform” (UT) [19,40] to approximate the statistics of the random LoS variables transformed by the nonlinear function that determines depth from stereo. The uncertainty in the image plane is a three-dimensional Gaussian distribution $\mathcal{N}\left({\mu }_x,C_x\right)$ that also accounts for camera calibration error in the focal length $f$, where ${\mu }_x$ is the mean, and $C_x$ is the covariance. This Gaussian distribution defines seven sigma points [40]. Figure 11 shows the 3D sigma points along the three axes of the i-th camera due to uncertainty in the detected centroid location in the image plane that gives rise to the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w={}_c{}^w\mathit{R}{}^{i\prime }{\mathit{X}}_{t,c}^i$ represented in the world coordinate system {W}.

Given that the dimension of ${\mathit{X}}_{t,c}^i$ is $L=3$, then $2L+1=7$ sigma points in three-dimensions are generated in the image plane and along the optical axis, as shown in Figure 12. Consequently, seven constraints ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ are found for the same t-th target. The sigma points in the image plane are generated for each target from information that camera $C^i$ either detects locally or receives information about from a neighboring camera $C^j$. The mean ${\mu }_x$ of the Gaussian distribution is the LoS vector ${\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w={}_c{}^w\mathit{R}{}^{i\prime }{\mathit{X}}_{t,c}^i$ shown in Figure 11, represented by a red arrow.

The proximity metric determines the distance (a nonlinear function) between each candidate constraint pair, i.e., ${\overrightarrow{\mathit{r}}}_{t,k,i}^w$ and ${\overrightarrow{\mathit{r}}}_{t,k,j}^w$ (amounting to a total of 49 candidate pairs, but only the 7 points with minimum distance are considered for each constraint pair). Any point in 3D space along constraint ${\overrightarrow{\mathit{r}}}_{t_a,k,i}^w$ whose distance to another candidate constraint ${\overrightarrow{\mathit{r}}}_{t_b,k,j}^w$ falls within the proximity tolerance $d_{min}\le \delta$ indicates that targets $t_a$ seen by $C^i$ and $t_b$ seen by $C^j$ are the same target, and a set of points in 3D space are thus obtained. Figure 13 shows the result of proximity determination between a constraint to a target detected locally by Camera 1 and each of the constraints to targets received in information vectors from neighboring cameras. In this case, Cameras 2, 3, 4, and 5 send constraints ${\overrightarrow{\mathit{r}}}_{t,k,j}^w$ to Camera 1 within time window ${\mathit{K}}_k^1$ and some of them are relative to the same Target 1.

Each set of 3D points with corresponding UT mean and covariance (${\mu }_{fk},C_{fk})$ after averaging refers to a time instant within the time window ${\mathit{K}}_k^i$. Notice that more than one such set may relate to the same instant k when the UT processes simultaneous constraints that originate in information vectors transmitted by n neighboring cameras tracking the same target at instant k. The representative point in 3D space and respective covariance at instant k, ${\mu }_{fk}$, and $C_{fk}$ are the average over the n UT means and covariances ${\mu }_{jk},C_{jk}, j\in {\mathcal{N}}^{j,i}$ of simultaneous sets of points in 3D space at instant k.

The final 3D mean, ${\mu }_{fk}$, averages over the n simultaneous UT means ${\mu }_{1k},\dots ,{\mu }_{nk}$ in 3D space at instant k as follows:

$${\mu }_{fk}=\frac{{\mu }_{1k}+{\mu }_{2k}+\dots +{\mu }_{nk}}{n}.$$

(9)

Likewise, $C_{fk}$ averages over simultaneous UT covariances $C_{1k},\dots ,C_{nk}$. Figure 14 depicts this situation, where five UT means are obtained at instant $k=13, \mathrm{one} \mathrm{at} \mathrm{instant} k=14, \mathrm{and} \mathrm{one} \mathrm{at} \mathrm{instant} k=15$. The result is the average of the UT means at each $k$ time instant, as depicted in Figure 15. The corresponding UT covariance average is not shown in Figure 15.

Thus, 3D points are measured at each instant $k$ within period $[k_c-{\alpha }_1,k_c+{\tau }_{k_c}^i]$ of the time window ${\mathit{K}}_k^i$. Two conditions must be met for these measurements to be generated: (1) at least one information vector ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^j,j}^w$ is received from a neighboring camera at any instant $k$ within the time window period $[k_c-{\alpha }_1,k_c+{\tau }_{k_c}^i]$, and (2) this information vector relates to the same target observed at the i-th camera. These two conditions are not always met; in that case, the 3D position of the target centroid is not measured at all instants within the aforementioned period. At least two 3D position measurements at different instants within the time window period $[k_c-{\alpha }_1,k_c+{\tau }_{k_c}^i]$ are necessary to derive a linear polynomial and interpolate from the target 3D position measurements forward or backward in time until the capture instant $k_c$ to compute the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$. This interpolation process occurs at the instant $k_c+{\tau }_{k_c}^i$, and ${\mathit{z}}_{t,k_c,3D}^i$ is the average of the 3D UT means measured within the period $[k_c-{\alpha }_1,k_c+{\tau }_{k_c}^i]$ and interpolated in time until the capture instant $k_c$. The corresponding fused covariance ${\mathit{C}}_{t,k_c,3D}^i$ is the average of UT covariances interpolated in time until the capture instant.

In essence, our approach provides the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$ and respective covariance ${\mathit{C}}_{t,k_c,3D}^i$ from the following steps:

(1) The 3D position measurements ${\mathit{z}}_{t,k,3D}^i$ and covariances ${\mathit{C}}_{t,k,3D}^i$ are output by the i-th camera local proximity algorithm at each k instant within period $[k_c-{\alpha }_1,k_c+{\tau }_{k_c}^i]$ if the two conditions above are met;

(2) Then, a linear polynomial interpolation propagates ${\mathit{z}}_{t,k,3D}^i$ and ${\mathit{C}}_{t,k,3D}^i$ backward or forward in time until alignment with the i-th camera local capture instant $k_c$;

(3) Finally, the fused 3D position measurements ${\mathit{z}}_{t,k_c,3D}^i$ and covariances ${\mathit{C}}_{t,k_c,3D}^i$ result from averaging the interpolated measurements and covariances at capture instant $k_c$.

A remark is that when just one 3D position measurement is generated at an instant that differs from $k_c$, the measurement is discarded because there is no other point to derive a linear interpolating polynomial, and the next time window is awaited. By contrast, when just a single 3D position measurement occurs at frame capture instant $k_c$, this measurement is accepted as ${\mathit{z}}_{t,k_c,3D}^i$ with the corresponding covariance ${\mathit{C}}_{t,k_c,3D}^i$.

The fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$ and covariance ${\mathit{C}}_{t,k_c,3D}^i$ at instant $k_c$ are the required information for our decentralized approach with either the local estimation phase at instant $k_c+{\tau }_{k_c}^i$ or further fusion-based estimation with BAF or SA-PF at instant $k_c+{\alpha }_2$.

2.2. Dynamic Target Model: Linear Gaussian Model

The discrete-time dynamic model for each target [41] at each camera, is given by

$${\mathit{x}}_{t,k_c+\mathsf{\Delta }{k}_c,3D}^i=\mathit{F}\left(k_c,k_c+\mathsf{\Delta }{k}_c\right){\mathit{x}}_{t,k_c,3D}^i+\mathit{w}\left(k_c,k_c+\mathsf{\Delta }{k}_c\right)$$

(10)

where $\mathit{F}\left(k_c,k_c+\mathsf{\Delta }{k}_c\right)$ is the state transition matrix from instant $k_c$ to $k_c+\mathsf{\Delta }{k}_c$, defined as

$$\mathit{F}\left(k_c,k_c+\mathsf{\Delta }{k}_c\right)=\left[\begin{array}{cccccc}1& \mathsf{\Delta }{k}_c& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& \mathsf{\Delta }{k}_c& 0& 0\\ 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 1& \mathsf{\Delta }{k}_c\\ 0& 0& 0& 0& 0& 1\end{array}\right]$$

(11)

and $\mathit{w}\left(k_c,k_c+\mathsf{\Delta }{k}_c\right)$ is the cumulative noise between time instants $k_c$ and $k_c+\mathsf{\Delta }{k}_c$, which is considered to be Gaussian with zero mean and covariance $\mathit{Q}\left(k_c, k_c+\mathsf{\Delta }{k}_c\right)$ expressed as follows:

$$\mathit{Q}\left(k_c, k_c+\mathsf{\Delta }{k}_c\right)={\sigma }_u^2\left[\begin{array}{cccccc}\frac{\mathsf{\Delta }{k}_c{}^3}{3}& \frac{\mathsf{\Delta }{k}_c{}^2}{2}& 0& 0& 0& 0\\ \frac{\mathsf{\Delta }{k}_c{}^2}{2}& \mathsf{\Delta }{k}_c& 0& 0& 0& 0\\ 0& 0& \frac{\mathsf{\Delta }{k}_c{}^3}{3}& \frac{\mathsf{\Delta }{k}_c{}^2}{2}& 0& 0\\ 0& 0& \frac{\mathsf{\Delta }{k}_c{}^2}{2}& \mathsf{\Delta }{k}_c& 0& 0\\ 0& 0& 0& 0& \frac{\mathsf{\Delta }{k}_c{}^3}{3}& \frac{\mathsf{\Delta }{k}_c{}^2}{2}\\ 0& 0& 0& 0& \frac{\mathsf{\Delta }{k}_c{}^2}{2}& \mathsf{\Delta }{k}_c\end{array}\right],$$

(12)

where ${\sigma }_u^2$ is the variance of process noise in the nearly-constant-velocity model [41]. The measurement model for the i-th camera is given by ${\mathit{z}}_{t,k_c,3D}^i=\mathit{H}{\mathit{x}}_{t,k_c,3D}^i+{\mathit{\nu }}^i$, where ${\mathit{\nu }}^i$ is the measurement noise vector with dimensions 3 × 1, which is assumed to be Gaussian with zero mean and covariance matrix ${\mathit{R}}^i$. The latter is the fused covariance matrix ${\mathit{C}}_{t,k_c,3D}^i$ resulting from interpolation into alignment with frame capture instant $k_c$ and averaging as described in Section 3.1, and $\mathit{H}$ is given by

$$\mathit{H}=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 1& 0\end{array}\right]$$

(13)

2.3. Target Information Fusion within the Fusion Window [15]

Our proposal is the use of fused 3D position measurements derived from stereoscopy in combination with local state estimation to track actual moving subjects undergoing free 3D motion within the fields of view of asynchronous, calibrated cameras with known poses and located far apart. In certain circumstances, very distinct parts of a target surface can be viewed from very different viewing directions; thus, the 3D centroid position of such a target lies at some point inside and not on the viewed surfaces. Local estimation uses either the Kalman filter in information form or particle filtering. Then we test additional fusion with either BAF or SA-PF for comparison with just local estimation and investigate the tradeoff with the additional communication load.

For that purpose, SA-PF and BAF have their algorithms modified as follows. After the local estimation at the i-th camera used the fused 3D position measurement from the asynchronous LoS vectors received within the time window ${\mathit{K}}_k^i$, further fusion occurs with the additional content received asynchronously from the neighboring cameras within a fusion window with a configurable p size. This additional content being exchanged is either the information pair (${\tilde{\mathit{y}}}_{t,k_c^{″},3D}^{i,i},{\tilde{\mathit{Y}}}_{t,k_c^{″},3D}^{i,i}$) in BAF or the fused 3D position measurement and covariance (${\mathit{z}}_{t,k_c,3D}^i$, ${\mathit{C}}_{t,k_c,3D}^i$) in SA-PF. Figure 16 represents the actions occurring in two time windows in sequence. The motivation to use SA-PF is that it is not limited to the linear Gaussian assumption of the information filter.

We then investigate whether an improvement occurs with the additional communication exchange in the fusion window. Define $k_c^{″}$ and $k_c^{‴}$ as, respectively, transmit and reception instants when camera $C^i$ exchanges with neighboring cameras $C^j$ either the information pair in BAF or the fused 3D position measurement and covariance in SA-PF. Fusion with additional communication exchange in BAF and SA-PF is as follows:

(1)

BAF fusion: The t-th target information pair is propagated from frame capture instant $k_c$ to $k_c^{″}$, thus yielding (${\tilde{\mathit{y}}}_{t,k_c^{″},3D}^{i,i},{\tilde{\mathit{Y}}}_{t,k_c^{″},3D}^{i,i}$) for $C^i$ transmission. Define $k_c^{‴}$ to be the reception instant at which camera $C^i$ receives an information pair estimate (${\tilde{\mathit{y}}}_{t,k_c^{‴},3D}^{i,j},{\tilde{\mathit{Y}}}_{t,k_c^{‴},3D}^{i,j}$) in the fusion window. We use the Euclidean distance applied to the corresponding 3D centroid position to indicate whether the target is the same. Then, all the received information pairs’ estimates are propagated backward in time to $k_c$ for Kullback–Leibler Average (KLA) fusion with the local i-th camera information pair estimate (${\tilde{\mathit{y}}}_{t,k_c,3D}^{i,i},{\tilde{\mathit{Y}}}_{t,k_c,3D}^{i,i}$), and the result propagated to the next frame capture instant $k_c+1$.

(2)

SA-PF fusion: As stated before, $C^i$ sends the fused 3D measurement ${\mathit{z}}_{t,k_c,3D}^i$ and covariance ${\mathit{C}}_{t,k_c,3D}^i$ at instant $k_c^{″}$ to the neighboring cameras $C^j$ without time propagation from $k_c$ to $k_c^{″}$. At various instants $k_c^{‴}$, $C^i$ receives (${\mathit{z}}_{t,k_c^{‴},3D}^j$,${\mathit{C}}_{t,k_c^{‴},3D}^j$) from neighboring cameras, only once from each $C^j$. Here as well we use Euclidean distance applied to 3D position measurement to indicate whether the target is the same. Fusion occurs in this filter by propagating particles’ state estimates from the capture instant $k_c$ to the distinct receiving instants $k_c^{‴}$ and sequentially updates the respective weights with (${\mathit{z}}_{t,k_c^{‴},3D}^j$,${\mathit{C}}_{t,k_c^{‴},3D}^i$). At the end of the fusion window, $C^i$ updates the state estimate at $k_c$ and propagates to the next frame capture instant $k_c+1$.

3. Asynchronous Bayesian State Estimation

Figure 16 shows how we changed the timeline of the Batch Asynchronous Filter (BAF)-Lower Load Mode (LLM) [15] and the Sequential Asynchronous Particle Filter (SA-PF) [17] to accommodate further fusion. Our purpose here is to investigate whether improved target tracking across the asynchronous camera network results from further communication other than just exchanging LoS vectors among neighboring cameras to compute fused 3D position measurements and then use local estimation.

Neighboring cameras transmit either the information pairs’ estimates (BAF) or the fused 3D position measurement and covariance (SA-PF) at instant $k_c^{″}$ in the end of the time window ${\mathit{K}}_k^i$, and introduce a fusion window with configurable p duration after ${\mathit{K}}_k^i$. At instant $k_c^{‴}$ within the fusion window, neighboring cameras receive asynchronously either information pairs’ estimates (BAF) or fused 3D position measurements and covariances (SA-PF).

Relative to local estimation with fused 3D position measurements, we investigate (a) the KLA-based fusion of received information pairs’ estimates temporally aligned in batch with the local frame capture instant $k_c$ (BAF), or (b) the sequential update of particles’ weights by sampling from a likelihood density function with the received fused 3D position measurements and covariances (SA-PF).

The following Algorithm 1 describes the steps to investigate the effect of additional communications and fusion other than exchanging LoS vectors among neighboring cameras for computing fused 3D position measurements:

Algorithm1. Algorithm to investigate the effect on target tracking of further fusion other than exchanged LoS vectors and local estimation

Input: I–images taken by camera $C^i$ Output: estimated state ${\mathit{x}}_{t,k_c,3D}^i$ and the root mean square error computation ${ϵ}_{t,k_c,2D}^i$

for r Monte-Carlo simulation runs for each camera 1-If t-th target detection is confirmed after three occurrences in images, its position estimate ${\widehat{\mathit{z}}}_{t,k_c,3D}^i$ is computed as follows: for each detected target 2-Detect target centroids in image plane ${\mathit{z}}_{t,k,2D}^i$ within the time window period ${\mathit{K}}_k^i$; 3-Transform measurements ${\mathit{z}}_{t,k,2D}^i$ into stored information vectors ${\mathit{i}}_{t,k,i}^w=\{k;{\mathit{O}}_{c,i}^w;{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k,i}^w\}$; 4-Asynchronous transmission/reception of information vectors ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^i,i}^w=\{k_c+{\tau }_{k_c}^i;{\mathit{O}}_{c,i}^w;{\overrightarrow{\mathit{L}\mathit{o}\mathit{S}}}_{t,k_c+{\tau }_{k_c},i}^w\}$ among cameras $C^i$ and $C^j$, $j\in {\mathcal{N}}^{j,i}, i\ne j$; 5-Compute 3D position measurements ${\mathit{z}}_{t,k,3D}^i$ and covariances ${\mathit{C}}_{t,k,3D}^i$ within the time window ${\mathit{K}}_k^i$ from received LoS vectors, and interpolate all of them into temporal alignment with the frame capture instant $k_c$ to account for frame processing delay and camera asynchrony; 6-Local estimation of the target position and velocity in 3D space ${\mathit{x}}_{t,k_c,3D}^i$; 7-Investigate the added fusion phase with either the BAF or the SA-PF Asynchronously transmit/receive information pairs’s estimates (BAF) or fused 3D position measurements (SA-PF) among cameras $C^i$ and $C^j$, $j\in {\mathcal{N}}^{j,i}, j\ne i$; 8-${\widehat{\mathit{z}}}_{t,k_c,3D}^i$ is the resulting 3D position component of the target state estimate; 9-Perspective projection of the 3D position estimate of each target being tracked onto the 2D image planes of each neighboring camera, expressed in pixel coordinates: ${\widehat{\mathit{z}}}_{t,k_c,3D}^i\to {\widehat{\mathit{z}}}_{t,k_c,2D}^i$; end for; end for; 10-Average root mean square error ${ϵ}_{t,k_c,2D}^i=\sqrt{\parallel {\widehat{\mathit{z}}}_{t,k_c,2D,r}^i-{\mathit{z}}_{t,k_c,2D}^i{\parallel }_2^2}$ to compare performance in steps 6 and 7. end for.

3.1. Batch Asynchronous Filter (BAF)-Lower Load Mode (LLM) [15]

The initialization of each target involves the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$ and respective covariance ${\mathit{C}}_{t,k_c,3D}^i$. Each camera $C^i$ uses Kalman filtering in information form [7]. The fused 3D position measurement pair $({\mathit{z}}_{t,k_c,3D}^i,{\mathit{C}}_{t,k_c,3D}^i)$ is transformed in the measurement information pair $\left({\mathit{H}}^T{\mathit{C}}_{t,k_c,3D}^i{}^{-1}{\mathit{z}}_{t,k_c,3D}^i,{\mathit{H}}^T{\mathit{C}}_{t,k_c,3D}^i{}^{-1}\mathit{H} \right)$.

Local estimation uses the measurement information pair and the information pair (${\widehat{\mathit{y}}}_{t,k_c,3D}^i$,${\widehat{\mathit{Y}}}_{t,k_c,3D}^i$) predicted from the previous frame capture instant to update locally the information pair to $\left({\mathit{y}}_{t,k_{c,},3D}^{i,i},{\mathit{Y}}_{t,k_c,3D}^{i,i}\right)$ of the t-th target at camera $C^i$ where ${\mathit{y}}_{t,k_{c,},3D}^{i,i}={\widehat{\mathit{y}}}_{t,k_c,3D}^i+{\mathit{H}}^T{\mathit{C}}_{t,k_c,3D}^i{}^{-1}{\mathit{z}}_{t,k_c,3D}^i$ and ${\mathit{Y}}_{t,k_c,3D}^{i,i}={\widehat{\mathit{Y}}}_{t,k_c,3D}^i+{\mathit{H}}^T{\mathit{C}}_{t,k_c,3D}^i{}^{-1}\mathit{H}$. Then, ${\widehat{\mathit{z}}}_{t,k_c,3D}^i=\mathit{H}{\left({\mathit{Y}}_{t,k_c,3D}^{i,i}\right)}^{-1}{\mathit{y}}_{t,k_{c,},3D}^{i,i}$ is the 3D position estimate of the target for later use to evaluate performance in step 6 of the Algorithm 1 with the average root mean square error in the image plane.

To investigate the added fusion phase in step 7 of the Algorithm 1, for the purpose of transmission $C^i$ predicts the information pair at instant $k_c^{″}$ within the fusion window illustrated in Figure 16 as follows:

$$\begin{array}{c}{\tilde{\mathit{Y}}}_{\mathrm{t},k_c^{″},3D}^{i,i}={\left(\mathit{F}\left(k_c,k_c^{″}\right){\mathit{Y}}_{t,k_c,3D}^{i,i}{}^{-1}\mathit{F}{\left(k_c,k_c^{″}\right)}^T+\mathit{Q}\left(k_c,k_c^{″}\right)\right)}^{-1}\\ {\tilde{\mathit{y}}}_{\mathrm{t},k_c^{″},3D}^{i,i}={\tilde{\mathit{Y}}}_{\mathrm{t},k_c^{″},3D}^{i,i} \mathit{F}\left(k_c,k_c^{″}\right)\left({\mathit{Y}}_{t,k_c,3D}^{i,i}{}^{-1}{\mathit{y}}_{t,k_c,3D}^{i,i}\right).\end{array}$$

(14)

$C^i$ transmits the predicted information pair $\left({\tilde{\mathit{y}}}_{t,k_c^{″},3D}^{i,i},{\tilde{\mathit{Y}}}_{t,k_c^{″},3D}^{i,i}\right)$ to all cameras $j\in {\mathcal{N}}^{j,i}$. Furthermore, $C^i$ receives at the local instant $k_c^{‴}$ within the fusion window the information pair $\left({\mathit{y}}_{t,k_c^{″},3D}^{j,j},{\mathit{Y}}_{t,k_c^{″},3D}^{j,j}\right)$ from a neighboring camera $C^j$, as depicted in Figure 16. Camera $C^i$ has no knowledge of the frame capture instant of neighboring camera $C^j$. Therefore, $C^i$ reads the information pair received from $C^j$ as $\left({\mathit{y}}_{t,k_c^{‴},3D}^{i,j},{\mathit{Y}}_{t,k_c^{‴},3D}^{i,j}\right)$, which undergoes propagation backward in time to the local frame capture instant $k_c$:

$$\begin{array}{c}{\tilde{\mathit{Y}}}_{t,k_c ,3D}^{i,j}={\left(\mathit{F}\left(k_c^{‴},k_c \right){\mathit{Y}}_{t,k_c^{‴},3D}^{i,j}{}^{-1}\mathit{F}{\left(k_c^{‴},k_c \right)}^T+\mathit{Q}\left(k_c^{‴},k_c \right)\right)}^{-1}\\ {\tilde{\mathit{y}}}_{t,k_c ,3D}^{i,j}={\tilde{\mathit{Y}}}_{t,k_c ,3D}^{i,j} \mathit{F}\left(k_c^{‴},k_c \right)\left({\mathit{Y}}_{t,k_c^{‴},3D}^{i,j}{}^{-1}{\mathit{y}}_{t,k_c^{‴},3D}^{i,j}\right).\end{array}$$

(15)

Fusion of local and received information pairs at camera $C^i$ for each target at instant $k_c$ is obtained with the Kullback–Leibler Average (KLA) [20] as follows:

$$\begin{array}{c}{\widehat{\mathit{Y}}}_{t,k_c,3D}^i=\frac{1}{n\left({\mathcal{N}}^{j,i}\right)+1}\left({\mathit{Y}}_{t,k_c,3D}^{i,i}+{\displaystyle {\displaystyle \sum }_{j,j\in {\mathcal{N}}^{j,i}}^{n\left({\mathcal{N}}^{j,i}\right)}}{\tilde{\mathit{Y}}}_{t,k_c,3D}^{i,j}\right)\\ {\widehat{\mathit{y}}}_{t,k_c,3D}^i=\frac{1}{n\left({\mathcal{N}}^{j,i}\right)+1}\left({\mathit{y}}_{t,k_c,3D}^{i,i}+{\displaystyle {\displaystyle \sum }_{j,j\in {\mathcal{N}}^{j,i}}^{n\left({\mathcal{N}}^{j,i}\right)}}{\tilde{\mathit{y}}}_{t,k_c,3D}^{i,j}\right).\end{array}$$

(16)

where $n\left({\mathcal{N}}^{j,i}\right)$ indicates the number of neighboring cameras in the set ${\mathcal{N}}^{j,i}$ whose information pairs of the t-th target are received within the fusion window of the i-th camera. Then, ${\breve{\mathit{z}}}_{t,k_c,3D}^i=\mathit{H}{\left({\widehat{\mathit{Y}}}_{t,k_c,3D}^i\right)}^{-1}{\widehat{\mathit{y}}}_{t,k_c,3D}^i$ is the fused 3D position estimate of the target in step 7 of the Algorithm 1 for later use to evaluate performance with the average root mean square error in the image plane.

We define as strategy (a) the local estimation in step 6, with exchange of just the information vectors ${\mathit{I}}_{t,k_c+{\tau }_{k_c}^i,i}^w$ conveying LoS vectors. This strategy predicts from the updated (${\mathit{y}}_{t,k_{c,},3D}^{i,i},{\mathit{Y}}_{t,k_c,3D}^{i,i}$) to $\left({\widehat{\mathit{y}}}_{t,k_c+1,3D}^i, {\widehat{\mathit{Y}}}_{t,k_c+1,3D}^i\right)$ in the next capture instant $k_c+1$, and computes the next measurement information pair $\left({\mathit{H}}^T{\mathit{C}}_{t,k_c+1,3D}^i{}^{-1}{\mathit{z}}_{t,k_c+1,3D}^i,{\mathit{H}}^T{\mathit{C}}_{t,k_c+1,3D}^i{}^{-1}\mathit{H}\right)$ based on the fused 3D position measurement and covariance, which we compute with LoS vectors received in the next time window ${\mathit{K}}_{k+1}^i$.

For comparison, we define strategy (b) in step 7 as adding KLA-based fusion with information pairs’ estimates received from neighboring cameras within the fusion window other than the LoS vectors received within ${\mathit{K}}_k^i$. This strategy predicts from the KLA-fused information pair (${\widehat{\mathit{y}}}_{t,k_c,3D}^i,{\widehat{\mathit{Y}}}_{t,k_c,3D}^i$) to $\left({\widehat{\mathit{y}}}_{t,k_c+1,3D}^i, {\widehat{\mathit{Y}}}_{t,k_c+1,3D}^i\right)$ in the next capture instant $k_c+1$ and, as in strategy (a), computes the next measurement information pair $\left( {\mathit{H}}^T{\mathit{C}}_{t,k_c+1,3D}^i{}^{-1}{\mathit{z}}_{t,k_c+1,3D}^i,{\mathit{H}}^T{\mathit{C}}_{t,k_c+1,3D}^i{}^{-1}\mathit{H} \right)$ from the fused 3D position measurement and covariance from LoS vectors received in the next time window ${\mathit{K}}_{k+1}^i$. For clarification, see Figure 17.

The position estimate of the t-th target resulting from strategies (a) and (b) at camera $C^i$ at frame capture instant $k_c$ are, respectively, ${\widehat{\mathit{z}}}_{t,k_c,3D }^i=\mathit{H}{\mathit{Y}}_{t,k_c,3D}^{i,i}{}^{-1}{\mathit{y}}_{t,k_{c,},3D}^{i,i}$ and ${\check{\mathit{z}}}_{t,k_c,3D }^i=\mathit{H} {\widehat{\mathit{Y}}}_{t,k_c,3D}^i{}^{-1}{\widehat{\mathit{y}}}_{t,k_c,3D}^i$.

3.2. Sequential Asynchronous Particle Filter [17]

The initial step for obtaining a fused 3D position measurement pair $({\mathit{z}}_{t,k_c,3D}^i,{\mathit{C}}_{t,k_c,3D}^i)$ at each camera $C^i$ is the same as previously described. Each camera has the same inter-frame period $T={\alpha }_1+{\alpha }_2$ between consecutive frame capture instants as depicted in Figure 10. The model of the t-th target is the same from Section 3.2.

Define $k_c^{\prime }$ at the previous frame capture instant ($k_c^{\prime }=k_c-1)$ and $m=1,\dots ,n\left({\mathcal{N}}^{j,i}\right)$, where we recall that $n\left({\mathcal{N}}^{j,i}\right)$ indicates the number of neighboring cameras $C^j, j\in {\mathcal{N}}^{j,i}$, whose fused 3D position measurements ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ of the t-th target are received within the fusion window of the i-th camera. Each camera $C^i$ receives measurements ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ at distinct instants $k_c^{‴}$ within the fusion window and computes the corresponding state estimates ${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m}$. The updated state estimate in $C^i$ is then computed from ${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m}$, $j_m=\left\{j_1,j_2,\dots ,j_{n\left({\mathcal{N}}^{j,i}\right)}\right\}$, and $j_0$ is the $C^i$ camera. As an example, let camera 2 ($i=2$) receive measurements from $n\left({\mathcal{N}}^{j,i}\right)=3$ neighboring cameras, say, cameras 5, 7, and 9; then $m=1,2,3$, ${\mathcal{N}}^{j,2}=\left\{j_1,j_2,j_3\right\}=\left\{5,7,9\right\}$, and $j_0=2$.

In the prediction process of SA-PF, new particles ${\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$ sample the state space from a proposal pdf $q\left({\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}|{\mathit{x}}_{t,k_c^{\prime },3D}^{i,\left(l\right)},{\mathit{z}}_{t,k_c,3D}^i\right)$ with the set of previous particles ${\mathit{x}}_{t,k_c^{\prime },3D}^{i,\left(l\right)}$ and weights ${\omega }_{t,k_c^{\prime }}^{i,\left(l\right)}$, and a normalization yields weights ${\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}$ associated with particles ${\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$. Then follows the update process; when $C^i$ asynchronously receives a measurement ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ from camera $C^{j_m}$ in the fusion window, the predicted particles ${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)}$ are computed, and the corresponding weights ${\widehat{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l\right)}$ are updated sequentially using measurements ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ and particles ${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)}$ in the likelihood function. After receiving the measurement from the last of the neighboring cameras within the fusion window, the updated state estimate ${\mathit{x}}_{t,k_c,3D}^i$ is then computed with the set of weights ${\omega }_{t,k_c}^{i,\left(l\right)}$ and particles ${\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$.

Prediction. At $k_c=0$, L particles initialize with ${\mathit{x}}_{t,0,3D}^{i,\left(l\right)},l=1,\dots ,L,$ sampled from the known pdf $p({\mathit{x}}_{t,0,3D}^i)$. The weights are initialized equally as ${\omega }_{t,0}^{i,\left(l\right)}=1/L$ for all $l$.

At instants $k_c\ge 1$ with ${\mathit{z}}_{t,k_c,3D}^i$ previously obtained, new particles ${\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$ are propagated from the previous particles ${\mathit{x}}_{t,k_c^{\prime },3D}^{i,\left(l\right)}$ with the deterministic part of the motion model in Equation (10). The corresponding weights ${\tilde{\omega }}_{t,k_c}^{i,\left(l\right)}$ use the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$:

$${\tilde{\omega }}_{t,k_c}^{i,\left(l\right)}={\omega }_{t,k_c^{\prime }}^{i,\left(l\right)}\frac{p({\mathit{z}}_{t,k_c,3D}^i|{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}) p({\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}|{\mathit{x}}_{t,k_c^{\prime },3D}^{i,\left(l\right)})}{q\left({\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}|{\mathit{x}}_{t,k_c^{\prime },3D}^{i,\left(l\right)},{\mathit{z}}_{t,k_c,3D}^i\right)}={\omega }_{t,k_c^{\prime }}^{i,\left(l\right)}p({\mathit{z}}_{t,k_c,3D}^i|{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)})$$

(17)

The above weights are normalized according to ${\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}={\tilde{\omega }}_{t,k_c}^{i,\left(l\right)}/\sum _{l^{\prime }=1}^L{\tilde{\omega }}_{t,k_c}^{i,\left(l^{\prime }\right)}$. The set of particles and respective normalized weights is ${\left\{{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)},{\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}\right\}}_{l=1}^L$. This is the local estimation defined as strategy (a), wherein SA-PF becomes just PF, which does not involve exchange other than LoS vectors, ${\widehat{\mathit{x}}}_{t,k_c,3D}^i=\sum _{l=1}^L{\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$, and follows the propagation to the next frame capture instant $k_c+1$ (step 6 of the pseudoalgorithm).

Update. Strategy (b) or step 7 of the Algorithm 1 involves the fusion window, when $C^i$ receives asynchronously measurements ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ from the neighboring cameras $C^j$ at distinct reception instants $k_c^{‴}$, then SA-PF predicts particles ${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)}$ and updates the weights ${\widehat{\omega }}_{t,k_c^{‴}}^{i,\left(l\right)}$ as in Equations (18) and (19) respectively.

Initially, ${\widehat{\omega }}_{t,k_c^{‴}}^{i,j_0,\left(l\right)}={\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}$ described above and the predicted particles ${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)}$ are:

$${\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)}=E\left\{{\mathit{x}}_{t,k_c^{‴},3D}^{j_m,\left(l\right)}|{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}\right\}=\mathit{F}\left(k_c,k_c^{‴}\right){\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$$

(18)

The corresponding weights ${\widehat{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l\right)},l=1,\dots ,L$ are updated sequentially according to:

$${\tilde{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l\right)}={\widehat{\omega }}_{t,k_c^{‴}}^{i,j_{m-1},\left(l\right)}p\left({\mathit{z}}_{t,k_c^{‴},3D}^{j_m}|{\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)}\right), m=1,\dots ,n\left({\mathcal{N}}^{j,i}\right)$$

(19)

where $p({\mathit{z}}_{t,k_c^{‴},3D}^{j_m}|{\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)})$ is the likelihood function in Equation (21). The weights in Equation (19) are normalized according to ${\widehat{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l\right)}={\tilde{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l\right)}/\sum _{l^{\prime }=1}^L{\tilde{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l^{\prime }\right)}$. The set of particles and normalized weights ${\left\{{\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)},{\widehat{\omega }}_{t,k_c^{‴}}^{i,j_m,\left(l\right)}\right\}}_{l=1}^L$ represent the “partial posterior” $p\left({\tilde{\mathit{x}}}_{t,k_c,3D}^{i\left(j_0:j_m\right)}|{\mathit{z}}_{t,k_c,3D}^{i\left(j_0:j_m\right)},{\mathit{z}}_{t,1:k_c^{\prime },3D}\right)$. This conditioning is on all previous measurements ${\mathit{z}}_{t,1:k_c^{\prime },3D}$, and all current measurements ${\mathit{z}}_{t,k_c,3D}^{i\left(j_0:j_m\right)}$ received at camera $C^i$ during the fusion window at distinct instants $k_c^{‴}$, including the local ${\mathit{z}}_{t,k_c,3D}^i$ and up to the neighboring camera $C^{j_m}$ as indicated by superscript $\left(j_0:j_m\right)$ [17].

This update in SA-PF is sequential until the last remaining measurement from the neighboring cameras is received within the fusion window, and then the “global posterior” weights [17] become ${\omega }_{t,k_c}^{i,\left(l\right)}={\widehat{\omega }}_{t,k_c^{‴}}^{i,j_{n\left({\mathcal{N}}^{j,i}\right)},\left(l\right)}$. The set of particles and respective normalized weights is ${\left\{{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)},{\omega }_{t,k_c}^{i,\left(l\right)}\right\}}_{l=1}^L$, and the state estimate is ${\mathit{x}}_{t,k_c,3D}^i=\sum _{l=1}^L{\omega }_{t,k_c}^{i,\left(l\right)}{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$. Finally, we propagate the particles ${\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}$ to the next frame capture instant $k_c+1$ with the deterministic part of the motion model in Equation (10).

4. Simulation Results

4.1. Evaluation Setup

We use MATLAB [42] and the PETS2009 database [27] with a camera network composed of $N_s=8$ cameras. The targets move within the surveillance volume of a network of calibrated cameras, whose construction (intrinsic) parameters and poses are given in the header of the PETS2009 image sequence dataset file. This dataset is challenging for tracking tasks because targets change direction, and clusters and scattering occur frequently. Scene illumination and obstacles also hinder tracking. In this experiment, the frame sampling rate was 7 frames/s $\approx 143 \mathrm{ms}$, with a total of 794 frames per camera. Each $k$ time step takes approximately $143 \mathrm{ms}$. Figure 18 shows all FoVs projected on the ground reference plane for each of the eight cameras. Portions of some FoVs do not project on the ground and instead cover the scenario above ground. The surveillance area of the cameras is approximately 60 m × 90 m as depicted in Figure 18, and their FoVs overlap. Moreover, the neighborhoods for each camera are defined by the network communication topology also shown. The chosen topology reflects two camera neighborhoods (clusters), one with cameras having large FoVs areas projected on the ground and the other one consisting of cameras with smaller projections. Communications between the clusters occur by means of cameras 4 and 8 and cameras 1 and 5.

BAF assumes the information pdfs are Gaussian whereas PF makes no such assumption. They are evaluated under the two strategies (a) and (b) previously explained.

The random process modeling the processing delay in frame units is ${\tau }_k^i\in \mathcal{U}\left\{0,{\tau }^{max}\right\}$, where $\mathcal{U}$ is the uniform distribution. A uniform distribution is also used for the offset ${\alpha }^{ij}$ between cameras: $\mathcal{U}\left\{0,{\alpha }^{max}\right\}$. Denote ${\mathsf{\Lambda }}^{i,r}\subset \left[1,749\right]$ as the set of frame capture instants during the r-th run of a Monte Carlo simulation, r = 1, …, M of camera $C^i$ tracking a target and ${\mathsf{P}}_k^i\subset \left\{1,\dots ,N_t\right\}$ as the set of targets in the FoV of camera $C^i$ at capture instant $k_c$. The evaluation is based on the average root mean square error:

$$\epsilon =\sqrt{\frac{1}{M{N}_c}{\displaystyle \sum }_{r=1}^M{\displaystyle \sum }_{i=1}^{N_c}\frac{1}{\left|{\mathsf{\Lambda }}^{i,r}\right|}{\displaystyle \sum }_{\forall k\in {\mathsf{\Lambda }}^{i,r}}\frac{1}{\left|{\mathsf{{\rm P}}}_k^i\right|}{\displaystyle \sum }_{\forall t\in P_k^i}||{\widehat{\mathit{z}}}_{t,k_c,2D,r}^i-{\mathit{z}}_{t,k_c,2D}^i{||}_2^2}$$

(20)

where ${\widehat{\mathit{z}}}_{t,k_c,2D,r}^i$ is the estimated 3D position of the t-th target projected according to the calibrated pinhole model onto the image plane of camera $C^i$ at the instant $k_c$ during the r-th round, and ${\mathit{z}}_{t,k_c,2D}^i$ corresponds to the actual measurement in the image plane as provided by the target detection algorithm. The latter does not change with $r$, since the image sequence dataset is given and processed previously by the target detection algorithm. ${\sigma }_{\mathit{u}}^{\mathit{2}}=4$ in Equation (12), and ${\mathit{R}}^i={\mathit{C}}_{t,k_c,3D}^i$ in the information filter with the measurement information pair $\left({\mathit{H}}^T{\mathit{C}}_{t,k_c,3D}^i{}^{-1}{\mathit{z}}_{t,k_c,3D}^i,{\mathit{H}}^T{\mathit{C}}_{t,k_c,3D}^i{}^{-1}\mathit{H} \right)$ and in the likelihood function in Equation (21). $\mathit{F}$ and $\mathit{Q}$ are given by the dynamic model of the target. The error $\epsilon$ statistics converged with $M=12$ runs.

We skip dataset frames to simulate asynchronous frame capture and frame processing delay. $\epsilon$ is analyzed by increasing the relative offset $0\le {\alpha }^{max}\le 40$ (see Figure 19) with a fixed offset ${\alpha }^{max}$ and processing delay ${\tau }_k^i={\tau }^i$ at each i-th camera and r-th run. Each camera has ${\tau }^i$ sampled once from $\mathcal{U}\left\{0,{\tau }^{max}\right\},$ $0$ $\le {\tau }^{max}\le$ $3$ and remains the same throughout all runs.

The same process is applied to analyze the effect on $\epsilon$ of increasing $0\le {\tau }^{max}\le 40$ (see Figure 19) with a fixed processing delay ${\tau }^{max}$ and relative offset ${\alpha }^i$ at each i-th camera, and r-th run. Each camera has ${\alpha }^i$ sampled once from $\mathcal{U}\left\{0,{\alpha }^{max}\right\},$ (synchronous case) $0\le {\alpha }^{max}\le 4$ (asynchronous case) and remains the same throughout all runs.

Concerning the particle filter, the likelihood function in Equation (17) is the multivariate Gaussian pdf [43]:

$$\begin{array}{c}p\left({\mathit{z}}_{t,k_c,3D}^i|{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}\right)=\\ \frac{1}{{\left(2\pi \right)}^{\frac{N}{2}}de{t}^{\frac{1}{2}}\left({\mathit{C}}_{t,k_c,3D}^i\right)}exp\left[-\frac{1}{2}{\left(\mathit{H}{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}-{\mathit{z}}_{t,k_c,3D}^i\right)}^T{\left({\mathit{C}}_{t,k_c,3D}^i\right)}^{-1}\left(\mathit{H}{\mathit{x}}_{t,k_c,3D}^{i,\left(l\right)}-{\mathit{z}}_{t,k_c,3D}^i\right)\right],\end{array}$$

(21)

$N=3$ since ${\mathit{z}}_{t,k_c,3D}^i$ and ${\mathit{C}}_{t,k_c,3D}^i$ refer to the fused 3D position measurement. Equation (19) $p({\mathit{z}}_{t,k_c,3D}^{j_m}|{\tilde{\mathit{x}}}_{t,k_c^{‴},3D}^{i,j_m,\left(l\right)})$ shares the same structure of Equation (21) but changes the corresponding variables. $L=1000$ particles are used.

Figure 19 shows the simulation results for strategies (a) and (b). (a) exchanges LoS vectors among neighboring cameras followed by local estimation using fused 3D position measurements (BAF and PF), and (b) employs a fusion window after the time window ${\mathit{K}}_k^i$ with duration $p=\left\{1,2,4\right\}$ in frames to process either exchanged information pairs’ estimates (BAF) or fused 3D position measurements (SA-PF) other than LoS vectors.

4.2. Discussions

Figure 19 shows that BAF and PF filters present similar performance when varying ${\alpha }^{max}$ and ${\tau }^{max}$ in all tested situations for strategy (a) where LoS vectors measurements exchanged among cameras in the time window ${\mathit{K}}_k^i$ generate the fused 3D position measurement ${\mathit{z}}_{t,k_c,3D}^i$ and both filters use it to estimate the tracked targets’ positions and velocities. The BAF-LLM filter presents a better performance than the SA-PF filter in strategy (b) when ${\tau }^{max}={\alpha }^{max}\ge 16$. The initial weights ${\widehat{\omega }}_{t,k_c^{‴}}^{i,j_0,\left(l\right)}={\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}$ in the update phase consider the weights ${\widehat{\omega }}_{t,k_c}^{i,\left(l\right)}$ at instant $k_c$ in the receiving instant $k_c^{‴}$, and the instant of the received measurements ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ is considered as the frame capture instant of the neighboring camera, since SA-PF is not informed about the frame capture time at the neighboring camera. Moreover, just one significantly delayed measurement from a neighboring camera yields a small likelihood value, thus compromising the remaining particle weight updates in the sequential fusion step of SA-PF. The small likelihood in such condition is due to the particle state estimate undergoing time propagation in Equation (19) from capture instant to reception instant, whereas the neighboring camera measurement received asynchronously in the fusion window presents a large delay relative to the local camera capture instant. Then, if $k_c^{‴}$ $\gg$ $k_c$, the updated estimate is negatively affected by the delay. Therefore, it is justified that for ${\tau }^{max}$ and ${\alpha }^{max}\le 16$ (small values of processing delay and asynchrony), the SA-PF filter produces an average root mean square error response similar to that of the BAF-LLM filter; i.e., small ${\alpha }^{max}$ and ${\tau }^{max}$ do not degrade SA-PF filter performance in strategy (b).

As depicted in Figure 19 for all tested p = {1,2,4}, the average root mean square error of SA-PF undergoes sudden degradation relative to BAF-LLM when either asynchrony or frame processing delay increase significantly (blue and magenta curves—strategy (b)). BAF-LLM is not susceptible due to its prediction step from the frame capture instant to the transmission instant.

One problem is to validate the target information received by camera $C^i$ because, as stated previously, different Ids can be assigned to the same target in different cameras. This problem is solved in BAF by evaluating the Euclidean distance between the target 3D position estimate in camera $C^i$ with the position estimate received from the neighborhood. In the SA-PF filter, we use the Euclidean distance between measurement ${\mathit{z}}_{t,k_c,3D}^i$ and measurement ${\mathit{z}}_{t,k_c^{‴},3D}^{j_m}$ received from the neighborhood; i.e., for each information exchange between cameras in the fusion window of a potential target, a distance tolerance ensures the correct fusion of data arising from the same target.

Target tracking by a camera is lost if the camera capture interval $T={\alpha }_1+{\alpha }_2$ exceeds the target duration within the camera FoV. Different from [15] where a fixed number of targets remain within sight of the camera network during the experiment, our distributed target tracking algorithm can track a varying number of targets as they move within the FoV of at least 2 cameras.

Cameras in close spatial proximity degrade the 3D estimation accuracy of targets at long range, since the 3D measurement from imaging as proposed here borrow the concept of depth estimation from baseline length and the epipolar constraint in stereoscopy.

Additionally, we conducted further evaluation of (1) the effect of camera pose uncertainty, with focus on camera attitude error, and (2) computational complexity of our proposed 3D position measurement approach.

The PETS2009 image sequence dataset is a real benchmark scenario for target tracking with fixed cameras whose calibration parameters and pose data are accurate. In spite of that, we conduct a Monte Carlo simulation with M = 20 runs wherein uncertainties are added in the extrinsic calibration parameters of each camera (rotation and translation parameters). To the calibrated rotation angles ${\theta }_z,{\theta }_y,{\theta }_x$ of each camera in radians, we add a Gaussian uncertainty $\mathcal{N}\left(\mu =0.05 \mathrm{rad},\sigma =0.001 \mathrm{rad}\right)$. Additive Gaussian uncertainty $\mathcal{N}\left(\mu =10 \mathrm{cm},\sigma =1 \mathrm{cm}\right)$ in camera location is also applied. We evaluate the mean square error in Equation (20). The simulation investigated processing delays $\tau$= (0, 2, 4, 6, 8, 10, 12, 14, 16, 18), maximum asynchrony among cameras ${\alpha }_{m\mathrm{a}x}=4$, and the particular asynchrony value for each camera sampled from the uniform distribution $\mathcal{U}\left\{0,{\alpha }^{max}\right\}$, i.e., each camera received $0\le \alpha \le {\alpha }_{max}$.

Figure 20 indicates uncertainties in cameras’ poses degrade the fused 3D position measurement of targets when compared with exact poses, as expected.

We evaluate numerically the computational complexity by increasing the number $n_i$ of targets observed in the field of view of the i-th camera and the number $n_j$ of targets received by the i-th camera of the j-th cameras, thus resulting a total of $n_i·n_j$ targets. The processing tasks considered for each of the targets are:

(a)

Variable $n_i$—each target has 7 sigma points assigned to the local LoS vector pointing to the t-th target in the image plane;

(b)

Variable $n_j$—each target with its 7 sigma points assigned to the received LoS vector pointing to the received t-th target in the image plane;

(c)

Intersection search of pairs of LoS from the set of 49 LoS vectors providing points in 3D space (Equations (6) and (8));

(d)

Checks which 3D points pass the distance criterion $d\le 0.3$ m;

(e)

Selects the 7 LoS vectors from the set of 49 LoS vectors that pass the above test with smallest distance $d$; and

(f)

UT computation: 3D position measurement mean and covariance obtained at a $k_c$ capture instant of the i-th camera.

We noticed three regions in Figure 21:

(a)

For the number of targets varying from 1 up to 90 targets (i.e., from 1 × 1 up to 90 × 90 targets), the elapsed processing time shows an exponential tendency;

(b)

From 91 up to 262 targets (i.e., from 91 × 91 up to 262 × 262 targets) we observe a linear tendency, and over 262 × 262 targets, the elapsed processing time presents an exponential tendency.

In the PETS2009 image sequence dataset, the number of observed targets at any $k_c$ capture instant does not reach above 10 targets simultaneously. Therefore, our approach yields a very low processing time to carry out the 3D position measurements.

5. Conclusions

We propose position measurement in 3D space with intersecting vector constraints derived from exchanged LoS vectors in a camera network with known relative poses. Solving for the 3D intersection of vector constraints is based on computing the proximity of reverse straight lines. Each camera does not need the construction parameters of the other ones to obtain the 3D measurements. We then applied our approach to tracking simultaneous targets with a network of asynchronously communicating cameras with frame processing delays. The targets enter and leave the fields of view of the cameras in the image sequence dataset.

We use the average root mean square error across all image planes of the asynchronous camera network to measure tracking performance under two conditions: fused 3D position measurement from exchanged LoS vectors and local state estimation (strategy (a)), and fusion of additional information other than the exchanged LoS vectors (strategy (b)).

BAF and PF present similar performance under strategy (a), which uses local estimation. With regard to strategy (b) involving further fusion, BAF-LLM is superior to SA-PF under large asynchrony and processing delay. Comparing between strategies (a) and (b), BAF, PF, and BAF-LLM yield approximately the same performance in a wide range of asynchrony and processing delay. We remark that local estimation in BAF and PF demands less communications than further fusion used in BAF-LLM and SA-PF. Further investigation should focus on the use of consensus to improve target tracking subject to asynchronous exchange in the camera network.