# Multi-Target Tracking Using Windowed Fourier Single-Pixel Imaging

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Principle and Method

#### 2.1. Single-Target Tracking Method Using FSI

#### 2.2. Multi-Target Tracking Method Using WFSI

#### 2.3. Independent Estimation Approach

#### 2.4. Joint Estimation Approach

## 3. Results

#### 3.1. Simulation

#### 3.1.1. Multi-Target Locating

#### 3.1.2. Multi-Target Tracking Using the Independent Estimation Approach

#### 3.1.3. Multi-Target Tracking Using the Joint Estimation Approach

#### 3.2. Experiment

## 4. Discussion

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Abbreviations

SPI | single-pixel imaging |

FSI | Fourier single-pixel imaging |

WFSI | windowed Fourier single-pixel imaging |

STFT | short-time Fourier transform |

DMD | digital modulating devices |

## References

- Lu, H.; Li, Y.; Li, H.; Lv, R.; Lang, L.; Li, Q.; Song, G.; Li, P.; Wang, K.; Xue, L.; et al. Ship Detection by an Airborne Passive Interferometric Microwave Sensor (PIMS). IEEE Trans. Geosci. Remote Sens.
**2020**, 58, 2682–2694. [Google Scholar] [CrossRef] - Zhang, J.; Hu, T.; Li, W.; Li, L.; Yang, J. Research on Millimeter-wave Radiation Characteristics of Stereo Air Target. In Proceedings of the 2020 IEEE MTT-S International Microwave Workshop Series on Advanced Materials and Processes for RF and THz Applications (IMWS-AMP), Suzhou, China, 29–31 July 2020; pp. 1–3. [Google Scholar] [CrossRef]
- Goodwin, A.; Glancey, M.; Ford, T.; Scavo, L.; Brey, J.; Heier, C.; Durr, N.J.; Acharya, S. Development of a low-cost imaging system for remote mosquito surveillance. Biomed. Opt. Express
**2020**, 11, 2560–2569. [Google Scholar] [CrossRef] [PubMed] - Liggins, M.; Chong, C.Y. Distributed fusion architectures and algorithms for target tracking. Proc. IEEE
**1997**, 85, 95–107. [Google Scholar] [CrossRef] [Green Version] - Case, E.E.; Zelnio, A.M.; Rigling, B.D. Low-Cost Acoustic Array for Small UAV Detection and Tracking. In Proceedings of the 2008 IEEE National Aerospace and Electronics Conference, Dayton, OH, USA, 16–18 July 2008; pp. 110–113. [Google Scholar] [CrossRef]
- Christnacher, F.; Hengy, S.; Laurenzis, M.; Matwyschuk, A.; Naz, P.; Schertzer, S.; Schmitt, G. Optical and acoustical UAV detection. In Proceedings of the Electro-Optical Remote Sensing X, Edinburgh, UK, 26–27 September 2016; International Society for Optics and Photonics: Bellingham, WA, USA, 2016; Volume 9988, p. 99880B. [Google Scholar]
- Magana-Loaiza, O.S.; Howland, G.A.; Malik, M.; Howell, J.C.; Boyd, R.W. Compressive object tracking using entangled photons. Appl. Phys. Lett.
**2013**, 102, 231104. [Google Scholar] [CrossRef] - Sun, S.; Lin, H.; Xu, Y.; Gu, J.; Liu, W. Tracking and imaging of moving objects with temporal intensity difference correlation. Opt. Express
**2019**, 27, 27851–27861. [Google Scholar] [CrossRef] - Zhu, S.; Ma, K.K. A new diamond search algorithm for fast block-matching motion estimation. IEEE Trans. Image Process.
**2000**, 9, 287–290. [Google Scholar] [CrossRef] [PubMed] - Barjatya, A. Block matching algorithms for motion estimation. IEEE Trans. Evol. Comput.
**2004**, 8, 225–239. [Google Scholar] - Zhou, D.; Cao, J.; Cui, H.; Hao, Q.; Chen, B.k.; Lin, K. Complementary Fourier single-pixel imaging. Sensors
**2021**, 21, 6544. [Google Scholar] [CrossRef] - Edgar, M.P.; Gibson, G.M.; Padgett, M.J. Principles and prospects for single-pixel imaging. Nat. Photonics
**2019**, 13, 13–20. [Google Scholar] [CrossRef] - Gopalsami, N.; Liao, S.; Elmer, T.W.; Koehl, E.R.; Heifetz, A.; Raptis, A.P.C.; Spinoulas, L.; Katsaggelos, A. Passive millimeter-wave imaging with compressive sensing. Opt. Eng.
**2012**, 51, 091614. [Google Scholar] [CrossRef] - Jiang, W.; Li, X.; Peng, X.; Sun, B. Imaging high-speed moving targets with a single-pixel detector. Opt. Express
**2020**, 28, 7889–7897. [Google Scholar] [CrossRef] [PubMed] - Zhou, F.; Shi, X.; Chen, J.; Tang, T.; Liu, Y. Non-imaging real-time detection and tracking of fast-moving objects. arXiv
**2021**, arXiv:2108.06009. [Google Scholar] - Xu, S.; Zhang, J.; Wang, G.; Deng, H.; Ma, M.; Zhong, X. Single-pixel imaging of high-temperature objects. Appl. Opt.
**2021**, 60, 4095–4100. [Google Scholar] [CrossRef] [PubMed] - Olivieri, L.; Totero Gongora, J.S.; Pasquazi, A.; Peccianti, M. Time-resolved nonlinear ghost imaging. ACS Photonics
**2018**, 5, 3379–3388. [Google Scholar] [CrossRef] [Green Version] - Olivieri, L.; Gongora, J.S.T.; Peters, L.; Cecconi, V.; Cutrona, A.; Tunesi, J.; Tucker, R.; Pasquazi, A.; Peccianti, M. Hyperspectral terahertz microscopy via nonlinear ghost imaging. Optica
**2020**, 7, 186–191. [Google Scholar] [CrossRef] [Green Version] - Chen, S.C.; Feng, Z.; Li, J.; Tan, W.; Du, L.H.; Cai, J.; Ma, Y.; He, K.; Ding, H.; Zhai, Z.H.; et al. Ghost spintronic THz-emitter-array microscope. Light. Sci. Appl.
**2020**, 9, 1–9. [Google Scholar] [CrossRef] - Zanotto, L.; Piccoli, R.; Dong, J.; Caraffini, D.; Morandotti, R.; Razzari, L. Time-domain terahertz compressive imaging. Opt. Express
**2020**, 28, 3795–3802. [Google Scholar] [CrossRef] [Green Version] - Jiao, S.; Sun, M.; Gao, Y.; Lei, T.; Xie, Z.; Yuan, X. Motion estimation and quality enhancement for a single image in dynamic single-pixel imaging. Opt. Express
**2019**, 27, 12841–12854. [Google Scholar] [CrossRef] - Monin, S.; Hahamovich, E.; Rosenthal, A. Single-pixel imaging of dynamic objects using multi-frame motion estimation. Sci. Rep.
**2021**, 11, 7712. [Google Scholar] [CrossRef] - Shi, D.; Yin, K.; Huang, J.; Yuan, K.; Zhu, W.; Xie, C.; Liu, D.; Wang, Y. Fast tracking of moving objects using single-pixel imaging. Opt. Commun.
**2019**, 440, 155–162. [Google Scholar] [CrossRef] - Zhang, Z.; Ye, J.; Deng, Q.; Zhong, J. Image-free real-time detection and tracking of fast moving object using a single-pixel detector. Opt. Express
**2019**, 27, 35394–35401. [Google Scholar] [CrossRef] [PubMed] - Deng, Q.; Zhang, Z.; Zhong, J. Image-free real-time 3-D tracking of a fast-moving object using dual-pixel detection. Opt. Lett.
**2020**, 45, 4734–4737. [Google Scholar] [CrossRef] [PubMed] - Zha, L.; Shi, D.; Jian, H.; Yuan, K.E.; Meng, W.; Yang, W.; Runbo, J.; Yafeng, C.; Wang, Y. Single-pixel tracking of fast-moving object using geometric moment detection. Opt. Express
**2021**, 29, 30327–30336. [Google Scholar] [CrossRef] [PubMed] - Zhang, Z.; Ma, X.; Zhong, J. Single-pixel imaging by means of Fourier spectrum acquisition. Nat. Commun.
**2015**, 6, 6225. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zhang, Z.; Wang, X.; Zheng, G.; Zhong, J. Fast Fourier single-pixel imaging via binary illumination. Sci. Rep.
**2017**, 7, 12029. [Google Scholar] [CrossRef]

**Figure 1.**Sketch of multi-target tracking using WFSI. The process is as follows: Firstly, locate the initial multi-target positions. Secondly, design the window functions according to the positions. Thirdly, implement $6K$ measurements to calculate $2K$ Fourier coefficients. After that, using the multi-target tracking method with the independent approach and joint approach, the displacements can be estimated. Then, according to the estimated result, redesign the window functions to continue multi-target tracking.

**Figure 2.**Illustration of multi-target locating via the projected curves method. ${f}_{Y}\left(x\right)$ and ${f}_{X}\left(y\right)$ are the projection of the scene image on the X axis and the Y axis. $({x}_{1},{x}_{2})$ and $({y}_{1},{y}_{2})$ are the regions where targets exist on the X axis and the Y axis. $({L}_{x1},{L}_{x2},{L}_{y1},{L}_{y2})$ are the length of the regions. $({w}_{1},{w}_{2})$ represents the window functions designed for the two targets.

**Figure 3.**Illustrations of the measurement process. (

**a**) is the target image with $256\times 256$ pixels, (

**b**) is the designed window function with $256\times 256$ pixels. (

**c**–

**e**) are the binary Fourier patterns ${P}_{bin}(\frac{2}{N},0,\phi )$ with $512\times 512$ pixels used to calculate the Fourier coefficient ${F}_{I2}(\frac{2}{N},0)$.

**Figure 4.**Simulation scenes: (

**a**) a complex background, (

**b**) a scene image at moment ${t}_{1}$, and (

**c**) a scene image at moment ${t}_{2}$. There are four moving targets in the scene, denoted as ${T}_{1},{T}_{2},{T}_{3},{T}_{4}$.

**Figure 5.**Projected curves of the image with four targets. (

**a**) is the projected curve of image scene on X axis and (

**b**) is the projected curve on Y axis. $(\widehat{{x}_{1}},\widehat{{x}_{2}},\widehat{{x}_{3}})$ are the three regions on the X axis where the target may exist, and $(\widehat{{y}_{1}},\widehat{{y}_{2}},\widehat{{y}_{3}})$ are the three regions on the Y axis. The length ${L}_{x}$ and ${L}_{y}$ of the regions determine the size of window function.

**Figure 6.**The four window functions for four targets: (

**a**) is the four-window function, (

**b**) is the windowed image at moment ${t}_{1}$, and (

**c**) is the windowed image at moment ${t}_{2}$.

**Figure 7.**The four window functions: (

**a**–

**d**) are the images at the moment ${t}_{1}$, and (

**e**–

**h**) are the images at the moment ${t}_{2}$; (

**a**,

**e**) represent target ${T}_{1}$, (

**b**,

**f**) represent target ${T}_{2}$, (

**c**,

**g**) represent target ${T}_{3}$, and (

**d**,

**h**) represent target ${T}_{4}$.

**Figure 8.**Illustrations of the windowed image at moment ${t}_{2}$ and ${t}_{3}$: (

**a**–

**d**) represent the images at moment ${t}_{2}$, and (

**e**–

**h**) represent the images at moment ${t}_{3}$, (

**c**,

**g**) are the images of target ${T}_{2}$, and (

**d**,

**h**) are the images of target ${T}_{3}$. (

**h**) contains the interference caused by target ${T}_{2}$.

**Figure 9.**Experiment scene: (

**a**) is the 240th frame in the video, and (

**b**) is the image of 240th frame with background subtraction.

**Figure 10.**Estimated results in five different frames from the 260th frame to the 340th frame. The first row shows the scene images, and the second row shows the results of multi-target tracking. The real-time estimated position of Target 1 (representing the yellow ball on the left in the 240th frame) is marked with a red dot, and Target 2 (representing the red ball on the right in the 240th frame) is marked with a red triangle.

**Figure 12.**The estimated results of coordinates between the 240th frame and the 630th frame: (

**a**) is the yellow ball, and (

**b**) is the red ball.

Targets | ${\mathit{T}}_{1}$ | ${\mathit{T}}_{2}$ | ${\mathit{T}}_{3}$ | ${\mathit{T}}_{4}$ |
---|---|---|---|---|

Position coordinate | $(20,20)$ | $(20,200)$ | $(100,200)$ | $(200,300)$ |

Estimation result | $(19,20)$ | $(20,199)$ | $(98,199)$ | $(198,299)$ |

Targets | ${\mathit{T}}_{1}$ | ${\mathit{T}}_{2}$ | ${\mathit{T}}_{3}$ | ${\mathit{T}}_{4}$ |
---|---|---|---|---|

True value | $(5,5)$ | $(20,0)$ | $(-10,20)$ | $(20,-10)$ |

Estimation result | $(4.82,4.60)$ | $(19.67,0.08)$ | $(-8.89,19.65)$ | $(21.15,-8.55)$ |

Error | 0.43 | 0.34 | 1.16 | 1.85 |

Independent Estimation | Joint Estimation | True Value | |
---|---|---|---|

T2 | (19.66, −0.14) | (19.48, 0.15) | (20, 0) |

Error of T2 | 0.37 | 0.29 | |

T3 | (−23.34, −6.92) | (−11.29, 19.82) | (−10, 20) |

Error of T3 | 30.04 | 1.30 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zhang, J.; Hu, T.; Shao, X.; Xiao, M.; Rong, Y.; Xiao, Z.
Multi-Target Tracking Using Windowed Fourier Single-Pixel Imaging. *Sensors* **2021**, *21*, 7934.
https://doi.org/10.3390/s21237934

**AMA Style**

Zhang J, Hu T, Shao X, Xiao M, Rong Y, Xiao Z.
Multi-Target Tracking Using Windowed Fourier Single-Pixel Imaging. *Sensors*. 2021; 21(23):7934.
https://doi.org/10.3390/s21237934

**Chicago/Turabian Style**

Zhang, Jinyu, Taiyang Hu, Xiaolang Shao, Mengxuan Xiao, Yingjiao Rong, and Zelong Xiao.
2021. "Multi-Target Tracking Using Windowed Fourier Single-Pixel Imaging" *Sensors* 21, no. 23: 7934.
https://doi.org/10.3390/s21237934