Vision without the Image

Abstract

Novel image sensors transduce the stream of photons directly into asynchronous electrical pulses, rather than forming an image. Classical approaches to vision start from a good quality image and therefore it is tempting to consider image reconstruction as a first step to image analysis. We propose that, instead, one should focus on the task at hand (e.g., detection, tracking or control) and design algorithms that compute the relevant variables (class, position, velocity) directly from the stream of photons. We discuss three examples of such computer vision algorithms and test them on simulated data from photon-counting sensors. Such algorithms work just-in-time, i.e., they complete classification, search and tracking with high accuracy as soon as the information is sufficient, which is typically before there are enough photons to form a high-quality image. We argue that this is particularly useful when the photons are few or expensive, e.g., in astronomy, biological imaging, surveillance and night vision.

1. Introduction

Current computer vision algorithms start with a high-quality image as input. While such images may be acquired almost instantly in a well-lit scene, dark environments demand a significantly longer acquisition time. This long acquisition time is undesirable in many applications that operate in low-light environments: in biological imaging, prolonged exposure could cause health risks [1] or sample bleaching [2]; in autonomous driving, the delay that is imposed by image capture could affect a vehicle’s ability to stay on-course and avoid obstacles; in surveillance, long periods of imaging could delay response, as well as produce smeared images. When light is low, the number of photons per pixel is small and images become noisy. Computer vision algorithms are typically not designed to be robust vis-a-vis image noise, thus practitioners face an uneasy tradeoff between poor performance and long response times.

Novel sensor technology offers a new perspective on image formation: as soon as a photon is sensed it should be transmitted to the host Central Processing Unit (CPU), rather than wait until a sufficient number of photons has been collected to form good quality image. Thus an image, in the conventional sense, is never formed. Designs and prototypes of photon-counting image sensors, such as the quantum sensors [3], single-photon avalanche detectors [4], quanta image sensors [5,6], and the giga-vision camera [7], have been proposed recently. These sensors are capable of reliably detecting single photons, or a small number of photons. Instead of returning a high-quality image after a long exposure, photon-counting sensors report a stream of photon counts densely sampled in time.

Currently, the dominant use for photon-counting image sensors is image reconstruction [8]: the stream of photon counts is used to synthesize a high-quality image to be used in consumer applications or computer vision. However, the goal of vision is to compute information about the world (class, position, velocity) from the light that reaches the sensor. Thus, reconstructing the image is not a necessary first step. Rather, one should consider computing information directly from the stream of photons [9,10]. This line of thinking requires revisiting the classical image-based paradigm of computer vision, and impacts both the design of novel image sensors and the design of vision algorithms.

Computing directly from the stream of photons presents the advantage that some information may be computed immediately, without waiting for high-quality image to be formed. In other words, information is computed incrementally, enabling the downstream algorithms to trade off reaction times with accuracy. This is particularly appealing in low-light situations were photon arrival times are widely spaced. As the hardware for computation becomes faster, this style of computation will become practical in brighter scenes, especially when response times are crucial (e.g., in vehicle control).

Here we explore three vision applications: classification, search and tracking. In each application, we will propose an algorithm that makes direct use of the stream of photons, rather than an image. We find that each one of these algorithms achieves high accuracy with only a tiny fraction of the photons required for capturing high-quality images. We conclude with a discussion of what was learned.

2. Results

2.1. Simplified Imaging Model

Assume that the scene is stationary and photon arrival times follow a homogeneous Poisson process. Within an interval of length ${\delta }_t$, the observed photon count $X_i$ at pixel location i is subject to Poisson noise whose mean rate depends on maximum rate ${\lambda }_{max}\in {\mathbb{R}}^{+}$, the true intensity at that pixel $I_i\in [0,1]$ and a dark current rate ${ϵ}_{dc}\in [0,1]$ per pixel [8]:

$$\begin{array}{cc}\hfill P(X_i=k)& =Poisson(k;{\lambda }_{max}\frac{(I_i+{ϵ}_{dc})}{(1+{ϵ}_{dc})}{\delta }_t)\hfill \end{array}$$

(1)

(a model including sensor read noise is described in Section 3.1).

The sensor produces a stream of images ${\mathbf{X}}_1,{\mathbf{X}}_2,\dots$, where ${\mathbf{X}}_t\in {\mathcal{N}}^d$ contains the photon counts from d pixel locations from the time interval $[(t-1){\delta }_t,t{\delta }_t]$ (Figure 1a). We use ${\mathbf{X}}_{1:t}$ to represent the stream of inputs $\{{\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_t\}$.

When the illuminance of the environment is constant, the expected number of photons collected by the sensor grows linearly with the exposure time. Hence we use the number of photons per bright pixel (PPP) as a proxy for the exposure time t. PPP $=1$ means that the a pixel with maximum intensity has collected 1 photon. Additionally, since PPP is linked to the total amount of information content in the image regardless of the illuminance level, we will use PPP when describing the performance of vision algorithms. Figure 1b, c shows two series of inputs ${\mathbf{X}}_{1:t}$ with increasing PPP.

2.2. Classification

Distinguishing objects of different categories hinges upon the extraction of “features”, which are structural regularities in pixel values such as edges, corners, contours, etc. For example, the key feature that set apart a handwritten digit “3” from a digit “8” (Figure 1b,c, last column) is the fact that a “3” has open loops and “8” has closed loops—This corresponds to different strokes on the left side of the digit. In normal lighting conditions, these features are fully visible, may be computed by, e.g., convolution with an appropriate kernel, and fed into a classifier to predict the category of the image.

In low light, classification is hard because the features are corrupted by noise. A closed contour may appear broken due to stochastically missing photons. The noise in the features in turn translates to uncertainties on the classification decision. This uncertainty diminishes as the exposure time increases. It is intuitive that a vision algorithm that is designed to compute from a minimal number of photons should keep track of said uncertainties, and dynamically determine the exposure time based on the desired accuracy.

In particular, one wishes to predict the category $Y\in \{1,2,\dots ,C\}$ of an image based on photon counts ${\mathbf{X}}_{1:t}$. The predictions must minimize exposure time while being reasonably accurate, i.e.,

$$\begin{array}{c}\hfill min\mathbb{E}\left[T\right]s.t.\mathbb{E}[\widehat{Y}\ne Y]\le \gamma \end{array}$$

(2)

where T is a random variable denoting the exposure time required to classify an image, $\widehat{Y}$ ∈ $\{1,2,\dots ,C\}$ is the prediction of the class label, γ is the maximum tolerable misclassification error, and the expectation is taken over all images in a dataset.

2.2.1. Classification Algorithm

In order to make the most efficient use of photons, we first assume that a conditional probabilistic model $P\left(Y\right|{\mathbf{X}}_{1:t})$ is available for any $t\ge 0$ (we will relax this assumption later) and for all possible categories of the input image. An asymptotically optimal algorithm that solves the problem described in Equation (2) is Sequential Probability Ratio Testing (SPRT) [12] (Figure 2a,b):

$$\begin{array}{cc}& \text{Choose an appropriate error threshold}\theta \hfill \\ & c^{*}=\underset{c\in \{1,2,\dots ,C\}}{argmax}P(Y=c|{\mathbf{X}}_{1:t})\hfill \\ & \left\{\begin{array}{cc}\text{report}Y=c^{*}& \text{if}log\frac{P(Y=c^{*}|{\mathbf{X}}_{1:t})}{P(Y\ne c^{*}|{\mathbf{X}}_{1:t})}>\theta \\ \text{increase}t& \text{otherwise}\end{array}\right.\hfill \end{array}$$

(3)

Essentially, SPRT keeps accumulating photons by increasing exposure time until there is predominant evidence in favor of a particular category. Due to the stochasticity of the photon arrival events and the variability in an object’s appearance, the algorithm observes a different stream of photon counts each time. As a result, the exposure time T, and equivalently, the required PPP, are also different each time (see Figure 3).

The accuracy of the algorithm is controlled by the threshold θ. When a decision is made, the declared class $c^{*}$ satisfies that $log\frac{P(Y=c^{*}|{\mathbf{X}}_{1:t})}{P(Y\ne c^{*}|{\mathbf{X}}_{1:t})}>\theta$, which means that class $c^{*}$ has at least posterior probability $Sigm\left(\theta \right)\stackrel{▵}{=}\frac{1}{1+e^{-}\theta }$ according to the generative model, and the error rate of SPRT is at most $1-Sigm\left(\theta \right)$. For instance, if the maximum tolerable error rate is $10\%$, θ should be set so that $1-Sigm\left(\theta \right)=0.1$, or $\theta \approx 2.2$, while an error rate of $1\%$ would drive θ to $4.6$. Since higher thresholds lead to longer exposure times, the threshold serves as a knob to trade off speed versus accuracy, and should be set appropriately based on γ (Equation (2)).

The assumption that the conditional distribution $P(Y=c|{\mathbf{X}}_{1:t})$ is known is rather restrictive. Fortunately, the conditional distribution may be directly learned from data. In particular we train a recurrent neural network [13] $f_c\left({\mathbf{X}}_{1:t}\right)\approx P(Y=c|{\mathbf{X}}_{1:t})$ to approximate the conditional distribution. This network has a compact representation, and takes advantage of the sparseness of the photon-counts for efficient evaluation. Details of the network may be found in Section 3.2 and [9].

2.2.2. Experiments

We evaluate the low-light classification performance of the SPRT on the MNIST dataset [11], a standard handwritten digits dataset with 10 categories. The images are $28\times 28$ in resolution and in black and white. We simulate the outputs from a photon-counting sensor according to the full noise model (Section 3.1). The images are stationary within the imaging duration. We do not assume a given conditional distribution $P\left(Y\right|{\mathbf{X}}_{1:t})$ but train a recurrent network approximation $f\left({\mathbf{X}}_{1:t}\right)$ from data (Section 3.2).

Recall that classification correctness in each trial and the required exposure time (or PPP) are random variables. We therefore characterize SPRT performance based on the tradeoff between error rates (ER, $\mathbb{E}[\widehat{Y}\ne Y]$) and the median PPP in Figure 3a. The tradeoff is generated by sweeping the thresholds $\theta \in [-2.2,9.2]$. For comparison we tested the performance of models that were trained to classify images from a single PPP. We call these models “specialists” for the corresponding PPP. The specialists are extended to classify images at different light levels by scaling the image to the specialized PPP. To get a sense of the intraclass and interclass PPP variability, we also visualize the PPP histograms for multiple runs of different images in the same class (Figure 3b), and the overall PPP histograms for a few classes (Figure 3c). Lastly, we analyze how SPRT’s performance is sensitive to sensor noises in Figure 4. Details of the analysis procedure are found in Section 3.1.

2.3. Search

Search is a generalization of classification into multiple locations. The task is to identify whether a target object (e.g., keys, a pedestrian, a cell of a particular type) is present in a scene cluttered with distractors (e.g., a messy desk, a busy street at night or a cell culture). Note that despite the multiple candidate positions for a target to appear, we consider search as a binary task, where the two hypotheses are denoted $C=1$ (target-present) and $C=0$ (target-absent). We assume for simplicity that at most one target may appear at a time (for multiple targets, see [14]).

The difficulty of search in low-light conditions may be attributed to the following factors. (1) There are multiple objects in the display, and each object is subject to photon count fluctuations. (2) Long range constraints, such as the prior knowledge that at most one target is present in the visual field, must be enforced. (3) Properties of the scene, such as the amount of clutter in the scene and the target and distractor appearance, may be uncertain. For example, we may know that there may be either three or twelve objects in the scene, and intuitively the search strategy for these two scenarios should be drastically different. Therefore, scene properties must be inferred for optimal performance.

We assume that a visual field consists of L non-overlapping locations, out of which M locations may contain an object. M represents the amount of clutter in the scene. The objects are simplified to be oriented bars and the only feature that separates a target from a distractor is the orientation. The orientation at location l is denoted $Y^{\left(l\right)}$. The target orientation and the distractor orientation are denoted $y_T$ and $y_D$, respectively. The scene properties are collected denoted $\varphi =\{M,y_T,y_D\}$. The scene properties may be unknown for many search tasks, thus φ is a vector of random variables. The variable of interest is $C\in \{0,1\}$: $C=1$ iff $\exists l\in \{1,\dots ,L\},Y^{\left(l\right)}=y_T$, (i.e., $C=1$ iff there exists a location that contains a target).

We also assume that a low-light classifier discussed in Section 2.2 has been developed for classifying bar stimulus: the classifier computes $f_y({\mathbf{X}}_{1:t}^{\left(l\right)})\approx P(Y^{\left(l\right)}=y|{\mathbf{X}}_{1:t}^{\left(l\right)})$, the probability that the bar orientation at location l is y conditioned only on the local photon counts ${\mathbf{X}}_{1:t}^{\left(l\right)}$.

2.3.1. Search Algorithm

Similar to the low-light classification problem, an asymptotically optimal search algorithm is based on SPRT. The detailed algorithm is [12,14]:

$$\begin{array}{cc}& \text{Choose two error thresholds}{\theta }_0<0,{\theta }_1>0\hfill \\ & \text{Compute}S\left(t\right)\stackrel{▵}{=}log\frac{P(C=1|{\mathbf{X}}_{1:t})}{P(C=0|{\mathbf{X}}_{1:t})}\hfill \\ & \left\{\begin{array}{cc}\text{report}C=1& \text{if}S\left(t\right)>{\theta }_1\\ \text{report}C=0& \text{if}S\left(t\right)<{\theta }_0\\ \text{increase}t& \text{otherwise}\end{array}\right.\hfill \end{array}$$

(4)

where $S\left(t\right)$ is the log likelihood ratio between the two competing hypotheses, target-present ($C=1$) and target-absent ($C=0$). This algorithm is a binary version of the classification algorithm in Equation (3). Similar to Equation (3), the two thresholds ${\theta }_0$ and ${\theta }_1$ controls the amount of false reject errors (i.e., declare target-absent when target-present) and false accept errors (i.e., declaring target-present when target-absent).

The key for SPRT is to compute $S\left(t\right)$ from photon counts ${\mathbf{X}}_{1:t}$. The inference procedure may be implemented by two circuits, one infers the scene properties φ, and the other computes $S\left(t\right)$ (see [14]):

$$\begin{array}{cc}\hfill S\left(t\right)& =log\frac{1}{L}\sum _{l,\varphi }{R}^{(l,\varphi )}\left({\mathbf{X}}_{1:t}^{\left(l\right)}\right)P\left(\varphi \right|{\mathbf{X}}_{1:t})\hfill \end{array}$$

(5)

where

$$\begin{array}{cc}\hfill R^{(l,\varphi )}\left({\mathbf{X}}_{1:t}^{\left(l\right)}\right)& \stackrel{▵}{=}\frac{{\sum }_y{f}_y\left({\mathbf{X}}_{1:t}^{\left(l\right)}\right)P(Y^{\left(l\right)}=y|\varphi ,C^{\left(l\right)}=1)}{{\sum }_y{f}_y\left({\mathbf{X}}_{1:t}^{\left(l\right)}\right)P(Y^{\left(l\right)}=y|\varphi ,C^{\left(l\right)}=0)}\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill P\left(\varphi \right|{\mathbf{X}}_{1:t})& \propto P\left(\varphi \right)\prod _l\sum _y{f}_y\left({\mathbf{X}}_{1:t}^{\left(l\right)}\right)\frac{P(Y^{\left(l\right)}=y|\varphi ,C^{\left(l\right)}=0)}{P(Y^{\left(l\right)}=y)}\hfill \end{array}$$

(7)

Therefore, $S\left(t\right)$ may be computed by composing the low-light classifiers $f_y\left({\mathbf{X}}_{1:t}^{\left(l\right)}\right)$ according to Equations (5)–(7). The probabilities used in Equations (6) and (7), such as $P(Y^{\left(l\right)}$ = $y)$ and $P(Y^{\left(l\right)}$ = $y|\varphi ,C=2)$, may be estimated from past data.

2.3.2. Experiments

We choose a simple setup (Figure 5a) to illustrate how the performance of the search algorithm is affected by scene properties: the amount of clutter M, the target/distractor appearances $y_T$ and $y_D$, as well the degree of uncertainty associated with them. The setup contains $L=14$ locations, each occupying a $7\times 7$ area from which the sensor collects photons. The area contains a $3\times 7$-pixel bar with intensity 1 and background pixels with intensity 0. The max emission rate is ${\lambda }_{max}=3$ photons/s, and the dark current is $50\%$ (causing the background to emit 1 photon/s). Examples of the lowlight search setup are shown in Figure 6d.

We conduct two experiments, one manipulates the scene complexity M and the other target/distractor appearances. In the first experiment M is either chosen uniformly from $\{3,6,12\}$, or fixed at one of the three values (Figure 6a,b): (1) Despite the high dark current noise, a decision may be made quickly with less than 2 photons per pixel. (2) The amount of light required to achieve a given classification error increases as M. (3) Not knowing the complexity further increases the required photon count. (4) Target-absent conditions requires more photons than target-present conditions. In the second experiment the target-distractor appearance difference $\delta y=|y_T-y_D|$ is either chosen uniformly from $\{{20}^{\circ },{30}^{\circ },{90}^{\circ }\}$ or fixed at one of the three. Figure 6c suggests that target dissimilarity heavily influences the ER-PPP tradeoff, while uncertainty in the target and distractor appearances does not.

2.4. Tracking

Finally, we demonstrate the potential of photon-counting sensors in tracking under low-light conditions. The goal of tracking is to recover time-varying attributes (i.e., position, velocity, pose, etc.) of one or multiple moving objects. It is challenging because, unlike classification and search, objects in tracking applications are non-stationary by definition. In low-light environments, as the object transitions from one state to another, it leaves only a transient footprint, in the form of stochastically-sprinkled photons, which is typically insufficient to fully identify the state. Instead, a tracker must postulate the object’s dynamics and integrate evidence over time accordingly. The evidence in turn refines the estimates of the dynamics. Due to the self-reinforcing nature of this procedure, the tracker must perform optimal inference to ensure convergence to the true dynamics.

Another challenge that sets low-light tracking apart from regular tracking problems is that the observation likelihood model is not only non-Gaussian, but also often unavailable, as it is commonly the case for realistic images. This renders most Kalman filter algorithms [15] ineffective.

2.4.1. Tracking Algorithm

The tracking algorithm we have designed is a hybrid between the Extended Kalman Filter [15] and the Auxiliary Particle Filter [16]. Let ${\mathbf{Z}}_t$ denote the state of the object, F the forward dynamics that govern the state transition: ${\mathbf{Z}}_{t+1}=F\left({\mathbf{Z}}_t\right)$, which are known and differentiable, and $P_t\left(Z\right)$ the posterior distribution over the states at time t: $P_t\left(Z\right)\stackrel{▵}{=}P({\mathbf{Z}}_t=Z|{\mathbf{X}}_{1:t})$.

We make two assumptions: (1) $P_t$ may be approximated by a multivariate Gaussian distribution; (2) A low-light regressor $f\left(Z\right|{\mathbf{X}}_t)\approx P({\mathbf{Z}}_t=Z|{\mathbf{X}}_t)$ is available to compute a likelihood score of ${\mathbf{Z}}_t$ given only the snapshot ${\mathbf{X}}_t$ at time t. $f\left(Z\right|{\mathbf{X}}_t)$ does not have to be normalized. We justify assumption (1) and describe algorithms for realizing assumption (2) in Section 3.3. As we will see in Equation (17), the Poisson noise model (Equation (1)) ensures that $f\left(Z\right|{\mathbf{X}}_t)$ exists and takes a simple form.

Given a prior probability distribution $P_0$, our goal is to compute the posterior distribution $P_t$ for all t. The tracking algorithm starts with $t=0$ and repeat the following procedure (Figure 7b).

$$\begin{array}{cc}& 1.\text{Compute the predictive distribution}P\left({\mathbf{Z}}_{t+1}\right|{\mathbf{Z}}_t)\text{from}{P}_t\hfill \\ & 2.\text{Draw}K\text{samples}{Z}_s^{\prime }\text{from}P\left({\mathbf{Z}}_{t+1}\right|{\mathbf{Z}}_t)\hfill \\ & 3.\text{Observe}{\mathbf{X}}_{t+1}\text{and compute}{W}_s=f\left(Z_s^{\prime }\right|{\mathbf{X}}_{t+1})\hfill \\ & 4.\text{Approximate}{P}_{t+1}\left(Z^{\prime }\right)\text{as using samples}{Z}_s^{\prime }\text{weighted by}{W}_s\hfill \\ & 5.\text{Increase}t=t+1\hfill \end{array}$$

(8)

Under the Gaussian assumption for $P_t$, both steps 1 and 4 may be computed in close-form (Section 3.3). This is in sharp contrast to regular particle filters, which do not assume any parametric form for $P_t$ and accomplish steps 1 and 4 using samples. Empirically we found that the Gaussian assumption is reasonable and often leads to efficient solutions with less variability.

2.4.2. Experiments

We choose the 1D inverted pendulum problem (Figure 7a) that is standard in control theory. A pendulum is mounted via a a massless pole on a cart. The cart can move horizontally in 1D on a frictionless floor. The pendulum can rotate full circle on a fixed 2D plane perpendicular to the floor. The pendulum is released at time $t=0$ at an unknown angle ${\alpha }_0\in [0,{360}^{\circ })$ from the vertical line, while the cart is at an unknown horizontal offset ${\beta }_0\in \mathbb{R}$. The task is to identify how the angle ${\alpha }_t$ and the offset ${\beta }_t$ change through time from the stream of photon counts ${\mathbf{X}}_{1:t}$. The state of the pendulum system is ${\mathbf{Z}}_t=\{{\alpha }_t,{\dot{\alpha }}_t,{\beta }_t,\dot{{\beta }_t}\}$. The system’s forward dynamics is well-known [17].

In our simulations, only the pole of the pendulum is white and everything else is dark. The highest photon emission rate of the scene is ${\lambda }_{max}$ and the dark current rate is ${ϵ}_{dc}$. We systematically vary ${\lambda }_{max}$ and ${ϵ}_{dc}$ and observe the amount of estimation error in the angle ${\alpha }_t$ and cart position ${\beta }_t$. See Section 3.4 for the simulation procedure.

We see that (1) estimation errors decrease over time (Figure 8a,b), (2) smaller ${ϵ}_{dc}$ leads to faster reduction in estimation error on average (Figure 8a,b), and (3) the tracker’s convergence time, i.e., the time it takes to achieve a certain level of estimation accuracy, decreases with illuminance (Figure 8c). The time required to satisfy high accuracy requirements (e.g., <1${}^{\circ }$ for α estimation) does NOT follow a simple inverse proportional relationship with illuminance. Instead, the convergence time plateaus, potentially due to the noise in the sampling procedure (Equation (8), step 2).

3. Materials and Methods

3.1. Imaging Model Including Noise Sources

Within an interval $\delta t$, the sensor readout x is corrupted by a series of noise sources.

• The amount of photons N incident on the pixel is subject to Poisson noise (shot noise). The noise level is determined by the true intensity I and the dark current ${ϵ}_{dc}\sim \mathcal{N}(0,{\sigma }_{ϵ})$.
• The photon counts are corrupted by an additive Gaussian read noise ${ϵ}_r\sim \mathcal{N}(0,{\sigma }_r)$ and a multiplicative fixed pattern noise ${ϵ}_{fpn}\sim \mathcal{N}(0,{\sigma }_{fpn})$.

$$\begin{array}{r}N\sim Poisson(·|I+{ϵ}_{dc})\end{array}$$

(9)

$$\begin{array}{r}x=max(0,(N+{ϵ}_r)(1+{ϵ}_{fpn}))\end{array}$$

(10)

Sensors designed for low light applications (e.g., [5]) have promised low read noise and low fixed pattern noise. Therefore, we focus on modeling the shot noise and dark current, and assume that algorithms have access to N when the algorithms are trained. We then test the algorithm using realistic values of read noise and fixed pattern noise to study robustness against noise (see Figure 4).

3.2. Low-Light Classifier

A low-light classifier $f\left({\mathbf{X}}_{1:t}\right)$ that approximates the conditional distribution $P\left(Y\right|{\mathbf{X}}_{1:t})$ for Y = $\{1,2,\dots ,C\}$ is developed in [9]. The classifier is a recurrent neural network consisting of multiple layers $h^{\left(1\right)}\left(t\right),\dots ,h^{\left(L\right)}\left(t\right)$ where the activation at layer l is:

$$\begin{array}{cc}\hfill h_j^{\left(l\right)}\left(t\right)& =\sum _{\tau =1}^t{W}_j^{\left(l\right)}{\mathbf{X}}_{\tau }+b_{i}t\hfill \end{array}$$

(11)

$$\begin{array}{cc}& =h_j^{\left(l\right)}(t-1)+W_j^{\left(l\right)}{\mathbf{X}}_t+b_i\hfill \end{array}$$

(12)

where $W_j^{\left(l\right)}\in {\mathbb{R}}^d$ and $b_j^{\left(l\right)}\in \mathbb{R}$ are the weights and the biases of the j-th unit. Equation (12) suggests that $h_j^{\left(l\right)}\left(t\right)$ may be computed incrementally from its old value $h_j^{\left(l\right)}(t-1)$.

The hidden units at layer l are organized into non-overlapping groups and pooled. A pooling unit $h_k^{\left(l\right)}$ oversees the hidden units at block $G_k$, and its activation is computed by:

$$\begin{array}{c}\hfill m_k^{\left(l\right)}\left(t\right)=max(0,\underset{j\in G_k}{max}(h_j^{\left(l\right)}\left(t\right))\end{array}$$

(13)

Let $j^{*}\left(t\right)=arg{max}_{j\in G_k}(h_j^{\left(l\right)}\left(t\right))$ denote the index of the max unit at time $t-1$ ($j^{*}\left(t\right)=0$ denote the event that the max value is 0). If within time interval $\delta t$ only a small set $G_k^{\prime }\subseteq G_k$ of hidden units within group $G_k$ are updated, $m_k^{\left(l\right)}\left(t\right)$ may also be computed incrementally:

$$\begin{array}{c}\hfill m_k^{\left(l\right)}\left(t\right)=\underset{j\in G_k^{\prime }\cup j^{*}(t-1)}{max}(h_j^{\left(l\right)}\left(t\right))\end{array}$$

(14)

Both Equations (12) and (14) are critical for an efficient implementation of the classifier. For example, if only a tenth of the units are updated in each layer, the computation time for $f\left({\mathbf{X}}_{1:t}\right)$ may be reduced by a factor of 10.

Finally, the output of the classifier is given by:

$$\begin{array}{cc}\hfill f_c\left({\mathbf{X}}_{1:t}\right)& =\frac{exp(m_c^{\left(L\right)}\left(t\right))}{{\sum }_{c^{\prime }}exp(m_{c^{\prime }}^{\left(L\right)}\left(t\right))},c\in \{1,2,\dots ,C\}\hfill \end{array}$$

(15)

Since $f\left({\mathbf{X}}_{1:t}\right)$ approximates the conditional likelihood $P(Y=c|{\mathbf{X}}_{1:t})$, the parameters ${\{W_j^{\left(l\right)},b_j^{\left(l\right)}\}}_{j,l}$ of $f(·)$ may be learned by maximum likelihood from a dataset of photon counts. However, it is expensive to keep track of a high number of photon count streams. Fortunately, Equation (11) also suggests that the network’s prediction at time t only depends on the cumulative photon counts ${\mathbf{S}}_t\stackrel{▵}{=}{\sum }_{\tau =1}^t{\mathbf{X}}_{\tau }$. Therefore, one only needs a dataset of $\{{\mathbf{S}}_t,t,Y\}$ tuples to perform maximum likelihood learning.

In detail, we simulated a lowlight MNIST dataset ${\{{\mathbf{S}}^{\left(i\right)},PP{P}^{\left(i\right)},Y^{\left(i\right)}\}}_i$ where the PPPs are sampled uniformly from $\{0.22,2.2,22,220\}$. Note that we are using PPP instead of the exposure time t for reasons discussed in Section 2.1. At PPP$=220$ each image pixel contains around 5 bits of information (log signal-to-noise-ratio $\approx 5$). Our implementation uses the MatConvNet package [18] and its default hyper-parameters for training. We train a model with the same connectivity as the LeNet [11] denoted: 784-20-50-500-10. The model contains 784 input units, followed by two convolutional hidden layers with 20 and 50 filters, respectively, of size $5\times 5$. Inputs to a convolutional layer is convolved with the filters, and then pooled over $2\times 2$ non-overlapping windows. After the convolutional layers are a fully connected hidden layer with 500 units and a fully connected softmax layer with 10 output categories. We minimize the negative log likelihood with a $L2$ weight decay:

$$\begin{array}{c}\hfill -\sum _{i}log{f}_{Y^{\left(i\right)}}({\mathbf{S}}^{\left(i\right)})+\eta \sum _l\left|\right|W^{\left(l\right)}{\left|\right|}_2^2\end{array}$$

(16)

where $\eta =0.0005$ is the strength of the weight decay. We use stochastic gradient descent with mini-batches of 100 and train for 60 epochs with learning rate $=0.001$ and momentum $=0.9$.

3.3. Tracking Algorithm

3.3.1. Low-Light Regressor

The photon count ${\mathbf{X}}_t$ at time t depends only on the angle ${\alpha }_t$ and cart position ${\beta }_t$ and not their time derivatives, so our low-light regressor can only predict ${\alpha }_t$ and ${\beta }_t$: $f\left({\mathbf{Z}}_t\right|{\mathbf{X}}_t)=f({\alpha }_t,{\beta }_t|{\mathbf{X}}_t)$. Since we simulate the scene using a generative model, we can compute the exact form of the low-light regressor. Let $P\left({\mathbf{I}}_t\right|{\alpha }_t,{\beta }_t)$ be the generative model where ${\mathbf{I}}_t$ is the intensity value of the image, the photon emission rate ${\lambda }_i$$({\alpha }_t,{\beta }_t)$ for every pixel may be computed using Equation (1). As a result the log likelihood of observing ${\mathbf{X}}_t$ is:

$$\begin{array}{cc}\hfill logP\left({\mathbf{X}}_t\right|{\alpha }_t,{\beta }_t)& =log\prod _{i}Poisson(X_{ti};{\lambda }_i({\alpha }_t,{\beta }_t))=Const.+{\mathbf{X}}_t^{T}log\left(\lambda ({\alpha }_t,{\beta }_t)\right)-1^T\lambda ({\alpha }_t,{\beta }_t)\hfill \end{array}$$

(17)

Note that this likelihood model is linear in the intensity image λ given the parameters ${\alpha }_t$ and ${\beta }_t$, and not linear in terms of the parameters themselves. In addition, the likelihood is Poisson, not Gaussian. Hence Kalman filters are not applicable here.

Using Bayes rule we have:

$$\begin{array}{c}\hfill f\left({\mathbf{Z}}_t\right|{\mathbf{X}}_t)=P({\alpha }_t,{\beta }_t|{\mathbf{X}}_t)\propto P\left({\mathbf{X}}_t\right|{\alpha }_t,{\beta }_t)P({\alpha }_t,{\beta }_t)\end{array}$$

(18)

When a generative model is not available, the regressor may be trained discriminatively on a dataset using maximum likelihood, similar to Section 3.2.

3.3.2. Approximating the Predictive Distribution (Step 1 of Equation (8))

Let $P_t\approx \mathcal{N}({\mu }_t,{\Sigma }_t)$ and F be the dynamics of the inverted pendulum. The predictive distribution may be approximated as a gaussian by linearizing F:

$$\begin{array}{c}\hfill {\mathbf{Z}}_{t+1}|{\mathbf{Z}}_t\approx \mathcal{N}(F\left({\mu }_t\right),(▿F{|}_{{\mu }_t})\Sigma {(▿F{|}_{{\mu }_t})}^T)\end{array}$$

(19)

where ${▿F|}_{{\mu }_t}$ is the Jacobian of the dynamics F evaluated at the prior mean ${\mu }_t$.

3.3.3. Approximating the Posterior Distribution (Step 4 of Equation (8))

Given weighted samples ${\left\{Z_s^{\prime }\right\}}_{s=1}^K$ and normalized weights ${\left\{W_s\right\}}_{s=1}^K$ (i.e., ${\sum }_s{W}_s=1$), the posterior $P_{t+1}$ may be approximated by a Gaussian with: $P_{t+1}\approx \mathcal{N}({\mu }_{t+1},{\Sigma }_{t+1})$, where

$$\begin{array}{ll}{\mu }_{t+1}& =\sum _s{W}_s{Z}_s^{\prime }\end{array}$$

(20)

$$\begin{array}{ll}{\Sigma }_{t+1}& =\frac{{\sum }_s{W}_s(Z_s^{\prime }-{\mu }_{t+1}){(Z_s^{\prime }-{\mu }_{t+1})}^T}{1-{\sum }_s{W}_s^2}\end{array}$$

(21)

3.4. Tracking Experiment

The pole of the pendulum has an intensity of 1 while everything else (background, cart, etc) has intensity 0. The pole is $3\times 30$ pixels and the entire scene is $80\times 80$ pixels. The pendulum is half the mass of the cart. The maximum photon emission rate is ${\lambda }_{max}$, and dark current is ${ϵ}_{dc}$. The pendulum is released at the origin (${\beta }_0=0$) with ${\alpha }_0$ randomly chosen from $\{{10}^{\circ },{20}^{\circ },{30}^{\circ },{40}^{\circ }\}$. Each initial state is simulated 25 times and then aggregated, yielding a total of 100 trials for each condition (i.e., for each pair of ${\lambda }_{max}$ and ${ϵ}_{dc}$). The distribution on ${\alpha }_0$ and ${\beta }_0$ used for simulation is not available to the tracking algorithm, which instead assumes uniform distributions for both.

3.4.1. Internal Noise Due to Sampling

For Figure 7a,b we used $K=1000$ samples from the predictive distribution (Equation (8), step 2). Repeating the experiment using $K=300$ samples shows the same trend with larger error bars. One counterintuitive finding is that as the exposure time increases, the estimation error first decreases and then diverges (both higher mean and deviation are visible, Figure 7a,b). The degree of divergence is aggravated by increasing signal-to-noise ratio (or reducing dark current ${ϵ}_{dc}$). This may be explained by the internal noise in the tracking algorithm. The algorithm relies on samples from the predictive distribution $P\left({\mathbf{Z}}_{t+1}\right|{\mathbf{Z}}_t)$ (Equation (8), step 1) to coincide with states that have a high observation likelihood $f\left(Z\right|{\mathbf{X}}_t)$ (Equation (8), step 3). This coincidence is less likely to happen when the observation likelihood becomes precise, or as the posterior becomes sharply-peaked. Similarly, for Figure 7c we used $K=1000$, and repeated the experiment with $K=300$. They obtain exactly the same trends except that smaller K corresponds to a higher plateau for the convergence time (Figure 7c), indicating that sampling noise may be limiting the speed for accurate tracking.

4. Discussion and Conclusions

The advent of photon-counting sensors motivates us to reconsider the prevalent paradigm in computer vision: Rather than first capturing an image and then analyzing it, we should design algorithms that incrementally compute information from the stream of photons that hits the sensor, without any attempt to reconstruct the image. This style of thinking is particularly attractive in low light conditions, where the exposure time required for capturing a high-quality image is prohibitively lengthy.

Photon-counting sensors deliver small increments of the image at short delays and high frequencies. We show that this incremental input could in principle be applied to solve a variety of vision problems with a short exposure time. Algorithms that are inspired by the asymptotically optimal SPRT appear particularly well suited for minimizing photon counts while satisfying a desired accuracy bound.

Our first finding is that useful information may be computed in a short amount of time, well ahead of the integration time that is required for forming (or reconstructing) a high quality image. In Figure 1b,c, we see that a low-light classification algorithm can achieve $1\%$ classification error of handwritten digits before one photon per pixel has been collected. The low-light classifier may be viewed as reconstructing the features (instead of the image), and carrying the uncertainty of the features all the way to classification. This uncertainty is essential in a sequential decision making setting to determine when to stop collecting more photons. In comparison, conventional approaches simply reconstruct the image, and pass it to a classifier trained on high-quality images. The conventional approach suffers from two issues. (1) Since the conventional approach discards the uncertainty information, it is not clear how to determine the required exposure time; and (2) statistics of the reconstruction may be different from that of high-quality images, hence the classifier’s performance may not be guaranteed.

Second, algorithms for classification and search from streams of photons are photon-efficient: they stop as soon as a confident decision is made. This efficiency is critical for domains such as astrophysics where each photon is precious [19], and cell imaging applications where the dies that are employed to visualize cell structures are phototoxic [20,21]. As an example of the photon efficiency, Figure 3a shows that at PPP $=1$ the low-light classifier based on SPRT can already achieve a better performance than a classifier using PPP $=220$. Additionally, contrary to the conventional paradigm that obtains images with a fixed duration, low-light classifiers and search algorithms uses different exposure times depending on the specific photon arrival sequence (Figure 3b) and on the overall classification difficulty of the example (Figure 3b,c).

Third, algorithms become faster when more light is present. For classification and search where the input image is stationary, time is synonymous with the amount of photons. Higher illuminance therefore translates to faster decisions. This simple relationship is useful in that a low-light system trained for classification or search at one illuminance level may be easily applied at another illuminance level. The transition only requires knowing the illuminance level of the new scene, which may be estimated either via an explicit illuminance sensor or from the total photon count across the image [22]. In addition, the ER vs. median PPP tradeoff (Figure 3a and Figure 6a–c) is an illuminance-independent characteristic of the algorithm and the task.

Last, the relationship between illuminance and speed is not always simple in tracking. The dynamics governing the object movement/state transition has its own time scale. A regressor $f\left({\mathbf{X}}_t\right)$ thus has only a finite duration for integrating information before the object moves too far. A tracker relies on accurate prediction of the regressor to postulate the object’s next position. An inaccurate prediction due to short exposure time may cause tracking failure. In addition, internal noise in the tracking algorithm (Section 3.4) may cause the speed to plateau after a certain illuminance level. As a result, the relationship between illuminance and convergence time is not a simple inversely proportional relationship, as shown in Figure 8c.

Although we have provided proof-of-concept illustrations of low-light vision applications with photon-counting sensors, many challenges still remain. (1) We are not aware of any hardware specialized at processing streams of photon counts at high speeds. Nonetheless, current Field-programmable Gate Array (FPGA) implementations have achieved over 2000 Hz throughput for classifying images of a similar resolution as those in Section 2.2 [23]. In addition, the low-light classifiers implemented as a recurrent neural network (Section 2.2) can be updated incrementally, i.e., $f\left({\mathbf{X}}_{1:t}\right)$ can be computed from the internal states of $f\left({\mathbf{X}}_{1:t-1}\right)$ and ${\mathbf{X}}_t$ with sparse updates. The sparseness may be key to expedite computation. (2) We do not yet have datasets collected directly from photon-counting sensors to verify the robustness of the proposed methodology, as many such sensors are still in the making [5,6,7]. (3) Our noise model (Section 3.1) may be a crude approximation to handle moving objects. For example, we do not model motion induced blur or input disturbances due to camera self-motion. Nonetheless, motion induced blur may not be an issue if the sensor is collecting a single photon at a time, such as in low light and/or high-frequency imaging scenarios. In these scenarios even though the amount of photons is so low that full image reconstruction is difficult, our algorithm can still make correct and full use of the where and when of photon arrivals. This is precisely the advantage of image-free vision.

In conclusion, we propose to integrate computer vision with photon-counting sensors to address the challenges facing low-light vision applications. We should no longer wait for a high-quality image to be formed before executing the algorithm. Novel algorithms and hardware solutions should be developed to operate on streams of photon-counts. These solutions should also sidestep image reconstruction and focus directly on the task at hand.