Images from Bits: Non-Iterative Image Reconstruction for Quanta Image Sensors

Abstract

A quanta image sensor (QIS) is a class of single-photon imaging devices that measure light intensity using oversampled binary observations. Because of the stochastic nature of the photon arrivals, data acquired by QIS is a massive stream of random binary bits. The goal of image reconstruction is to recover the underlying image from these bits. In this paper, we present a non-iterative image reconstruction algorithm for QIS. Unlike existing reconstruction methods that formulate the problem from an optimization perspective, the new algorithm directly recovers the images through a pair of nonlinear transformations and an off-the-shelf image denoising algorithm. By skipping the usual optimization procedure, we achieve orders of magnitude improvement in speed and even better image reconstruction quality. We validate the new algorithm on synthetic datasets, as well as real videos collected by one-bit single-photon avalanche diode (SPAD) cameras.

1. Introduction

1.1. Quanta Image Sensor

Since the birth of charge coupled devices (CCD) in the late 1960s [1] and the complementary metal-oxide-semiconductor (CMOS) active pixel sensors in the early 1990s [2], the pixel pitch of digital image sensors has been continuously shrinking [3]. Shrinking the pixel pitch is intimately linked to the need of increasing image resolution, reducing power consumption and reducing the size and weight of cameras. However, as pixel pitch shrinks, the amount of photon flux detectable by each pixel drops, leading to reduced signal strength. In addition, the maximum number of photoelectrons that can be held in each pixel, known as the full-well capacity, also drops. Small full-well capacity causes reduced maximum signal-to-noise ratio and lowers the dynamic range of an image [4]. Therefore, pushing for smaller pixels, although feasible in the near future, will become a major technological hurdle to new image sensors.

A quanta image sensor (QIS) is a class of solid-state image sensors originally proposed by Eric Fossum as a candidate solution for sub-diffraction-limit pixels. The sensor was first named the digital film sensor [5] and later the quanta image sensor [6,7,8,9] (see [10] for a more comprehensive discussion of the history). A similar idea to QIS was developed a few years later by EPFL (École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland), known as the Gigavision camera [11,12,13]. In the past few years, research groups at the University of Edinburgh (Edinburgh, UK) [14,15,16], as well as EPFL [17,18] have made new progresses in QIS using binary single-photon avalanche diode (SPAD) cameras. In the industry, Rambus Inc. (Sunnyvale, CA, USA) is developing binary image sensors for high dynamic range imaging [19,20,21]. For the purpose of this paper, we shall not differentiate these sensors, but refer to them generally as the QIS, because their underlying mathematical principles are similar.

The working principle of QIS is as follows: In CCD and CMOS, one considers each pixel as a “bucket” that collects and integrates photoelectrons. The bucket is partitioned in QIS into thousands of nanoscale cells referred to as “jots”. Each jot is capable of detecting a single photon to generate a binary response indicating whether the photon count is above or below a certain threshold q. If the photon count is above q, the sensor outputs a “1”; If the photon count is below q, the sensor outputs a “0”. QIS has a very small full-well capacity because it is not designed to accumulate photons. Since the binary response is generated as soon as the number of photons exceeds the threshold, QIS can be operated at very high speed. For example, using single-photon avalanche diodes (SPAD), one can achieve 10k frames per second with a spatial resolution of $320\times 240$ pixels [14] or even 156k frames per second with a spatial resolution of $512\times 128$ pixels [17]. For a higher spatial resolution, Massondian et al. [9] reports a QIS operating at 1000 frames per second for a spatial resolution of $1376\times 768$ pixels.

From a signal processing perspective, the challenge of QIS is the extremely lossy process using binary measurements to acquire the light intensity. In order to compensate for the loss, QIS over-samples the space by using a large number of jots and takes multiple exposures in time. This technique is similar to the classic approach in oversampled analog-to-digital conversions.

1.2. Scope and Contribution

The theme of this paper is about how to reconstruct images from the one-bit quantized measurements. Image reconstruction is a critical component of QIS, for without such an algorithm, we will not be able to form images. However, unlike classical Poisson image recovery problems where solutions are abundant [22,23,24,25,26,27], the one-bit quantization of QIS makes the problem uniquely challenging, and there is a limited number of existing methods [28,29,30,31]. Another challenge we have to overcome is the complexity of the algorithm, which has to be low enough that we can put them on cameras to minimize power consumption, memory consumption and runtime. Numerical optimization algorithms are generally not recommended if they are iterative and require intensive computation for every step.

The main contribution of this paper is a non-iterative image reconstruction algorithm for QIS data. We emphasize the non-iterative nature of the algorithm as it makes the algorithm different from existing methods. The new algorithm is based on a transform-denoise framework. The idea is to apply a variance stabilizing transform known as the Anscombe transform [32] to convert a sum of one-bit quantized Poisson random variables to binomial random variables with equal variances. When variance is stabilized, standard image denoising algorithms can be applied to smooth out the noise in the image. Transform-denoise is a single-pass algorithm with no iteration. Empirically, we find that the new algorithm achieves two orders of magnitude improvement in speed and provides an even better reconstruction result than existing iterative algorithms.

The rest of the paper is organized as follows. First, in Section 2, we present the imaging model of QIS. We discuss how the light intensity of the scene is over-sampled and what statistics do the one-bit quantized measurements have. Unlike existing works, which typically assume a quantization level $q=1$, we make no assumption about q, except that it is a positive integer. This requires some discussion about the incomplete Gamma function. We also discuss why a simple summation over a local spatial-temporal volume is insufficient to reconstruct images. Section 3 presents the main algorithm. We discuss the concept of variance stabilizing transform, its derivation and its limitations. We also discuss how various image denoising algorithms can be plugged into the framework. In Section 4, we present experimental results. There are two sets of data we will discuss. One is a synthetic dataset in which we can objectively measure the reconstruction quality. The other one is a set of real videos captured by SPAD cameras. We compare the proposed algorithm with existing methods. Proofs of major theorems are given in the Appendix A.

2. QIS Imaging Model

In this section, we provide a quick overview of the QIS imaging model. A similar description of the model was previously discussed in [13,29]. For notational simplicity, we consider one-dimensional signals. The extension to the two-dimensional images is straightforward. Furthermore, we follow [5] by referring to sub-pixels of a QIS as “jots”.

The mathematics of QIS is built upon two concepts: (1) a spatial oversampling process to model the acquisition by the sensor; and (2) a quantized Poisson process to model the photon arrivals. The block diagram of the model is summarized in Figure 1.

2.1. Oversampling Mechanism

We represent the light intensity of a scene using a digital signal $\mathit{c}={[c_0,c_1,\dots ,c_{N-1}]}^T$. To avoid the ambiguity of the scaling, we assume that $c_n$ is normalized so that $c_n\in [0,1]$ for $n=0,1,\dots ,N-1$. A fixed and known constant $\alpha >0$ is multiplied to scale $c_n$ to the proper range.

QIS is a spatial oversampling device. For an N-element signal $\mathit{c}$, QIS uses $M\gg N$ number of jots to acquire $\mathit{c}$. Thus, every $c_n$ in the signal $\mathit{c}$ is sampled by $M/N$ jots. The ratio $K\stackrel{\mathrm{def}}{=}M/N$ is called the spatial oversampling factor. To model the oversampling process, we follow [13] by considering an upsampling operator $(\uparrow K)$ and a low-pass filter with filtering coefficients $\left\{g_k\right\}$, as shown in Figure 1. Since upsampling and low-pass filtering are linear operations, the overall process can be compactly expressed using a matrix-vector multiplication

$$\mathit{s}=\alpha \mathit{G}\mathit{c},$$

(1)

where $\mathit{s}={[s_0,s_1,\dots ,s_{M-1}]}^T$ is the light intensity arriving at those M jots, and the matrix $\mathit{G}\in {\mathbb{R}}^{M\times N}$ encapsulates the upsampling process and the linear filtering.

There are a variety of choices for the low-pass filter depending on which filter provides a better model of the light intensity. In this paper, we make the following assumption about the low-pass filter.

Assumption 1.

We assume that $\left\{g_k\right\}$ is a boxcar function, such that $g_k=1/K$ for $k=0,\dots ,K-1$. Consequently, the matrix $\mathit{G}$ is

$$\mathit{G}=\frac{1}{K}{\mathit{I}}_{N\times N}\otimes {\mathbf{1}}_{K\times 1},$$

(2)

where ${\mathbf{1}}_{K\times 1}$ is a K-by-one all one vector, ${\mathit{I}}_{N\times N}$ is an N-by-N identity matrix and ⊗ denotes the Kronecker product.

Intuitively, what Assumption 1 does is to assume that the light intensity is piecewise constant. As will be discussed in Section 3, this is important for us to derive simple algorithms.

2.2. Quantized Poisson Observation

At the surface of the m-th jot, the light intensity $s_m$ generates $y_m$ photons according to the Poisson distribution

$$\mathbb{P}\left\{Y_m=y_m;s_m\right\}=\frac{s_m^{y_m}{e}^{-s_m}}{y_m!},$$

(3)

where $Y_m$ denotes the Poisson random variable at the m-th jot and $y_m$ denotes the realization of $Y_m$.

The one-bit quantization of QIS is a truncation of $y_m$ with respect to a threshold $q\in {\mathbb{Z}}^{+}$. Precisely, the observed one-bit measurement at the m-th jot is

$$b_m\stackrel{\mathrm{def}}{=}\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{y}_m\ge q,\hfill \\ 0,\hfill & \mathrm{if}\mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

Denoting $B_m$ the random variable of the one-bit measurement at the m-th jot, it follows that the probability of observing $B_m=1$ is

$$\mathbb{P}\left\{B_m=1;s_m\right\}=\mathbb{P}\left\{Y_m\ge q;s_m\right\}=\sum _{k=q}^{\infty }\frac{s_m^k{e}^{-s_m}}{k!},$$

(5)

and the probability of observing $B_m=0$ is $\mathbb{P}\left\{B_m=0;s_m\right\}={\sum }_{k=0}^{q-1}\frac{s_m^k{e}^{-s_m}}{k!}$.

For general q, keeping track of the sum of exponentials in Equation (5) could be cumbersome. To make our notations simple, we adopt a useful function called the incomplete Gamma function [33], defined as follows.

Definition 1.

The incomplete Gamma function ${\mathsf{\Psi }}_q:{\mathbb{R}}^{+}\to [0,1]$ for a fixed threshold $q\in {\mathbb{Z}}^{+}$ is defined as

$${\mathsf{\Psi }}_q\left(s\right)\stackrel{\mathrm{def}}{=}\frac{1}{\mathsf{\Gamma }\left(q\right)}{\int }_s^{\infty }{t}^{q-1}{e}^{-t}dt=\sum _{k=0}^{q-1}\frac{s^k{e}^{-s}}{k!},$$

(6)

where $\mathsf{\Gamma }\left(q\right)=(q-1)!$ is the standard Gamma function evaluated at q.

The incomplete Gamma function is continuous in s, but discrete in q. For every fixed s, ${\mathsf{\Psi }}_q\left(s\right)$ is the cumulative distribution of a Poisson random variable evaluated at $q-1$. When q is fixed, ${\mathsf{\Psi }}_q$ is the likelihood function of the random variable $B_m$. In this paper, we will focus on the latter case where q is fixed so that ${\mathsf{\Psi }}_q$ is a function of s. Since ${\mathsf{\Psi }}_q$ is continuous, the derivative of ${\mathsf{\Psi }}_q\left(s\right)$ is available:

Property 1.

The first order derivative of ${\mathsf{\Psi }}_q\left(s\right)$ is

$$\frac{d}{ds}{\mathsf{\Psi }}_q\left(s\right)=-\frac{e^{-s}{s}^{q-1}}{\mathsf{\Gamma }\left(q\right)}$$

(7)

With the incomplete Gamma function, we can rewrite Equation (5) as

$$\begin{array}{c}\hfill \mathbb{P}\left\{B_m=1;s_m\right\}=1-{\mathsf{\Psi }}_q\left(s_m\right),\mathrm{and}\mathbb{P}\left\{B_m=0;s_m\right\}={\mathsf{\Psi }}_q\left(s_m\right).\end{array}$$

(8)

Example 1.

When $q=1$, the incomplete Gamma function becomes ${\mathsf{\Psi }}_q\left(s_m\right)=e^{-s_m}$. In this case,

$$\begin{array}{c}\hfill \mathbb{P}\left\{B_m=1;s_m\right\}=1-e^{-s_m}and\mathbb{P}\left\{B_m=0;s_m\right\}=e^{-s_m}.\end{array}$$

2.3. Image Reconstruction for QIS

We consider a multiple exposure setting. Assuming that the scene is stationary, the measurement we obtain is a sequence of binary bit maps $\left\{b_{m,t}\right|m=0,\dots ,M-1,\mathrm{and}t=0,\dots ,T-1\}$, where the first index m runs through the M jots spatially, and the second index t runs through the T frames in time. The image reconstruction task is to find the signal $\mathit{c}={[c_0,\dots ,c_{N-1}]}^T$ that best explains $\left\{b_{m,t}\right\}$. Translating into a probabilistic framework, this can be formulated as maximizing the likelihood:

$$\begin{array}{cc}\hfill \widehat{\mathit{c}}& =\underset{\mathit{c}}{\mathrm{argmax}}\prod _{t=0}^{T-1}\prod _{m=0}^{M-1}\mathbb{P}{[B_{m,t}=1;s_m]}^{b_{m,t}}\mathbb{P}{[B_{m,t}=0;s_m]}^{1-b_{m,t}},\mathrm{subject}\mathrm{to}\mathit{s}=\alpha \mathit{G}\mathit{c},\hfill \\ & =\underset{\mathit{c}}{\mathrm{argmax}}\prod _{t=0}^{T-1}\prod _{m=0}^{M-1}{(1-{\mathsf{\Psi }}_q\left(s_m\right))}^{b_{m,t}}{\mathsf{\Psi }}_q{\left(s_m\right)}^{1-b_{m,t}},\mathrm{subject}\mathrm{to}\mathit{s}=\alpha \mathit{G}\mathit{c},\hfill \end{array}$$

(9)

where the constraint $\mathit{s}=\alpha \mathit{G}\mathit{c}$ follows from Equation (1). Taking the logarithm on the right-hand side of Equation (9), the maximization becomes

$$\widehat{\mathit{c}}=\underset{\mathit{c}}{\mathrm{argmax}}\sum _{t=0}^{T-1}\sum _{m=0}^{M-1}\left\{b_{m,t}log(1-{\mathsf{\Psi }}_q\left(s_m\right))+(1-b_{m,t})log{\mathsf{\Psi }}_q\left(s_m\right)\right\},\mathrm{subject}\mathrm{to}\mathit{s}=\alpha \mathit{G}\mathit{c}.$$

(10)

The optimization in Equation (10) is known as the maximum likelihood estimation (MLE). As shown in [13], the objective function of Equation (10) is concave (for any q), and therefore, the global maximum exists and is unique. When $\mathit{G}$ satisfies Assumption 1, the closed form solution is available, and we will show it in Section 3. However, for general $\mathit{G}$ or if we include additional constraints to enforce the smoothness of the solution, e.g., using total variation [34] or sparsity [35], then numerical algorithms are required. Some existing methods include the gradient descent method [13], Newton method [28] and the alternating direction method of multipliers [29]. These algorithms are iterative, and the computation for each iteration is costly.

3. Non-Iterative Image Reconstruction

In this section, we present a non-iterative image reconstruction algorithm for QIS. There are three components behind the algorithm. The first is the closed-form solution of Equation (10), which is coherent to [13]. The second component is a variance stabilizing transform that transforms the sum of one-bit quantized Poisson random variables to a binomial random variable with equal variances. The transformation is known as the Anscombe transform, named after the English statistician Frank Anscombe (1918–2001). The third idea is the application of an off-the-shelf image denoiser.

3.1. Component 1: Approximate MLE

Because of the piecewise constant property of Assumption 1, the sequence $\left\{b_{m,t}\right\}$ can be partitioned into N blocks where each block contains $K\times T$ binary bits. This leads to a decomposition of Equation (10) as

$$\widehat{\mathit{c}}=\underset{\mathit{c}}{\mathrm{argmax}}\sum _{t=0}^{T-1}\sum _{n=0}^{N-1}\sum _{k=0}^{K-1}\left\{b_{Kn+k,t}log\left(1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\right)+(1-b_{Kn+k,t})log{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\right\},$$

(11)

where the subsequence ${\mathcal{B}}_{n,t}\stackrel{\mathrm{def}}{=}\{b_{Kn,t},\dots ,b_{Kn+(K-1),t}\}$ denotes the n-th block of the t-th frame. Define

$$S_n\stackrel{\mathrm{def}}{=}\sum _{t=0}^{T-1}\sum _{k=0}^{K-1}{b}_{Kn+k,t}$$

(12)

be the sum of the bits (i.e., the number of one’s) in ${\mathcal{B}}_{n,t}$. Then, Equation (11) becomes

$$\begin{array}{c}\hfill \widehat{\mathit{c}}=\underset{\mathit{c}}{\mathrm{argmax}}\sum _{n=0}^{N-1}{S}_{n}log\left(1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\right)+(L-S_n)log{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right),\end{array}$$

(13)

where $L=KT$. The maximum of Equation (13) is attained when each individual term in the sum attains its maximum. In this case, the closed form solution of Equation (13), for every n, can be derived as in Proposition Equation 1.

Proposition 1.

The solution of Equation (13) is

$${\widehat{c}}_n=\frac{K}{\alpha }{\mathsf{\Psi }}_q^{-1}\left(1-\frac{S_n}{L}\right),n=0,\dots ,N-1,$$

(14)

where $L=KT$, K is the spatial oversampling factor and T is the number of exposures.

Proof. 

See Appendix A. ☐

It would be instructive to illustrate Proposition 1 using a figure. Figure 2 shows the case when $T=1$, i.e., a single exposure, and $K=16$. The one-bit measurements are first averaged to compute the number of ones within a block of size K. Then, applying the inverse incomplete Gamma function ${\mathsf{\Psi }}_q^{-1}(·)$ and a scaling constant $K/\alpha$, we obtain the solution ${\widehat{c}}_n$.

Proposition 1 shows why a simple summation $S_n/L$ is inadequate to achieve the desired result, although such summation has been used in [18,36]. By only summing the number of ones, the resulting value $S_n/L$ is the empirical average of these one-bit measurements. Since the probability of drawing a one in QIS follows a quantized Poisson distribution and not a Bernoulli distribution, the nonlinearity due to the quantized Poisson distribution must be taken into account. A comparison of the ground truth image, the summation result and the MLE solution is shown in Figure 3.

As an immediate corollary of Proposition 1, we can simplify the inverse incomplete Gamma function when $q=1$. This result is sometimes known as the exponential correction function [37].

Corollary 1.

When $q=1$, the MLE solution of the n-th pixel ${\widehat{c}}_n$ is

$${\widehat{c}}_n=-\frac{K}{\alpha }log\left(1-\frac{S_n}{L}\right),n=0,\dots ,N-1,$$

(15)

where $S_n$ is the number of ones in the n-th block (See Equation. (12)) and $L=KT$ is the total number of pixels in the block.

Proof. 

The result follows from the fact that ${\mathsf{\Psi }}_q\left(s\right)=e^{-s}$ for $q=1$. Thus, ${\mathsf{\Psi }}_q^{-1}\left(s\right)=-logs$. ☐

3.2. Component 2: Anscombe Transform

The MLE solution $\widehat{\mathit{c}}={[{\widehat{c}}_0,\dots ,{\widehat{c}}_{N-1}]}^T$ computed through Proposition 1 is noisy, as illustrated in Figure 3. The reason is that for a relatively small K and T, the randomness in the one-bit measurement has not yet been eliminated by the summation in $S_n$. Therefore, in order to improve the image quality, additional steps must be taken to improve the smoothness of the image.

At first glance, this question seems easy because if one wants to mitigate the noise in $\widehat{\mathit{c}}$, then directly applying an image denoising algorithm $\mathcal{D}$ to $\widehat{\mathit{c}}$ would be sufficient, e.g., Figure 4a. However, a short afterthought will suggest that such an approach is invalid for the following reason. For the majority of image denoising algorithms in the literature, the noise is assumed to be independently and identically distributed (i.i.d.) Gaussian. In other words, the variance of the noise should be spatially invariant. However, the resulting random variable in Figure 2 does not have this property.

Our proposed solution is to apply an image denoiser before the inverse incomplete Gamma function as shown in Figure 4b. Besides the order of denoising and the Gamma function, we also add a pair of nonlinear transforms $\mathcal{T}$ and ${\mathcal{T}}^{-1}$ before and after the denoiser $\mathcal{D}$. The reasons for these two changes are based on the following observations.

Observation 1.

Under Assumption 1, the random variables $\left\{B_{Kn+k,t}\right|k=0,\dots ,K-1,andt=0,\dots ,T-1\}$ are i.i.d. Bernoulli of equal probability $\mathbb{P}[B_{Kn+k,t}=1]=1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)$ for $k=0,\dots ,K-1$ and $t=0,\dots ,T-1$.

The proof of Observation 1 follows immediately from Assumption 1 that if $\mathit{G}=(1/K){\mathit{I}}_{N\times N}\otimes {\mathbf{1}}_{K\times 1}$, we can divide the M jots into N groups each having $K\times T$ entries. Within the group, the one-bit measurements are all generated from the same pixel $c_n$.

The consequence of Observation 1 is that for a sequence of i.i.d. Bernoulli random variables, the sum is a Binomial random variable. This is described in Observation 2.

Observation 2.

If $\left\{B_{Kn+k,t}\right\}$ are i.i.d. Bernoulli random variables with probability $\mathbb{P}[B_{Kn+k,t}=1]=1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)$ for $k=0,\dots ,K-1$ and $t=0,\dots ,T-1$, then the sum $S_n$ defined in Equation (12) is a Binomial random variable with mean and variance:

$$\mathbb{E}\left[S_n\right]=L\left(1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\right),\mathrm{Var}\left[S_n\right]=L{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\left(1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\right).$$

Observation 2 is a classic result in probability. The mean of the Bernoulli random variables is specified by the incomplete Gamma function ${\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)$, which approaches one as K increases. Thus, for fixed T, the probability $1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\to 0$ as $K\to \infty$. When this happens, the binomial random variable $S_n$ can be approximated by a Poisson random variable with mean $L\left(1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)\right)$ [38]. However, as T also grows, the binomial random variable $S_n$ can be further approximated by a Gaussian random variable due to the central limit theorem. Therefore, for a reasonably large K and T, the resulting random variable $S_n$ is approximately Gaussian.

The variance of this approximated Gaussian is, however, not constant. The variance changes across different locations n because $\mathrm{Var}\left[S_n\right]$ is a function of $c_n$. Therefore, if we want to apply a conventional image denoiser (which assumes i.i.d. Gaussian noise) to smooth $S_n$, we must first make sure that the noise variance is spatially invariant. The technique used to accomplish this goal is called the variance stabilizing transform [39]. In this paper, we use a specific variance stabilizing transform known as the Anscombe transform [32]. Anscombe transform is best known in the image processing literature for Poisson denoising, where one transforms observed Poisson data to approximately Gaussian with equal variance [24]. For binomial random variables $S_n$, the Anscombe transform and its property are given in Theorem 1.

Theorem 1

(Anscombe transform for binomial random variables). Let $S_n$ be a binomial random variable with parameters $(L,p_n)$, where $p_n=1-{\mathsf{\Psi }}_q\left(\frac{\alpha c_n}{K}\right)$ and $L=KT$. Define the Anscombe transform of $S_n$ as a function $\mathcal{T}:\{0,\dots ,L\}\to \mathbb{R}$, such that

$$Z_n=\mathcal{T}\left(S_n\right)\stackrel{\mathit{def}}{=}\sqrt{L+\frac{1}{2}}{sin}^{-1}\left(\sqrt{\frac{S_n+\frac{3}{8}}{L+\frac{3}{4}}}\right).$$

(16)

Then, the variance of $Z_n$ is $\mathrm{Var}\left[Z_n\right]=\frac{1}{4}+\mathcal{O}\left(L^{-2}\right)$ for all n.

Proof. 

The proof of Theorem 1 is given in the Appendix A. It is a simplified version of a technical report by Brown et al. [40]. The original paper by Anscombe [32] also contains a sketch of the proof. However, the sketch is rather brief, and we believe that a complete derivation would make this paper self-contained. ☐

The implication of Theorem 1 is that regardless of the location n, the transformed random variable $Z_n$ has a constant variance $\frac{1}{4}$ when L is large. Therefore, the noise variance is now location independent, and hence, a standard i.i.d. Gaussian denoiser can be used.

Example 2.

To provide readers a demonstration of the effectiveness of Theorem 1, we consider a checkerboard image of $N=64$ pixels with intensity levels $c_0,\dots ,c_{N-1}$. The n-th pixel $c_n$ generates $K=100$ binary quantized Poisson measurements $\{B_{Kn},\dots ,B_{Kn+(K-1)}\}$ using $\alpha =100$, $q=1$, $T=1$ (so $L=100$). From each of these K measurements, we sum to obtain a binomial random variable $S_n={\sum }_{k=0}^{K-1}{B}_{Kn+k}$. We then compute the variance of $\mathrm{Var}\left[S_n\right]$ and $\mathrm{Var}\left[\mathcal{T}\left(S_n\right)\right]$ using ${10}^4$ independent Monte Carlo trials. The results are shown in Figure 5, where we observe that $\mathrm{Var}\left[S_n\right]$ varies with the location n, and $\mathrm{Var}\left[\mathcal{T}\left(S_n\right)\right]$ is nearly constant for all n.

Remark 1.

The inverse Anscombe transform is

$$S_n={\mathcal{T}}^{-1}\left(Z_n\right)=\left(L+\frac{3}{4}\right){sin}^2\left(\frac{Z_n}{\sqrt{L+\frac{1}{2}}}\right)-\frac{3}{8},$$

(17)

which we call the algebraic inverse. Another possible inverse of the Anscombe transform is the asymptotic unbiased inverse [32], defined as

$$S_n={\mathcal{T}}_{\mathrm{unbias}}^{-1}\left(Z_n\right)={\left(1+\frac{1}{2L}\right)}^{-1}\left[\left(L+\frac{3}{4}\right){sin}^2\left(\frac{Z_n}{\sqrt{L+\frac{1}{2}}}\right)-\frac{1}{8}\right].$$

(18)

The performance of the unbiased inverse is typically better for low noise (large L), whereas the algebraic inverse is better for high noise (small L). This is consistent with the Poisson denoising literature. (e.g., [24]).

Example 3.

Table 1. PSNR values using algebraic inverse ${\mathcal{T}}^{-1}$ and asymptotic unbiased inverse ${\mathcal{T}}_{\mathrm{unbias}}^{-1}$. The results are averaged over 10 standard images. In this experiment, we set $T=1$, $q=1$ and $\alpha =K$.

K	1	4	9	16	25	36	49	64
${\mathcal{T}}^{-1}$	20.51	23.08	25.00	26.47	27.49	28.40	29.09	29.71
${\mathcal{T}}_{\mathrm{unbias}}^{-1}$	19.43	23.64	25.30	26.62	27.57	28.45	29.12	29.73

shows the peak signal to noise ratio (PSNR) values of the reconstructed images using the algebraic inverse and the asymptotic unbiased inverse. In this experiment, we consider 10 standard images commonly used in the image processing literature: Baboon, Barbara, Boat, Bridge, Couple, Hill, House, Lena, Man and Peppers. The sizes of the images are either $256\times 256$ or $512\times 512$. For each image, we set $T=1$, $q=1$ and $\alpha =K$ and vary $K=\{1,4,9,16,25,36,49,64\}$. The results in Table 1 indicate that ${\mathcal{T}}_{\mathrm{unbias}}^{-1}$ is consistently better than ${\mathcal{T}}^{-1}$ for $K>1$, although the difference diminishes as K grows.

3.3. Component 3: Image Denoiser

An important feature of the proposed algorithm is that it can take any off-the-shelf image denoiser for the operator $\mathcal{D}$. Here, by image denoiser, we meant an image denoising algorithm designed to remove i.i.d. Gaussian noise from an observed image.

Image denoising is an important research topic by its own. In the following, we provide a few popular image denoising algorithms.

• Total variation denoising [34]: Total variation denoising was originally proposed by Rudin, Osher and Fatemi [34], although other researchers had proposed similar methods around the same time [41]. Total variation denoising formulates the denoising problem as an optimization problem with a total variation regularization. Total variation denoising can be performed very efficiently using the alternating direction method of multipliers (ADMM), e.g., [42,43,44].
• Bilateral filter [45]: The bilateral filter is a nonlinear filter that denoises the image using a weighted average operator. The weights in a bilateral filter are the Euclidean distance between the intensity values of two pixels, plus the spatial distance between the two pixels. A Gaussian kernel is typically employed for these distances to ensure proper decaying of the weights. Bilateral filters are extremely popular in computer graphics for applications, such as detail enhancement. Various fast implementations of bilateral filters are available, e.g., [46,47].
• Non-local Means [48]: non-local means (NLM) was proposed by Buades et al. [48] and, also, an independent work of Awante and Whitaker [49]. Non-local means (NLM) is an extension of the bilateral filter where the Euclidean distance is computed from a small patch instead of a pixel. Experimentally, it has been widely agreed that such patch-based approaches are very effective for image denoising. Fast NLM implementations are now available [50,51,52].
• BM3D [53]: 3D block matching (BM3D) follows the same idea of non-local means by considering patches. However, instead of computing the weighted average, BM3D groups similar patches to form a 3D stack. By applying a 3D Fourier transform (or any other frequency domain transforms, e.g., discrete cosine transform), the commonality of the patches will demonstrate a group sparse behavior in the transformed domain. Thus, by applying a threshold in the transformed domain, one can remove the noise very effectively. BM3D is broadly regarded as a benchmark of today’s image denoising algorithm.

The factors to consider in choosing an image denoiser are typically the complexity and quality. Low complexity algorithms, such as bilateral filter, are fast, but the denoising ability is limited. High end algorithms, such as BM3D and non-local means, produce very good images, but require much computation. The trade-off between complexity and performance is a choice of the user.

Readers at this point may perhaps ask a question: What will happen if we apply a denoiser after the MLE solution, like the one shown in the block diagram in Figure 4a? As we have explained in the Anscombe transform section, this will lead to a suboptimal result because the noise is not i.i.d. Gaussian. To illustrate the difference in terms of performance, we show in Figure 6 a comparison between applying image denoising using the two block diagrams shown in Figure 4. The denoiser we use in this experiment is BM3D. The metric we use to evaluate the performance is the peak signal to noise ratio (PSNR), which will be defined formally in Section 4. In short, a large PSNR value is equivalent to a low mean squared error comparing the estimated image and the ground truth image. The results are shown in Figure 6. Although denoising after the MLE (the conventional idea) generates some reasonable images, the PSNR values are indeed significantly lower than the proposed Anscombe approach. This is not surprising, because the denoiser tends to oversmooth the dark regions and undersmooth the bright regions due to the signal-dependent noise levels.

3.4. Related Work in the Literature

The proposed algorithm belongs to a family of methods we call the transform-denoise methods. The idea of transform-denoise is similar to what we do here: transform the random variable using a variance stabilizing transform, then denoise using an off-the-shelf image denoiser. Among the existing transform-denoise methods, perhaps the most notable work is the one by Makitalo and Foi [24], where they considered the optimal inverse of the Anscombe transform for the case of Poisson–Gaussian random variables. A more recent work by the same research group [27] showed that it is possible to boost the denoising performance by applying the transform-denoise iteratively. We should also mention the work by Foi [54], which considered the modeling and transformation for clipped noisy images. The problem setting of that work is for conventional sensors. However, the underlying principle using the transform-denoise approach is similar to that of QIS.

The approximate MLE solution in Section 3.1 is based on the piecewise constant assumption (Assumption 1). Under this assumption, summing of the Bernoulli random variables can be thought of as performing a “binning” of the pixels. Binning is a common technique in restoring images from Poisson noise, especially when the signal-to-noise ratio is low [23,25,26]. Binning can also be applied together with transform-denoise, e.g., in [27], to achieve improved results. For QIS, the result of binning is different from that of the Poisson noise, for the sum of QIS bits leads to a binomial random variables, whereas the sum of Poisson noise leads to a Poisson random variable.

4. Experimental Results

In this section, we provide further experimental results to evaluate the proposed algorithm.

4.1. Synthetic Data

In this experiment, we consider 100 natural images downloaded from the Berkeley Segmentation database [55]. The input resolution of these images is $481\times 321$ (or $321\times 481$), and all images are converted to a gray-scale image with values in the range $[0,1]$. To generate the synthetic data, for each image, we consider an oversampling factor of four along the horizontal and the vertical directions (so $K=16$). The sensor gain is set as $\alpha =16$, and the threshold level is $q=1$ to simulate a single photon sensor that triggers at one photon. We generate one-bit observations using a quantized Poisson statistics and use the proposed algorithm to reconstruct. As a comparison, we also test the MLE solution, i.e., a summation followed by the inverse incomplete Gamma transform (see Theorem 1), and an ADMM algorithm using a total variation regularization [29,31]. For the proposed algorithm, we use BM3D as the image denoiser.

We report two results in this experiment. First, we consider the peak signal to noise ratio (PSNR) as an evaluation metric. The PSNR of an estimated image $\widehat{\mathit{c}}$ compared to the ground truth image ${\mathit{c}}^{*}$ is defined as

$$\mathrm{PSNR}=-20{log}_{10}\left(\frac{\parallel \widehat{\mathit{c}}-{\mathit{c}}^{*}{\parallel }^2}{N}\right),$$

(19)

where N is the number of pixels in $\mathit{c}$. Typically, higher PSNR values imply better image reconstruction quality. The PSNR values of these 100 images are shown in Figure 7. In this figure, we observe that the proposed algorithm is better than the MLE solution and the ADMM solution by 10.20 dB and 2.75 dB, respectively, which are very substantial amounts from a reconstruction perspective.

Apart form the PSNR values, we also report the runtimes of the algorithms. The runtimes of the algorithms are recorded by running the methods on the same machine and the same platform, which is an Intel i7-6700 3.4-GHz desktop with Windows 7/MATLAB 2014. As shown in Figure 8, the runtime of the proposed algorithm is approximately two orders of magnitude ($100\times$) faster than the ADMM algorithm.

As for the influence of the oversampling factor K on the reconstruction quality, we show in Figure 9 the reconstructed results using $K=4,16,64$ and the binary one-bit measurements. While it is clear from the figure that the reconstructed image improves as K increases, we observe that most of the visual content has been recovered even at $K=4$.

4.2. Real Data

In this experiment, we consider two single-photon avalanche diode (SPAD) cameras for capturing high speed videos. The first camera is a CMOS SPAD-based image sensor developed by Dutton et al. [14,15,16]. This camera has a resolution of $320\times 240$, with a frame rate of 10k frames per second. The second camera is the SwissSPAD camera developed by Burri et al. [17,18]. This camera has a resolution of $512\times 128$, with frame rate of 156k frames per second. Both cameras capture one-bit measurements from a scene containing a stationary background with a rapidly moving foreground. Our experimental goal is to test if the proposed algorithm can resolve the spatial content with minimal trade-off in the temporal resolution.

There are several points of this experiment on which we should comment. First, since the spatial resolution of these two cameras is relatively small (as compared to the synthetic case), we do not assume any spatial oversampling, i.e., $K=1$. Instead, we use T temporal frames to reconstruct one output image. To ensure smooth transitions across adjacent frames, we use a temporally-sliding window as we progress to the next output image. Second, since these real videos do not have a ground truth, we can only compare the quality of the resulting images visually.

We first look at the results of the SPAD camera by Dutton et al. [14,15,16]. Figure 10 shows several snapshots of the “fan” sequence and the “milk” sequence. To generate this result, we run the proposed algorithm using $T=16$ frames in a sliding window mode. That is, we use Frames 1 to 16 to recover Frame 1 and Frames 2 to 17 to recover Frame 2, etc. The quantization level q is set as $q=1$, and the sensor gain α is adjusted to produce the best visual quality. For the “fan” sequence, we set $\alpha =4$, and for the “milk” sequence, we set $\alpha =0.75$. As we can see from the figures, the proposed algorithm recovers most of the content from the scene, even revealing the textures of the milk in the scene.

As for the SwissSPAD camera, we consider a video sequence “oscilloscope” captured at a frame rate of 156k frames per second. The goal is to track the sinusoid shown on the oscilloscope’s screen. We consider four values of T = 4, 16, 64 and 256 for the number of frames. The results are shown in Figure 11. As one may expect, when $T=256$, the image quality improves because we are effectively summing 256 frames to reconstruct one output frame. However, since we are summing the 256 frames, the temporal resolution is severely distorted. In particular, the trace of the sinusoid signal disappears because of the strong averaging effect. When we reduce the number of frames to $T=16$, we observe that the trace of the signal can be observed clearly. The same image obtained by the MLE (i.e., simple summation) is still highly noisy.

5. Conclusions

We present a new image reconstruction algorithm to recover images from one-bit quantized Poisson measurements. Different from existing algorithms that are mostly iterative, the new algorithm is non-iterative. The algorithm consists of three key components: (1) an approximation to the standard maximum likelihood estimation formulation that allows us to decouple the dependency of pixels; (2) a nonlinear transform known as the Anscombe transform that converts a sum of one-bit quantized Poisson random variables to a Gaussian random variable with equal variance; (3) an off-the-shelf image denoising algorithm that performs the smoothing. Experimental results confirm the performance of the proposed algorithm. The algorithm demonstrates two orders of magnitude improvement in speed compared to existing iterative methods and shows several dBs of improvement in terms of the peak signal to noise ratio (PSNR) as a metric of image quality.