1.1. Evolution of the CCD-ISIS
Image sensors for high-speed image capturing may be categorized as follows:
Parallel/Partial Readout Imaging Scheme (PPR):
This is the most common technique to increase the frame rate simply by reading out image signals in parallel through many readout taps, and/or partially only from a selected area in the imaging area.
In situ Storage Imaging Scheme (ISIS):
A number of storage elements are placed within or nearby each pixel to store a number of consecutively captured images. The ISIS camera operates at an ultra-high frame rate with a frame interval equal to the time required for signal electrons generated in photo-diodes move to the nearby storage elements [
1–
6].
The PPR cameras can provide higher spatial resolution when operating at reasonably high frame rates. Currently, advanced PPR cameras achieve several thousand frames per second (fps) for more than 1,000,000 pixels [
7]. However, the pixel count reduces when the cameras operate at a higher frame rate. For example, a camera of this type can capture images at more than 1,000,000 fps, but the pixel count reduces to less than 64 × 16. The number of continuous frames is practically unlimited, since the image signals are continuously read out from the image sensor and stored in a large size buffer memory device outside the sensor.
An ISIS camera has achieved 1,000,000 fps (1 Mfps) [
2,
3]. The frame rate is not limited by the pixel count. However, the number of continuous frames is severely limited by the number of storage elements installable in a small area nearby each pixel. The number of
in situ memory elements should be more than 100, considering that replay at 10 fps makes moving images look smooth and replay for more than 10 seconds at 10 fps enables scientists and engineers fully recognize dynamic events. Image signals stored in the
in situ storage are read out slowly from the sensor after cease of each continuous image acquisition phase.
Most of the
in situ storage image sensors are CCD imagers [
1–
6], whereas some are based on the CMOS imager technology [
8]. Kosonocky
et al. proposed and fabricated a CCD ISIS, and proved the validity of the in-pixel storage scheme [
1]. Each pixel is equipped with an SPS (Series-Parallel-Series) CCD register, which transfer signal charges in two directions in a pixel,
i.e., horizontally, vertically and then horizontally again during image capturing. The sensor is capable of recording 30 consecutive images at 833,000 fps.
In 2001, Etoh
et al. developed a CCD image sensor in which a slanted linear CCD register is installed in each pixel to achieve a simplest one-direction charge transfer. The camera records 103 consecutive 81,120-pixel images at a speed of 1 Mfps [
2,
3]. Ohtake
et al., supervised by Etoh, developed the color version with 300,000 pixels capable of recording 144 consecutive frames while maintaining the maximum frame rate at 1 Mfps (ISIS-V4) [
4]. These cameras have been extensively applied to high-speed fluid dynamics [
9].
In 2007, Lazovsky
et al. proposed a CCD structure that can record only 16 consecutive images of 64 × 64 pixels at the speed of 100 Mfps [
6]. Due to inexistence of continuous overwriting mechanism, low fill factor and low quantum efficiency, the sensor has limited practical uses.
Some technologies can capture consecutive images at much higher frame rates. Shiraga
et al. achieved a frame interval down to several picoseconds by using electronic streak combined with a linear slanted imaging scheme, which is similar to the slanted linear CCD storage [
10]. By implementing an innovative holographic imaging technology, Kubota
et al. successfully captured three-dimensional moving images of flying light of a frame interval in the order of femtoseconds [
11]. Goda
et al. developed a technology that can take images at a frame rate of 6.1 millions fps with a single photo-detector [
12]. Even conventional multi-framing cameras operate at frame rates higher than 1 Mfps.
However, the ISIS cameras [
2–
5] provide much better image quality, higher spatial resolution and wider applicability with the simple and compact structure. ISIS cameras have been tested for applicability in bio- and nano-science and engineering. For the tests, the cameras were mounted on microscopes and TEMs. However, most of the attempts failed. The sensitivity of the image sensors is insufficient; the incident light is too weak due to a very short frame interval and a very high magnification rate.
Therefore, the authors started development of new image sensors that satisfy the two requirements: ultra-high speed and ultra-high sensitivity. The sensor is a backside illuminated ISIS (BSI-ISIS) [
5]. The very high frame rate is achieved by employment of the ISIS structure on the front side; the very high sensitivity achieved by a combined use of three existing technologies: backside illumination, cooling and CCM,
i.e., the multi-step impact ionization [
14]. CCD image sensors equipped with CCM are referred as EM-CCD and widely used for fluorescent imaging in bio-science.
The test BSI-ISIS has been fabricated and evaluated (ISIS-V12) [
5,
13]. The pixel count is 200,000; the full well capacity is 10,000e-. The maximum frame rate is 250,000 fps and 1 Mfps, respectively for full well capacity and reduced charge handling capacity. The sensitivity is very high: images with a 4.9e- signal level in the 7.7e- noise floor was observed. It is noteworthy that the BSI-ISIS which utilizes the CCM can detect signals with levels lower than the noise floor.
The authors thus have started designing in parallel the following two new BSI-ISISes:
The first one is currently in the final stage of fabrication. The frame rate and the sensitivity are expected to be 10 Mfps and sub-ten-photon per pixel, respectively (ISIS-MV12). The design has been briefly explained elsewhere [
5]. The theoretical maximum frame rate for the BSI-ISIS is estimated about 100 Mfps, as explained later. The design has been completed. It was named ISIS-100M after its targeted frame rate. The design requires three metal layers. In addition, to achieve 100 Mfps, two metal layers are necessary on the carrier substrate to send driving voltages through stacked differential wires.
Development of the CCD-ISIS family is summarized in
Table 1.
Current standard CMOS processes utilize four or more metal layers. CCD can be fabricated by advanced CMOS processes with the fine design rule with slight modification [
15]. Therefore, the authors started the modification of the ISIS-100M to fit an advanced CMOS process. The CMOS-based ISIS adds another function, PPR, the parallel and partial readout, to the ultra-high speed and very high sensitivity of the CCD-ISIS. The sensor is named as RA-ISIS, after its function: the random-access and the
in situ storage image signal. Reducing the chip size is another way to increase the frame rate. A fine CMOS process effectively reduces the pixel size of the CCD-ISIS. The RA-ISIS will realize the ultimate ultra-high-speed image sensor.