High-Dynamic-Range Spectral Imaging System for Omnidirectional Scene Capture

.


Introduction
Omnidirectional imaging is a useful technology which has been widely used in daily life.For instance, omnidirectional images are useful for landscape archiving applications such as Google Street View [1] and the "Aftermath of the 2011 Tohoku earthquake and tsunami" (A project by The University of Tokyo and Tohoku University) [2].Omnidirectional images are also currently applied to virtual reality (VR) technologies through head-mounted display systems [3][4][5].According to the popularization of these applications, high-quality omnidirectional imaging system is required in terms of color information, dynamic range information and resolutions.
The dynamic range of luminance values in natural scenes is quite large and the scenes also include various spectral power distributions.The technology of an omnidirectional spectral imaging system for such a high dynamic range (HDR) natural scene is crucial in recent computer vision, computer graphics, and color imaging.Until now, imaging systems have been developed separately as spectral imaging, omnidirectional imaging, and HDR imaging with different applications.For instance, with regards to spectral imaging, various types of multiband camera systems were proposed for the purposes of digital archiving, medicine, and natural scene analysis [6][7][8][9][10][11][12].Omnidirectional imaging systems were proposed mainly for surveillance, intelligent transport systems, scene archiving [13][14][15][16][17][18].Recently, video streaming services such as YouTube handle omnidirectional video streams since consumer-grade omnidirectional cameras have been released [19][20][21].The basic technology of HDR imaging has been already developed and the systems are widely used in current digital cameras and computer graphics rendering [22,23].
Hybrid systems with two functions were also proposed for specific purposes.The LadyBug camera system is available for capturing red-green-blue (RGB) images of omnidirectional HDR scenes [16].Hirai et al. proposed an imaging method to recover spectral information from HDR scenes [24].Martínez-Domingo et al. recently proposed an HDR spectral imaging system for material classifications [25].However, there was only a system based on the integration of spectral imaging, omnidirectional imaging, and HDR imaging technologies [26].
The integrated system of spectral, omnidirectional, and HDR imaging is useful not only for natural scene archiving and analysis [27], but also multispectral rendering using environment mapping [28].Tominaga et al. developed an HDR spectral imaging system for capturing an omnidirectional natural scene [26].This system was constructed with an RGB digital camera, a fisheye lens, two-color filters and a rotating table.Given that the system used a three-band RGB camera and two additional color filters attached to the camera, it realized a multiband imaging system with six spectral bands in the visible wavelength range.This system works well for static scenes where objects and light sources are unchanged over time.However, the essential drawback is that the system is not available for time-lapse scenes where objects and light sources move with time.The conventional system of one color camera and two-color filters is based on manual operation.Operators have to exchange the color filters and rotate the camera table manually, which renders the image acquisition time-consuming.Therefore, if objects move during image acquisition, misregistration occurs between the spectral channels.As a result, the combined multiband image includes ghosting and color artifacts.Figure 1 demonstrates a color image combined from the multiband images with misregistration when a cloudy sky was captured by the same system.Color artifacts and ghosts are caused by the random movement of clouds.It is difficult to align or compensate for the spectral images of randomly-moving objects.Reliable image data of dynamic scenes are valuable for many fields including natural scene analysis [27,29], image rendering [28], and time-lapse image editing [30].A solution method for reliably capturing dynamic natural scenes is to construct a one-shot multi-band imaging system.In practice, the system can be constructed using a stereo system [31,32] or multispectral filter array [33,34].However, these proposed multiband imaging systems are neither omnidirectional nor HDR.
In this paper, we propose a new system for acquiring HDR spectral images for capturing omnidirectional scenes.In order to implement quick image acquisition without color artifacts due to the filter exchanges, the system is constructed using two programmable high-speed video cameras with specific lenses and a programmable rotating table.Two different types of color filters are mounted on the two-color video cameras for six-band image acquisition.We present several algorithms for HDR image synthesis, lens distortion correction, image registration, and omnidirectional image synthesis.We reconstruct spectral power distributions (color signals) from the six-band images based on the Wiener estimation algorithm.Based on this imaging system, we present two types of applications: time-lapse imaging and gigapixel imaging.These imaging techniques can be realized by replacing the mounted lenses.In other words, our imaging system is applied to the extensions of either temporal resolution or spatial resolution.The performances of the proposed system are evaluated in detail in terms of the system configurations, acquisition time, color artifacts, and spectral estimation accuracy.Finally, we present the experimental results for acquiring natural scenes based on time-lapse or gigapixel imaging approaches.Additionally, as applications of the proposed imaging system, we implement time-lapse spectral computer graphics renderings and spectral-based gigapixel image relighting of an indoor archived scene.
Compared with the previous manual system [26] for acquiring HDR omnidirectional spectral images, the time-lapse imaging by the proposed system realizes the short-time image acquisition without any color artifacts thanks to the automatic-controlled stereo-based omnidirectional imaging system.In addition, we apply the proposed system to gigapixel imaging.The conventional system was developed for scene color analysis and image-based lighting.The resolution of the conventional system is poor for the high-quality VR reproduction.On the other hand, the gigapixel imaging by the proposed system provides high-resolution 360 • VR images with accurate color reproduction.In Section 5, we also discuss the performance comparison between the proposed and the conventional systems.

Related Studies
Multi-band imaging is a useful technology that is now widespread in all fields related to visual information.Research covers a broad range of areas including multispectral image acquisition, spectral reflectance estimation, illuminant estimation, accurate color reproduction, and computer graphics rendering.The application fields of multi-band imaging include medicine, digital archiving, wide gamut technology and natural scene analysis.Thus far a variety of multi-band imaging systems and methods have been proposed for acquiring spectral information from a scene.A well-known technique for realizing multi-band imaging is to combine a monochrome camera and color filters with different spectral bands [7,8].For acquiring accurate colors of art paintings, Haneishi et al. proposed an algorithm to select optimal color filters in a multi-band system that consisted of a monochrome camera and color filters [35].Tominaga et al. reproduced accurate colors of CG art paints by using such a six-band camera system [36].These systems usually require a time-consuming process for manually changing the color filters.Another well-known technique is to employ a camera system with six or more bands by combining one or two RGB digital cameras with additional color filters.The conventional HDR omnidirectional multi-band imaging system used a three-band RGB camera and two additional color filters for realizing six-band imaging [26].Shrestha et al. introduced a stereo-based imaging system that consisted of two RGB cameras with two different color filters [31].Tsuchida et al. developed a stereo-based eleven-band imaging system with one RGB camera and eight monochromatic cameras with eight different color filters [32].Given that these multi-band imaging systems are typically based on commercial digital cameras and color filters, the equipment costs can be reduced.Multi-band imaging techniques with unique optical systems have been also proposed.Ohsawa et al. presented a six-band imaging system using two three-band image sensors and a beam splitter [37].Yata et al. proposed twelve-band imaging by mounting four color filters on the optical axis between a lens and an RGB sensor [33].Similarly, Manakov et al. developed a 27-band imaging technique using such an optical system and nine color filters [34].In addition, several studies have addressed to modulate color filter arrays on image sensors for developing multi-band imaging systems.Tanida et al. mounted nine color filters on an image sensor with a microlens array [38].Monno et al. used an image sensor with a five-colored filter instead of a conventional RGB Bayer filter [39].Sajadi et al. proposed an RGB-CMY imaging technique by using two switchable CMY filters [40].Shrestha et al. simulated six-band imaging by placing chromagenic filters on top of conventional RGB filters [41].Multi-band imaging has been often realized based on multi-color illuminations.Park et al. proposed a multi-band imaging system based on an RGB camera and five different color LEDs [42].Hirai et al. developed a system using a monochromatic camera and six types of color LEDs for archiving art paints [43].Nakahata et al. used a spectral light source for capturing multi-band images [44].In general, to recover the spectra from multi-band images, spectral estimation methods can be applied [24,35,45].
A variety of omnidirectional imaging systems have also been developed.As a simple way, fisheye lenses have been used for taking 180 • images.Recently, omnidirectional stereo imaging systems based on fisheye (or wide-angle) lenses have been also developed for studying scene depth analysis and stereo VR technologies [15,46,47].In addition, camera systems with mirrors have been proposed.Nayar et al. presented a catadioptric omnidirectional camera by using hemisphere mirrors [13].Omnidirectional stereo imaging using multi-mirrors has been also proposed [48][49][50].In the research field of computer graphics, a spherical mirror ball is often used to apply environment mapping [18].Since only one image is sufficient for recovering a hemisphere image, these mirror-based imaging techniques provide the short-time image acquisition.However, the recovered omnidirectional image generally has low resolution with some mirror distortions.In addition, the scene region occluded by the camera is not recovered.As another approach for omnidirectional imaging, image stitching algorithms are well known [51].The stitching technique is often applied to gigapixel panorama imaging.For example, an omnidirectional gigapixel imaging system has been developed based on image stitching techniques [17].This gigapixel imaging system takes a huge amount of images by using a telescopic camera and an automatically-controlled rotating table.As shown in such examples, stitching-based omnidirectional imaging provides a high-resolution omnidirectional image with less distortion.However, the image acquisition time of stitching-based imaging is longer than that of mirror-based imaging due to the number of captured images.For reducing the acquisition time in the stitching-based techniques, imaging systems with multi-directional cameras have been developed.Akin et al. proposed a hemispherical multiple camera system for high-resolution omnidirectional imaging [52].Perazzi et al. presented a panorama imaging system using an unstructured camera array [53].However, these imaging systems does not cover a 360 • spherical field of view.For synthesizing a 360 • spherical image, recently, commercial-grade omnidirectional cameras with two or more image sensors have been released [19,20].The popularization of these reasonable commercial cameras, we can easily upload omnidirectional videos on video streaming services and develop VR scenes [4,21].
As described above, various multi-band and omnidirectional imaging systems have been proposed.However, there are few systems for capturing both multi-band and omnidirectional images simultaneously.Tominaga et al. had developed a multi-band omnidirectional imaging system for archiving and analyzing natural scenes [26].The main contribution of this system is to realize a simple and reasonable system for 360 • omnidirectional images with spectral information.In addition, the spatial resolution of the images captured is much better than the mirror-based systems thanks to the uses of several images.However, as described in the Introduction, this system had a problem with the time consumed for capturing a scene.In addition, this system encountered a loss and a non-uniformity of spatial resolutions because of the use of a fisheye lens.
Until now, various imaging systems have been developed for capturing omnidirectional, HDR or spectral images.Some conventional systems have been proposed for satisfying two of the three functions.The imaging system in Ref. [26] is the only conventional system developed for capturing HDR omnidirectional spectral images.However, the conventional system requires much acquisition time due to the manual control.There are no practical imaging systems to concurrently satisfy these three functions.In addition, though time-lapse and gigapixel imaging technologies will be required by users for next-generation omnidirectional scene archiving, these issues have not been addressed.

Proposed System for HDR Omnidirectional Spectral Imaging
Figure 2 shows the proposed imaging system for acquiring HDR spectral images of time-varying omnidirectional scenes.The system consists of two programmable high-speed RGB video cameras with two wide-angle (or telephoto) lenses, and a programmable rotating table.Two different types of color filters are mounted on the two video cameras, respectively.
The video camera is a Baumer HXG20 (Baumer Electric AG: Frauenfeld, Switzerland) with the image size of 2048 × 1088 pixels and the bit depth of 12 bits, which can capture images over 100 frames per second (fps).The cameras also have linear response characteristics.These camera specifications are enough for reducing the acquisition time.The lens is a Spacecom wide-angle lens for time-lapse imaging, which covers a 95 • horizontal angle and a 61 • vertical angle.In addition, we employ a Kowa telephoto lens for gigapixel imaging, which covers a horizontal angle of 14.5 • and a vertical angle of 10.8 • .The rotating table is a CLAUSS RODEON VR station HD (Dr.Clauss: Zwönitz, Germany), which can be automatically controlled by computer.The table allows for a 360 • movement in the horizontal plane and 180 • movement in the vertical plane, respectively.The blue broken lines shown in Figure 2 indicate the rotational axes of the horizontal and vertical rotations.The two cameras are mounted on the table by rotating 90 degrees.The optical axis of the proposed imaging system is set to the optical axis of the camera (A).Optimal filter selections are important for recovering accurate spectra.The proposed six-band imaging system consists of two RGB cameras and two different color filters.One of practical and effective filter selections in six-band imaging based on the combination of RGB cameras and two-color filters is to use following two-color filters [6,26,35]: one is for shifting the RGB spectral sensitivities to the short and long wavelength, and the other is for shifting the sensitivities to the middle wavelength.Based on the above observations, we collected 36 commercially-available color filters and measured their spectral transmittance.Figure 3 shows the transmittance of the collected color filters.Then, through simulation and actual experiments of spectral recoveries, the two additional color filters of KODAK No.34A (Kodak: Rochester, USA) and FUJIFILM SP-18 (Fujifilm: Tokyo, Japan) were selected for applying to our image acquisition.By combining these color filters to the camera sensitivities, we can obtain effective six channel-spectral sensitivity functions.Figure 4 shows the overall spectral sensitivity functions of the proposed imaging system.The KODAK No.34A filter is effective for shifting the spectral sensitivities to the short and long wavelength in the visible range as shown in the solid curve of Figure 4.The FUJIFILM SP-18 filter is effective for shifting the spectral sensitivities to the middle wavelength as shown in the broken curve.To generate six-band images from the stereo cameras shown in Figure 2, we align the captured images based on a Phase Only Correlation algorithm [54] (Please refer to the next section as well).

Synthesizing HDR Omnidirectional Spectral Images
Figure 5 shows the flow of all of the computational algorithms for estimating HDR omnidirectional spectral images.The details of the algorithms are described below.

HDR Image Synthesis
A natural scene has a wide luminance range from dark shadowed areas to a highly bright sky.The HDR imaging technique by Debevec et al. [55] was adopted for capturing a natural scene.In our system, an HDR image is created by combining fifteen low dynamic range (LDR) images captured at different exposure times from 252 ms. to 0.015 ms.An HDR image can be synthesized by: where I HDR ij is the pixel value at a coordinate (i, j) of the synthesized HDR image, t k is k-th exposure time, I k ij is a pixel value at (i, j) of the LDR image at the k-th exposure time, and w(I) is a weighting function for the pixel values of the LDR images.
Note that a ghost removal technique is not applied in this study [22,51,56].It is known that the above multi-exposure synthesizing technique often causes motion artifacts called "ghost".However, in our experimental cases, the synthesized HDR images include no noticeable ghost artifacts, thanks to the short image acquisition time, i.e., approximately 503 ms. in total.If a synthesized HDR image includes ghost artifacts, a ghost removal technique will be required.

Correction of Lens Distortion
For creating accurate omnidirectional images, it is crucial to correct the lens distortion of captured images.We measured several lens distortion characteristics by using Zhang's method [57] in advance.Then we corrected the captured images based on the measured distortion characteristics.

Image Registration
Given that the original images are captured by the stereo camera system, there is a disparity (displacement) between one image with a KODAK No.34A filter and the other with a Fujifilm SP-18 filter.Then, we have to completely align the two images for the combined six-band images with no registration error.We have implemented the Phase Only Correlation (POC) technique [54] for the image registration.The POC calculates the displacement by using the correlation between Fourier transformed images.The algorithm is given as: where F(u, v) and G(u, v) are the Fourier transformation of two input images, G(u, v) is a complex conjugation of G(u, v).The displacement can be obtained from the spatial coordinates with a maximum value in the inverse Fourier transformation of R(u, v).As described in the previous section, the optical axis of the proposed imaging system is standardized by one of the cameras (A) in Figure 2. Therefore, the images captured by camera (A) are used as the reference image for the image registration.Figure 6 shows an example of the image registration results in creating a six-band image from original two-color images.

Polar Coordinate Transformation
We use the polar coordinate system for representing omnidirectional images.All captured images are transformed into the polar coordinate system by the following equation [58].
where f is a focal length, (x, y, f ) indicates the 3D coordinates and (θ, φ) are 2D spherical coordinates.Figure 7 displays an example of the transformed polar coordinate image.In the present system for time-lapse imaging, the polar coordinate image in Figure 7 covers 61 • horizontal and 95 • vertical angles.

Omnidirectional Image Synthesis
For synthesizing an omnidirectional image, we utilize a total of 36 images (twelve images in the horizontal plane and three images in the vertical plane) for time-lapse imaging and 1260 images (sixty images in the horizontal plane and twenty-one images in the vertical plane) for gigapixel imaging.The number of images were decided based on the lens specifications, which were described in Section 2. The wide-angle lens covers 61 • horizontal and 95 • vertical angles, and the telephoto lens does 14.5 • horizontal and 10.8 • vertical angles.In order to avoid significant vignetting effects and leave image stitching regions with keeping the synthesized image quality, we empirically decided the number of images (directions).The use of the programmable rotating table allows us to control and record the horizontal and vertical directional angles of each captured image.In other words, we are able to identify the relative polar coordinates among the polar coordinate images.In addition, for achieving accurate spherical projections and image stitching results, we set the imaging system on the ground by using a spirit level.Then, we can synthesize an omnidirectional image from 36 or 1260 polar coordinate images based on the recorded directional angles and an image stitching algorithm [59].Given that each polar coordinate image covers wider angles, the overlapping regions between the neighboring images are blended linearly.Figure 8 shows an example of the six-band omnidirectional images synthesized from the 36 images.

Spectral Estimation
The input spectral power distributions of the camera consist of two types of color signals: (1) direct illumination from the sky and (2) indirect illumination of the reflected light from object surfaces in the scene.The camera outputs for the color signals are then described with noisy observations such as: where E(λ) denotes the incident color signal with a continuous spectrum in the visible range [400, 700 nm], and R i (λ), as the spectral sensitivity functions, s i are the signal components, and σ i are the noise components of the i-th sensor.x denotes the spatial coordinates on an image.We can rewrite Equation ( 5) in a matrix form: where ρ is the 6-dimensional sensor output, e denotes the n-dimensional vector representing the spectrum E(λ) when sampling spectral functions at n points in the visible wavelength range.R is a diagonal matrix with the size of 6 × n for the spectral sensitivity function, σ is the six-dimensional noise vector, and s is the six-dimensional signal.
The statistical estimation theory is adopted for recovering spectra from multi-band images [60].Our spectral estimation method is based on the Wiener estimator [24,35], which has been often utilized for recovering spectral information from noisy observations.However, we note that this estimator requires prior statistical knowledge such as the covariance matrix of spectral distributions and the covariance matrix of observation noises.Here, we present the Wiener estimation and our parameter settings.When e and σ are statistically uncorrelated, the estimated signal ê can be given by: where e is an average spectrum of the database, σ 2 is the noise variance, I is a 6 × 6 unit matrix, C ss is the covariance matrix of a spectral database that is statistically determined as C ss = E[(e − e)(e − e) t ].
In this study, we preliminarily determined the noise variance by minimizing the estimation error between the estimates and measured spectral power distributions of the Macbeth Color Checker under various light sources.To determine C ss , we used a spectral database which is generated by multiplying 1378 surface spectral reflectances of various natural and artificial objects by nine spectral power distributions of different light sources.

Performances of the Proposed Imaging System
Tables 1 and 2 summarize the proposed system configurations and the performance.Six-band images were acquired using two-color video cameras with two different types of color filters.In our image acquisition, we require no filter exchange by employing a multiband imaging based on a stereo one-shot technique.For time-lapse imaging, an omnidirectional image is synthesized by thirty-six images (twelve images in the horizontal plane and three images in the vertical plane), and the pixel size is 7260 × 3630.For gigapixel imaging, an omnidirectional image is synthesized by 1260 images (sixty images in the horizontal plane and twenty-one images in the vertical plane), and the pixel size is 60,000 × 32,000.The acquisition time for each directional image is approximately 5 s which includes both the capturing and the rotating processes.Then, it takes approximately 180 s for the time-lapse imaging and approximately 2 h for the gigapixel imaging, in total, to capture the multiband HDR images for an omnidirectional scene.As a performance comparison, we used a conventional integrated system proposed in Ref. [26].We note that the conventional system requires a color filter exchange.The system control is based on manual operation.Then, the conventional system requires at least 10 min to acquire all images in the pixel sizes of 6000 × 3000.Moreover, the proposed system needs no filter exchange and is fully-automatic.Hence, the image acquisition can become much shorter than that of a conventional system in time-lapse imaging, which is slightly superior in spatial resolution.

Color and Spectral Accuracy
To validate the accuracy of the recovered spectral power distributions, we compare the estimated results with the ground truth.In this experiment, we estimated the spectra of the X-rite ColorChecker under D65.The ground truth data was measured by a spectrophotometer.Figure 9 displays the estimation results of the spectral power distributions.The average normalized RMSE of 24 colors in the ColorChecker is 0.0148, and the average color difference ∆Eab is 1.49.Compared with the conventional multi-band imaging systems and spectral estimation results [26], our results demonstrate that the proposed system can estimate the spectral power distribution accurately.We also estimated spectral power distributions using a same RGB camera shown in Section 2. The average normalized RMSE and the average color difference ∆Eab of the 24 colors are 0.0530 and 6.12, respectively.As shown in these estimation results, the proposed six-band imaging system can recover accurate spectra compared with an RGB imaging system.
Figure 10 shows the comparative results of evaluating color artifacts around a moving cloud.In Figure 10a, the multi-band images of the conventional system were synthesized using the images captured at different times, where the time interval between one image capturing and the other capturing was approximately 6 min.In this scene, clouds in the sky moved quickly.For these reasons, the misregistration and color artifacts caused around the clouds are as shown in Figure 10a.Conversely, the image captured by the proposed system has no color artifacts around the moving clouds.Figure 11 shows the estimated color signal ê (solid line) from our six-band imaging and the direct measurements e m (broken line) for a sky area and a green tree in Figure 8b.The measurements were obtained by using a spectro-radiometer.In these figures, the spectral power distributions are normalized as e m 2 = ê 2 = 1.The estimation accuracy of color signals was examined at five points in the scene of Figure 8b.The average RMSE in the proposed system was 0.0149.The accuracy would be within an acceptable range in the spectral estimation.In addition, we compared the estimation results from the proposed six-band imaging with those from a same RGB camera.The average RMSE of RGB imaging was 0.0319.Even in an actual scene, the proposed six-band imaging is superior to the RGB imaging in terms of the spectral recovery.

Spectral Image Characterization of a Time-Lapse Scene
Figure 12 shows the HDR spectral image obtained for a time-lapse omnidirectional scene of our university campus.Note that the sun regions in Figure 12 are saturated, so that the colors of the sun are incorrectly reproduced.The image sequence was captured successively from 6:00 a.m. to 5:00 p.m. at 5-min intervals.The entire capturing procedure could be automatically carried out.Thus, the present system is effective in obtaining a time-lapse omnidirectional spectral image sequence.Figure 13 represents the time-variations of the spectral power distributions observed in daytime from the light source of the fixed sky region and the reflected surface of the building wall, which are shown as red squares in Figure 12, respectively.The spectral power distributions of the sky vary dramatically throughout the day.In particular, we can confirm the morning sky produces a reddish color.In addition, it is shown that the spectral power distributions of the white wall as a reflectance object are influenced by those of the sky.

Spatial Resolution Characterization of Gigapixel Imaging
As a resolution comparison, we compare the proposed system based on the gigapixel imaging with the one based on time-lapse imaging.Figure 14 displays the comparative results.The resolution based on the gigapixel imaging is approximately 1.9 gigapixels.Whereas the resolution based on the time-lapse imaging is approximately 26 megapixels, because the images based on time-lapse imaging have been developed to acquire time-lapse spectral images of omnidirectional natural scenes.As shown in Figure 14, we can observe the details of distant objects.If immersive virtual experiences through head mounted display systems are required, omnidirectional images with such gigapixel resolutions will be one of the solutions.

Trade-Off between the Number of Captured Images, Acquisition Time and Image Quality
The proposed system requires a lot of directional images for synthesizing an omnidirectional image.In particular, the proposed gigapixel imaging requires approximately 2 h and 17 min in the image capturing step.As a practical implementation for reducing the number of captured images and the acquisition time, we consider reducing pixels of overlapping regions that are used in the image synthesis (image stitching).Figure 15a shows the relationship between overlapping pixels and the relative acquisition time (or the relative number of captured images, or capturing directions).The red circle in Figure 15a represents the current setting of the gigapixel imaging shown in Table 2.The blue circle in Figure 15a shows the image capturing setting with no overlapping regions.In this case, it takes approximately 46 min in the image acquisition.However, no overlapping regions cause significant fails in the image stitching.We empirically found that at least 12% overlapping pixels were required for achieving the image synthesis (the green circle in Figure 15a).In this case, it takes approximately 1 h and 19 min in the image acquisition.However, the case of 12% overlapping still causes some miss-alignments in the image stitching.Figure 15b,c show the image stitching results under the red and green circle settings of Figure 15a, respectively.Compared with the current setting (the red circle of Figure 15a and a synthesized result shown in Figure 15b), the case of 12% overlapping (the green circle of Figure 15a and a result in Figure 15c) often distorts the image quality around stitching regions.As shown in these results, we need to determine the capturing setting by considering the trade-off between the number of images, the acquisition time and the image quality.

Time-Lapse Spectral Computer Graphics Rendering
Environmental mapping is a significant rendering technique in computer graphics.For realistic image rendering, HDR omnidirectional images are used as environmental lighting [18,61].Recently omnidirectional multi-spectral images also have been applied for accurate color reproduction [27,62].However, time-lapse omnidirectional spectral images have not been applied for spectral CG rendering yet.Then we implemented spectral CG rendering using our time-lapse omnidirectional spectral images.
Figure 16 shows the spectral CG rendering results using time-lapse image sequence in Figure 12.In this scene rendering, a mirror ball on a gray plate is used as a computer graphics object.As shown in these results, our imaging system provides time-lapse spectral CG rendering with accurate color information.

Gigapixel Image Relighting of Indoor Arichived Scene
Gigapixel images have been applied to scene archiving.In particular, gigapixel imaging technologies are significant for archiving insides of cultural heritages and museums.Then we tested our system for indoor scene archiving.In addition, we relighted an omnidirectional indoor scene, since spectral imaging is useful for illumination simulation [42,44,63,64].
Figure 17 shows the experimental results of an indoor scene and gigapixel image relighting.In this experiment, we measured illumination spectra by using a white reference which was located in the scene.Then, we implemented the image relighting under an incandescent illumination (CIE standard illuminant A).We also show the close-up images of an art painting which exists at the left-side in the scene.The close-up images have the resolution of approximately 2400 × 1600 pixels.If our system is applied to omnidirectional indoor scene archiving such as museums, high-quality digital museums under various illumination conditions will be provided through VR technologies.

Conclusions and Discussion
This paper has proposed an HDR spectral imaging system for omnidirectional natural scenes.The system was constructed using two programmable high-speed video cameras with wide-angle (or telephoto) lenses and a programmable rotating table.Two different color filters were mounted on each video camera to acquire six-band images.We presented algorithms for HDR image synthesis, lens distortion correction, image registration, omnidirectional image synthesis, and spectral recovery.The proposed system was applied to the extensions of either time resolution or spatial resolution for time-lapse imaging or gigapixel imaging.The performances of the proposed system were discussed in terms of the system configurations, acquisition time, spatial resolution, artifacts, and spectral estimation accuracy.Experimental results in an outdoor natural scene showed that the proposed system was feasible and powerful for acquiring time-lapse HDR spectral images or high-resolution omnidirectional scenes.Finally, we applied the captured omnidirectional images to time-lapse spectral CG renderings and spectral-based gigapixel image relighting.
As shown in the experimental results, the proposed system can provide accurate spectra and colors compared with the conventional system.However, the proposed system of HDR omnidirectional six-band imaging still has two problems in the spectral and color recovery.One is a ghost artifact.As described in Section 4.1, we do not implement a ghost removal technique.Therefore, if a target scene includes fast-moving objects (such as a car, bicycle, or running people), the proposed system will cause ghost artifacts in a synthesized image.The other is an occlusion artifact.The stereo-based omnidirectional imaging causes color artifacts due to the scene occlusions by the stereo cameras.As shown in the bottom side of Figure 8b, we can see color artifacts around a computer which existed within one meter from the cameras.The color artifacts are caused by the occlusions around close-range objects and the image registration fails.This is the disadvantage of stereo-based six-band imaging compared with a single camera system.Similarly, recent stereo-based VR technologies using two omnidirectional RGB cameras have carefully handled such occlusion problems [46,47].The occlusion problem in omnidirectional six-band imaging is a remaining challenge for improving spectral and color reproductions.
In addition, the acquisition time of the proposed system based on gigapixel imaging should be reduced.Similar to conventional gigapixel imaging systems [65,66], we should carefully discuss and address the considerable acquisition time.This causes changes in spectral power distributions of the sunlight.In this paper, we discussed the relationship between the acquisition time, capturing directions, overlapping regions, and the synthesized image quality in Section 5.5.If further acquisition-time reduction is required, it is necessary to develop a novel hardware configuration and an image capturing algorithm for realizing the short-time and high-quality gigapixel imaging.

Figure 1 .
Figure 1.Color artifacts in the captured multispectral image of moving clouds.

Figure 4 .
Figure 4. Spectral sensitivity functions of the proposed six-band imaging system.

Figure 6 .
Figure 6.Image registration results for a natural scene, where the right image represents an aligned six-band image.(a) Before image registration; (b) After image registration.

Figure 7 .
Figure 7. Effective images of the polar coordinate transformation.(a) Before transformation; (b) After transformation.

Figure 10 .
Figure 10.Comparison of color images with moving clouds produced by the two systems.(a) Conventional system; (b) Proposed system.

Figure 11 .
Figure 11.Spectral estimation results of an actual scene.(a) Sky; (b) Green tree.

Figure 15 .
Figure 15.Trade-off between acquisition time, overlapping pixels and image quality.(a) Relationship between acquisition time and overlapping pixels; (b) Synthesized image under the red circle setting of (a); (c) Synthesized image under the green circle setting of (a).

Figure 17 .
Figure 17.Experimental result of indoor scene capture and gigapixel image relighting using illuminant A (incandescent illumination).(a) Captured gigapixel indoor scene; (b) Close-up of an art painting in (a); (c) Relighted gigapixel image of (a); (d) Close-up of an art painting in (c).

Table Programmable Table 2 .
Performances of the Proposed Imaging System.