A Do-It-Yourself Hyperspectral Imager Brought to Practice with Open-Source Python

Abstract

Commercial hyperspectral imagers (HSIs) are expensive and thus unobtainable for large audiences or research groups with low funding. In this study, we used an existing do-it-yourself push-broom HSI design for which we provide software to correct for spectral smile aberration without using an optical laboratory. The software also corrects an aberration which we call tilt. The tilt is specific for the particular imager design used, but correcting it may be beneficial for other similar devices. The tilt and spectral smile were reduced to zero in terms of used metrics. The software artifact is available as an open-source Github repository. We also present improved casing for the imager design, and, for those readers interested in building their own HSI, we provide print-ready and modifiable versions of the 3D-models required in manufacturing the imager. To our best knowledge, solving the spectral smile correction problem without an optical laboratory has not been previously reported. This study re-solved the problem with simpler and cheaper tools than those commonly utilized. We hope that this study will promote easier access to hyperspectral imaging for all audiences regardless of their financial status and availability of an optical laboratory.

1. Introduction

Hyperspectral imaging based on unmanned aerial vehicles (UAVs), especially drones, has gain a lot of attention in recent years, for example, in precision agriculture [1,2] and precision forestry [3,4,5,6]. Compared to more traditional aircraft- and spacecraft-based operations, drones offer ease of operation, cost-efficiency, and small ground-pixel size. In addition, drone-based imaging is possible even in cloudy weather [2].

Commercial hyperspectral imagers (HSIs) are expensive, which restricts their usage for low-budget operators, such as individual researchers or landowners, as even the cheapest HSIs cost tens of thousands of euros [7,8]. Commercial HSIs are also heavy and bulky, which introduces additional requirements for drones utilized in UAV campaigns [8]. Several cheap HSIs have been introduced in the literature: some are made lightweight for UAV operations [8,9], while others target laboratory usage [7]. While several of these HSIs can be manufactured in do-it-yourself manner, calibration and aberration correction methods usually require access to an optical laboratory.

Our study builds on do-it-yourself HSI presented by Sigernes et al. [9], which we call the Sigernes design. It is a push-broom type HSI that can be manufactured with off-the-shelf optical components and cameras, which are embedded in a 3D-printed casing. Its manufacturing process does not require access to special equipment other than a thermoplastic 3D printer. A photograph of our adaptation of the Sigernes design is shown in Figure 1.

A push-broom HSI forms a hyperspectral image by scanning the area being imaged line by line. The front lens sees a circular area, which is restricted to a line by a narrow slit. Light passing through the slit is collimated before diffracting it into its wavelengths by a diffraction grating. Finally, diffracted light is focused to the sensor of a video camera. In each captured frame, each row represents a spectrograph of a slit pixel. Stacking all scanned frames together forms the final hyperspectral image, often called an image cube. The image cube then consists of two spatial dimensions and one spectral dimension.

Manual manufacturing process of the Sigernes design gives rise to a certain aberration that we call the tilt. The tilt originates from misalignment of the slit, i.e., as the slit is cut into a circular plate of metal that is inserted by hand, it is very difficult to align it perfectly with the grating and the camera. In our experience, 1–2 degree accuracy can be achieved with reasonable effort. As the tilt is specific for this design, we did not find any suggestions how to correct it in the literature, although Sigernes et al. [9] noted its existence.

Aside from the problems caused by the manufacturing process, there exist several optical aberrations that are unavoidable in push-broom imagers in general. On an ideal push-broom HSI, given an evenly illuminated area of single material, any spectral emission lines would form perfectly vertical lines of even length on the sensor. On real devices, the lines are not straight or of even length, if not corrected by optical design or software solutions [10].

Spectral smile is an aberration that causes the emission lines to have a curved shape. It is caused by dispersion when the slit image is spread over the sensor [11]. Figure 2 shows an example of a raw, uncorrected frame acquired from our imager side by side with an ideal synthetic frame that is free of aberrations. In addition, the well-known keystone aberration is barely visible in the real frame. It causes the emission lines to shorten towards the red end (right-hand side in the figure) of the spectrum. Tilt and smile aberrations are more easily distinguishable in figures in Section 3.

Optical aberrations are usually expressed in terms of point spread functions (PSFs) [10,12], but determining a PSF for a given instrument is not a trivial task and requires laboratory equipment, such as a monochromator [12] or spectral calibration lamps [13]. There exist several corrections for spectral smile [11,12,13,14], but we found none that could be carried out without an optical laboratory. For illustration of the most common optical aberrations present in push-broom imagers and suggestions on how to minimize their effect by optical design, please refer to [10]. For correction of keystone and smile of the Sigernes design when optical laboratory is available, please refer to [13].

In Section 2, we provide an improved casing for the Sigernes design as a printable 3D model (available at the project’s OSF repository at https://osf.io/3uhkb) along with an open-source software artifact capable of imaging and aberration correction (available at https://github.com/silmae/desmiler). Section 3 presents two experiments: the first one shows the effectiveness of our correction method with synthetic data, and the second one demonstrates imaging without an optical laboratory (dataset for experiment replication is also available at the OSF repository). The results of the experiments are shown in Section 4 and discussed in Section 5. Section 6 concludes the study.

2. Materials and Methods

The working principle of the Sigernes design is quite simple: light is first gathered and restricted to a narrow beam, which is then diffracted over the video camera’s sensor. The 3D-printed casing keeps the optical components immobile relative to each other and prevents stray light entering the sensor. The design of the optics is thoroughly described in [9]. Here, we highlight some changes and improvements made to the original casing. The presented 3D models are available at https://osf.io/3uhkb. Modifiable versions of the models can be opened and modified using the open-source computer aided design (CAD) software FreeCAD (https://www.freecadweb.org).

The slit needs to be aligned with the grating and the camera sensor in order to obtain correct spectral data. Any misalignment causes imaged emission lines to be tilted. In the original casing in [9], the slit could rotate freely, which we found difficult to control. Our casing fixes the slit in place when the casing is closed, which provides consistent slit orientation over imaging sessions. Furthermore, fixing the slit part of the S-tubes (2 in Figure 1), along with the collimator lens part (3 in Figure 1), allows adjustment of the focus tube during operation. A thumb hole was cut to the casing for the purpose.

Fixing holes in the bottom half of the casing fit to optical breadboards with 50 mm hole spacing for easy integration into optical laboratory equipment. For tripod usage, an adapter plate was devised (printable 3D model available at https://osf.io/srzgf).

The used video camera was Grasshopper3 model GS3-U3-91S6M-C (Flir) machine vision camera with ICX814 (Sony) monochromatic CCD sensor of $3376\times 2704$ pixels. The 3D model of the casing must be modified if a different camera is being used. The camera heats up during operation, so a connection to air must be established to allow heat to dissipate. The orientation of the camera is forced to be parallel with the grating by the casing, so problems in orientating the camera with the slit image, as mentioned in [15], are avoided.

The used optical components are identical to those used in [9]. They are listed in Appendix A for completeness. From a monetary point of view, the used system costs less than 2000 euros, of which the machine vision camera costs roughly 1000 euros. Using cheaper camera and diffraction grating can bring the total price down to few hundred euros [8]. Recent development in 3D-printing imaging quality lenses [16] may lower the cost further in the future. A further benefit of using 3D-printed optics would be that, as it does not matter if normal or aberration corrected lens is printed, one could use better lenses in do-it-yourself imagers without additional cost and thus reduce the amount of aberrations.

The presented software artifact (available at https://github.com/silmae/desmiler) is capable of scanning hyperspectral image cubes, correcting spectral smile and tilt, and showing side-by-side comparisons of corrected and uncorrected data. For image cube comparison, there is a graphical user interface (GUI) with band selection and spectral angle mapping [17,18] views. Raw camera feed preview for tuning camera parameters and focus is also provided.

An effort was made to produce maintainable and modifiable program code, for example by using separate parameter script for dataset naming conventions and using an easily changeable interface for camera commands. We aimed for good coverage in documentation of the code. The software is written in Python programming language and it relies on publicly available libraries, most notably numpy [19], scipy [20], and xarray [21] for matrix calculations and data analysis, as well as open-source in-house camera controller library camazing (https://pypi.org/project/camazing).

2.1. Software Usage

The user interface (UI) of the software is implemented as an object of UI class used interactively through ipython [22] command shell. Instancing an object is simply done by running an ipython magic command %run ui.py, which will provide an UI object with name ui. Several synthetic examples (single frames and full image cubes) can also be generated and inspected using the UI object.

The flowchart in Figure 3 illustrates the operation of the software artifact. The only hardware operation is adjusting the focus with the help of the preview functionality. Operations used through the UI object are presented in the center column. There are several useful views helping the user with estimating the goodness of performed task (diamond shapes in the flowchart). Parameter tuning for scanning and aberration correction is done by writing desired values in a control file (right-hand column), which is reloaded from disk automatically when needed.

Preview is used to adjust the focus of the imager and cropping the raw sensor image to illuminated area. Scanning length and velocity can be given in arbitrary units, such as mm/s and mm, respectively. Acquisition overhead and exposure time affect how much time is allocated to acquire each frame and thus, the total frame count of the scan. The actual scanning time may differ if the imager cannot provide frames at requested rate, in which case the parameters need to be adjusted by increasing acquisition overhead or decreasing scanning velocity. Next, emission line location estimates and parameters for peak finding algorithm are given. After recording dark and white reference frames, aberration corrected reflectance image can be calculated.

2.2. Aberration Correction

Correcting the spectral smile is a well-researched subject (see, e.g., [11,12,13,14]), but we did not find any methods that could be applied without an optical laboratory. Our correction method is similar to that of Esmonde-White et al. [15] in that we search each row for pixels belonging to a certain emission line and transform the image so that curved lines become straight again. The major difference is that we do not need optical laboratory equipment for the correction. Our algorithm also takes the tilt effect into account and corrects both aberrations in a single pass. The main working principle is presented here, and an in-depth description is provided in Appendix B.

Our smile correction method consists of three distinguishable steps: locating the emission lines of a frame, constructing a shift matrix, and applying the shift. In emission line location, we use a single frame with sharp emission lines and run a peak-finding algorithm for every row. The $(x,y)$ positions of the peaks are ordered in sets, each belonging to a single emission line. A circular arc is then fitted to each set of pixel positions and its radius and center point are used to calculate an appropriate shift along each row needed to straighten the emission lines. These per-pixel shifts are stored in a shift matrix, which is used to alter the original frame to form a new, corrected frame. Figure 4 illustrates the matter. Some numerical considerations on the circular arcs and using curvature as a metric for straightness are presented in Appendix C.

Let us assume a photon that should have been registered in pixel $p=(x_p,y_p)$ was actually registered in pixel $p^{\prime }=(x_{p^{\prime }},y_p)$. Note that we assume the shift occurring only in spectral dimension, so the spatial y-coordinate remains constant.

Essentially, we want to know the distance between detected and true column of the photon $d=\parallel x_p-x_{p^{\prime }}\parallel$. Using the natural circle parameters, circle center $(a,b)$ and radius r, which are given by the circle fitting, we get that the vertical distance between the circle center and pixel p is $\parallel y_p-b\parallel$. Using simple trigonometry, the distance from the circular arc to the line is

$$d=r\left(1-cos\left(arcsin\frac{y_p-b}{r}\right)\right),$$

(1)

which is illustrated in Figure 4. It is easy to see that, if the circle center resides in the lower half of the frame, the distance d will be greater near the top of the grid than in the bottom. This asymmetry corrects the tilt effect as a side product.

We assume that the radius r of the circular arc is much greater than $\parallel y_p-b\parallel$, so the arcus sine is defined. This is a valid assumption, as, if it were not true, the frame would be distorted useless. With that assumption, we can say that, if $sgn\left(a\right)$ is positive, the emission line curves to the right and the correct value can be obtained from pixel $(x_p+d,y_p)$, otherwise from pixel $(x_p-d,y_p)$.

Regardless of the position we move the emission line to, distances between emission lines are likely to change, which means that any wavelength calibrations must be done after the smile correction.

3. Experiments

To quantify the effectiveness of our method and show that the imager can be used without an optical laboratory, we conducted two experiments. In the first experiment, synthetic data were generated to show that our algorithm truly decreases the amount of spectral smile and the tilt. In the second experiment, a Macbeth color checker was scanned to show that hyperspectral image cubes can be acquired using just common office appliances.

3.1. Synthetic Frames

First, to construct an ideal, undistorted synthetic frame, a fluorescence light was imaged. The mean of several sensor rows was calculated to form a single smooth spectrogram, which acted as a base for the synthetic frame generation. This spectrum was then repeated on the synthetic sensor’s imaging area similar to real sensor. To generate non-identical, but similar spectra, each row was multiplied by a small random value drawn from a Gaussian distribution. In addition, random noise drawn from uniform distribution was added to each pixel value. The rest of the sensor rows, which would not get illuminated in the real sensor, were filled with random noise drawn from uniform distribution. The process is similar to the one used in [15]. A side-by-side comparison of a full synthetic frame and a real frame can be seen in Figure 2. For the rest of this section, we use only the illuminated (cropped) area of the sensor, as we would in the case of real frames.

The tilt and the spectral smile were generated using the correction algorithm in reverse, i.e., first generating a distortive shift matrix and interpolating each row with it. The tilt was set to $1^{\circ }$ and the curvature to $3\times {10}^{-5}$ $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\mathrm{px}$}\right.$, which are common values found in real frames acquired by our imager. Examples of the generated frames cropped to the imaging area with and without distortions can be seen in Figure 5.

For statistical analysis, 1000 frames were generated. The smile correction procedure was run for each frame, and the emission line search and curve fitting was used to determine the amount of tilt and spectral smile before and after the correction. Emission line search was successful for 937 frames. It is not likely that the actual correction had failed, but rather that the emission lines were not recognized well enough. However, the matter was not investigated further.

Tilt of the emission lines was estimated with linear least square error fit in $(y,x)$ coordinate system; as the lines are nearly vertical, a line fit in $(x,y)$ coordinates would have caused nearly infinite slope. Curvature of the emission lines was estimated using parabolic fit as in Equation (A5). Other possible curvature estimation methods are further considered in Appendix C. This experiment can be fully replicated using the program code available at https://github.com/silmae/desmiler.

3.2. Color Checker

In the color checker experiment, a Macbeth color checker (ColorChecker Classic, X-Rite) was selected as a test target. It was taped onto the side of an office locker along with a four-fold strip of common copy paper as a white reference target. The imager was placed on an office desk against a metallic ruler fastened to the desk with a clamp and some tape. Instead of using a motorized scanning platform, the imager was slid along the ruler by hand. Distance from the front lens to the target was approximately 30 cm, while the length of the scan was 14 cm. From that distance the imager could cover the squares and some of their black boundaries in across scan direction. Imaged area consisted of the red, green, and blue color tiles (4 cm × 4 cm each). 798 frames with 110 ms exposure time were recorded. Cropping was set to 2000 pixels in spectral dimension and 800 pixels in along slit direction. The scene was illuminated by two halogen lamps with a diffuser. The lamps were set above the imager on both sides, and they were immobile relative to the target. The experiment setup is shown in Figure 6.

4. Results

Comparing the ideal frame in Figure 5a with corrected frame in Figure 5e shows that our method corrects the tilt and spectral smile to imperceptible levels. The emission lines are shifted out of the original locations, as expected. Statistical results in

Table 1. Mean values of tilt and curvature of 937 synthetic frames. Frames were generated with $1^{\circ }$ tilt and $3\times {10}^{-5} \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\mathrm{px}$}\right.$ curvature, which correlate well with mean values shown in original columns. Used error estimate is the standard deviation of the mean. Corrected values show that they are zero at the limit of the used metrics, as the error is greater than the values themselves. The four emission lines selected for the correction cover wavelengths approximately from 400 to 600 nm.

EL	Band	Tilt		Curvature
		Original	Corrected	Original	Corrected
1	629	$(902\pm 13)\times {10}^{-3}$	$(1\pm 7)\times {10}^{-3}$	$(29.4\pm 2.5)\times {10}^{-6}$	$(0.1\pm 1.2)\times {10}^{-6}$
2	762	$(899\pm 19)\times {10}^{-3}$	$(3\pm 79)\times {10}^{-3}$	$(29.0\pm 3.5)\times {10}^{-6}$	$(0.9\pm 10.3)\times {10}^{-6}$
3	980	$(887\pm 35)\times {10}^{-3}$	$(13\pm 107)\times {10}^{-3}$	$(27.3\pm 6.7)\times {10}^{-6}$	$(3.5\pm 23.3)\times {10}^{-6}$
4	1517	$(902\pm 7)\times {10}^{-3}$	$(1\pm 5)\times {10}^{-3}$	$(29.7\pm 1.2)\times {10}^{-6}$	$(0.1\pm 0.8)\times {10}^{-6}$
Total mean		$(897\pm 14)\times {10}^{-3}$	$(5\pm 14)\times {10}^{-3}$	$(28.9\pm 2.5)\times {10}^{-6}$	$(1.2\pm 2.6)\times {10}^{-6}$

show that, on average, the tilt was reduced from ${0.9}^{\circ }$ to ${0.005}^{\circ }$, and spectral smile from $30\times {10}^{-6}$ to $1\times {10}^{-6}$. The uncorrected values in the table show the tilt is underestimated compared to the $1^{\circ }$ tilt used in the generation process.

One may notice that the error estimate values, the standard deviation of the mean, of the corrected frames are greater than the actual values, which implies that the correction reduces the aberrations to zero in terms of the used metrics. Numerical considerations of the curvature metric, presented in Appendix C, show that the curvature metric begins to fluctuate when values decrease below ${10}^{-6}$, so this behavior is expected.

The correction was based on four emission lines covering the range approximately from 400 to 600 nm. Emission Line 3 shows significantly greater error estimates than the other three lines. We suspect that the algorithm had troubles recognizing the line properly on all occasions, but the matter was not investigated further.

The correction shifted each emission line by 5 pixels on average on the used four emission lines. The shift of lines other than those used for the correction was not investigated, but some of the emission lines in Figure 5 seem to have shifted considerably more than this. The mere existence of the location shift implies that any wavelength calibration must be performed after the smile correction and not before.

The false color reconstruction of the color checker scan in Figure 7 shows that the imager can provide decent scan quality without using optical laboratory. The black boundary between red and green tile is somewhat blurry, which is caused by the imager not being fully connected to the ruler. Vertical discontinuities near the right edge of the green tile are probably caused by the imager not being fully connected to the table during the scan. The faint horizontal stripes originate from unevenness of the slit.

5. Discussion

Our aberration correction method reduced both the spectral smile and tilt to zero within the accuracy of used metrics. The least squares fitting used as the core of both metrics sets the limit of the accuracy that can be achieved. The results should be further verified using well-established calibration methods, which is impossible with the equipment available in our laboratory.

Using the imager completely without laboratory equipment was shown to be possible, but using at least a motorized scanning platform is recommended for better quality scan results. In addition, as imaging in low lighting requires a long scanning time for reasonable signal-to-noise ratio, driving the scan manually can be quite straining. Using spectral calibration lamps with well-separated emission lines for light reference would lessen the manual labor required to find good emission line location estimates.

The software artifact is relatively easy to use, even if the user interface (UI) is a bit of a patchwork combining graphical elements, ipython console, and text file modification. However, using a very light UI makes it easy to add new features to the software and change the existing ones. The software was written in the viewpoint of correcting the spectral smile and lacks many features full imaging software should have; most notably, it lacks wavelength calibration, which is not essential for smile correction.

We made an effort to construct the software code in understandable and maintainable manner for ease of modification for different purposes. One such purpose might be to use this study as a student project for university level physics and information technology courses. For other than scientific audiences, we hope that easy-to-use software makes hyperspectral imaging more accessible and lessens the anxiety to try out hyperspectral imaging for various tasks that it can be useful in. The presented correction method removes the need for accessing an optical laboratory for spectral smile correction, and thus improves the image quality that can be expected from such low-cost imagers. We hope that this helps, in part, in adaptation of hyperspectral imaging as common tool rather than something only selected few with great resources can utilize.

For further research, the same circle fitting idea should be investigated in correcting the keystone aberration as well, as, even if it is a linear effect in itself, combined with the scaling effect that causes the red-end emission lines stretch in spatial dimension, the ends of the emission lines can be considered to form an arc. This is barely visible in Figure 2 as a sideways hourglass shape. We will continue to develop the software in the hope of eventually achieving full open-source imaging software beneficial to a greater group of users and adaptable to a large set of imagers.

6. Conclusions

In this study, we showed that spectral smile aberration of the push-broom hyperspectral imager presented by Sigernes et al. [9] can be corrected without using an optical laboratory. We also corrected an aberration, which we named tilt. The tilt is caused by the misalignment of the entrance slit and is specific for this particular imager design. We demonstrated that the imager can be used in acquisition of hyperspectral image cubes without access to an optical laboratory.

We provided simple open-source imaging software with the ability to correct spectral smile and the tilt and inspect the results. The software contains several built-in examples of spectral smile and tilt aberrations, and different kinds of examples can be generated by varying involved parameters. The program code is made publicly available through Github repository at https://github.com/silmae/desmiler under MIT license.

For those readers who wish to build their own imager, we provided required 3D models of the casing, in both print-ready and modifiable formats through https://osf.io/3uhkb. The repository also contains the dataset (hyperspectral image cube and reference frames for corrections) for reproducing the presented color checker experiment.