Next Article in Journal
Local Knowledge and Professional Background Have a Minimal Impact on Volunteer Citizen Science Performance in a Land-Cover Classification Task
Next Article in Special Issue
Capability Assessment and Performance Metrics for the Titan Multispectral Mapping Lidar
Previous Article in Journal
Analysis of MABEL Bathymetry in Keweenaw Bay and Implications for ICESat-2 ATLAS
Previous Article in Special Issue
Voxel-Based Spatial Filtering Method for Canopy Height Retrieval from Airborne Single-Photon Lidar
Article Menu
Issue 9 (September) cover image

Export Article

Remote Sens. 2016, 8(9), 767;

Evaluation of Single Photon and Geiger Mode Lidar for the 3D Elevation Program
U.S. Geological Survey, Reston, VA 20192, USA
Woolpert, Arlington, VA 22206, USA
Dewberry Consultants LLC, Fairfax, VA 22031, USA
U.S. Geological Survey, Lakewood, CO 80225, USA
Author to whom correspondence should be addressed.
Academic Editors: Jie Shan, Juha Hyyppä, Lars T. Waser and Prasad S. Thenkabail
Received: 28 June 2016 / Accepted: 8 September 2016 / Published: 19 September 2016


Data acquired by Harris Corporation’s (Melbourne, FL, USA) Geiger-mode IntelliEarth™ sensor and Sigma Space Corporation’s (Lanham-Seabrook, MD, USA) Single Photon HRQLS sensor were evaluated and compared to accepted 3D Elevation Program (3DEP) data and survey ground control to assess the suitability of these new technologies for the 3DEP. While not able to collect data currently to meet USGS lidar base specification, this is partially due to the fact that the specification was written for linear-mode systems specifically. With little effort on part of the manufacturers of the new lidar systems and the USGS Lidar specifications team, data from these systems could soon serve the 3DEP program and its users. Many of the shortcomings noted in this study have been reported to have been corrected or improved upon in the next generation sensors.
lidar; Geiger-mode; single photon

1. Introduction

While not new in terms of technology, the commercialization of two new types of lidar instruments, Geiger-mode Lidar (GML) and Single Photon Lidar (SPL) offer the promise of utilizing high-altitude lidar collections for large area mapping across the United States for the 3D Elevation Program (3DEP). The 3D Elevation Program is accelerating the rate of three-dimensional (3D) elevation data collection in response to a call for action to address a wide range of urgent needs nationwide [1]. With constantly changing lidar technology, the 3DEP needs to keep up with emerging trends and instruments that could fulfill the goals of the program at a reduced cost to the taxpayer. Currently, the 3DEP relies on the mature discrete multiple-return lidar systems for data collection; for the purposes of this paper, we will refer to these as “linear-mode lidar” (LML) systems.
Both GML and SPL utilize focal plane array detectors, where the returned pulse is recorded using an array of receivers instead of single receiver as is the case in LML systems. The transmitted laser pulses for both GML and SPL are low energy. These low energy pulses are detected by receivers that are sensitive to individual photons; thereby enabling the added advantage of higher flying altitudes.
Over the past 15 years, the Massachusetts Institute of Technology, Lincoln Laboratory (MIT/LL), Defense Advanced Research Projects Agency (DARPA) and private industry have been developing airborne lidar systems based on arrays of Geiger-mode Avalanche Photodiode (GmAPD) detectors capable of detecting a single photon [2,3]. The extreme sensitivity of GmAPD detectors allows operation of lidar sensors from very high altitudes and acquisition efficiency rates in excess of 1000 km2/h. Up until now the primary emphasis of this technology has been limited to defense applications, despite the significant benefits of applying this technology to non-military uses such as mapping, monitoring critical infrastructure and disaster relief. The first commercial Geiger-mode lidar system, the IntelliEarth™ system by Harris Corporation, began advertising its capability in early 2015.
Photon-counting lidar systems were in development shortly after some of the first commercially available linear-mode systems began operating [4]. The first successful photon-counting airborne laser altimeter was demonstrated in 2001 under NASA’s Instrument Incubator Program (IIP) [5]. This instrument was flown from altitudes up to 6700 m above ground level (AGL), operated at a wavelength of 532 nm, and imaged terrestrial and shallow water bathymetry to depths of a few meters over the Atlantic Ocean and Assawoman Bay off the Virginia coast. This SPL system modeled the solar noise background and developed simple algorithms, based on post-detection Poisson filtering (PDPF) to optimally extract the weak altimeter signal from a high noise background during daytime operations [6]. The theoretical results are reinforced by data from an airborne microlaser altimeter, developed under NASA’s Instrument Incubator Program. This instrument was designed primarily to produce, over a mission life of three years, a globally contiguous map of the Martian surface, with 5 m horizontal resolution and decimeter vertical accuracy, from an altitude of 300 km. This type of SPL instrument is planned to replace the Geoscience Laser Altimeter System (GLAS) on the ICESAT-II mission, slated for launch in 2017 [7,8,9]. Currently, the only commercially available SPL system is the High-Resolution Quantum Lidar System (HRQLS) by Sigma Space Corporation.
While there are numerous potential benefits of these newly commercialized systems to collect data for 3DEP, such as increased flying height and higher point density, there are documented issues of concern for these types of systems as well. Williams Jr. [10] demonstrated through physics-based Monte Carlo simulations of avalanche photodiode (APD) lidar receivers that under typical operating scenarios, GmAPD with array-based receivers may often be ineffective in detecting partially occluded targets, such as bare earth under vegetation. Due to their ability to detect only one photon per laser pulse, the target detection efficiency of GmAPD receivers was shown to respond nonlinearly to the specific conditions including range, laser power, detector efficiency, and target occlusion, which caused the GmAPD target detection capabilities to vary unpredictably over standard mission conditions. In the detection of partially occluded targets, Williams Jr. [10] found that GmAPD lidar receivers performed optimally within only a narrow operating window of range, detector efficiency, and laser power; outside this window performance degraded sharply. He concluded that the inability of the GmAPD to detect target signal present at the receiver’s aperture may lead to a loss of operational capability, may have undesired implications for the equivalent optical aperture, laser power, and/or system complexity, and may incur other costs that can affect operational efficacy.
Past history of using GmAPD detectors suggest an issue specifically in their ability to penetrate foliage [11]. Most APDs operating in Geiger-mode report only one range measurement per transmitted laser pulse. If a GmAPD makes a foliage range measurement, it cannot make a range measurement to a target concealed by the foliage. When too much laser energy is received, the vast majority of range measurements are from the foliage and only a small percentage are from the target.
Given the ability of these detectors to measure a single photon, minimizing solar background is an important consideration during the design of the system. There is a general concern by traditional LML users that these data are too noisy for commercial use, especially during daylight operations. The choice of system wavelength also has a significant impact on the solar background level and overall system efficiency. Other effective methods of reducing solar background is minimizing the system aperture, installation of a narrow bandpass filter in the receive path, reducing the detector instantaneous field of view, minimizing the range gate duration, or by simply operating at night. By designing and operating the system such that solar background is minimized significantly increases data quality while increasing the operating range of the sensor and reducing or eliminating the need for noise filtering.
Lower energy also translates into fewer photons reaching the receiver, and as a result these systems do not have full waveform digitization capability. Current LML technology relies on a flux of photons (500 to 1000 photons) to record the returned signal. This energy is much larger than the one needed for the GML and SPL technologies. Such large fluxes of energy make it possible to digitize the full waveform and to produce an intensity image with reasonable radiometric resolution to represent a black and white image at the lidar wavelength. This is not the case with the GML and SPL systems. To overcome such problem while capitalizing on the high density of its point cloud, the GML system in particular produce what is called a “reflectivity image” or “relative reflectivity image”, by computing the ratio of the numbers of incident photons to the returned photons.
Currently, there are only two companies with instruments that are commercially available that claim they can collect data that meets 3DEP’s requirements in terms of data quality and utility. These are the SPL High-Resolution Quantum Lidar System (HRQLS) system by Sigma Space Corporation, and the Harris IntelliEarth™ GML lidar. While both instruments are unique as described above, both claim that they can provide data that meets the requirements for the 3D Elevation Program. This paper will evaluate these particular sensors as they relate to the 3D Elevation Program, and are not necessarily reflective of the potential of both GML and SPL in general.

1.1. Sigma Space Single Photon Technology

Sigma Space utilized its single photon sensor, HRQLS, to collect the data for the study [12]. HRQLS has a 10 × 10 array of 100 beamlets, generated by a passive diffractive optical element (DOE) in front of a 25 kHz laser. The returned 100 beamlets are imaged into 10 × 10 micro-channel plate photomultiplier detectors with low jitter and very fast recovery time. The fast 1.6 nanosecond/24 centimeter recovery time/range of the combined detector and timing receiver allow the systems to view multiple photon events per pixel per pulse, making them capable of daylight operation. They also can penetrate semi-porous volumetric scatterers, such as tree canopies, turbid water bodies, thin clouds or fog. In addition, the 10 × 10 beamlet array, combined with a proper choice of aircraft velocity and altitude, allows the generation of contiguous few decimeter resolution maps on a single overflight while operating at altitudes up to 9000 m AGL and at speeds up to 250 knots yielding an aerial coverage of 1340 km2/h. Due to the finite speed of light and the high ground speed of the conical scanner, initially co-aligned transmit and received field of views (FOVs) become increasingly displaced at the higher altitudes. As a result, the scanner optical design and scan speed must compensate for the displacement over a wide range of potential operating altitudes. As an added benefit, the surface returns and receiver range gates are automatically paired with the correct laser start pulse even when multiple laser pulses are simultaneously in flight. Table 1 lists the technical specifications for the HRQLS sensor used by Sigma Space Corporation to collect the data.

1.2. Harris IntelliEarth™ Technology

The Harris IntelliEarth™ Geospatial Solutions Geiger-mode lidar sensor is the first commercial airborne lidar system that takes advantage of the single photon capabilities of the Geiger-mode avalanche photodiode [3]. Harris’ system uses an array of 32 × 128 detectors. This concept of pulse splitting results in a much higher point density as compared to the current linear lidar, where the pulse may have few returns in vegetated areas as a best case scenario.
The primary components of the system include:
  • 128 × 32 InP/InGaAsP Geiger-mode camera capable of readout rates in excess of 100 kHz.
  • Compact Nd:YAG diode pumped solid state laser.
  • 270 mm Holographic Optical Element (HOE) scanner with 15° scan half angle capable of rotational speeds in excess of 2000 rotations per minute. (US Patent US 2015/0029571 A1 29 January 2015).
  • Real time transmit line of sight adjustment which compensates for scanner motion during pulse round trip time.
  • High efficiency narrow bandpass filter that reduces solar background noise.
  • Transmit beam shaping optics that optimizes illumination pattern on the ground.
  • Nadir looking Ritchey—Chrétien telescope for collection of returned light.
  • Inertial navigation system including an inertial measurement unit.
  • High speed flash detector for precision laser pulse timing.
  • Data acquisition electronics.
  • Sensor controller.
The Harris Geiger-mode lidar sensor uses a conical Palmer scan pattern produced by a direct drive, hub driven HOE scanner. HOE scanners have been used in the past [13] but this particular implementation has several advantages over previous designs. When flown with 50 percent overlap, the scan pattern provides four looks from four different directions. With a scan half angle of 15°, the sides as well as roofs of structures are sampled, which significantly increases the potential for interpretability of the point-cloud data.
There have been very few studies comparing the performance of these new sensors to LML or ground survey measurements [12,14], and this study is the first of its kind to evaluate the performance of LML to GML and SPL systems as they relate to collecting data that is adequate for the 3D Elevation Program. We evaluated both commercial instruments: the SPL HRQLS system by Sigma Space Corporation, and the Harris IntelliEarth™ GML. Comparing data collected by these instruments over an area in Northern Connecticut with both existing and new LML and good survey ground measurements helped determine the current capability of these instruments to meet 3DEP’s requirements.

2. Data Sets

We designed a 500 mi2 project area for this evaluation study that overlapped the northern third of Connecticut Sandy QL2 Lidar collected for the 3DEP, which was flown in April/May 2014 by Dewberry. This coverage included all desired land cover and terrain variability, portions of Hartford including the main airport, and rivers and lakes to test hydroflattening in select areas. Three sub-areas were selected within the main project area—Urban, Mixed Use, and Forested (Figure 1). For all three sub-areas, two independent teams (Woolpert and Dewberry) processed the data acquired by Sigma Space and Harris using their own internal proprietary methods to create final deliverables that were intended to meet the United States Geological Survey (USGS) v1.2 specifications [15], including classified LAS v1.4 point cloud data, 1-m hydroflattened DEMs, intensity images (where applicable) and associated reports and metadata.
Data processing and analysis was done independently by Dewberry and Woolpert. Woolpert acquired leaf-on data in a small (~77 km2) area in the Forested sub-area for this study around the same time as the HRQLS and IntelliEarth™ data acquisition. Woolpert acquired lidar data according to the USGS QL2 specifications over an area densely covered by vegetation. The purpose of this data acquisition was to have QL2 data during leaf-on condition close in time to the IntelliEarth™ and HRQLS data acquisition. Woolpert used a Leica ALS70 500 kHz Multiple Pulses in Air lidar sensor system.
Sigma Space and Harris Corporation acquired data in the project area during leaf-on conditions in August/September 2015. Calibrated, unclassified LAS data and supporting acquisition reports were delivered for this evaluation study. Harris acquired IntelliEarth™ data in the project area during nighttime conditions at 7950 m above ground; as requested, they provided data acquired in daytime conditions along two overlapping swaths in the Urban sub-area albeit acquired at a lower altitude (2293 m above ground) (Figure 2). Sigma Space acquired HRQLS data in nighttime and daytime conditions at 2293 m above ground, and delivered one set of LAS v1.2 data for the entire project area. In the Forested sub-area, there were some areas acquired in the early morning where fog was present. As a result, these areas were not considered for the evaluation. Table 2 lists all the data sets used in this evaluation study; the abbreviated names for each dataset are used throughout this paper.
Harris Corporation acquired IntelliEarth™ data on 2 September and 16 September 2015 during leaf-on conditions. The primary evaluation data were collected using the Harris IntelliEarth™ sensor 1 from an altitude of 7950 m above ground. This dataset is referred to as GMHarLO15_26k (Table 2). The flights were conducted at night and with an overlap of approximately 55%. It should be noted, however, that while the IntelliEarth™ sensor collects data in swaths, it processes all swaths together for a final solution, and as a result Harris states it is not possible to identify the swaths within the final point cloud. In addition to the high altitude collection, Harris also collected a low altitude dataset during the day for a small overlapping area. These data were collected at 2293 m above ground in the Urban sub-area (Figure 2) and referred to as GMHarLO15_7.5kDT (Table 2). These data were tested for improved foliage penetration and differences between day and night collections. The unclassified data were received on 1 November 2015 for this evaluation study. The following data/reports were delivered by Harris for this study:
  • Control Points and Report for Control
  • Intensity Imagery
  • Acquisition and Processing Report
  • SBET files
  • Raw Tiled LAS v1.4
The IntelliEarth™ data were processed to a LAS point-cloud and calibrated using a photogrammetric bundle adjustment method.
In addition to the leaf-on data acquired in September 2015, Harris also acquired IntelliEarth™ data in leaf-off conditions at 7950 m on 6 December 2015 within a small sample forested area using their new IntelliEarth™ Geiger-mode sensor 2. The first sensor (serial number 1), had a slight optical blur which defocuses the returning light into roughly a 3 × 3 pixel area. This blur made it more difficult for the system to match returning photons from the multiple looks in to a final XYZ point solution. It also reduced the spatial resolving power for data reflecting off of natural objects, buildings, and other man-made structures. Sensor 2 used a new Holographic Optical Element (HOE) that focused the light correctly onto 1 pixel and thereby significantly improving canopy penetration. Although this leaf-off dataset, referred to as GMHarLF15_26k (Table 2) was not part of the original scope, we evaluated the sample data for canopy penetration.

Data Classification and Editing

The data were processed by using Global Mapper (v16.2.7), GeoCue (v14.1.21.2), TerraScan (v15.031), TerraModeler (v15.007), and Microstation (v8i v08.02.04) software utilizing independent proprietary methods by both Dewberry and Woolpert. The acquired 3D laser point clouds, in LAS binary format, were imported and tiled according to the project tile grid. Once tiled, the points were classified using proprietary routines. These routines classify any obvious low outliers in the dataset to class 7 and high outliers in the dataset to class 18. After points that could adversely affect the ability to derive a ground model were removed from class 1, the ground layer was extracted from this remaining point cloud. The ground extraction process built an iterative surface model. For this evaluation study, it was important to maintain consistency between the ground models in order to ensure the comparisons were not skewed by incorrect classification. As a result, the classified ground points from the Dewberry 2014 lidar collection were used as a starting point in the macro and classified points that were within 20 cm to an initial ground class. This step minimized the amount of manual editing required. It was noted that for the IntelliEarth™ dataset, the density of points actually penetrating through the foliage was minimal and a traditional macro would have difficulty discerning ground and non-ground points with such low density. By using the 2014 data, we were able to retain valid ground points within the forested area with reduced manual effort.
Each tile was then imported into proprietary software and surface models were created to examine the ground classification. Analysts visually reviewed the ground surface models and corrected errors in the ground classification such as vegetation, buildings, bridges, and noise caused by random photon events that were present after the automated classification. 3D visualization techniques were employed to view the point-cloud data at multiple angles and in profile/transect mode to ensure that non-ground points were removed from the ground classification. After the ground classification corrections were completed, the dataset was processed through a water classification routine that utilized breaklines compiled to automatically classify hydro features. The water classification routines selected ground points within the breakline polygons and automatically classified them as class 9 (water). During these water classification routines, points that were within 1 × nominal point spacing (NPS) or less of the hydrographic features were moved to class 10 (ignored ground) due to breakline proximity. The lidar tiles were classified to the following classification schema:
  • Class 1 = Unclassified, used for all other features that do not fit into the Classes 2, 7, 9, or 10 including vegetation, buildings, etc.
  • Class 2 = Bare − Earth Ground
  • Class 7 = Low Noise
  • Class 9 = Water, points located within collected breaklines
  • Class 10 = Ignored Ground due to breakline proximity
After manual classification, the LAS tiles were peer reviewed and then underwent a final QA/QC. After the final QA/QC and corrections, the LAS files were then converted from LAS v1.2 to LAS v1.4. All headers, appropriate point data records, and variable length records, including spatial reference information, were updated and verified using proprietary Dewberry and Woolpert tools.
All three sets of lidar data (IntelliEarth™, HRQLS and linear mode) derived products were processed using Woolpert’s and Dewberry’s individual workflows and production procedures and processes.
Once all hydro-flattened data was imported, surveyed ground control data was imported and calculated for positional accuracy assessments. As a quality control measure, Woolpert and Dewberry have developed routines to generate accuracy statistical reports by comparisons against the points and the DEMs using surveyed checkpoints of higher accuracy.

3. Methods

In total 83 ground checkpoints were acquired: Thirty-four were acquired in 2014 during the leaf-off survey and 49 additional checkpoints were acquired in 2015 to support this study (Figure 3).
Data from both the HRQLS and IntelliEarth™ systems were checked to determine if they complied with requirements in the USGS Lidar Base Specification v1.2 [15]. Assessments included testing attributes contained the required values, determining if the point density met USGS base specification QL2 levels (at least two points per meter squared), if relative accuracies were sufficient, and if absolute accuracies were within specification. Other qualitative tests included an assessment of day versus night data quality, quality of reflectance images (for IntelliEarth™ only), and the amount of noise points inherent in the data. The Non-vegetative Vertical Accuracy (NVA) and Vegetated Vertical Accuracy (VVA) of the data were evaluated according to the guidelines and the recommendations of the “ASPRS Positional Accuracy Standards for Digital Geospatial Data” using all the available checkpoints [16].
As the primary derivative product for the 3DEP is a bare earth DEM, an assessment of the differences between an accepted QL2 bare earth DEM versus derived bare earth DEMs were also performed independently by both Dewberry and Woolpert, and then results of those determinations were checked by USGS to make sure that both assessments and processing of bare earth DEMs were truly independent.

4. Results and Discussion

4.1. Attributes Required to Meet USGS Lidar Base Specification 1.2

Data from both the HRQLS and IntelliEarth™ systems were checked to determine if they complied with requirements in the USGS Lidar Base Specification v1.2 [15]. While compliant in many categories, there were several requirements that neither dataset met (Table 3).

4.2. Data Density

Both IntelliEarth™ and HRQLS datasets had Aggregate Nominal Pulse Spacings (ANPS) that greatly exceeded USGS QL1 requirements. The IntelliEarth™ data acquired at 7540 m above ground produced 25 points per square meter (ppsm), and the HRQLS data acquired at 2293 m above ground produced 23 ppsm. Although both these systems produced data analogous to a “single return only” linear mode sensor, the data density was much higher at these high altitudes than typical linear mode sensors currently being used for the 3DEP program. Current state-of-the-art linear mode sensors will produce data at 2–4 ppsm at a flying altitude of 2100 m above ground or lower.

4.3. Data Smoothness/Relative Accuracy

Both sensors used different processing and calibration procedures that are based on detecting individual photons using an array-based detector to determine the range to a target. LML sensors determine the range to the target by detecting the return signal from the entire transmitted pulse. As a result, the typically used swath-based relative accuracy method was not used for this evaluation study. Instead, the entire dataset was treated as a single swath, and the relative accuracy QL1/QL2 requirement of 6 cm was verified in a sample area. The IntelliEarth™ data easily met the 6 cm requirements in flat and gently sloping terrain and there was very little variability within the test areas (except over bright reflective targets summarized below). The HRQLS data displayed greater variability over flat terrain with significant low noise points. Since there were no timestamps in the LAS data, there was no way to differentiate each swath after the data were merged to a tile grid. Furthermore, while the single swath accuracy requirement was more stringent than the accuracy assessment between swaths, this method was chosen to ensure that the data could easily be compared with the IntelliEarth™ evaluation. The smooth surface repeatability test require the data to meet 6 cm relative accuracy for QL1 or QL2 data. This test was performed by computing the maximum difference within a 0.5 m cell. Because the validation is only relevant on smooth surfaces with little to no slope, a 0.5 m cell size was sufficient to determine the repeatability on those surfaces. This included open terrain, roads, and flat roof tops. Based on the overall evaluation, the single photon data tended to contain slightly more noise within the swath than would be typically seen in a linear mode system. On average the differences were between 5 and 7 cm within areas where the slope was less than 10 degrees. The HRQLS data required more manual editing to remove outliers and some of this variability in the data. There was significant variability over highly reflective flat targets with the HRQLS data as summarized below. The variability in the HRQLS data exceeded the relative accuracy of 6 cm in these locations.
The IntelliEarth™ and HRQLS data were examined to determine effects of range walk over highly reflective surfaces such as roadway markings and runway paint stripes (Figure 4). Range walk occurs when highly reflective targets appear higher than the surrounding less reflective targets. In traditional LML sensors, range is calculated by measuring the time of flight of laser pulses reflected from a target. The timing of the return is typically calculated by first converting the returning photon pulse into a proportional current and then precisely measuring some position along the rise of the current pulse, for example by simple power threshold, finding the Full Wave at Half Maximum of the strongest reflected pulses, or through various cross correlation techniques. When techniques such as level thresholding are used to determine range, reflections from extremely bright surfaces will cause current pulses rise above that threshold much sooner than for typical surfaces in the scene, leading to significant timing difference. As a result, those very bright surfaces appear to rise or “walk” above the surrounding surface area. Range walk is a very common source of range errors in linear mode sensors, and sensor manufacturers and production groups have various methods for adjusting their data.
The HRQLS and IntelliEarth™ data along Runway 33 at Bradley International Airport in Windsor Locks, CT in the “Mixed Use” sub-area were analyzed for range errors (Figure 4). The SPSigLO15_7.5k data showed a 50–60 cm offset across the paint stripes and lots of low noise points at the two ends and in the middle of the runway. There were also high noise points in the HRQLS data along the transect. The GMHarLO15_26k data showed some variation (~15 cm) due to range walk, but was able to define the shape of the runway along the transect. The LML dataset had a lower point density but clearly showed the shape of the runway without any range walk effects.

4.4. Absolute Accuracy Assessment

Two types of accuracy assessments were performed. First, we performed an assessment against point clouds. Secondly, we performed an assessment against derived bare earth DEMs (Table 4). For the point cloud data assessment, Vegetated Vertical Accuracy (VVA) were assessed only in areas that were processed to final classified point-cloud and where a valid lidar point was within 20 m of the checkpoint (Test 1). The accuracy assessment made against DEMs included interpolated areas as well (Test 2).
Both sensors met the absolute accuracy requirements for Non-vegetated Vertical Accuracy (NVA) and VVA for QL1/QL2 data in both tests. For IntelliEarth™ data, the NVA based on 32 checkpoints was 0.17 m and VVA based on 15 checkpoints was 0.256 m using the point clouds only. For HRQLS data, the NVA based on 31 checkpoints was 0.172 m and VVA based on 17 checkpoints was 0.174 m. The minimum requirements for QL1/QL2 accuracy are 0.196 m for NVA and 0.296 m for VVA. Using Test 2 shows that both HRQLS and IntelliEarth™ data are meeting the NVA of 19.6 cm at the 95 percent confidence level as required. However, both HRQLS and IntelliEarth™ datasets failed to meet the VVA requirements of 29.4 cm. The poor foliage penetration especially in the IntelliEarth™ data, and therefore the poor capability in delineating the ground under dense trees during leaf-on conditions compromised the quality of filtered DEM in vegetated areas.

4.5. Canopy Penetration

To understand canopy penetration, we analyzed the data that were collected by the IntelliEarth™ Sensor #1 at 7950 m above ground in leaf-on conditions (GMHar15LO15_26k) and the data acquired by the HRQLS sensor at 2293 m above ground in leaf-on conditions (SPSig15LO15_7.5k). These two datasets were compared with the linear-mode data acquired by Dewberry in 2014 leaf-off conditions (LMDewLF14) as well as the linear-mode data acquired by Woolpert in 2015 in leaf-on conditions (LMWptLO15) around the same time as the IntelliEarth™ and HRQLS collects (Figure 5).
Canopy penetration in the IntelliEarth™ data acquired using sensor 1 was very poor. The system was able to mostly capture only the top of the tree canopy in a vegetated terrain, with little or no returns from the canopy structure/understory or the ground. Although the acquisition was during leaf-on conditions, which can greatly reduce canopy penetration compared to a leaf-off collect, the canopy penetration was much poorer than the HRQLS and linear-mode collect during the same leaf-on season over the same area. Canopy penetration in the IntelliEarth™ data acquired using sensor 2 based on the sample data acquired in December 2015 and provided later was very good. The dataset produced by sensor 2 had an ANPS of over 107 ppsm that generated a bare earth (ground) density of 14 ppsm. The sensor 2 was able to generate returns from the canopy structure as well as the ground. It was unknown if canopy penetration improvement was due to the new sensor or the fact that data were collected during leaf-off conditions. It was hypothesized that it was both. Canopy penetration in the data acquired using the HRQLS sensor was qualitatively adequate and resembled the type of penetration expected from a LML dataset in leaf-on conditions; however, the HRQLS sensor did not pass the VVA assessment of bare earth DEMs, suggesting an influence from noisier points and larger than desired voids under vegetation.

4.6. Derivative Products

4.6.1. Evaluation of Intensity Imagery

Intensity values were not collected with the HRQLS system. The lack of intensity was not a significant issue for the processing of data to ground/non-ground, however it did present challenges in collection of breaklines for hydro features as discussed in the section below.
The IntelliEarth™ system stored intensity data as a 16-bit unsigned value and was created from multiple aggregated measurements. These measurements were scaled over the 16-bit range via the number of returns observed for a given reflectance. In the IntelliEarth™ system intensity was referred to as “Reflectance” and exhibited similar grayscale characteristics as linear mode systems. We reviewed the reflectance data acquired by IntelliEarth™ to determine its usability and applicability. This evaluation focused on using the images for the extraction of feature data such as hydrographic features, roads, and buildings. In each case, the reflectance image produced by the IntelliEarth™ sensor offered the same type of capabilities as an intensity image and could be used as such. We did not identify any areas where the reflectance imagery would be less useful than a standard LML intensity image. Figure 6 shows an example of the reflectance imagery produced from the GMHar15LO_26k data.
Similar to linear mode intensity images, the IntelliEarth™ images have variability in the intensity values from the same or similar targets within each data set. The linear mode intensity images typically have variability across different swaths. For IntelliEarth™, the reflectance images tended to have a gridded appearance, which is based on the aggregation of points during processing of the data.

4.6.2. Evaluation of Breakline Development

The lack of intensity imagery from the HRQLS system primarily impacted the collection of breaklines required to develop products that meet USGS/NGP Base Specifications v1.2. Typically, the breaklines are derived using a combination of intensity imagery and the elevation data. The intensity imagery allows the users to reliably determine if features are water bodies or not. Without the use of the intensity imagery the analyst acquiring the breaklines will have to rely more heavily on the elevation values and ancillary imagery sources that were not collected coincidentally with the lidar. The result is that areas of marsh may lean more heavily toward being collected as water or temporal changes between ancillary imagery and the lidar collection could result in incorrect classification of water. For this exercise, a few sample features were collected using only the data available from the HRQLS collect. Since the HRQLS system uses the green wavelength (532 nm) as its laser source, the data may include bathymetry in water bodies. The presence of bathymetric data can cause greater error in collecting breaklines for hydro features. Some of the differences may also be temporal in nature and represent the variation in the water levels between collections.

4.6.3. Evaluation of Day/Night Collection

We performed an evaluation of the differences between the night and day collections provided by IntelliEarth™ (GMHarLO15_26k and GMHarLO15_7.5kDT). Harris collected an overlapping area within the Urban sub-area for this evaluation. The daytime dataset (GMHarLO15_7.5kDT) were acquired at 2293 m above ground compared to the 7950 m above ground of the nighttime collection. Based on a review of the overlapping area acquired in the night and day, we did not identify any major differences in canopy penetration or excessive noise due to solar radiance. The point density was higher on the daytime collection, but that difference is likely due to the difference in flying height and processing and is not a result of additional solar noise in the data. There was very little difference in the canopy structure or penetration of the data to the ground. There also did not appear to be any additional noise points in the daytime dataset.

4.7. Comparing Independent Evaluations

Both Woolpert and Dewberry independently bare earth processed the IntelliEarth™ and HRQLS data, and created bare earth DEMs from these points using internal proprietary methods. These DEMs were then differenced from the accepted QL2 Sandy 1m DEMs to determine differences between accepted DEMs and these test DEMs. We then tested the correlations between four difference grids to determine if they were truly created independently. Correlations between HRQLS DEM differences and IntelliEarth™ DEM differences were r = 0.74 and 0.55, respectively (Figure 7). These low correlations suggest that the derived bare earth DEMs developed by each group were created independently.

5. Conclusions

Elevation data are essential to a broad range of applications, including flood risk management, natural resource management, forest resources management, agriculture and precision farming, wildlife and habitat management, national security, recreation, and many others. The 3DEP is a national partnership program managed by the USGS to acquire high-resolution elevation data for the United States. The initiative is backed by a comprehensive assessment of requirements conducted by Dewberry in 2011 and is in the early stages of implementation. As part of the 3DEP program, we evaluated two new technologies that have the potential for rapid data collection rates with improved data densities and accuracy. These high-altitude airborne lidar sensors can acquire high-density data at much higher collection rates, thereby providing the possibility of QL1 data of 8 ppsm or higher at considerably lower costs.
Data acquired by Harris Corporation’s IntelliEarth™ sensor and Sigma Space Corporation’s HRQLS sensor were evaluated to assess the suitability of these new technologies for the 3D Elevation Program. While not able to collect data currently to meet USGS lidar base specification, some of this has to do with the fact that the specification was written for linear-mode systems specifically, and the next major version of the USGS lidar base specification will become more flexible to allow these instruments to be included. With little effort on part of the manufacturers of the new lidar systems and the USGS lidar specifications team, data from these systems could soon collect data that meets 3DEP requirements and serve the 3DEP program and its users. Many of the shortcomings noted in this study have been reported to have been corrected or improved upon by both companies in their next generation sensors.


We would like to thank both Harris Corporation and Sigma Space Corporation for flying and providing the point cloud data for this evaluation. We would also like to thank staff at Dewberry and Woolpert for flying linear mode lidar, as well as processing and helping assess these data.

Author Contributions

Q. Abdullah and A. Nayegandhi were responsible for processing and providing independent assessments. J. Stoker was responsible for validating results and checking for independence. J. Winehouse was responsible for directing the contract for both Woolpert and Dewberry.

Conflicts of Interest

The authors declare no conflict of interest. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.


  1. Sugarbaker, L.J.; Constance, E.W.; Heidemann, H.K.; Jason, A.L.; Lukas, V.; Saghy, D.L.; Stoker, J.M. The 3D Elevation Program initiative—A call for action. U.S. Geol. Surv. Circ. 2014. [Google Scholar] [CrossRef]
  2. Aull, B.F.; Loomis, A.H.; Young, D.J.; Stern, A.; Felton, B.J.; Daniels, P.J.; Landers, D.J.; Retherford, L.; Rathman, D.D.; Heinrichs, R.M.; et al. Three-dimensional imaging with arrays of geiger-mode avalanche photodiodes. Proc. SPIE 2004, 5353, 105–116. [Google Scholar]
  3. Clifton, W.E.; Steele, B.; Nelson, G.; Truscott, A.; Itzler, M.; Entwistle, M. Medium altitude airborne geiger-mode mapping LIDAR system. Proc. SPIE 2015, 9465. [Google Scholar] [CrossRef]
  4. Priedhorsky, W.C.; Smith, R.C.; Ho, C. Laser ranging and mapping with a photon-counting detector. Appl. Opt. 1996, 35, 441–452. [Google Scholar] [CrossRef] [PubMed]
  5. Degnan, J.; Wells, D.; Machan, R.; Leventhal, E. Second generation airborne 3D imaging lidars based on photon counting. Proc. SPIE 2007, 6771. [Google Scholar] [CrossRef]
  6. Degnan, J.J. Photon-counting multikilohertz microlaser altimeters for airborne and spaceborne topographic measurements. J. Geodyn. 2002, 34, 503–549. [Google Scholar] [CrossRef]
  7. Moussavi, M.S.; Abdalati, W.; Scambos, T.; Neuenschwander, A. Applicability of an automatic surface detection approach to micro-pulse photon-counting lidar altimetry data: Implications for canopy height retrieval from future ICESat-2 data. Int. J. Remote Sens. 2014, 35, 5263–5279. [Google Scholar] [CrossRef]
  8. Awadallah, M.; Abbott, A.L.; Thomas, V.; Wynne, R.H.; Nelson, R. Estimating Forest Canopy Height and Biophysical Parameters using Photon-counting Laser Altimetry. In Proceedings of the 13th International Conference on LiDAR Applcation for Assessing Forest Ecosystems (SilviLaser 2013), Beijing, China, 9–11 October 2013; pp. 137–144.
  9. Abdalati, W.; Zwally, H.J.; Bindschadler, R.; Csatho, B.; Farrell, S.L.; Fricker, H.A.; Harding, D.; Kwok, R.; Lefsky, M.; Markus, T.; et al. The ICESat-2 Laser Altimetry Mission. Proc. IEEE 2010, 98, 735–751. [Google Scholar] [CrossRef]
  10. Williams, G.M., Jr. Limitations of geiger-mode arrays for flash ladar applications. Proc. SPIE 2010, 7684. [Google Scholar] [CrossRef]
  11. Johnson, S.E. Foliage penetration optimization for geiger-mode avalanche photodiode lidar. Proc. SPIE 2013, 8731. [Google Scholar] [CrossRef]
  12. Li, Q.; Dengan, J.; Barrett, T.; Shan, J. First Evaluation on Single Photon-Sensitive Lidar Data. Photogramm. Eng. Remote Sens. 2016, 82, 495–503. [Google Scholar] [CrossRef]
  13. Schwemmer, G.K.; Wilkerson, T.D.; Guerra, D.V. Compact scanning lidar systems using holographic optics. Proc. SPIE 1998. [Google Scholar] [CrossRef]
  14. Kim, A.M.; Runyon, S.C.; Olsen, R.C. Comparison of full-waveform, single-photon sensitive, and discrete analog LIDAR data. Proc. SPIE 2015, 9465. [Google Scholar] [CrossRef]
  15. Heidemann, H.K. Lidar base specification (ver. 1.2, November 2014). In U.S. Geological Survey Techniques and Methods; Book 11, Chapter B4; USGS: Reston, VA, USA, 2014. [Google Scholar]
  16. ASPRS Positional Accuracy Standards for Digital Geospatial Data. Available online: (accessed on 28 June 2016).
Figure 1. Areas of interest used for processing and assessing bare earth.
Figure 1. Areas of interest used for processing and assessing bare earth.
Remotesensing 08 00767 g001
Figure 2. Location of leaf-on linear mode collection (LMWptLO15), IntelliEarth™ daytime collection (GMHarLO15_7.5kDT), and IntelliEarth™ sensor 2 leaf-off collection (GMHarLF15_26k).
Figure 2. Location of leaf-on linear mode collection (LMWptLO15), IntelliEarth™ daytime collection (GMHarLO15_7.5kDT), and IntelliEarth™ sensor 2 leaf-off collection (GMHarLF15_26k).
Remotesensing 08 00767 g002
Figure 3. Location of checkpoints used for Vertical Accuracy Assessments. Plus signs are NVA checkpoints. Triangles are VVA checkpoints.
Figure 3. Location of checkpoints used for Vertical Accuracy Assessments. Plus signs are NVA checkpoints. Triangles are VVA checkpoints.
Remotesensing 08 00767 g003
Figure 4. Example of range walk.
Figure 4. Example of range walk.
Remotesensing 08 00767 g004
Figure 5. Cross section used for comparisons, overlaid on imagery (image top). Sample profiles of HRQLS, leaf on (top profile); linear-mode lidar, leaf-on (middle-top profile); linear-mode lidar, leaf-off; (middle-bottom profile); and IntelliEarth™ lidar, leaf on (bottom profile).
Figure 5. Cross section used for comparisons, overlaid on imagery (image top). Sample profiles of HRQLS, leaf on (top profile); linear-mode lidar, leaf-on (middle-top profile); linear-mode lidar, leaf-off; (middle-bottom profile); and IntelliEarth™ lidar, leaf on (bottom profile).
Remotesensing 08 00767 g005
Figure 6. Example of an intensity image from: a linear-mode system (top); and the IntelliEarth™ system (bottom).
Figure 6. Example of an intensity image from: a linear-mode system (top); and the IntelliEarth™ system (bottom).
Remotesensing 08 00767 g006
Figure 7. Correlations between Dewberry’s and Woolpert’s IntelliEarth™ DEM differences (top); and HRQLS DEM differences (bottom) from accepted 3DEP lidar. r = 0.74 and 0.55, respectively.
Figure 7. Correlations between Dewberry’s and Woolpert’s IntelliEarth™ DEM differences (top); and HRQLS DEM differences (bottom) from accepted 3DEP lidar. r = 0.74 and 0.55, respectively.
Remotesensing 08 00767 g007
Table 1. Technical specifications for the HRQLS sensor.
Table 1. Technical specifications for the HRQLS sensor.
Number of beams100
Wavelength532 nm
Laser Repetition Rate25 kHz
Laser Pulsewidth700 ps
Laser Output Power1.5 W
Pixels/s2.5 Million
Eye SafetyEye safe by FAA standards
Multiple Return CapabilityYes
Pixel Recovery Time1.6 ns
RMS Range precision±5 cm
Scan PatternsLinear, conical
Scan Width0 to 40 degrees (selectable)
Operational Altitude Range6500–10,000 ft
Swath vs. AGL (at maximum scan angle)1.3 to 2 km
Area Coverage versus AGL (at maximum scan angle and 200 knots)400 to 640 km2/h single pass
Mean Point Density12 to 8 per square meter, single pass with 15% reflectivity
Size19 W × 25 D × 33 H inches
Weight50 lbs
Prime Power555 W
Table 2. List of datasets and their attributes that were used in this study.
Table 2. List of datasets and their attributes that were used in this study.
AbbreviationData TypeAcquired ByType of CollectDate/Year CollectedCollection Altitude (AGL)
LMDewLF14Linear ModeDewberryLeaf-OffApril/May 2014917 m
LMWptLO15Linear ModeWoolpertLeaf-OnSeptember 20152140 m
GMHarLO15_26kGeiger ModeHarrisLeaf-OnSeptember 20157950 m
GMHarLO15_7.5kDTGeiger ModeHarrisLeaf-On, Day TimeSeptember 20152293 m
SPSigLO15_7.5kSingle PhotonSigma SpaceLeaf-OnAugust 20152293 m
GMHarLF15_26kGeiger ModeHarrisLeaf-OffDecember 20157950 m
Table 3. Compliance to USGS Lidar Base Spec v1.2.
Table 3. Compliance to USGS Lidar Base Spec v1.2.
LAS Version 1.4LAS v1.4LAS v1.2Both data sets LAS v1.4 compatible
Point Data FormatCompliantCompliant
Coordinate Reference SystemCompliantCompliant
Global Encoder bitCompliantCompliant
Time StampCompliantNot CompliantIntelliEarth—unique but not based on acquired swaths.
HRQLS—none provided.
System IDCompliantCompliant
Multiple ReturnsNot CompliantNot CompliantBoth systems do not produce multiple returns.
Point Source IDNot CompliantCompliantIntelliEarth—No flight swaths.
IntensityReflectanceNot CompliantIntelliEarth—similar to linear-mode
HRQLS—no intensity data
Overlap and withheldNot CompliantCompliantIntelliEarth—No flight swaths.
Scan AngleNot CompliantNot CompliantSpec not compatible with these sensors.
XYZ CoordinatesCompliantCompliant
Table 4. Absolute accuracy tests using points and DEMs (* leaf on data).
Table 4. Absolute accuracy tests using points and DEMs (* leaf on data).
Dataset/SensorTest #1—PointsTest #2—DEM
HRQLS (2293 m AGL) *17.2 cm17.4 cm14.1 cm40.6 cm
IntelliEarth (7950 m AGL) *17.0 cm25.6 cm15.2 cm92.0 cm
Existing, accepted 3DEP QL2 data (917 m AGL)12.3 cm19.8 cm14.6 cm25.0 cm
Remote Sens. EISSN 2072-4292 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top