Statewide USGS 3DEP Lidar Topographic Differencing Applied to Indiana, USA

Abstract

Differencing multi-temporal topographic data (radar, lidar, or photogrammetrically derived point clouds or digital elevation models—DEMs) measures landscape change, with broad applications for scientific research, hazard management, industry, and urban planning. The United States Geological Survey’s 3D Elevation Program (3DEP) is an ambitious effort to collect light detection and ranging (lidar) topography over the United States’ lower 48 and Interferometric Synthetic Aperture Radar (IfSAR) in Alaska by 2023. The datasets collected through this program present an important opportunity to characterize topography and topographic change at regional and national scales. We present Indiana statewide topographic differencing results produced from the 2011–2013 and 2016–2020 lidar collections. We discuss the insights, challenges, and lessons learned from conducting large-scale differencing. Challenges include: (1) designing and implementing an automated differencing workflow over 94,000 km² of high-resolution topography data, (2) ensuring sufficient computing resources, and (3) managing the analysis and visualization of the multiple terabytes of data. We highlight observations including infrastructure development, vegetation growth, and landscape change driven by agricultural practices, fluvial processes, and natural resource extraction. With 3DEP and the U.S. Interagency Elevation Inventory data, at least 37% of the Contiguous 48 U.S. states are already covered by repeat, openly available, high-resolution topography datasets, making topographic differencing possible.

1. Introduction

Big Data has become a fact of life for a variety of Earth observation activities that require high spatial or temporal resolution and coverage to monitor changes to the Earth’s surface [1,2]. The increasing availability of airborne and spaceborne datasets supports geospatial studies of regional or larger spatial scales and/or high temporal resolution [3,4,5]. These new scales of observation support potentially transformative, synoptic, and detailed assessments of broad-scale trends in hazards, agriculture, and infrastructure assets, as well as the physical processes or anthropogenic activity that drives the phenomena operating over these features [6,7,8,9]. Common challenges to most Earth observation studies include accessing the appropriate datasets, developing processing algorithms, meeting the computational requirements demanded by the problem, and analyzing the results [10,11].

Differencing of multi-temporal topographic data is a powerful technique to measure natural and anthropogenic change at the Earth’s surface [12,13,14,15,16,17,18,19]. A limitation to performing topographic differencing over large spatial extents is often the lack of sufficiently high-quality and resolution (one meter per pixel or better) data over the area of interest. The 3D Elevation Program (3DEP) led by the United States Geological Survey is an ambitious effort to collect lidar and IfSAR data over the entire United States by 2023 [5]. This nationwide topography collection brings immense opportunities and associated challenges of expanding topographic differencing from the site scale to the regional, state, and possibly even the national scale.

We present the first U.S. statewide high-resolution topographic differencing results over the state of Indiana, as well as the derived products and insights. We chose Indiana due to the availability of two statewide datasets, anticipated interesting change observations from agricultural, forest, urban, riparian, and coastal environments, and the previous collaboration between OpenTopography and Indiana for data hosting [20]. We produced digital terrain models (DTM) that consist of bare-earth elevations, as well as digital surface models (DSM) that measure the landscape’s upper elevation. Our derived products include the vertically differenced DTMs and DSMs over the decadal time scale (Figure 1) as well as canopy height models (CHM; Figure 2) for each of the two datasets.

Vertical differencing is the subtraction of raster-based DTMs and DSMs collected on different dates, and the CHM can be produced by differencing a DSM with a DTM derived from the same topographic dataset.

The 94,000 km² area of Indiana is two to five orders of magnitude larger than the typical footprint in topographic differencing studies in the literature [21,22,23,24]. Differencing at this scale brings new Big Data challenges not present for many smaller-scale studies: (1) the workflow must be highly automated, especially when differencing statewide datasets with varying point density and noise characteristics; (2) computational resources must be available to handle multiple terabytes of intermediate and final point cloud and raster products and have adequate processing capability to compute results in an appropriate timeframe; (3) there is also a requirement for appropriate horizontal and vertical coordinate system transformations that place both datasets in the same coordinate system and epoch; (4) as with all lidar topography, the Indiana datasets contain errors from sources such as instrument calibration and positioning and point cloud classification; hence, metadata detail and quality are particularly important. At the state scale, correcting these errors would be prohibitively time-intensive, so these errors must be considered when interpreting the results. Here, we discuss our workflow as applied to Indiana to address the challenges of performing statewide differencing, show the differencing results at the statewide and local scales, and identify several remaining challenges for national scale differencing.

OpenTopography is a U.S. National Science Foundation-funded facility that democratizes the use of high-resolution topography data by co-locating these datasets with processing tools in the cloud that can be accessed by geospatial beginners and experts via a user-friendly web portal [25,26,27]. In recent years, OpenTopography has expanded its processing capabilities to include on-demand, user-initiated, vertical, and 3D differencing [28]. The availability of these tools allows users to perform on-demand vertical and 3D topographic differencing on spatially overlapping datasets hosted by OpenTopography. The success of this tool in the production system required standardizing the differencing workflow, and, in particular, selecting the appropriate software libraries and optimizing the resolution of the differencing products based on the input topography datasets (modern versus legacy) and sometimes hybrid (lidar and photogrammetry) dataset pairs. The ground area of a single on-demand differencing job available for users via OpenTopography is limited to typically ~1–20 km² (~50–350 million points per epoch) due to the required compute capacity, which is sufficient for many applications and spans the majority of overlapping datasets hosted by OpenTopography. The regional, statewide, and national scale of the datasets collected through the U.S. Geological Survey’s 3DEP program motivates larger-scale differencing that requires overcoming the technical challenges associated with processing large and heterogeneous datasets.

2. Background

2.1. Topographic Differencing

We provide a brief overview of topographic differencing methods, including those used by OpenTopography, and refer the reader to [28] for a full discussion. There are multiple ways to compute change between two topographic datasets: vertical or raster-based differencing of DTMs or DSMs works well when the dominant change is vertical and/or the topography is flat. We apply this approach to Indiana, where topographic change is dominantly vertical and driven by changes in vegetation, new infrastructure, river activity, mining, and quarrying. However, lateral shifts in a landscape with relief produce topographically correlated errors when the datasets are no longer co-registered [29]. Three-dimensional differencing via a windowed implementation of the iterative closest point (ICP) algorithm mitigates these errors but works best when the landscape only translates or rotates and does not change shape [30,31]. While this approach has been successfully applied to the 3D displacement fields of earthquakes and creeping landslides [16,32], it is not appropriate for Indiana statewide differencing where the landscape changes often reflect a change in shape (e.g., new buildings or river erosion). There are other differencing techniques, including the cloud-to-cloud tool in CloudCompare and the Multiscale Model to Model Cloud Comparison (M3C2, [33]).

Topographic differencing can be performed with data from a variety of sources, including space-based radar global topography, such as Shuttle Radar Mission Topography (SRTM, [34]), optical satellite photogrammetry [21], airborne lidar, sUAS lidar or photogrammetry [23], or with hybrid datasets [24,35]. The dataset choice often depends on the dataset availability and the resolution required to resolve the signal of interest given the signal’s amplitude and the dataset noise characteristics. For the Indiana statewide differencing effort, we chose lidar data because of the statewide coverage and lidar’s sensitivity to both vegetation and the bare earth.

Because we perform differencing with lidar data, we discuss the errors in this data type only. Due to rapid advances in lidar scanning systems, older datasets typically have larger errors and lower point density than newer datasets. Some earlier airborne lidar point clouds have vertical errors of 5–25 cm, and horizontal errors are often five times larger [36,37,38,39,40]. Lidar dataset quality has generally improved over time, and the later dataset that we use for the differencing meets the U.S. Geological Survey’s Quality Level 2 requirements of 10 cm vertical accuracy. Lidar surveys consist of data acquired along multiple parallel paths or flight-lines. The Indiana datasets were collected with north–south- and east–west-oriented flight directions. Flight-line offsets due to georeferencing errors [41] create linear artifacts in the differencing results aligned with the flight direction [42], and these artifacts are quite prominent in the Indiana differencing results.

2.2. Digital Elevation Model Generation

A digital elevation model (DEM) is a generic term for a raster or grid-based set of elevation values [43]. We use the term DEM generally to describe the methodology for DTMs and DSMs. Methods for DEM generation from point cloud datasets have been reviewed by [44]. To generate the DEMs, we use the triangular irregular network (TIN) algorithm, which is a local neighborhood algorithm that represents topography as a set of neighboring triangles. The DEM’s optimal resolution is based on the point density of the point cloud dataset [45]:

$$Grid resolution=\frac{1}{\sqrt{point density}}$$

(1)

DEMs can be produced at a lower resolution, particularly for differencing datasets with a different point density. Errors can result from producing the DEM at too great a resolution, terrain variability, classification errors, and errors in the original dataset acquisition and point cloud processing [46].

2.3. Vertical Raster-Based Differencing

Vertical raster-based differencing can be performed on raster DTMs and DSMs with identical origins, boundaries, and resolution. The earlier dataset is the ‘compare,’ and the later dataset is the ‘reference’ [19]. The vertical difference is a raster subtraction:

$$Z_{Difference=Z_{reference}-Z_{compare},}$$

(2)

where $Z_{reference}$ and $Z_{compare}$ are the DSMs or DTMs acquired at different dates, and $Z_{Difference}$ is the differenced raster. Because topographic change can be on the same order of magnitude as dataset errors, errors in the surveying, point classification, and metadata (i.e., coordinate reference system (CRS) and epoch) often become pronounced.

2.4. Canopy Height Model

A canopy height model (CHM) or normalized DSM (nDSM) measures the height of vegetation and buildings above the ground. A common and straightforward approach to calculate a CHM is to subtract the DTM from the DSM acquired at the same time:

CHM = DSM – DTM

(3)

We use this approach, which follows naturally from the vertical differencing. However, this approach has errors when the DTM and DSM do not represent the ground surface [47] and the top of the landscape, respectively. There are other approaches to calculate the CHM—for example, the interpolation of the full point cloud dataset to calculate the position of the ground beneath non-ground points [48].

3. Datasets

The 2011–2013 Indiana lidar dataset [20] was collected by IndianaMap and became part of the U.S. Geological Survey’s Lidar Point Cloud project, which preceded their 3DEP project. It was collected county by county from 2011 to 2013 and covers a total of 94,000 km² with an average point density of 1.6 pts/m². The dataset includes 149 × 10⁹ lidar returns, and 44% of these points are classified as ground (i.e., LAS class 2). The dataset was originally delivered in State Plane CRS with units of U.S. feet and with different EPSG codes for the eastern and western portions of the state. We accessed the data in Entwine Point Tiles (EPT) format (https://entwine.io (accessed on 1 July 2021)) via the Amazon Web Services (AWS) Simple Storage Service (S3) Public Dataset bucket (https://registry.opendata.aws/usgs-lidar/ (accessed on 1 July 2021)). The dataset was published by the USGS with a horizontal CRS of WGS84 Web Mercator (EPSG: 3857) and a vertical CRS of the North American Vertical Datum (NAVD) 88 with the GEOID09 datum. The dataset is also available through OpenTopography as well as The National Map, where it is saved on the county scale, as shown in Figure 3A.

The USGS 3DEP Indiana lidar dataset was collected between 2016 and 2020 and is also available through the AWS Public Dataset S3 bucket in EPT format. The dataset is USGS Quality Level 2, implying a 10 cm vertical accuracy and at least 2 pts/m². The horizontal CRS is the same as the 2011–2013 dataset, WGS84 (EPSG: 3857). The vertical CRS is the North American Vertical Datum (NAVD) 88 with the GEOID12 datum and units of meters. The 2016–2020 lidar collection is saved in multiple files, each of which typically spans multiple counties, although the dataset boundaries do not strictly follow county lines, as shown in Figure 3B.

4. Methods

We developed a largely automated workflow to generate DTM and DSM change and CHM maps of Indiana from the classified lidar point cloud datasets. In this section, we outline the main steps of the workflow, as shown in Figure 4. The resulting statewide differencing maps are shown in Figure 1, statewide CHM maps are shown in Figure 2, and Figure 5, Figure 6 and Figure 7 show example areas of notable DTM, DSM, and CHM change, respectively.

We accessed both Indiana lidar point cloud topography datasets from an AWS public dataset S3 bucket using a Point Data Abstraction Library (PDAL) processing pipeline [49]. At the download step, we reprojected the data in the native Web Mercator (EPSG 3857) horizontal CRS to Universal Transverse Mercator (UTM) Zone 16 (WGS84 Ellipsoid), EPSG 32616. Web Mercator (EPSG 3857) is a standard projection for web mapping applications, but is not ideal for most mapping applications because of the high area distortions. The UTM CRS has very little distortion (>1/1000 m/m) within individual zones. We found that downloading the data in 4 km² tiles from the AWS bucket was optimal and avoided rare time-out issues with any larger areas, which would in turn require a complete restart of that process.

For efficient memory management, we divided each county into four parts and downloaded the 4 km² tiles of data that spanned each of the quarters. We used LAStool’s ‘lastile’ [50] to retile the quarter county data into 1 km² tiles. Although LAStool’s ‘lastile’ could handle large datasets well, the command sometimes failed without an error or warning message when applied to full counties.

We generated 1 km² DSMs and DTMs for both acquisitions. Both lidar datasets were classified by the dataset vendor prior to delivery with common point classifications, including unclassified (LAS class 1), ground (class 2), low to high vegetation (classes 3–5), building (class 6), noise (class 7), water (class 9), rail (class 10), road surface (class 11), overlap (class 12), wire (class 13), and transmission tower (class 15). We did not pursue point cloud reclassification because both datasets underwent a quality control review by the State of Indiana and USGS prior to delivery. To extract the ground points, we used LAStools’ ‘las2las’ [50] with the option to keep ground points. We generated one-meter-resolution raster DSMs and DTMs with LAStool’s ‘las2dem’, which uses a triangular irregular network (TIN) approach [51]. We preferred the 1 km² tile area generated with lastile because the gridding step was performed efficiently, and 1 km-square tiles worked well in the visualization steps. We ensured that the datasets had the same bounds even along county boundaries with GDAL’s ‘gdalwarp’ [52].

The geoid epochs of the two datasets differ: the 2011–2013 dataset is in GEOID09 while the 2016–2020 dataset is in GEOID12B. Using GDAL’s ‘gdalwarp’, we performed a vertical CRS transformation to remove the appropriate GEOID from the raster-based heights, resulting in ellipsoidal height elevations.

We generated topographic differencing (Equation (2)) CHM (Equation (3)) maps with GDAL’s ‘gdal_calc’. Areas of no data occur above water in DTMs and immediately outside of county boundaries and have values of −9999.

There are inconsistent dataset names and several partially empty datasets in the AWS S3 public bucket. Manually correcting these dataset names makes the algorithm less automated. We performed no differencing over the western portion of the counties Vigo, Wayne, and Wells where data are missing.

Easy access and visualization of large statewide differencing datasets poses its own set of challenges. Our objective was to make the data browsable and explorable through an easy-to-use web interface that can be viewed concurrently by users with varying geospatial ability and experience. We utilized the open-source application server Geoserver (geoserver.org) to publish a Web Mapping Service (WMS) of the statewide differencing data products, as shown in Figure 8. To leverage Geoserver and its visualization capabilities, we prepared each raster dataset (i.e., two DTMs, two DSMs, DTM change, DSM change, and two CHMs) by applying compression, developing a tiling scheme, and adding overviews. File tiling reduces access time by eliminating the need to read through an entire file before subsetting the necessary pixels. With tiling, access requests can simply find the appropriate tile(s) and efficiently subset an area of interest from those tiles. Overviews allow the user to quickly pan and zoom within the map by setting up a ‘pyramid’ of tiled mosaics. The overviews improve performance, which is critical for the end user experience when visualizing large data volumes. Our WMS of the Indiana differencing products is available here: https://doi.org/10.5069/G9MG7MQV (accessed on 1 July 2021).

5. Results

In this section, we discuss the differencing and CHM results at the statewide and site scales. The visually most dominant signal in the statewide DTM change map (Figure 1a) is the north–south- and east–west-oriented blue and red stripes. These stripes do not represent real topographic change but instead indicate misalignment between data collected along adjacent flight paths. These errors are likely from the 2011–2013 dataset because the errors correlate with the county boundaries that guided the earlier acquisition (Figure 3A). While these errors can be corrected [22,42], the correction benefits from flight trajectory data are very time-intensive and are out of scope for most large-scale studies. Topographically correlated errors vary with slope direction—for example, in Brown County in South–Central Indiana. This apparent lateral shift reflects horizontal georeferencing errors.

Physical changes visible on both statewide differencing maps include mining and quarrying concentrated change near the western and southern state borders. The Wabash and White rivers create south–west-oriented patterns that stretch across the central and southern portions of the state and are produced by fluvial, riparian, and anthropogenic activity. In the south–central portion of the state DSM change map, the color variation along a north–south boundary reflects the seasons of the lidar acquisitions. Purple (west) indicates data acquisition during similar seasons, and red (east) indicates leaf-on to leaf-off conditions.

The statewide CHM maps (Figure 2) indicate landscape type, including agriculture in the center and north, forests such as the Hoosier National Forest in South–Central Indiana and in the northwest by Lake Michigan, and the riparian zones along major rivers. The variability between the two CHM maps in Southeast Indiana reflects a foliage change associated with the seasons of dataset acquisition.

In the remainder of the section, we discuss site-scale examples of DTM change, DSM change, and the CHM. For the DTM change, rivers and riparian zones are very dynamic (Figure 5A–C), as illustrated for the Wabash River in the southwestern-most part of Indiana. Oxbow lakes are developing, and the change map shows a signal that is likely a combination of sediment deposition, erosion along the meander core, as well as flight alignment errors. A shale and limestone quarry located north of Louisville, Indiana, also produces height changes up to a few tens of meters (Figure 5D–F). The change map shows the movement and removal of stone in the active quarry, and, as expected, the overwhelming red color indicates that material is net-removed. The Indiana Dunes National Park and other areas along Lake Michigan’s shoreline (Figure 5G–I) have been impacted by nearby industry, including coal-fired power plants, logging, farming, and suppressed burns, and thus have a high propensity for erosion, especially during recent high-water levels in the lake [53]. The National Parks Service is working to restore portions of the wetland areas, and the differenced DTM shows that the largest changes occurred along the beach where there is no vegetation. Sinkholes near Mitchell, Indiana, also produce significant change [54], as shown in Figure 5J–L. These sinkholes are represented by the circular features in the hillshades. The red and blue circles in the differencing plot show that sinkholes, sometimes ~20 m deep, have formed or have been filled in between the lidar acquisitions.

Here, we discuss DSM change. Figure 6A–C shows a new housing development in an Indianapolis suburb, where the maximum height of the new houses is ~10 m. Change is created by highway construction, where an overpass was replaced with a partial clover intersection and ramps bend approaching the intersection (Figure 6D–F). Figure 6G–I show a crane adjacent to ongoing construction at the Memorial Stadium at Indiana University. The Indiana Harbor and Ship Canal is home to one of the largest steel-making facilities in North America (Figure 6K,L), and the change reflects the movement of material. Agriculture causes landscape change (Figure 6M–O), and the blue squares are new farm buildings. A decrease in elevation along vegetation implies vegetation loss.

The CHM derived from the 2012 lidar acquisition over Fountaintown, Indiana, shows a field of ~2.5 m tall crops and surrounding ~5–8 m tall trees (Figure 7C), reflecting the summer acquisition. The CHM from the March–April 2017 lidar shows the same trees but does not show any crops due to acquisition before the summer (Figure 7F). Comparisons of CHM acquired during different years also show changes in land use and timber harvesting: the CHM from 2013 over La Porte, Indiana, shows trees that are absent in the 2017 CHM, indicating that the trees had been removed between the acquisitions (Figure 7I,L). The 2011 CHM acquired near Houston, Indiana, shows dense forest cover and trees up to ~10 m tall (Figure 7O). The 2017 CHM shows a reduction in vegetation density and tree coverage (Figure 7R). These observations reflect leaf-on (2011) and leaf-off conditions (2017).

6. Discussion

6.1. Large-Scale Topographic Differencing Challenges

We demonstrated the successful statewide topographic differencing of meter-scale resolution lidar topography. As with most Big Data Earth observation projects, we faced multiple challenges in accessing, processing, and analyzing the data. These include:

(1) The statewide DTMs, DSMs, DTM and DSM differencing, and CHM products together generated ~4 terabytes of final raster products. The intermediate products were of a similar scale because we saved the point clouds as compressed laz and not as larger las products. This required ~10 TB of local disk space that had to be readily available throughout the project.

(2) The intermediate point cloud products required careful management to ensure efficient allocation of the fixed memory available for the processing. Even popular software packages such as LAStools can have difficulty working with larger county-scale datasets. This fact emphasizes the Big Data challenges that are common to the processing and analysis of large-scale and high-resolution datasets.

(3) Once the processing workflow was established and associated technical challenges overcome, we estimated that the statewide differencing workflow would have taken approximately one year of compute time if it were running on a commodity server or local workstation. This aspect emphasizes the need for dedicated high-performance computing resources with large memory and multicore CPUs for faster processing and more reasonable timelines for analysis. Although relatively infrequent, the processing did stop, sometimes requiring a manual restart, which benefited from a modular workflow design that enabled relatively efficient failure recovery.

(4) The nature of the project management for the lidar acquisitions meant that the data were acquired over a range of dates, in irregular spatial patterns, and with inconsistent data characteristics. A dedicated data acquisition effort for state scale differencing could be designed to provide consistent data quality and organization that would more efficiently yield the differencing products discussed here.

(5) The largest source of noise is the offset between flight-lines in the lidar data, which appear as east–west- and north–south-oriented swaths most noticeable in the differenced DTM (Figure 1a). This type of error can be corrected using additional flight trajectory information [42]. However, this process is at best very time-intensive and requires flight information that is rarely delivered with the topography data, and thus would likely be challenging to locate at the state scale. Georeferencing errors also produce topographically correlated noise. We made the explicit choice to not correct these errors because the correction is out of scope for this project and the majority of the change signals are larger than the noise amplitude. This choice was also an opportunity to explore the overall quality of these statewide datasets and to better understand their suitability for multi-temporal analyses without additional and labor-intensive adjustments. In the case of Indiana, these flight-line errors appear to be largely present in the earlier lidar dataset. This observation is consistent with the general observation that older and legacy lidar datasets have larger errors and lower resolution. Thanks to improvements in lidar technology, as well as the data quality control and review processes, these types of errors are likely to be less prevalent in future dataset collections. We expect that this topographic differencing exercise presents a common set of challenges that would face any large-scale analysis of 3DEP or similar regional to national scale topography datasets.

These noise sources impose a limitation to the utility of the Indiana statewide datasets. We expect that our results are useful for detecting change and conducting scientific analyses at regional or statewide scales of signals whose amplitude exceeds that of the noise with spatially heterogeneous characteristics. Studies on the site scale or of lower-amplitude signals are likely to benefit from focused approaches to mitigate noise, such as the correction of the flight alignment, georeferencing, or point classification errors [22,42,44,50].

(6) Visualizing Big Data results is challenging, especially for those without an extensive geospatial background. Our Web Mapping Service visualization (Figure 8) demonstrates an approach to sharing Big Data studies, which is important for results such as these to be well communicated and possibly used to inform policy [2]. We expect that the web-based, easy-to-access visualization of our results will lead to a broader utilization of our work, especially in education [55] and by state and local policy makers.

6.2. Future Topographic Differencing

Thanks to the USGS 3DEP and other large-scale lidar mapping initiatives, there are now significant portions of the United States and the world covered by repeat, high-resolution, airborne and spaceborne topography datasets collected over the past decade. As shown in Figure 9, our analysis of USGS 3DEP and the U.S. Interagency Elevation Inventory (ISIEI) datasets shows that at least ~37% of the lower 48 of the United States is covered by open, repeat lidar. Topographic differencing is likely possible in additional areas by differencing 3DEP with other open data datasets, by incorporating datasets that are nominally public yet difficult to access, or by performing hybrid differencing with satellite topography datasets [21] that typically have a larger footprint than most lidar acquisitions.

While this growing body of open data presents a great opportunity for performing topographic differencing analyses, the large-scale integration of types and vintages of datasets brings new challenges. First, the repeat topographic data will likely have different characteristics, including resolution, noise, and sensitivity to vegetation. These differences are especially likely to be exacerbated for hybrid datasets. An approach for differencing modern and legacy lidar includes setting the resolution of the differencing product based on the lower-resolution dataset [28], as we do here. Second, the lack of a single, centralized repository for topographic data makes finding, accessing, and processing topography datasets originally acquired by a range of organizations challenging. Initiatives such as the 3DEP, the USIEI, and OpenTopography are improving this situation, but significant funding and political challenges exist in centralizing access to high resolution topography, particularly for legacy datasets collected and housed by numerous governmental entities. Third, performing differencing on increasingly large scales is a classic Big Data challenge with the requisite requirements for compute and storage. High-performance computing (HPC) and cloud infrastructure provide a pathway, but fundamental updates are necessary to data archive, access, and processing systems to fully enable big topographic data analysis. In addition, technical issues such as quality and detailed metadata, coordinate reference systems, consistency of point cloud classifications, and improvement of legacy dataset alignment can present serious obstacles if not dealt with correctly.

6.3. Other Large-Scale Processing

Big Data analyses of remote sensing datasets are becoming increasingly common and necessary to efficiently analyze large-scale geospatial data. For example, Constantini et al. [11] developed a time-series of surface deformation rates from Interferometric Synthetic Aperture Radar (InSAR) over all of Italy and discussed the importance of national databases of terrain movement and the associated challenges related to workflow development, efficient use of HPC resources, noise correction, and ground truthing. Bhangale et al. [10] implemented HPC resources to detect oil spills from multi-temporal Landsat datasets and characterized the computational gains from graphics processing units (GPUs).

Insights into Big Data processing from studies such as those discussed above and our work can provide support for a variety of activities in Earth observation. For example, these and similar methods could be applied to the mapping of geologic hazards including landslides [56], sinkholes [57], flood zones [7], tectonic faults [58,59], and coastal erosion [9]. They can also be applied to understand anthropogenic interaction with the environment—for example, biomass or biomass change [6,47,60,61], agriculture [62], and infrastructure development [8]. Increasing the complexity from the quantification of change to semantic variation over time (types and drivers of vertical variation in topography or canopy height structure) can come from additional analysis using machine learning tools [63].

7. Conclusions

In this paper, we presented the first statewide topographic differencing analysis using publicly available lidar high-resolution topography from the State of Indiana (2011–2013 dataset) and U.S. Geological Survey’s 3DEP (2016–2020 dataset). We discussed a range of Big Data challenges that we overcame, including accessing the datasets, developing an automated differencing workflow, allocating sufficient compute resources, and managing the data products and their visualization. The statewide differencing results show flight-line alignment errors that are likely from the earlier lidar acquisition and highlight the challenges that these errors pose in large-scale analyses. We documented landscape changes from fluvial activity, vegetation changes, and infrastructure development. As the 3DEP topography collection grows and increasingly more areas are covered with single and repeat topography, large-scale analyses such as ours will be critical to characterizing the Earth’s landscape and its changes over time.