Unmanned Aerial Vehicles ( UAV ) Photogrammetry in the Conservation of Historic Places : Carleton Immersive Media Studio Case Studies

The increasing commercialization of unmanned aerial vehicles (UAVs) has opened the possibility of performing low-cost aerial image acquisition for the documentation of cultural heritage sites through UAV photogrammetry. This paper presents two case studies that illustrate the use of the DJI Phantom 4 normal UAV for aerial image acquisition, and the results that can be achieved using those images. A general workflow procedure of oblique image capturing and data processing of large data sets has been illustrated in the Prince of Wales Fort case study to create photogrammetric models and to generate orthophotos for condition assessment applications. The second case study provides insight on the possibility of using UAVs for post-disaster documentation when the accessibility and the availability of high cost equipment is of major concern. The results that were obtained from UAV photogrammetry of Nyatapola Temple and Bhairabnath Temple in Taumadhi Square in Nepal, which were damaged by the 2015 Gorkha earthquake, are presented and discussed.


Introduction
The unmanned aerial vehicle (UAV) has quite literally taken the cultural heritage field to new heights.UAVs are changing the way that we look at the world.They are becoming more of a common tool and are not shrouded in secrecy as they once were.As UAVs can be equipped with cameras, they are also able to capture aerial photographs and video; a prospect that makes them a very appealing choice as a digital documentation tool.
According to the Canadian Aviation Regulations, the term 'unmanned aerial vehicle' is defined as "a power-driven aircraft, other than a model aircraft, that is designed to fly without a human operator on board" [1].UAVs themselves are known by many different names, including remotely piloted vehicles (RPV), drones, and remotely piloted aircraft system (RPAS) [2,3].The International Civil Aviation Organization (ICAO) uses the term RPAS as their international standard [4].For the context of this review, the term UAV will be used throughout.
The practice of using UAVs for photogrammetric applications can be traced back to the 1979 and 1980 tests by Przybilla and Wester-Ebbinghaus on the Schwebebahn (monorail) Wuppertal [5][6][7].
The advent of the specific term 'UAV photogrammetry' can be attributed to the work of Henri Eisenbeiss, whose dissertation in 2009 provided new insight into the use of UAVs for aerial photogrammetric applications [5].
The main goal of this research paper is to illustrate how UAV photogrammetry is being used for the documentation of cultural heritage sites.The manner in which this goal is accomplished is through an international case study analysis of two differing sites: Prince of Wales Fort (Churchill, MB, Canada) and Bhaktapur Durbar Square (Kathmandu, Nepal).Both of these sites have been previously documented, however, this is the first time that UAVs have been used.Specifically, the way in which the images have been acquired, how the data has been processed, and the deliverables that are generated from the data will be discussed.

UAV Photogrammetry and Cultural Heritage Documentation
The 2009 dissertation by Eisenbeiss can be considered as one of the first major work in regards to using UAVs for cultural heritage purposes [5].Since that time, the popularity of using UAVs for documentation through UAV photogrammetry has only increased due to technological advances and the ease in which UAVs can be purchased.UAV photogrammetry is described as "a photogrammetric measurement platform, which operates remotely controlled, semi-autonomously, or autonomously, without a pilot sitting in the vehicle [5]."The most common UAV platform used for UAV photogrammetric applications in relation to historical building documentation is rotary wing aircraft.This is because of the usability as compared to fixed wing platforms, and the ability to capture images at a close range.Literature that examines rotary wing UAVs for documentation applications is shown in [2,[8][9][10][11][12][13][14].
UAV photogrammetry provides advantages over terrestrial photogrammetry because of the ability to capture views that are not possible from the ground [15].In particular, oblique images are becoming more important in the UAV photogrammetric process.Through the advent of UAV systems with built in cameras, oblique images are easy to capture because it only involves adjusting the camera to the desired angle while using the remote control.Or, if the camera is not built in, it can be adjusted before each flight.Using oblique images is shown in [8,[15][16][17].Figure 1 is an adapted workflow from [3] that serves as a basis for UAV image acquisition and processing.It is important to note that a separate category, "Safety Procedures", was added to mission planning in the adapted figure.This is because safety should always be at the forefront of any UAV mission.By understanding the safety procedures and protocols, it helps to minimize the possibility that the UAV will cause damage to both humans on the ground (i.e., the drone becomes out of control and injures someone) and other aircraft in the sky.Additional information, such as completing a Flight Log after each flight, is also included in this category.The basic flight logbook contains pertinent information about the flight (i.e., flight pattern, time of flight, battery life, wind speed, temperature, etc.).

Gaps Identified
One gap in relation to UAV photogrammetry that was identified in literature had to do with the need for expanded education in regards to UAVs.In the article, Integrating UAVs into Geomatics Curriculum [18], it is argued that as UAVs are becoming more available for commercial purchase, there is a lack of formal training courses to be able to use them correctly.Al-Tahir [18] proposes three alternatives, which have been expanded upon in relation to their inclusion at Carleton University:

•
A design project-This can be similar to the Capstone that Carleton University does during the fourth year of engineering undergraduate degrees.

•
A short module within a larger course-This can be a few lectures in specific courses about cultural heritage documentation, such as Historic Site Recording and/or Directed Reading at Carleton University.

•
A complete advanced course in UAV photogrammetry including detailed photogrammetric processing and hands on training.

Case Study 1: Prince of Wales Fort
Prince of Wales Fort is the first case study to illustrate the use of UAV Photogrammetry.Prince of Wales Fort is located in Churchill, Manitoba, at the mouth of the Churchill River.Flights were conducted over the course of three days: 9-11 August 2016.The overall goals of this case study were:

•
to test the capabilities of the commercially available DJI Phantom 4 normal UAV for the creation of a photogrammetric model using oblique images; • to test how well large data sets of oblique images would be processed and oriented, and the quality of results they would give; and, • to see if the quality of the generated orthophotos from the model are sufficient for condition assessment applications and crack monitoring/identification.

Site History and Significance
The site itself was constructed over a forty year period from 1731-1771 by the Hudson's Bay Company in order to protect their northern trade routes [19].It was damaged by the French in 1782, and it was left dormant and neglected for 150 years until conservation initiatives were put forth by the Historic Sites and Monuments Board of Canada.The conservation efforts of the 1930's and 50's focused on the stabilization of the exterior rampart walls, as some sections were in danger of collapse.As the site was open to the public, the safety of the site was an imminent priority as to not put any visitors at risk of injury.Though the initial interventions did indeed help to ensure the continued longevity of the site, the repair techniques used negatively impacted aspects of the site's authenticity.Specifically, cement was poured behind the walls as a consolidation technique, which erased any signs of 18th century craftsmanship in the specific areas that were commissioned for repair [19].The site has been greatly affected in recent years due to climate change.Visible cracks and masonry deterioration are visible on site, and monitoring programs have been put in place in order to keep a detailed record of any changes.An overview image of the Fort is shown in Figure 2.

Site Restrictions
When working with UAVs in Canada, there are a series of protocols to follow.These are mandated by Transport Canada, the federal agency responsible for regulating transportation related activities in Canada.The most important regulation is the Special Flight Operations Certificate (SFOC), which is governed by the enabling legislation, the Canadian Aeronautics Act [20].The SFOC is the document allowing the use of UAVs for any application other than flying it for fun and pleasure [21].It helps to ensure that the operators will use their UAVs safely and contains information, such as how, when, and where the UAV will be used, maximum flight altitude, the operating area, and safety risk plans [22].An SFOC application was required so that this project could take place.
The most important factor on site was that the unpredictable weather dictated the working patterns.If the wind speed was greater than 32 km/h, then the flights could not be conducted.Additionally, if there was any sign of rain, flying could not commence.While on site, the wind speeds were approximately 20 km/h, and there were only a few instances of rain that shut down the operation for a few hours.
The UAV flights were conducted underneath an airplane flight path, which presented another restriction when trying to capture overall site context images at high altitudes.Any time that airplanes were spotted, the UAV was brought back to the operator as an extra safety precaution.Additionally, the safety mode was turned on, which meant that the maximum altitude of the UAV was 30 m AGL (above ground level).This would help to ensure that the UAV would be within the operator's line of sight at all times.

Equipment Used
The DJI Phantom 4 normal was used to acquire all of the aerial images.It is equipped with a built in camera and gimbal system, with a 1/2.3"sensorsize and a 3.61 mm focal length.In terms of other hardware used, there was one backup battery, and an IPhone 5s, which was used to control and monitor the UAV through the use of the DJI Go app.There was a generator on site, which was used to charge the UAV batteries and phone battery after each flight.

Data Collection
Ten rounds of flights were conducted when on site, all with varying purposes.The flights were done through the manual piloting by Alex Federman and all of the images captured were also done manually.Shown in Table 1 is a summary of the flight tests.

Data Processing
The general workflow followed during the data processing stage is shown in Figure 3.Each of the sections will be described in greater detail in the following subheadings.

File Organization
The photos and video footage was manually inspected and named in Adobe Bridge based on the date and flight number for posterity record keeping purposes.Additionally, the metadata from the UAV was imported to the online inventory airdata.com[23].Airdata.comprovides a way of quantifying the metadata by displaying attributes from the conducted flights, such as the route that the UAV took to capture the images, the altitude that each of the images was taken at, and notifications about the battery life.Figure 4 illustrates an example flight path pattern in airdata.com,while Figure 5 displays the photograph locations from the UAV.This information can be exported as a kml file to be used in a Google Maps application for the easy sharing of flight data.

Image Orientation and Dense Cloud Processing
Image orientation and dense cloud processing was done using Agisoft PhotoScan.A total of 843 oblique images were originally captured.These images were manually inspected and checked in order to ensure that they were sharp and that they would help in the alignment of the photogrammetric model.Those that were not were deleted.This brought the total number of kept images to 767.
Because of the very large field of view of the Phantom 4, it resulted in a large amount of overlap between the images, which helped with the image orientation process.The drawback to this large field of view is that images focusing on the exterior walls also captured some parts of the inner courtyard in the background of the photographs.Therefore, photos were masked to reduce this residual noise, in the sparse and dense cloud models.
161 Ground Control Points (GCPs) were added to the PhotoScan model before image alignment in order to help with the orienting images of the interior courtyard and the exterior walls, as well as providing scale to the model.These GCPs were surveyed by John Gregg and Shawn Kretz for their own terrestrial photogrammetric survey of the Fort.These GCPs are permanent targets on the walls of the Fort, as seen in Figure 6.The sparse cloud was processed on a high setting for accuracy and a generic pair presentation.0 key points (indefinite) and 5000 tie points were also used.The dense cloud processing, it was processed on a high quality setting with aggressive depth filtering.
In terms of the accuracy error of the model, 20 check points were selected to help to validate the results.One checkpoint was chosen for each exterior wall surface.The RMS error of the model checkpoints calculated by the PhotoScan software was 7.87 mm.As the intended drawing scale for this case study was 1:50, Andrews et al., cites that the accuracy for a 1:50 drawing set is 9 mm (from Section 4.5.2 in [24]).Therefore, this value is greater than the 7.87 mm error that was obtained by PhotoScan, thus validating the results.Table 2 displays the checkpoints RMS error from PhotoScan, while Table 3 displays the control points RMS error.

Mesh Generation
A mesh was necessary in the data processing because it is a required step before generating orthophotos in PhotoScan.Generating a mesh model from PhotoScan was unsuccessful as the software kept crashing due to the constraints from the computer hardware.32 GB of RAM was not enough to process the mesh.An alternative strategy was used where the images were imported into ContextCapture.The benefit of using ContextCapture is that for mesh and dense cloud processing, the software is RAM dependent.For the case of this model, 16 GB was the targeted RAM usage (half of the available RAM on the computer).The camera parameters from the PhotoScan model were imported into ContextCapture as an Omega Phi Kappa text file to aid in the Aerotriangulation step for image orientation.Importing the text file meant that the GCPs did not have to be reselected again in ContextCapture, thereby saving processing time.
The final dense cloud and the mesh models consisted of sixteen separate tiles being created instead of one whole model, like what is generated in PhotoScan.The resulting mesh and point clouds were automatically cleaned of excess noise points from the sky, which was a benefit that did not occur in PhotoScan.This was also done with no masking involved, as that function does not exist in ContextCapture.

Mesh Merging
CloudCompare was used to merge the sixteen resulting tiles from ContextCapture into one whole model.The tiles had to be merged because only one tile can be imported back into PhotoScan at a time per individual chunk.When being imported back into PhotoScan, the mesh file was aligned in the same coordinate system as the dense cloud model because the same camera parameters were used in the ContextCapture model generation.

Orthophoto Generation and Point Cloud Exporting
A series of orthophotos were created for individual wall sections of interest.The importance of adding the mesh back into the PhotoScan file was because of the usability of PhotoScan for orthophoto generation.The orthophotos that were created had an approximate GSD of 5 mm per pixel.This value is comparable to the theoretical GSD of the raw images that were used, because when calculations were made using an average flight distance of 12 metres away from the Fort walls, the resulting GSD was 5 mm or less.Though that result is higher than the 3 mm value for a 1:50 scale, as noted by Historic England's Metric Survey Specifications for Cultural Heritage, from visual inspection of the image, the mortar joints and the individual stones were all clearly visible to be traced for an elevation drawing [24].Figure 7 shows the generated orthophoto.

Discussion of Deliverables
There were two primary deliverables that were generated based off of the recording data.These include measured drawings that were based off of orthophotos that can be used for condition assessment purposes, and the establishment of a multi-temporal planar displacement analysis for specific sections of the fort that were captured with close-range UAV photogrammetry.

AutoCAD Measured Drawings
From the orthophotos that were generated from PhotoScan, a measured drawing application has been done in AutoCAD in order to draw out the individual stones that were to be used for condition assessment purposes the next time that work is done at the site.As there was difficulty scaling the exported GeoTIFF orthomosaic file from PhotoScan, the point cloud generated for the site was used to scale the orthoimage in AutoCAD.The reason as to why the drawing was not drawn directly from the point cloud is because the orthoimage was easier to visualize and provided a more detailed view of the stone locations.The decided scale was 1:50, as it is a standard size for elevation drawings and all of the stones and mortar joints can be seen.

CloudCompare Deformation Analysis
Due to the effects of climate change, a multi-temporal deformation analysis was deemed as a beneficial deliverable to make the best use of the data collected.The purpose behind this type of structural health monitoring strategy is that information collected from different time periods can be compared based on the same datum.This can be done using the open source software CloudCompare.
The information that is used for this analysis comes from point clouds generated in PhotoScan by using images from Flight 10, which were photographs that were taken at an angle perpendicular to the wall surface.This type of UAV photography methodology was done in four separate locations around the site.The datum used for the analysis is a best-fit plane, as the wall is rectangular in shape.A screenshot of the plane is shown in green in Figure 8.This same plane can be used to compare the point clouds from future interventions because it was generated in the same reference system as the point cloud.The plane can be imported into a new CloudCompare project along with the current epoch's point cloud.After the plane is imported, a cloud to mesh algorithm is run in order to determine the relative distance between the point cloud and the reference primitive.Figure 9 shows the resulting relative distances for the wall surface based on the first epoch.The values measurement unit for the C2M distances is meters.The areas in red and orange in Figure 9 are indicative of the outward bulging of the wall by 0.1 to 0.16 m, as that direction was deemed to be the positive axis.Areas in yellow and light green represent a planar surface, while dark green and blue represent areas where the wall turns back in on itself.This type of analysis can be beneficial in developing wall threshold limits in regards to intervention strategies, i.e., if the wall bulge reaches a certain value, then conservation work is required immediately.After a second data set is compiled, then it can be compared to the first one using the above methodology because it would be in the same reference coordinate system.

Case Study 2: Taumadhi Square
During the Gorkha earthquake of 2015, many of the monuments were heavily damaged in the Kathmandu Valley World Heritage Property [25,26].The restoration phase of the collapsed monuments has been very slow due to the lack of proper documentation.The purpose of this project was to record two monuments in Taumadhi Square: Nyatapola Temple and Bhairabnath Temple.The scope and objective of the case study included:

•
using UAV photogrammetry to document the Nyatapola Temple and Bhairabnath Temple; • to understand and implement quick recording techniques for condition assessments of the structures after the disaster; and, • to record the damages for posterity record keeping using UAV photographs.

Site History and Significance
The Kathmandu Valley World Heritage property is inscribed as seven monument zones, with Bhaktapur Durbar Square being one of them.Taumadhi Square, which is known as Ta:Maari in local language, is one of the most important squares among the four that compose the Bhaktapur Durbar Square monument zone.Taumadhi Square consists of two important monuments: Nyatapola Temple and Bhairabnath Temple.An aerial orthophoto of Taumadhi Square is shown in Figure 10.
Construction of Nyatapola Temple was dated back to 1701/1702 AD and it is one of the tallest tiered temple in existence within the Kathmandu Valley in Nepal.Bhairabnath Temple was constructed in 1686AD [25,27].These unique tiered temples were constructed with traditional materials including massive kiln burnt brick masonry in mud mortar for structural walls, and timber elements for the floors and roof.The roofs were then covered with small overlapping terracotta tiles with gilded brass ornamentation.The windows, doorways, and roof struts have rich decorative carvings.
The importance of these monuments increased due to their seismic resilient characteristics, especially in the major 2015 Gorkha Earthquake, which registered a magnitude of 7.6.Both structures survived with partial damages in the upper tier only [25].In addition, the Square is an important open space that has been used for centuries in different traditional festivals.The Bisket Jatra, which is one of the important festivals celebrated in the Nepalese New Year, starts with pulling the chariot of Bhairabnath Temple from the Taumadhi Square.In this sense, the square has high heritage significance in both tangible and intangible aspects.
In this case study, results that were obtained from the UAV photogrammetry in the process of recording and documenting the cultural heritage structures of Taumadhi Square is presented.

Equipment Used
In this case study, both terrestrial and UAV photogrammetry were performed.For terrestrial photogrammetry, the Nikon D800 DSLR Camera was used to capture the images from the ground level, which consisted of the plinth and ground floor.Ground-based image acquisition was limited in capturing information for higher elevations due to the steep angled field of view of the camera.To solve this problem and therefore capture information at higher elevations, the DJI Phantom 4 UAV was used.

Data Collection
Due to highly dense residential area surrounding both monuments, UAV photography was quite challenging.Both of the structures also consisted of multiple sloped roofs that extended approximately two metres away from the wall surface.Because of these roofs, the manual flying of the drone was conducted versus that of an automatic flight plan.The drawback to this method is that the UAV was not able to capture a clear image of the areas beneath the roof due to the shading on wall caused by the extended roofs, as well as higher variation in exposure condition.In spite of these limitations, the photos that were obtained were very useful to assess the condition of exterior surface of the structure.Figure 11 shows where images were taken when documenting the Bhairabnath Temple.

Data Processing
Agisoft PhotoScan was used for the post-processing of data to obtain the dense point clouds.The data was processed separately for the Nyatapola Temple and Bhairabnath Temple, and were then merged using the GCPs that were surveyed on site.Figure 10 4 and 5.The error is much higher for the Bhairabnath Temple than the Nyatapola Temple because there were less control points used when scaling the model.

Discussion of Deliverables
Primarily, the data obtained from the UAV photogrammetry was used to assess the structural condition of the exterior surfaces in the higher levels of the Nyatapola and Bhairabnath Temples.The benefit of using a UAV for condition assessment applications was that it minimized the overall costs on site.This was because a normal condition assessment would involve using scaffolding in order to reach the higher levels for proper inspection.This scaffolding also comes with a large cost, which might not be in the budget of every project.
Cracks in the exterior surface of the structural wall were recorded and the extent of damage in the upper level was recorded using the visual assessment of the pictures taken from UAV.The analysis was more qualitative than quantitative to come to conclusions about the damages.The result that was obtained from the condition assessment is presented in Figure 12.The georeferenced point clouds were used to generate the high quality orthophotos, with a GSD of 5 mm per pixel.These orthophotos were then used to generate the measured drawings of the structure, with an example being shown in Figure 13.

Discussion
This endeavour allowed for the to gain valuable experience with both acquiring as well as processing data collected through UAV imagery.For both case studies, in terms of equipment, three batteries and two chargers would be the optimal number for future UAV photogrammetry projects.This would ensure that 60+ min of continuous flight time could occur, helping to speed up the data capturing process.This was because there were periods of time on site where no work could be done because the batteries were still charging, thereby slowing down the project.
In regards to Prince of Wales Fort, a new methodology should be tried the next time that UAVs are used for photogrammetric applications, as the 5 mm per pixel GSD, though acceptable for a 1:50 elevation drawing set, was too high for a crack monitoring program.Therefore, the new strategy employed should be similar to that of the Flight 10 test, whereby the camera is oriented at 90 degrees towards the object like in terrestrial photogrammetry.The pilot should then perform a vertical linear pattern over the face of each wall, changing the camera positioning to an oblique angle at the top of the walls in order to capture the top details.
It is important to note that more extensive GSD planning would also be beneficial in order to ensure that the UAV is close enough to the object of interest to obtain the resulting resolution.One factor that should be planned in this type of strategy is the time cost involved, as flying closer to the object means more photographs, resulting in a longer flight time.One application that should be used the next time that flights like this are conducted (i.e., in open areas where there are no obstacles to the UAV) is the use of flight-path planning applications to ensure that the rate of capture is even, versus that of manual image acquisition, whereby the photographs are not at an even distance away from the object.
In regards to data processing for large data sets with a limited amount of computer processing power, a combination approach of using PhotoScan for initial image orientation and dense cloud processing, and then ContextCapture for mesh generation, proved to be a successful workflow based on the quality of the orthophotos produced for the Prince of Wales Fort case study.
Overall, UAV photogrammetry proved to be advantageous because the surrounding landscape, whether it be terrain in the Prince of Wales Fort case study, or an urban setting, like in Nepal, could also be captured and properly aligned into the model.This helps to show the site in its surroundings environment, which provides a better context to the overall models because it displays the connection between these historic sites and their environments.

Conclusions
It can be concluded that UAVs offer another platform for image acquisition for photogrammetric purposes.This paper helped to contribute to the knowledge base of using UAVs for documentation applications by examining their role in the image acquisition process.It presented two case studies that illustrated the use of the Phantom 4 normal UAV.For the Prince of Wales Fort, a photogrammetric model was successfully generated with a checkpoint RMS error of 7.87 mm, which was an acceptable error for a 1:50 drawing set.The large data set was successfully processed through a combination approach of PhotoScan and ContextCapture in order to generate orthophotos with a GSD of 5 mm per pixel, which is a sufficient error for condition assessment applications.For Taumadhi Square, the results obtained by the UAV were used for the creation of a series of photogrammetric models to generate orthophotos with a GSD of 5 mm per pixel.These orthophotos were used to develop drawings of the Nyatapola Temple and the Bhairabnath Temple.Additionally, the images that were captured were important for qualitative damage analysis.The results obtained will contribute in the development of the restoration and rehabilitation plan of the damaged structures.
Author Contributions: Alex Federman, Sujan Shrestha, Davide Mezzino, and Mario Santana Quintero wrote the paper.John Gregg, Shawn Kretz, and Christian Ouimet, provided valuable feedback during the revision process, as well as the original documentation results for the Prince of Wales Fort case study.

Figure 4 .
Figure 4. Example of flight path pattern for Prince of Wales Fort-Screenshot from airdata.com [23].

Figure 8 .
Figure 8. Screenshot of best-fit plane applied to point cloud in CloudCompare.

Figure 10 .
Figure 10.Orthophoto showing GCP locations of Taumadhi Square obtained from UAV Photogrammetry.

Figure 11 .
Figure 11.Photograph locations used in the processing of Bhairabnath Temple-Right image shows camera locations, while left image shows the Bhairabnath Temple.
displays the control point locations for the Taumadhi Square survey.The Nyatapola Temple and Bhairabnath Temple were processed on a high quality setting with aggressive depth filtering.No checkpoints were used because of the limited amount of GCPs available.The RMS errors from PhotoScan for the Nyatapola Temple and Bhairabnath Temple are shown in Tables

Figure 12 .
Figure 12.Damage recording from the UAV photographs.

Table 1 .
Summary of Flight Tests at Prince of Wales Fort.
Oblique images(at a gimbal pitch of 20-47 degrees down) of the west side of the interior courtyard area, as well as the interior parts of the tops of the rampart walls at 8-13 m high flight 10 20 min, 33 s Oblique images (gimbal perpendicular to the wall surface, and at a gimbal pitch of 7 degrees down) of specific areas of the fort for test photogrammetry purposes.Images taken at 3 m away from wall

Table 4 .
Root-mean-square error (RMSE) of Control Points from PhotoScan for Nyatapola Temple.

Table 5 .
Root-mean-square error (RMSE) of Control Points from PhotoScan for Bhairabnath Temple.