Article Individual Tree Species Classification by Illuminated—Shaded Area Separation

A new method, called Illumination Dependent Colour Channels (IDCC), is presented to improve individual tree species classification. The method is based on tree crown division into illuminated and shaded parts on a digital aerial image. Colour values of both sides of the tree crown are then used in species classification. Tree crown division is achieved by comparing the projected location of an aerial image pixel with its neighbours on a Canopy Height Model (CHM), which is calculated from a synchronized LIDAR point cloud. The sun position together with the mapping aircraft position are also utilised in illumination status detection. The new method was tested on a dataset of 295 trees and the classification results were compared with ones measured with two other feature extraction methods. The results of the developed method gave a clear improvement in overall tree species classification accuracy.


Use of Species Classification in Forest Management
Tree species classification has traditionally required an expert inspector working manually with aerial imagery.However, a demand for more precise and automatic forest attribute estimation has arisen.One of the keys towards more precise estimates is more accurate knowledge of the species involved.The forest management field is presented with a growing number of demands from both the industrial and non-industrial sectors.The industrial sector needs better information of the material quality and quantity of raw wood.The non-industrial sector needs more focused knowledge for such objectives as preservation of biodiversity, sequestration of carbon, creation of recreational opportunities, and hunting considerations.In order to meet these requirements, more precise information from forest inventories is needed.Thus, airborne laser scanning (ALS) is increasingly used for operative, stand-wise inventory in Scandinavia.The two main approaches used to derive forest information from ALS data have been based on laser canopy height distribution e.g., [1,2] and individual tree detection e.g., [3].In both approaches, the two main development areas are: (1) practical solution for tree species classification and (2) improvement in accuracy and quality of the reference sample plots.
The knowledge of the tree species information is needed in forest management planning.Biological studies on forest habitat mapping benefit from species specific forest information, since for example, the preferred tree species of some endangered species could be located using remote sensing [4].Knowledge of tree species is also needed in the forest industry as the species information determines the usability of the wood material.Both the tree growth and the timber volume estimates are species dependent.Very fine level information on the forest is especially important in wood procurement planning and in forest protection surveys [5].
Species specific reference measurements can be made with stand-wise precision on the field, but this requires a massive amount of work.The number of assessments per stand is also low, which lowers the estimation precision of the tree species specific timber range [5].
Species classification from aerial images is usually achieved using one of the following approaches: object-based or pixel-based.In object based species classification, trees or a group of trees are first detected, delineated and extracted from data.Features of a single tree object are then computed for classification.In pixel based classification, either image pixels or integrated data raster cells are classified.This approach is closely related to the methods used in land cover classification and is mainly used in determining the forest type or the main species in large forested areas [6,7].

LIDAR and Aerial Image Based Methods
One of the main reasons to use LIDAR intensity data is the fact that there is no shadowing in the LIDAR measurement.Range corrected LIDAR intensity data has been used lately for species classification [8,9].In Donoghue et al. [8] range corrected intensity measures were computed for different height quantiles.These quantiles were used to quantify the volume of spruce in even aged, mixed spruce and pine stands.Ørka et al. [9] used structural features together with the range corrected, first return pulse intensity data.The overall classification accuracy for classifying spruce and birch was 88%.Korpela et al. noted that the LIDAR intensity data was not sufficient to separate the three main species of forest trees in Finland [10].
Different approaches have been developed to integrate aerial images and LIDAR data in several studies [10][11][12].Persson et al. [11] integrated LIDAR data and aerial colour-infrared (CIR) imagery to classify tree species into three classes: spruce, pine and deciduous.LIDAR data was used to segment trees.Segments were mapped to the corresponding aerial image.The classification was done using 10% of the brightest pixels of each tree crown.Each chosen pixel was represented by two angle values, which were calculated from the green, red, and infrared components of the pixel.A sample tree was represented by the mean of the pixel angle values within the tree segment.Spectral band ratio filtering was suggested for the reduction of shadowing effects.An overall classification accuracy of 90% was reported for the training set.A spectral rationing algorithm and formation of a hybrid colour composite image has been also used to reduce shadow effects in other studies, e.g., in Bork et al. [13].
LIDAR data were used to delineate tree crowns in a study, where five tree species were classified from aerial images taken with ADS40 and RS30 digital cameras [14].The training set consisted of CIR images.An overall classification accuracy of 86% was reached.LIDAR collected digital surface model (DSM) data was also used to delineate trees in Heinzel et al. [12].Histogram linearized CIR true orthophotos were transformed into hue (H), saturation (S), and intensity (I) channels.The detected tree polygon was fitted to the spectral data and shaded areas with very low intensity values were removed.The classification was done in two steps: first using the hue channel histogram and second using NIR band.The overall classification accuracy for tree classes of oak / hornbeam, beech, and conifer was 84%.
Korpela et al. integrated LIDAR data with aerial images and used them to classify seedling trees in a raster cell setup with an approximate resolution of 0.5 m [10].Conifers, deciduous broad-leaved trees, other low vegetation, and abiotic surfaces were used as reference classes.The achieved classification accuracy varied between the study stands with minimum 61.1% and maximum 77.8%.Respective minimum and maximum accuracies changed to 61.6 and 78.9 percent, when the used tree samples were limited to those in direct sunlight.
Temporal LIDAR intensity data has been also applied in tree species classification [15].Both leaf-off and leaf-on datasets were measured from the same forest site located in Washington Park Arboretum, Seattle, Washington, USA.Eight deciduous and seven coniferous tree species were included in the study.Resulting classification accuracies between deciduous and coniferous trees were 73.1% for leaf-on dataset, 83.4% for leaf-off dataset, and 90.6%, when all datasets were used.
Successful tree species classification results have been also reported using only aerial images [16].Healthy and damaged spruce, pine, fir and beech trees were classified semi-automatically using CIR images of 0.5 m resolution.The achieved average classification accuracy was 80%.
According to the recent studies reviewed above, the most typical approach to object based tree species classification is the use of LIDAR data for tree crown delineation and for selection of the corresponding image pixels.The species classification is typically done either by combining only features based on image colour channels or by combining them with LIDAR based structural features.Shadowing problems are handled using filtering and pixel selection.

Methodology Comparison Studies
New forest type and tree species classification methods are usually developed using individual datasets.This makes it difficult to compare their results and performance with each other.However, some comprehensive projects have been done during last few years.
A EuroSDR tree extraction project, where different extraction methods were tested on freely available datasets, took place in 2008 [17].Twelve different groups participated in the project.Only two participants classified tree species.The tree species classification results were 78% correctly classified trees using airborne photographs (57% of the trees were classified) and 54% correctly classified trees using laser data (64% of the trees were classified).The abovementioned results are of interest because of the great variation between their classification percents and the ones published in articles presenting classification methods with over 80% classification accuracy.We assume that the good previous results have been obtained by having controlled conditions.The EuroSDR test showed that the tree classification accuracies published before (e.g., in tree finding) did not match with the results obtained in the joint test.Thus, methods that work in nonoptimal forest conditions are still needed.More research should be also focused on method comparison.

Tree Canopy Division into Illuminated and Shaded Parts
We assume that there could be features with measurable differences between different tree species when the illuminated and shaded parts of the tree canopies are first separated and then compared with each other.This assumption is based on the fact that foliages of different tree species have different light scattering properties due to their general shape and leaf properties, e.g., [18].We anticipate that the transmittance of the tree canopy affects the image brightness on the shadowed side of the tree.Separation of the illuminated and the shaded parts of a tree canopy should also allow better utilisation of the available dataset as heavy filtering is not needed for shadow removal.Different viewing geometries are also taken into account as long as the locations of both the camera and the sun are known.The separation of an individual tree canopy into illuminated and shaded parts has been done before in Leaf Area Index (LAI), Normalized Difference Vegetation Index (NDVI) and canopy radiative transfer studies [19][20][21].This approach has been utilised in single tree species classification as well [22,23].Sharp contrast differences between the shaded and illuminated parts of the trees were used in automatic individual tree crown delineation and tree top finding already in the 1990s from aerial photographs [24][25][26] where it gave good results.
In this study, we test to see if we are able to find classification aiding features by separating a single tree dataset into illuminated and shaded parts.The dataset and the methods used in data extraction are introduced in Section 2. Tested features and the used classification methods are presented in Section 3. The classification results are given in Section 4 and the discussion of the results and possible further studies are in Section 5.

Test Area
The test area was located in Espoo city, southern Finland (N60° 9.0', S24° 39.4').Vegetation in the test area was a mix of common city lawn, planted deciduous trees (linden, alder, for example), and mixed natural growing stock consisting of original domestic trees like birches, pines, and spruces.The test area had a varying contoured topography with heights between 0-30 meters above sea level.

ALS Da
The ALS Optech ALT flown 1.000 overlapping All scann DTM) of th Finland).Th DTM, heigh height value cell size w EUREF-FIN h's m ch al length of the cameras was 25 mm.The multispectral colour cameras were sensitive in the following spectral bands: Blue (400-580 nm), Green (500-650 nm), Red (590-675 nm), and NIR (675-850 nm).One pixel footprint size on the ground was approximately 0.25 × 0.25 m 2 .

Tree Sample Data
The dataset used for tree species classification consisted of 295 sample trees.Location and species of each sample tree were verified in the field.In some cases a sample consisted of more than one tree of the same species.In such a case trees in the sample were growing so closely to each other that they had a common canopy.Tree samples were chosen from the three most common tree species (birch, pine, and spruce) in Finnish forests.Both individual trees and ones growing in copses of various sizes were included in the dataset.The total number of tree samples was 151 birches, 99 pines, and 45 spruces.Samples were chosen so that they represented different ages and sizes of their species.

Tree Crown Delineation
Data for each tree sample were manually extracted and registered from ALS data and a digital aerial image.The sample extraction was done using an interactive interface built with MATLAB (Mathworks, Natick, MA, USA).Manual delineation was chosen as it was considered the most accurate method for a limited number of tree samples.
A dataset of a tree sample consisted of data cells.The position coordinates, elevation, canopy height, RGBNIR colour values, visibility to the camera, and shading status of each data cell were saved.The number of data cells for each tree sample varied from tens to several hundreds, depending on the size of the crown.Metadata describing surroundings and the used extraction parameters were also saved for each tree sample in addition to the data stored in data cells.
Height and position values for data cells located in each selected sample canopy were extracted from both the CHM and the DSM.A 3 × 3 median filter was applied to all CHM and DSM raster cells within the selected canopy.Height values were smoothened to avoid cases, where the laser beam had penetrated the canopy giving a raster cell a low height value compared to its neighbours.Unit normal vectors pointing towards the sun and the camera were also calculated for each data cell after height and position extraction.

Data Cell Visibility and Shading Determination
Different viewing angle geometries were considered after height value extraction.Both DSM and CHM were created as if the viewer would always be straight above each data cell.In an aerial image each pixel was viewed from a different angle depending on the location of the sensor and the pixel's footprint on the ground.This meant that a varying number of data cells in each side of the selected tree sample were not seen by the sensor.This situation is presented in Figure 2. The n works thus as an effective cutoff range.The length of a step was set to 30 cm for the test dataset.z(i,j), c z (i,j), z 0 , and z DSM (i,j) are the corresponding height components in the raster cell point (i,j).L is a preset number of occluding pixels that are blocking the line of sight between the sun and the data cell.It serves as an estimate for transmittance of direct sunlight through the canopy.It is used because the rasterized elevation model cannot make a difference between full and partial occlusion.Completely opaque materials are described with L = 0.This means, in terms of L, that even one blocking data cell in (2) causes a complete occlusion of the original data cell.Same type of visibility determination is also used in true orthophoto generation from several aerial images, for example, in Bang et al. [27].Individual data cell shading was inspected after the data cell visibility verification.The data cell shading inspection was analogous to that done in visibility verification, but this time the vector drawn from the inspected data cell was pointing towards the sun.If any of the other data cells along the vector's line were higher than the vector's height component in any point, the data cell was marked as shaded.Otherwise the data cell was marked as illuminated.This situation is shown in Figure 3.The location of the sun was calculated using the flight time records and a MATLAB routine presented in [28] by V. Roy.The routine uses algorithms presented in Reda et al. [29].Shadowing inspection was done only for the direct shading.Transmittance properties of different species and possible diffuse effects were not considered.In this study, trees were approximated as opaque objects in both visibility and shadow detections.Thus the value of L = 0 was used in both visibility and shading calculations.

Colour Value Extraction
Colour values were linked to each visible data cell by registering and extracting them from an original digital aerial image.All colour values for one tree sample were taken from a single aerial image.Registration and extraction were done by calculating the projected location of each data cell on the original image using collinearity equations [30].These equations are given in (3)., where (x,y) is the coordinate of a pixel on the aerial image corresponding the projected cell, (X,Y,Z) is the cell location on the DSM, and (X 0 ,Y 0 ,Z 0 ) is the sensor location.f is the focal length of the sensor, and r ij are the elements of the (ω, φ, κ) rotation matrix.[11].Features for this set were collected by filtering the brightest 10% of the pixels in tree sample data.Then a normalized (colour space) unit vector was formed for every tree sample by using the averaged intensities of the green, red and near-IR colour channels.Two descriptive angles, azimuth and elevation, were calculated between the colour vector components after the normalization.These two angles were then used as classification features.
We used this feature set as a reference as its extraction is straightforward.We did not use pan-sharpened images which were utilised in the referenced article [11].The colour channel analysis was done instead with the original multispectral aerial images to preserve extracted colour values as well as possible.The suggested feature extraction procedure was followed otherwise.

Classification Methods
Supervised parametric classification was used in this work with a cross validation setup.In cross validation, n groups consisting of m samples were formed.Then, n-1 groups were used as training sets and all trees in the one remaining group were classified.Classification was then repeated for each sample group in a loop with n rounds using each sample group as a test set.The division into training and test sets was the same for each of the tested feature spaces and algorithms.Cross validation was implemented as a leave-one-out setup, where sample size m = 1.Leave-one-out setup was chosen because the total number of sample trees was low.Some tests were also run using different k-fold cross-validations.Their results varied a lot due to small training set sizes and heterogeneous tree composition.
The used classification algorithms were quadratic, linear, and Mahalanobis distance based discrimination functions.The classification was done using MATLAB Statistics Toolbox.Only the best discrimination function results are presented for each feature set.This is because the optimal decision surfaces are dependent on the used features.

Classification Results
The error matrices for the tree species classification with the proposed and the reference feature sets are presented in Table 1.Only the best cases are shown for each feature set.The best results were achieved in all studied cases using all four colour channels (RGBNIR).The proposed feature set, IDCC, gave the best classification result when a combination of illuminated and shaded colour channel values and their ratios were used.The overall accuracy of the classification with this feature set was 70.8%.Coniferous and deciduous trees were separated from each other with the overall accuracy of 78.0%.These results were achieved with the quadratic discrimination function.The two angular features presented in the reference (Persson et al. [11]) resulted in an overall tree species classification accuracy of 64.4%.The classification accuracy between conifers and deciduous trees was 75.6%.Best results were achieved using linear discriminant analysis function.Both quadratic and mahalanobis distance using discrimination functions gave notably lower classification results.We also tested the validity of the proposed feature set by classifying extracted tree samples using only colour values of original aerial images with no extra processing.The DSM and CHM data were used to delineate tree crowns, but the height information was not used otherwise in tree species classification.The overall classification accuracy was relatively high: 67.8%.The accuracy of the deciduous and coniferous classification was 76.9%.These results were obtained with linear classification.
Classification results showed four interesting issues.Firstly, the reference feature set gave the lowest overall recognition percentage of all three feature sets.This resulted most likely from the strong filtration done to the tree data.Secondly, the strengths of the features were divided: raw colour channel data and the features derived from IDCC recognized pine trees with a good accuracy.The angular features suited the best for birch classification with a wide margin compared to the two other feature sets.None of the used feature sets gave a definite classification result for spruces and they were mixed mostly with pine trees.The third issue was that the classification accuracies between the deciduous and coniferous tree species were relatively close to each other in all used feature sets.
The used feature sets classified different tree species with a good accuracy.This result was utilised to develop a more accurate classification tree.The improved classification tree consisted of five steps.All the steps were based on the posterior probabilities given by the used MATLAB algorithm.The posterior probabilities gave an estimate of the classification's quality for each possible class.Before taking any steps in the decision tree all trees were classified separately with both IDCC and Persson feature sets.
In the first step all trees classified similarly by both feature sets were accepted as correct classifications.Birches were then searched from the unclassified trees using the following condition for the posterior probabilities derived from the Persson feature set: p Birch,Per > 0.5.This condition was based on the good performance of the Persson method feature set in birch classification.The third and fourth steps were performed in a similar fashion, but with different conditions: p Pine,IDCC > 0.5 and p Spruce,IDCC > 0.5.Trees that were not recognized after the first four steps were classified based on the largest posterior probability taken from the Persson feature set.
The combined five-step classifier gave a clear improvement in both the overall classification and in deciduous and coniferous tree separation.Table 1 shows that the combined classification tree managed to classify both birches and pines almost as well as the best feature sets.At the same time it gave the best classification for spruces.

Factors Affecting the Quality of Results
Aerial image data was taken on an afternoon hour in late summer.This means that the sun was already relatively close to the horizon, which was normal at the measurement latitude.The low position of the sun also caused more shadowing, both between the trees and their surroundings.The sun position was also changing most rapidly at this time of the year.The solar zenith angle changed 4.6° during flight in the test area.Aerial images were not radiometrically corrected so the sun movement had an effect on colour channel values between different images.
The flight time was not optimal from the spectral point of view.The colour of deciduous leaves gets dark green as the leaves age during summer.This is due to the increase of their chlorophyll concentration, e.g., Rautiainen et al. [31].Darkening of a leaf may not be such a notable issue in well illuminated conditions, but a shadowed dark leaf might get mixed with illuminated needles within the used colour channels.
Data synchronization between aerial images and elevation models was not exact as there was a month and a half between flights.Tree shapes had changed as the canopies had grown.Canopy shapes could have been affected by the local weather conditions, mainly by the wind.Different wind directions during flights cause notable changes in the canopy shape, e.g., Litkey et al. [32].Both aerial images and the laser scanning for elevation models need to be taken at the same time for optimal results.
The used classification algorithms may not have performed optimally.The size of the sample tree data was relatively small, only 295 trees in total, and trees were heterogeneous.Thus feature variation within all tree species was large.The spatial resolution of each data cell was quite low.This means that the data in each cell represents a general average of the area it is covering.The low resolution could also explain why the results of the reference method were similar with those gained from nonfiltered, original aerial image data.Heavy colour channel filtering seems to suit better for aerial images with high resolution, like the pan-sharpened images used in the reference article [11].
Extracted height values in data cells were mainly interpolated.The normal laser point coverage in the DSM region was 2-4 laser hits/m 2 .This meant that approximately 70% of rasterized height values were interpolated, when raster cell size was 30 × 30 cm 2 .In more densely covered areas (10-12 laser hits/m 2 ) most of the extracted data cells contained a measured value.However, these point densities seemed to be sufficient for the shading detection.The classification results showed a clear improvement compared to the cases, where height data were not used to extract features.It would be worthwhile to study how ALS derived point density scales with the general classification accuracy in this type of measurement.
Some additional uncertainty was also caused because the height data were median filtered to remove possible single gaps within the tree crowns.The filtration lowers the extracted height values within the tree crowns.It is also known that the ALS data tends to underestimate the actual heights of the scanned objects [17,33].These height uncertainties affected the projections on the original aerial images.Thus some of the pixels in the edge of illuminated and shaded sides might have been falsely flagged and adding the noise in the classification process.

Further Development
The ALS derived surface models were only utilised to determine shadowing of each tree sample in this study.It could be possible, however, to derive other canopy describing features out of them.Such a new feature could be the general inclination of canopy structure.
Tree crowns were delineated manually in this study.There should be, however, no limitations to implement an automated delineation, such as [34][35][36], method for single tree extraction.Automated delineation allows processing of large datasets in an efficient manner.
The calculation of DSM (and DTM/CHM) and registration errors have an effect on the shadowed area determination.The extent of the shadowed area depends on the original point density, the process which removes the penetrated hits, and the filtering method applied to the DSM.In practice, the laser always sees the trees smaller in size and height than what they are in reality due to the penetration of the laser hits inside the crown.The back projection of tree position and crown segment to images has therefore some error sources.These and other registration error effects should be studied further in the future.The approach used in five-step classification process could be further developed and optimized by adding or subtracting step number or by changing step order.However, the possibility to create a general classification tree might not be achievable, but the process must be calibrated every time for a new dataset.

Conclusions
A new feature set extraction method for tree species classification, named Illumination Dependent Colour Channels (IDCC), was presented in this research.The presented method is based on dividing a single tree crown into illuminated and shaded parts, and then using averaged colour values (RGBNIR) of both parts and their ratios as classification features.The division of a tree crown is done using height information collected from synchronized ALS data.Performance of the IDCC method was compared with another feature extraction method [11], which used filtering for colour image data, but no ALS data, and with the original, unprocessed, aerial image colour data.The comparison was performed by doing a leave-one-out classification for a test dataset comprising of 295 common Finnish trees.
The results showed that the IDCC gave a small, but notable, improvement of a few percentage units in overall classification accuracy over the two references.It was also noticed that the species-wise classification results varied between different methods.The IDCC had good performance in coniferous tree classification, while Persson's extraction method gave the best results in birch classification.This notification led to the development of a combined five-step classification tree.The combined classification tree utilised the strengths of both the IDCC and the Persson's reference method.The developed classification tree enhanced the overall classification accuracy of all tree species by an additional three percentage units.
The results from the developed feature set extraction method, IDCC, and the five-step classification tree are notably better when compared to the reference ones.Feature variations were large for all tree species within the used dataset.The tree samples of each species were of different ages and sizes, and they were located in different growing places with varying surroundings.Even though earlier studies have shown high classification accuracy for boreal forest tree species classification, the practically obtained accuracy has been from 50% to 70% at individual tree level.The proposed method should work especially well in high latitudes where aerial photographing must be often done with the sun close to horizon.
Generalisation of the proposed feature set extraction method should be straightforward for other types of cameras and hyperspectral sensors.Usage of hyperspectra, especially in the NIR region, is expected to yield more species dependent spectral features to improve a classification procedure.The effects caused by shadow movements during different times of the day are accounted for as the sun location is known at all times.Radiometric changes due to the sun movement as well as the atmospheric changes need to be noted separately and improvements in radiometry of images are expected to improve the classification results.
Figure which angle d field o the clo raster.
Figure and th A Another feature set was used as a reference.Its extraction procedure has been presented inPersson et al.

Table 1 .
Error matrices of the best classification results obtained with used methods.Classification type with the best result is given in brackets below the name of the method.