Combined Use of 3D and HSI for the Classiﬁcation of Printed Circuit Board Components

: Successful recycling of electronic waste requires accurate separation of materials such as plastics, PCBs and electronic components on PCBs (capacitors, transistors, etc.). This article therefore proposes a vision approach based on a combination of 3D and HSI data, relying on the mutual support of the datasets to compensate existing weaknesses when using single 3D- and HSI-Sensors. The combined dataset serves as a basis for the extraction of geometric and spectral features. The classiﬁcation is performed and evaluated based on these extracted features which are exploited through rules. The efﬁciency of the proposed approach is demonstrated using real electronic waste and leads to convincing results with an overall accuracy (OA) of 98.24%. To illustrate that the addition of 3D data has added value, a comparison is also performed with an SVM classiﬁcation based only on hyperspectral data.


Introduction
Humanity is producing more and more electronic waste. In 2019 worldwide, 53.6 million tons of electronic waste were produced. Technological innovations, short production cycles and, in particular, ever-shorter product life cycles are the reasons for this rapid increase. Only 17.4% of the electronic waste produced worldwide is recycled, while the rest ends up in waste incineration plants [1]. Due to existing toxic substances, this is not only harmful to the environment, but could also cause serious risks for the human health. This makes recycling processes that help to achieve sustainability more important.
Prerequisite for recycling is the sorting of electronic waste based on the type of material. Whether monitors, calculators, mobile phones or vacuums, these devices are shredded when they end up in landfills where they remain in the form of cables, plastics, metals and Printed Circuit Boards (PCBs). PCBs in particular represent a highly valuable resource consisting of polymers and precious metals [2]. Identifying the individual components on circuit boards (capacitors, integrated circuits, inductors, etc.) is a particular challenge. This challenging issue has already been addressed in a number of studies using different strategies. In [3], an approach was developed for the segmentation of Surface Mounted Devices (SMDs) on PCBs. The authors use a segmentation based on assembly paint and color distribution to detect two types of SMDs (small devices like resistors and integrated circuits). A further article dealing with the segmentation of SMDs can be found in [4] and they propose an algorithm which exploits shadows cast by SMDs to distinguish them from invalid counterparts. The segmentation of Through Hole Components (THCs) is part of [5] and is achieved by using a combination of RGB images and depth frames from a Microsoft Kinect sensor. A further study based also on depth frames obtained from the Microsoft The general concept of the proposed method consists of merging the datasets followed by extracting features in the 3D and hyperspectral domains. Especially the fusion of datasets is a challenging task and poses different requirements (e.g., calibration of line sensors and methods for registration) depending on the sensor technologies used. Hyperspectral sensors with high spectral and spatial resolution are usually line sensors and differ in their imaging properties from frame-based sensors. Accordingly, adapted models are necessary to perform the calibration, which is a required step for the registration of the datasets. Methodologies for the registration of line scanning HSI sensors and 3D Laser Scanners in outdoor applications can be found, for example, in [23][24][25]. The strategy adopted in this article focuses on a laboratory application and follows a simple registration concept that can similarly be applied to an industrial production/sorting process.
The basic steps proposed regarding the processing of the combined dataset can be easily generalizable or extended to other application fields. All objects typically have a shape and a material composition. This assures a generalization as 3D and HSI are in principle sensitive for features in these domains. The features obtained are next used as inputs for a rule-based approach. One of the main advantages of rule-based approaches is the simplicity. Once the knowledge on which rules are based has been worked out, conditions can be easily set up. Further advantages are the performance, the ability to handle redundant and irrelevant attributes and the flexible extensibility of rule sets [26]. Acquiring knowledge can seem effortful, but in view of the resulting advantages, it should be seen as a clear benefit which allows to structure and simplify problems by using expert knowledge. The rules used in this article are based on knowledge established for electronic waste sorting. A use of the procedure for other applications requires an adaptation of the rules to the respective application based on knowledge from the relevant field.

Materials and Methods
As shown in Figure 1, the general workflow of this work can be divided into four main parts. The data acquisition, the data fusion, the use of 3D data to simplify and structure the data basis and the final classification based on obtained HSI and 3D features. The individual parts are described in detail in the following subsections.
The general concept of the proposed method consists of merging the datasets lowed by extracting features in the 3D and hyperspectral domains. Especially the fus of datasets is a challenging task and poses different requirements (e.g., calibration of l sensors and methods for registration) depending on the sensor technologies used. Hyp spectral sensors with high spectral and spatial resolution are usually line sensors and d fer in their imaging properties from frame-based sensors. Accordingly, adapted mod are necessary to perform the calibration, which is a required step for the registration the datasets. Methodologies for the registration of line scanning HSI sensors and 3D La Scanners in outdoor applications can be found, for example, in [23][24][25]. The strate adopted in this article focuses on a laboratory application and follows a simple regis tion concept that can similarly be applied to an industrial production/sorting process.
The basic steps proposed regarding the processing of the combined dataset can easily generalizable or extended to other application fields. All objects typically hav shape and a material composition. This assures a generalization as 3D and HSI are in pr ciple sensitive for features in these domains. The features obtained are next used as inp for a rule-based approach. One of the main advantages of rule-based approaches is simplicity. Once the knowledge on which rules are based has been worked out, conditi can be easily set up. Further advantages are the performance, the ability to handle red dant and irrelevant attributes and the flexible extensibility of rule sets [26]. Acquir knowledge can seem effortful, but in view of the resulting advantages, it should be s as a clear benefit which allows to structure and simplify problems by using exp knowledge. The rules used in this article are based on knowledge established for e tronic waste sorting. A use of the procedure for other applications requires an adaptat of the rules to the respective application based on knowledge from the relevant field.

Materials and Methods
As shown in Figure 1, the general workflow of this work can be divided into f main parts. The data acquisition, the data fusion, the use of 3D data to simplify and str ture the data basis and the final classification based on obtained HSI and 3D features. T individual parts are described in detail in the following subsections.

Data Set Acquisition
The measurement setup shown in Figure 2 consists of two Specim Ltd. (Oulu, Finland) hyperspectral pushbroom cameras (Specim FX10 and Specim FX17), a 3D scanner, a linear stage and an illumination unit. This system was chosen for the acquisition of real electronic waste samples consisting of plastics and PCBs from the landfill. Accordingly, these objects are damaged and dirty. Special attention was paid to the choice of PCBs with different mounted electronic components. The required wavelength is limited to the NIR range, therefore only the Specim FX17, working in the range from 900 nm to 1700 nm (224 bands), is used for the measurements in this work. Acquired datasets are radiometrically normalized by using dark reference image for dark-current (closed shutter) and a white reference image to reduce the influence of the intensity variability. For the white calibration a 99% reflectance tile was used. a linear stage and an illumination unit. This system was chosen for the acquisitio electronic waste samples consisting of plastics and PCBs from the landfill. Acco these objects are damaged and dirty. Special attention was paid to the choice of PC different mounted electronic components. The required wavelength is limited to range, therefore only the Specim FX17, working in the range from 900 nm to 1700 bands), is used for the measurements in this work. Acquired datasets are radiom normalized by using dark reference image for dark-current (closed shutter) and reference image to reduce the influence of the intensity variability. For the white tion a 99% reflectance tile was used.
For the acquisition of the geometry, a structured light scanner from GOM (Braunschweig, Germany) is used. The Atos Core 500 is a high-resolution optica with a maximum resolution of 0.195 mm and a depth accuracy of 0.05 mm. This ner consists of two stereoscopic cameras with 5-megapixel resolution and a blue L projector which projects structured light onto the object. The projected light, whic tially encodes the surface of the object, is captured by the cameras and triangu used to obtain the necessary 3D information.

Combining Spectral and Spatial Data Sets
The first step of the proposed framework is the fusion of the datasets (show ure 3) consisting of HSI and 3D point cloud. This is achieved by using tie points the technical setup of the sensors (fixed sensors looking vertically downwards), th are successively captured during one measuring process in both hyperspectral a three-dimensional. The transformation parameters are estimated by using tie point existing in both datasets and allow the fusion in x-y plane. Due to the different re of the datasets, the assignment between 3D point and spectral signature is made pixel checking the neighborhood. The result is a 3D point cloud presented in Figu a spectral signature for each individual 3D point. Even though the sensor technol sented in Figure 2 would not be readily applicable in an industrial context due t sensor technology used, the potential for the overall concept of the measuremen and fusion strategy for use in industrial practice is given. For the acquisition of the geometry, a structured light scanner from GOM GmbH (Braunschweig, Germany) is used. The Atos Core 500 is a high-resolution optical system with a maximum resolution of 0.195 mm and a depth accuracy of 0.05 mm. This 3D scanner consists of two stereoscopic cameras with 5-megapixel resolution and a blue LED light projector which projects structured light onto the object. The projected light, which essentially encodes the surface of the object, is captured by the cameras and triangulation is used to obtain the necessary 3D information.

Combining Spectral and Spatial Data Sets
The first step of the proposed framework is the fusion of the datasets (shown in Figure 3) consisting of HSI and 3D point cloud. This is achieved by using tie points. Due to the technical setup of the sensors (fixed sensors looking vertically downwards), the objects are successively captured during one measuring process in both hyperspectral as well as three-dimensional. The transformation parameters are estimated by using tie points that are existing in both datasets and allow the fusion in x-y plane. Due to the different resolutions of the datasets, the assignment between 3D point and spectral signature is made pixel by pixel checking the neighborhood. The result is a 3D point cloud presented in Figure 4 with a spectral signature for each individual 3D point. Even though the sensor technology presented in Figure 2 would not be readily applicable in an industrial context due to the 3D sensor technology used, the potential for the overall concept of the measurement system and fusion strategy for use in industrial practice is given.

Processing of 3D Point Cloud
In these merged data sets the spatial component directly helps to simplify the problem as we can reduce the data set to regions of interest, such as the objects, and remove useless regions, such as the background. The second step of the proposed framework consists in removing the background. In our study case, the background can be easily removed using the depth information thanks to a plane estimation method. This has several advantages. Firstly, the amount of data are reduced, which has a positive effect on the further processing steps. Secondly, other disturbing effects such as dirt (e.g., on the conveyor belt) and shadow areas are removed. Areas of shadow in particular can have a

Processing of 3D Point Cloud
In these merged data sets the spatial component directly helps to simplify th lem as we can reduce the data set to regions of interest, such as the objects, and useless regions, such as the background. The second step of the proposed framewo sists in removing the background. In our study case, the background can be ea moved using the depth information thanks to a plane estimation method. This has advantages. Firstly, the amount of data are reduced, which has a positive effec further processing steps. Secondly, other disturbing effects such as dirt (e.g., on t

Processing of 3D Point Cloud
In these merged data sets the spatial component directly helps to simplify the problem as we can reduce the data set to regions of interest, such as the objects, and remove useless regions, such as the background. The second step of the proposed framework consists in removing the background. In our study case, the background can be easily removed using the depth information thanks to a plane estimation method. This has several advantages.
Firstly, the amount of data are reduced, which has a positive effect on the further processing steps. Secondly, other disturbing effects such as dirt (e.g., on the conveyor belt) and shadow areas are removed. Areas of shadow in particular can have a negative impact on the quality of the results. An example for disturbing shadow effects and the influence on the classification result is shown in Figure 5. Shadow areas occur mainly on the border of 3D objects and also on the border of 3D components on the top of PCBs. The most disturbing shadows are on the border of the objects. Shadow areas are generally a problem when processing datasets as they reduce the reliability and success rate of algorithms for object recognition, classification and other types of processing. The removal of shadow areas is therefore an essential step towards improving image quality [27,28] and thus also improve the base for further analysis steps. negative impact on the quality of the results. An example for disturbing shadow effects and the influence on the classification result is shown in Figure 5. Shadow areas occur mainly on the border of 3D objects and also on the border of 3D components on the top of PCBs. The most disturbing shadows are on the border of the objects. Shadow areas are generally a problem when processing datasets as they reduce the reliability and success rate of algorithms for object recognition, classification and other types of processing. The removal of shadow areas is therefore an essential step towards improving image quality [27,28] and thus also improve the base for further analysis steps.
(a) (b) Another advantage of adding 3D data is the possibility to partition the dataset by using cluster methods, which are addressing the shape content of objects. The aim of a cluster analysis is to find structures in a set of arbitrary data with different properties and to discover relationships in order to be able to group objects. This involves forming groups in such a way that the properties of the data within the group or cluster are approximately identical. A variety of cluster algorithms exists in the state of the art for this purpose. The task of cluster algorithms is to form clusters within datasets. Clustering methods differ in their approaches and strategies. This allows the separation of the individual objects in the dataset. Due to the homogeneous data available in our dataset, a cluster approach based on Euclidean distance was chosen for the separation of the individual objects. Cluster methods can be not only used for the individual object's separation (such as plastics and PCBs, as shown in Figure 6b), but also can be used for the separation of components on the boards (as shown in Figure 6c). Another advantage of adding 3D data is the possibility to partition the dataset by using cluster methods, which are addressing the shape content of objects. The aim of a cluster analysis is to find structures in a set of arbitrary data with different properties and to discover relationships in order to be able to group objects. This involves forming groups in such a way that the properties of the data within the group or cluster are approximately identical. A variety of cluster algorithms exists in the state of the art for this purpose. The task of cluster algorithms is to form clusters within datasets. Clustering methods differ in their approaches and strategies. This allows the separation of the individual objects in the dataset. Due to the homogeneous data available in our dataset, a cluster approach based on Euclidean distance was chosen for the separation of the individual objects. Cluster methods can be not only used for the individual object's separation (such as plastics and PCBs, as shown in Figure 6b), but also can be used for the separation of components on the boards (as shown in Figure 6c).

Rules-Based Classification
The third step of the proposed framework consists in defining rules to process 3D data and spectral data in order to classify individual 3D point clouds.
Based on the existing dataset, it can be stated that the hyperspectral point cloud contains valuable geometric information that is implicitly represented by the spatial arrangement of the individual 3D point clouds. In addition to the geometric information, the dataset also contains valuable spectral information to describe the physical property of the objects.
3D point clouds are a representation of spatial 3D information, which can be described by geometric characteristics. The derivation of these geometric characteristics is done by considering the local neighborhood of each 3D point. Based on this neighborhood, invariant moments can be calculated for each point, representing the geometric properties of local 3D structures [29]. A representation of some 3D features is shown in Figure 7. The 3D Appl. Sci. 2021, 11, 8424 7 of 19 features tested in this article are nowadays commonly used in lidar data processing. They are based on normal vectors and eigenvalue-based 3D features like linearity, verticality, planarity, scattering, omnivariance, anisotropy, eigenentropy and change in curvature [30].
in such a way that the properties of the data within the group or cluster are approximately identical. A variety of cluster algorithms exists in the state of the art for this purpose. The task of cluster algorithms is to form clusters within datasets. Clustering methods differ in their approaches and strategies. This allows the separation of the individual objects in the dataset. Due to the homogeneous data available in our dataset, a cluster approach based on Euclidean distance was chosen for the separation of the individual objects. Cluste methods can be not only used for the individual object's separation (such as plastics and PCBs, as shown in Figure 6b), but also can be used for the separation of components on the boards (as shown in Figure 6c).

Rules-Based Classification
The third step of the proposed framework consists in defining rules to process 3D data and spectral data in order to classify individual 3D point clouds.
Based on the existing dataset, it can be stated that the hyperspectral point cloud contains valuable geometric information that is implicitly represented by the spatial arrangement of the individual 3D point clouds. In addition to the geometric information, the dataset also contains valuable spectral information to describe the physical property of the objects.
3D point clouds are a representation of spatial 3D information, which can be described by geometric characteristics. The derivation of these geometric characteristics is done by considering the local neighborhood of each 3D point. Based on this neighborhood, invariant moments can be calculated for each point, representing the geometric properties of local 3D structures [29]. A representation of some 3D features is shown in Figure 7. The 3D features tested in this article are nowadays commonly used in lidar data processing. They are based on normal vectors and eigenvalue-based 3D features like linearity, verticality, planarity, scattering, omnivariance, anisotropy, eigenentropy and change in curvature [30].

Rules-Based Classification
The third step of the proposed framework consists in defining rules to process 3D data and spectral data in order to classify individual 3D point clouds.
Based on the existing dataset, it can be stated that the hyperspectral point cloud contains valuable geometric information that is implicitly represented by the spatial arrangement of the individual 3D point clouds. In addition to the geometric information, the dataset also contains valuable spectral information to describe the physical property of the objects.
3D point clouds are a representation of spatial 3D information, which can be described by geometric characteristics. The derivation of these geometric characteristics is done by considering the local neighborhood of each 3D point. Based on this neighborhood, invariant moments can be calculated for each point, representing the geometric properties of local 3D structures [29]. A representation of some 3D features is shown in Figure 7. The 3D features tested in this article are nowadays commonly used in lidar data processing. They are based on normal vectors and eigenvalue-based 3D features like linearity, verticality, planarity, scattering, omnivariance, anisotropy, eigenentropy and change in curvature [30]. For the characterization of the spectral components, shape-describing features introduced in [31] are used. Materials can be uniquely identified by their spectral signature. Changes in material compositions lead to changes in the spectral signature and conse- For the characterization of the spectral components, shape-describing features introduced in [31] are used. Materials can be uniquely identified by their spectral signature. Changes in material compositions lead to changes in the spectral signature and consequently to local changes in the geometric shape of these spectra. We have shown in [31]  that curvature is a suitable geometric feature and allows a detailed description of spectral signatures. Mathematically, the curvature can be determined by Equation (1) using the 1st and 2nd derivative for each point ( ), ( ) where x characterizes the wavelength (in nm), y the reflectance and t the sample value, respectively.
The approach to determine the significant parameters for the description of the spectral shape is presented in Figure 8. It is based on the computation of extreme values of the 2nd derivative. For a better quantification of absorption peaks, the spectral signatures are normalized by using Continuum Removal [32,33]. In Figure 9, the normalized spectra and the calculated curvatures for polystyrene (PS) and polypropylene (PP) are shown, which express a clear difference in the spectral footprint of both materials. The length of the green and red lines indicates the strength of the curvature. In order to take only significant curvatures into account, a threshold value must be used. The red lines represent the significant bands with a curvature value higher than the defined threshold and are derived on the basis of the curvature points presented in magenta and red colored dots. The direction of the lines (positive or negative) indicates the type of curvature behavior (concave and convex curve) and is also an important source of information to describe the shape of spectral signatures. Building on these extracted parameters regarding the spectra for each material and the 3D features describing the geometric domain, a collection of conditions can be used for classification. The combined use of both domains in particular allows to mutually support the content of each individual domain. For example, 3D features are mainly used in rule formation when the spectral information does not provide enough information (e.g., black objects on PCB). These include, for example, microchips. If the rules in the spectral domain do not produce any is a suitable geometric feature and allows a detailed description of spectral signatures. Mathematically, the curvature can be determined by Equation (1) using the 1st and 2nd derivative for each point P(x(t), y(t)) where x characterizes the wavelength (in nm), y the reflectance and t the sample value, respectively.
Appl. Sci. 2021, 11, x FOR PEER REVIEW that curvature is a suitable geometric feature and allows a detailed descrip tral signatures. Mathematically, the curvature can be determined by Equat the 1st and 2nd derivative for each point ( ), ( ) where x characteriz length (in nm), y the reflectance and t the sample value, respectively.
The approach to determine the significant parameters for the description tral shape is presented in Figure 8. It is based on the computation of extreme 2nd derivative. For a better quantification of absorption peaks, the spectral si normalized by using Continuum Removal [32,33]. In Figure 9, the normalized spectra and the calculated curvatures for poly and polypropylene (PP) are shown, which express a clear difference in the s print of both materials. The length of the green and red lines indicates the st curvature. In order to take only significant curvatures into account, a threshol be used. The red lines represent the significant bands with a curvature value the defined threshold and are derived on the basis of the curvature points magenta and red colored dots. The direction of the lines (positive or negati the type of curvature behavior (concave and convex curve) and is also an impo of information to describe the shape of spectral signatures. Building on the parameters regarding the spectra for each material and the 3D features descr ometric domain, a collection of conditions can be used for classification. Th x .
The approach to determine the significant parameters for the description of the spectral shape is presented in Figure 8. It is based on the computation of extreme values of the 2nd derivative. For a better quantification of absorption peaks, the spectral signatures are normalized by using Continuum Removal [32,33]. that curvature is a suitable geometric feature and allows a detailed description of spectral signatures. Mathematically, the curvature can be determined by Equation (1) using the 1st and 2nd derivative for each point ( ( ), ( )) where x characterizes the wavelength (in nm), y the reflectance and t the sample value, respectively.
The approach to determine the significant parameters for the description of the spectral shape is presented in Figure 8. It is based on the computation of extreme values of the 2nd derivative. For a better quantification of absorption peaks, the spectral signatures are normalized by using Continuum Removal [32,33]. In Figure 9, the normalized spectra and the calculated curvatures for polystyrene (PS) and polypropylene (PP) are shown, which express a clear difference in the spectral footprint of both materials. The length of the green and red lines indicates the strength of the curvature. In order to take only significant curvatures into account, a threshold value must be used. The red lines represent the significant bands with a curvature value higher than the defined threshold and are derived on the basis of the curvature points presented in magenta and red colored dots. The direction of the lines (positive or negative) indicates the type of curvature behavior (concave and convex curve) and is also an important source of information to describe the shape of spectral signatures. Building on these extracted parameters regarding the spectra for each material and the 3D features describing the geometric domain, a collection of conditions can be used for classification. The combined use of both domains in particular allows to mutually support the content of each individual domain. For example, 3D features are mainly used in rule formation when the spectral information does not provide enough information (e.g., black objects on PCB). These include, for example, microchips. If the rules in the spectral domain do not produce any In Figure 9, the normalized spectra and the calculated curvatures for polystyrene (PS) and polypropylene (PP) are shown, which express a clear difference in the spectral footprint of both materials. The length of the green and red lines indicates the strength of the curvature. In order to take only significant curvatures into account, a threshold value must be used. The red lines represent the significant bands with a curvature value higher than the defined threshold and are derived on the basis of the curvature points presented in magenta and red colored dots. The direction of the lines (positive or negative) indicates the type of curvature behavior (concave and convex curve) and is also an important source of information to describe the shape of spectral signatures. Building on these extracted parameters regarding the spectra for each material and the 3D features describing the geometric domain, a collection of conditions can be used for classification. The combined use of both domains in particular allows to mutually support the content of each individual domain. For example, 3D features are mainly used in rule formation when the spectral information does not provide enough information (e.g., black objects on PCB). These include, for example, microchips. If the rules in the spectral domain do not produce any results, geometric rules are then checked. In the case of microchips, which are geometrically rectangular and have a planar surface, the rule is defined as follows.
THEN class → rectangular box cally rectangular and have a planar surface, the rule is defined as follows.
(omnivariance > threshold A) AND (change in curvature < threshold B) THEN class → rectangular box A rule formation should follow a logical approach and be based on the characteris of the objects. As example, in the case of the microchip, the feature 'change in curvat is used to detect the planar surface of the object. This means that the threshold value this characteristic must be set low, because a planar surface has no curvature. The thr olds used in our approach (threshold A = 10 and threshold B = 0.045) were empiric determined by investigations based on the underlying dataset. In the case of a diffe data basis due to different sensor technology, it would only be necessary to adjust thresholds, as the features used reflect the characteristics of the objects. To better illustrate the rule formation in the spectral domain, we refer to the exam of the spectral curves of PP and PS from Figure 9 and express the resulting rule as sho in Listing 1. The rule for each material includes a number of conditions. Each of these condit is based on the Curvature Value (CV) at a specific band and the sign of the curva value. If the sign is negative, the curvature behavior is concave and CV must be sma A rule formation should follow a logical approach and be based on the characteristics of the objects. As example, in the case of the microchip, the feature 'change in curvature' is used to detect the planar surface of the object. This means that the threshold value for this characteristic must be set low, because a planar surface has no curvature. The thresholds used in our approach (threshold A = 10 and threshold B = 0.045) were empirically determined by investigations based on the underlying dataset. In the case of a different data basis due to different sensor technology, it would only be necessary to adjust the thresholds, as the features used reflect the characteristics of the objects.
To better illustrate the rule formation in the spectral domain, we refer to the example of the spectral curves of PP and PS from Figure 9 and express the resulting rule as shown in Listing 1.

Listing 1. Shape-based Rule for PS and PP.
The rule for each material includes a number of conditions. Each of these conditions is based on the Curvature Value (CV) at a specific band and the sign of the curvature value. If the sign is negative, the curvature behavior is concave and CV must be smaller than the defined threshold value. For a convex curvature behavior, the sign is positive, i.e., the curvature value must be greater than the defined threshold value.
As mentioned before, for rule formation, mainly the significant bands with high curvature value are considered (red lines). Each significant band leads to a condition. By analyzing the spectra beforehand, the number of conditions can be reduced to a minimum and not all significant curvature values (red lines) have to be taken into account. Therefore, only six conditions are defined for sample PS in Listing 1, while in Figure 9 a total of seven red lines are present. An overview of the used spectra for the rule-formation is given in Figure 10.
ppl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 19 than the defined threshold value. For a convex curvature behavior, the sign is positive i.e., the curvature value must be greater than the defined threshold value. As mentioned before, for rule formation, mainly the significant bands with high curva ture value are considered (red lines). Each significant band leads to a condition. By analyzing the spectra beforehand, the number of conditions can be reduced to a minimum and no all significant curvature values (red lines) have to be taken into account. Therefore, only six conditions are defined for sample PS in Listing 1, while in Figure 9 a total of seven red lines are present. An overview of the used spectra for the rule-formation is given in Figure 10.

Results
This section shows the steps of the rule-based classification. For a better understanding an overview of the process is shown in Figure 11. It can be understood as an iterative refine ment starting with simple and dominant conditions and continuing in the direction of more complex and composite conditions. The first step of the classification process is based on the knowledge about the spectral behavior of plastics, which is significantly different to those of PCBs. We use this knowledge to separate these two groups and check the average spec trum for each cluster. Since plastic objects are usually made of one material and not of a combination of materials as for PCBs (board and different components), the average spec trum provides a good approach to classify the objects in question (Figure 12). This procedure not only ensures a clean classification of these objects, but also offers advantages in terms o performance, as not every single pixel has to be checked.

Results
This section shows the steps of the rule-based classification. For a better understanding, an overview of the process is shown in Figure 11. It can be understood as an iterative refinement starting with simple and dominant conditions and continuing in the direction of more complex and composite conditions. The first step of the classification process is based on the knowledge about the spectral behavior of plastics, which is significantly different to those of PCBs. We use this knowledge to separate these two groups and check the average spectrum for each cluster. Since plastic objects are usually made of one material and not of a combination of materials as for PCBs (board and different components), the average spectrum provides a good approach to classify the objects in question (Figure 12). This procedure not only ensures a clean classification of these objects, but also offers advantages in terms of performance, as not every single pixel has to be checked.
By assuming that all unclassified objects are objects consisting of composite materials, the classification of clusters can be refined in the next step. The knowledge about the objects allows us to adapt our approach to the characteristics of the objects; in this case, circuit boards. That means, the PCBs are separated into individual parts consisting of the board (a plane surface) and the individual electronic components on the board (ecob). The separation between board and components is based on plane estimation [34], while the components on the board are again processed based on Euclidean distance clustering.
Next, once again, a rule check of the average spectra for each cluster (board and board components) is carried out. As a result (see example shown in Figure 13) the areas that were divided into boards and components can now be subdivided according to their material properties. In the example dataset, we thus obtain a classification of three different board types, three different types of capacitors and a connector consisting of the material polystyrene (PS).
Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19 Figure 11. Steps of the rule-based classification process using spectral and spatial features for the rule formation. The geometrical information is not only used for structuring but also supports the classification of components that cannot be classified from their spectral signature only. By assuming that all unclassified objects are objects consisting of composite materials, the classification of clusters can be refined in the next step. The knowledge about the objects allows us to adapt our approach to the characteristics of the objects; in this case, circuit boards. That means, the PCBs are separated into individual parts consisting of the board (a plane surface) and the individual electronic components on the board (ecob). The separation between board and components is based on plane estimation [34], while the components on the board are again processed based on Euclidean distance clustering. Next, once again, a rule check of the average spectra for each cluster (board and board components) is carried Figure 11. Steps of the rule-based classification process using spectral and spatial features for the rule formation. The geometrical information is not only used for structuring but also supports the classification of components that cannot be classified from their spectral signature only.
Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19 Figure 11. Steps of the rule-based classification process using spectral and spatial features for the rule formation. The geometrical information is not only used for structuring but also supports the classification of components that cannot be classified from their spectral signature only. By assuming that all unclassified objects are objects consisting of composite materials, the classification of clusters can be refined in the next step. The knowledge about the objects allows us to adapt our approach to the characteristics of the objects; in this case, circuit boards. That means, the PCBs are separated into individual parts consisting of the board (a plane surface) and the individual electronic components on the board (ecob). The separation All other existing components do not provide clean spectra that can be used for classification, either because of the black coloring (high absorption) or because of the material composition (metals). However, they are surrounded by detected and classified elements. This relation can be expressed by geometric and topological features allowing to further refine the classification. Figure 14 shows an example of results where a total of four additional groups of components were classified. The four groups include round capacitors, rectangular planar objects (such as microchips), resistors and other components (such as potentiometers and connectors). Especially in this case of classification based on geometric features, it can be seen that the classification of board components is challenging, particularly when dealing with dirty and damaged elements. Even though most of the components have been correctly classified, it can be seen, for example, that one of the connectors has been assigned to the class of radial capacitors (ecob 5). This is due to geometrical similarities shown in Figure 15.  All other existing components do not provide clean spectra that can be used for c sification, either because of the black coloring (high absorption) or because of the mate composition (metals). However, they are surrounded by detected and classified eleme This relation can be expressed by geometric and topological features allowing to furt refine the classification. Figure 14 shows an example of results where a total of four ad tional groups of components were classified. The four groups include round capacit rectangular planar objects (such as microchips), resistors and other components (such potentiometers and connectors). Especially in this case of classification based on geom ric features, it can be seen that the classification of board components is challenging, p ticularly when dealing with dirty and damaged elements. Even though most of the co The smaller the objects are, the more important it is to use 3D sensors that allow a high-resolution detection so that even the smallest objects can still be captured in detail. In this context, it is also important to mention that the sensors chosen for data acquisition should always be selected in accordance with the application. Capturing small objects with a low-resolution 3D scanner will not give satisfactory results. This is also reflected in the statistics computed for each class shown in Table 1. With decreasing object size, an exact classification becomes more difficult. For the calculation of the statistics, the underlying dataset was manually annotated and used for the computation of the confusion matrix.
The overall statistics for the entire dataset are shown in Table 2. It can be stated that with an Overall Accuracy (OA) of 98.24%, a satisfactory result is achieved. The smaller the objects are, the more important it is to use 3D sensors that allow a high-resolution detection so that even the smallest objects can still be captured in detail. In this context, it is also important to mention that the sensors chosen for data acquisition should always be selected in accordance with the application. Capturing small objects with a low-resolution 3D scanner will not give satisfactory results. This is also reflected in the statistics computed for each class shown in Table 1. With decreasing object size, an exact classification becomes more difficult. For the calculation of the statistics, the underlying dataset was manually annotated and used for the computation of the confusion matrix. The overall statistics for the entire dataset are shown in Table 2. It can be stated that with an Overall Accuracy (OA) of 98.24%, a satisfactory result is achieved.  assification for PCBs using 3D features (after step 3). Classification of , ecob 5, ecob 6 and ecob 7).
The smaller the objects are, the more important it i high-resolution detection so that even the smallest objec In this context, it is also important to mention that the se should always be selected in accordance with the appl with a low-resolution 3D scanner will not give satisfactor the statistics computed for each class shown in Table 1 exact classification becomes more difficult. For the calcu lying dataset was manually annotated and used for the c trix. The overall statistics for the entire dataset are show with an Overall Accuracy (OA) of 98.24%, a satisfactory   For comparison purposes, a SVM based classification (C-SVC, RBF-Kernel, C: 2048, gamma: 2.8284), which is one of the most widely used pixel-wise classifiers, was additionally performed. The SVM method was implemented in MATLAB using the LIBSVM library [35]. The optimization of the hyperparameters was determined by a 10-fold crossvalidation. For the training data set, 10% of samples of each class were randomly selected. The main purpose of the comparison is to show, that an approach, combining 3D and HSI, has advantages in terms of classification. The results of the SVM classification are also shown in Table 2. Considering the statistics, it is clearly evident that a classification processing 3D and HSI data outperforms a SVM classification based only on HSI.

Discussion
The results presented in Section 3 show significant added value of a combined use of 3D and HSI for classification of waste PCBs. In order to get the maximum benefit from such a combination, it is important for the strategic approach to analyze the underlying data sufficiently. A well-developed knowledge base with regard to the characteristics of objects helps to improve the results. Especially the addition of the 3D information is helpful in structuring and simplifying the datasets (background removal, removal of shadows, separation between objects). Hyperspectral data, on the other hand, help to determine the type of material and thus refines the classification. Even more important, each of the very different datasets supports the other and compensates the weaknesses of the other. Many of the components that are mounted on circuit boards have a black material color. The use of purely spectral information is not appropriate in these cases due to the high absorption of light. This is also confirmed by the SVM comparison (Table 2), where only HSI data were used for classification. A purely 3D-based classification can be successful in separating objects (e.g., board and components on board), but it would fail, for example, in differentiating between different board materials or compositions.
Especially with 3D data, the resolution and accuracy of the dataset used plays a major role. The higher the resolution, the more accurate the description of very fine, small objects on the board. We see in our experiments that we reach our limits with objects shown in Figure 16 and that a clean separation of small elements such as metals (red and blue box), resistors (green box) and capacitors (yellow box) is not successful. Nevertheless, even if a distinction between these elements does not seem possible in detail, these elements can be identified as components of PCBs through the given topological relationships, which could provide an advantage for further analyses (e.g., determination of different metals).
The classification map shown in Figure 14 is the result of a total of 13 defined rules and works well for the materials and components in the used dataset. An extension to further objects and materials needs the exploitation of their characteristics and the definition of additional rules. This has to be completed carefully in order to avoid conflicting definitions and needs the existence of spatial and spectral characteristics allowing to differentiate further object types. However, the iterative strategy going from more dominant to fewer dominant rules has shown, that even smallest elements can be separated with acceptable accuracy. This is already a reliable base for further extensions. Nevertheless, it is necessary to pay attention to correct modelling, and thus it is important to study the spatial and spectral properties of the objects beforehand and to investigate to what extent the acquired data can make these properties visible. Generally, when setting up rules, it is important to avoid overfitting. Thresholds that are overly sensitive can result in incorrect classifications when applied to new datasets and should therefore be chosen carefully. In addition, the underlying 3D data quality must be considered when forming rules based on 3D geometry. As example, in this article the parameters and thresholds for rule formation were chosen with regard to the used sensor system and the resulting point cloud. In the case of a different 3D sensor with another resolution, accuracy and noise behavior, it would be necessary to adjust the parameters.
Another aspect to consider is the performance of the system and the assessment of use in industrial practice. We are aware that the use in industrial practice poses certain challenges such as high-speed processing, an appropriate integration of the sensor technology into the sorting process and also the handling of dust and dirt in an industrial context is a factor that should not be underestimated. This means that the sensor technology used and the acquisition and processing strategy must meet these requirements. The approach proposed in this article has the potential to fulfil these demands, but certain modifications with regard to the 3D sensor technology are necessary. The strategy of the approach (sensors for acquisition, registration approach and data processing) is general and is mainly affected by the specifications and the capacity of the senor technology used. The structured light scanner used in this article is not advisable in the context of a sorting process due to the needed fast processing speed. However, it has the benefit to produce high quality 3D data, which in turn is advantageous in the context of small objects. The use of such a sensor system would therefore be recommended in the context of randomized quality checks where speed is not important.
However, there are alternatives that are suitable for use in production lines [36,37]. The study of [38] investigates the applicability of 3D sensor technologies in the context of production lines. As a result, four techniques for capturing 3D data in the context of industrial applications were identified, namely laser triangulation, time-of-flight (ToF), shape-from-focus (SFF) and stereovision [38]. Mainly laser scanners based on triangulation are widely used in production lines. Such systems are able to achieve depth resolutions down to the range of a few micrometers and are a robust and standardized technology that could also be used in the context of the proposed approach.
The use of hyperspectral sensors in industrial applications is also conceivable and is particularly widespread in the food industry [39][40][41]. The Specim FX17 sensor used was able to detect 670 FPS at full resolution. It is not always necessary to use the full resolution. By analyzing the data beforehand, spectral ranges or even only a small number of bands can be used, which ultimately leads to a further increase in the detection rate. Important in the context of hyperspectral sensor technology is the illumination unit. This must be installed in the measurement setup in such a way that the 3D technology used is not influenced. on 3D geometry. As example, in this article the parameter mation were chosen with regard to the used sensor system In the case of a different 3D sensor with another resolution it would be necessary to adjust the parameters. Figure 16. Small components such as metals (red and blue box), re (yellow box) are difficult to distinguish because of limits in reso classified as components of a PCB due to the topological relations Another aspect to consider is the performance of the use in industrial practice. We are aware that the use in ind challenges such as high-speed processing, an appropriate nology into the sorting process and also the handling of context is a factor that should not be underestimated. Figure 16. Small components such as metals (red and blue box), resistors (green box) and capacitors (yellow box) are difficult to distinguish because of limits in resolution. Nevertheless, they can be classified as components of a PCB due to the topological relationships provided by the 3D data.
At this point we would also like to briefly discuss the influence of object soiling on the measurements of the hyperspectral sensors, since in practice it is precisely such objects that are involved. Dirt on objects influences the reflectance on the surface of the objects. This inevitably leads to changes and shifts in the spectral signatures as can be seen in Figure 17 for one of the samples in the used dataset. That our proposed approach is robust to such influences can be seen from the achieved results. This is due to two main facts: First, we use spectral features that describe the shape of spectral signatures, even if the expression of absorption or reflectance peaks is low due to the weak reflectance, the shape of the spectral signature remains the same. Thus, the defined rules based on the shape apply in both cases. The second point is due to the 3D component. Through the 3D information we can group components in clusters and for the rule check we refer to the average spectral signature of this cluster. Thus, the influence of dirty areas is only significant if the object is completely soiled.
The processing time of the data is also a factor that needs to be taken into account. The 3D processing and point-by-point checking for conditions using MATLAB on a machine with an Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz and 16 GB RAM was 110.29 s for the background removed dataset (752.253 3D points and 224 bands) shown in Figure 4. For comparison, processing the same dataset (only HSI) on the same machine using SVM (LIBSVM for MATLAB) requires 2025.07 s.
For a practical application, 110.29 s is considered high. However, in this context it must also be considered that a full spectral and high-resolution 3D point cloud serves as the data basis. Optimizations in terms of processing time is conceivable and could be achieved by reducing the data basis (e.g., reduction of spectral resolution), by using more powerful hardware and by more efficient programming (e.g., parallel computing). The processing time of the data is also a factor that needs to be taken into account. The 3D processing and point-by-point checking for conditions using MATLAB on a machine with an Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz and 16 GB RAM was 110.29 s for the background removed dataset (752.253 3D points and 224 bands) shown in Figure 4. For comparison, processing the same dataset (only HSI) on the same machine using SVM (LIBSVM for MATLAB) requires 2025.07 s.

Conclusions
In this paper, a combined dataset consisting of a 3D point cloud and an HSI was used to classify different types of materials and objects of electronic waste. In addition to the classification of different plastic materials, it has been shown that a combination of geometric and physical information can also help to distinguish components on circuit boards, such as capacitors, transistors and other small parts on board. Using a rule-based classification approach leads to satisfying results with an OA of 98.24%. The comparison with an SVM approach based only on spectral information results in an OA of 88.65% and confirms that a combined dataset helps to improve the quality of classification. Funding: This research was funded by the European Union from the European Regional Development Fund and the state of Rhineland-Palatinate.

Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.