Deep-Learning-Based Three-Dimensional Detection of Individual Wave-Dissipating Blocks from As-Built Point Clouds Measured by UAV Photogrammetry and Multibeam Echo-Sounder
Abstract
:1. Introduction
- The 6D poses of individual blocks faithfully reproduce the as-built status of each block to improve the accuracy of estimates of new blocks’ quantities and their stacking plan in the supplemental work;
- The as-built status can be grasped block by block after the construction and supplemental works. Thus, the construction results can be recorded and visualized comprehensively compared to recording only the measured point clouds of existing block surfaces;
- By providing the pose and attribute information to each block model, it is possible to check the long-term change in the blocks, such as missing, sinking, and damaged blocks, and implement a more precise and sustainable maintenance activity.
- The originally proposed convolutional neural network called FPCCv2 enables the rapid segmentation of individual wave-dissipating block instances from a large-scale 3D measured point cloud captured from a scene of stacked blocks. This enables us to estimate 6D poses of multiple blocks at once and improve computational efficiency of the block pose estimation.
- A physics engine enables the synthetic and automatic generation of instance-labeled training datasets for the instance segmentation of blocks. It thus avoids laborious manual labeling work and secures rich training datasets for our convolutional neural network.
- Synthetic point cloud generation considering the difference in characteristics of measurement using UAV and MBES enhances the performance of instance segmentation.
- The combination of the 3D feature descriptor by PPF and point-to-point registration by ICP enables the precise estimation of 6D poses of individual blocks in a scene. Moreover, the difference in the type and size of individual blocks can be identified in a scene. This is useful for the as-built inspection and instance-level monitoring of wave-dissipating blocks.
- The performances of 6D pose estimation of individual wave-dissipating blocks are evaluated both in synthetic scenes and various real construction sites including undersea blocks.
2. Related Work
2.1. Three-Dimensional (3D) Monitoring of Wave-Dissipating Blocks in Breakwaters
2.2. Instance Segmentation on Point Cloud
2.3. Model-Based 3D Object Detection and 6D Pose Estimation
3. Overview of the Processing Pipeline
- First, the category-agnostic instance segmentation network FPCCv2 segments the input point cloud measured by UAV and MBES into the subsets of points corresponding to individual block instances. FPCCv2 is a kind of deep neural network, which is pre-trained by synthetic point clouds that mimic the point clouds of stacked blocks measured by UAV and MBES, respectively, using the stacked block CAD models and surface point sampling. The detailed algorithms are described in Section 4.
- Second, the 6D pose of an individual block is estimated from each segmented point cloud using a conventional descriptor-based 3D object detection algorithm that makes use of PPF [15] and ICP [16]. The detailed algorithm is described in Section 5.1.
- Finally, if the scene consists of multiple typed blocks, a fitness score corresponding to each type is calculated for each segmented point cloud to identify the type of detected individual block. The detailed algorithm is shown in Section 5.2.
4. Instance Segmentation of Measured Point Cloud Based on Deep Neural Network
4.1. FPCC and Its Limitation
4.2. Proposed Feature Extractor for FPCCv2
4.3. Generation of Data for Instance Segmentation
- A triangular mesh model of a wave-dissipating block is created from the surface tessellation of its 3D CAD model.
- Subsequently, a point set is densely sampled on every triangle face of the mesh model .
- A variety of penetration-free stacking poses of piled blocks are generated using a set of triangular mesh model instances for the model . Furthermore, a set of sampled points on blocks in stacking poses is calculated for each block as on the model instances , respectively.
- A subset of the sample points on the block model in the stacking poses is picked up as that are only visible from a given position of the measurement device.
- For every point , Gaussian noise at a certain level is imposed on the coordinates of q, which simulates the possible accidental error induced from the measurement device to create the final synthetic point cloud. The point cloud is used to train our instance segmentation network (FPCCv2).
5. Block Pose Estimation and Type Classification
5.1. Block 6D Pose Estimation Using 3D Feature Descriptor
5.2. Pose Refinement and Block-Type Classification
6. Results of Wave-Dissipating Block Detection
6.1. Experimental Sites
6.2. Block Instance Segmentation
6.2.1. Experimental Setting of CNN
6.2.2. Precision and Recall
6.3. Block Pose Estimation
6.3.1. Synthetic Data Set Creation
6.3.2. Pose Estimation Accuracy Using Synthetic Data Set
6.3.3. Pose Estimation Accuracy Using Real Scene Data
6.4. Block-Type Classification
6.5. Processing Time
7. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Appendix A
References
- Bueno, M.; Díaz-Vilariño, L.; Martínez-Sánchez, J.; González Jorge, H.; Arias, P. 3D reconstruction of cubic armoured rubble mound breakwaters from incomplete lidar data. Int. J. Remote Sens. 2015, 36, 5485–5503. [Google Scholar] [CrossRef]
- González-Jorge, H.; Puente, I.; Roca, D.; Martínez-Sánchez, J.; Conde, B.; Arias, P. UAV Photogrammetry Application to the Monitoring of Rubble Mound Breakwaters. J. Perform. Constr. Facil. 2016, 30, 04014194. [Google Scholar] [CrossRef]
- Stateczny, A.; Błaszczak-Bąk, W.; Sobieraj-Żłobińska, A.; Motyl, W.; Wisniewska, M. Methodology for Processing of 3D Multibeam Sonar Big Data for Comparative Navigation. Remote Sens. 2019, 11, 2245. [Google Scholar] [CrossRef] [Green Version]
- Kulawiak, M.; Lubniewski, Z. Processing of LiDAR and Multibeam Sonar Point Cloud Data for 3D Surface and Object Shape Reconstruction. In Proceedings of the 2016 Baltic Geodetic Congress (BGC Geomatics), Gdansk, Poland, 2 June 2016; pp. 187–190. [Google Scholar] [CrossRef]
- Alevizos, E.; Oikonomou, D.; Argyriou, A.V.; Alexakis, D.D. Fusion of Drone-Based RGB and Multi-Spectral Imagery for Shallow Water Bathymetry Inversion. Remote Sens. 2022, 14, 1127. [Google Scholar] [CrossRef]
- Wang, D.; Xing, S.; He, Y.; Yu, J.; Xu, Q.; Li, P. Evaluation of a New Lightweight UAV-Borne Topo-Bathymetric LiDAR for Shallow Water Bathymetry and Object Detection. Sensors 2022, 22, 1379. [Google Scholar] [CrossRef]
- Dąbrowski, P.S.; Specht, C.; Specht, M.; Burdziakowski, P.; Makar, A.; Lewicka, O. Integration of Multi-Source Geospatial Data from GNSS Receivers, Terrestrial Laser Scanners, and Unmanned Aerial Vehicles. Can. J. Remote Sens. 2021, 47, 621–634. [Google Scholar] [CrossRef]
- Sousa, P.J.; Cachaço, A.; Barros, F.; Tavares, P.J.; Moreira, P.M.; Capitão, R.; Neves, M.G.; Franco, E. Structural monitoring of a breakwater using UAVs and photogrammetry. Procedia Struct. Integr. 2022, 37, 167–172. [Google Scholar] [CrossRef]
- Lemos, R.; Loja, M.A.; Rodrigues, J.; Rodrigues, J.A. Photogrammetric analysis of rubble mound breakwaters scale model tests. AIMS Environ. Sci. 2016, 3, 541–559. [Google Scholar] [CrossRef]
- Puente, I.; Sande, J.; González-Jorge, H.; Peña-González, E.; Maciñeira, E.; Martínez-Sánchez, J.; Arias, P. Novel image analysis approach to the terrestrial LiDAR monitoring of damage in rubble mound breakwaters. Ocean Eng. 2014, 91, 273–280. [Google Scholar] [CrossRef]
- Gonçalves, D.; Gonçalves, G.; Pérez-Alvávez, J.A.; Andriolo, U. On the 3D Reconstruction of Coastal Structures by Unmanned Aerial Systems with Onboard Global Navigation Satellite System and Real-Time Kinematics and Terrestrial Laser Scanning. Remote Sens. 2022, 14, 1485. [Google Scholar] [CrossRef]
- Musumeci, R.E.; Moltisanti, D.; Foti, E.; Battiato, S.; Farinella, G.M. 3-D monitoring of rubble mound breakwater damages. Measurement 2018, 117, 347–364. [Google Scholar] [CrossRef] [Green Version]
- Shen, Y.; Lindenbergh, R.; Wang, J.; Ferreira, V.G. Extracting Individual Bricks from a Laser Scan Point Cloud of an Unorganized Pile of Bricks. Remote Sens. 2018, 10, 1709. [Google Scholar] [CrossRef] [Green Version]
- Xu, Y.; Arai, S.; Liu, D.; Lin, F.; Kosuge, K. FPCC: Fast point cloud clustering-based instance segmentation for industrial bin-picking. Neurocomputing 2022, 494, 255–268. [Google Scholar] [CrossRef]
- Drost, B.; Ulrich, M.; Navab, N.; Ilic, S. Model globally, match locally: Efficient and robust 3D object recognition. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13 June 2010; pp. 998–1005. [Google Scholar] [CrossRef] [Green Version]
- Besl, P.J.; McKay, N.D. Method for registration of 3-D shapes. In Sensor Fusion IV: Control Paradigms and Data Structures; Schenker, P.S., Ed.; International Society for Optics and Photonics, SPIE: Boston, MA, USA, 1992; Volume 1611, pp. 586–606. [Google Scholar] [CrossRef]
- Coumans, E.; Bai, Y. PyBullet, a Python Module for Physics Simulation for Games, Robotics and Machine Learning. 2016–2021. Available online: http://pybullet.org (accessed on 9 September 2022).
- Tulsi, K.; Phelp, D. Monitoring and Maintenance of Breakwaters Which Protect Port Entrances. 2009. Available online: http://hdl.handle.net/10204/4139 (accessed on 30 October 2022).
- Campos, Á.; Castillo, C.; Molina-Sanchez, R. Damage in Rubble Mound Breakwaters. Part I: Historical Review of Damage Models. J. Mar. Sci. Eng. 2020, 8, 317. [Google Scholar] [CrossRef]
- Campos, Á.; Molina-Sanchez, R.; Castillo, C. Damage in Rubble Mound Breakwaters. Part II: Review of the Definition, Parameterization, and Measurement of Damage. J. Mar. Sci. Eng. 2020, 8, 306. [Google Scholar] [CrossRef]
- Lemos, R.; Reis, M.T.; Fortes, C.J.; Peña, E.; Sande, J.; Figuero, A.; Alvarellos, A.; Laino, E.; Santos, J.; Kerpen, N.B. Measuring Armour Layer Damage in Rubble-Mound Breakwaters under Oblique Wave Incidence. 2019. Available online: https://henry.baw.de/handle/20.500.11970/106641 (accessed on 30 October 2022).
- Bueno, M.; Díaz-Vilariño, L.; González-Jorge, H.; Martínez-Sánchez, J.; Arias, P. Automatic modelling of rubble mound breakwaters from lidar data. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2015, 40, 9. [Google Scholar] [CrossRef] [Green Version]
- Wang, W.; Yu, R.; Huang, Q.; Neumann, U. SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake, UT, USA, 18 June 2018; pp. 2569–2578. [Google Scholar] [CrossRef]
- Lahoud, J.; Ghanem, B.; Oswald, M.R.; Pollefeys, M. 3D Instance Segmentation via Multi-Task Metric Learning. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October 2019; pp. 9255–9265. [Google Scholar] [CrossRef] [Green Version]
- Pham, Q.; Nguyen, T.; Hua, B.; Roig, G.; Yeung, S. JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15 June 2019; pp. 8819–8828. [Google Scholar] [CrossRef] [Green Version]
- Wang, X.; Liu, S.; Shen, X.; Shen, C.; Jia, J. Associatively Segmenting Instances and Semantics in Point Clouds. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15 June 2019; pp. 4091–4100. [Google Scholar] [CrossRef] [Green Version]
- Xu, Y.; Arai, S.; Tokuda, F.; Kosuge, K. A Convolutional Neural Network for Point Cloud Instance Segmentation in Cluttered Scene Trained by Synthetic Data Without Color. IEEE Access 2020, 8, 70262–70269. [Google Scholar] [CrossRef]
- Jiang, L.; Zhao, H.; Shi, S.; Liu, S.; Fu, C.W.; Jia, J. PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13 June 2020; pp. 4866–4875. [Google Scholar] [CrossRef]
- Chen, S.; Fang, J.; Zhang, Q.; Liu, W.; Wang, X. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Online, 10 March 2021; pp. 15467–15476. [Google Scholar]
- Armeni, I.; Sener, O.; Zamir, A.R.; Jiang, H.; Brilakis, I.; Fischer, M.; Savarese, S. 3D Semantic Parsing of Large-Scale Indoor Spaces. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27 June 2016; pp. 1534–1543. [Google Scholar] [CrossRef]
- Dai, A.; Chang, A.X.; Savva, M.; Halber, M.; Funkhouser, T.; Nießner, M. ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21 July 2017; pp. 2432–2443. [Google Scholar] [CrossRef] [Green Version]
- Hua, B.; Pham, Q.; Nguyen, D.T.; Tran, M.; Yu, L.; Yeung, S. SceneNN: A Scene Meshes Dataset with aNNotations. In Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA, 25 October 2016; pp. 92–101. [Google Scholar] [CrossRef]
- Zhang, F.; Guan, C.; Fang, J.; Bai, S.; Yang, R.; Torr, P.H.; Prisacariu, V. Instance Segmentation of LiDAR Point Clouds. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May 2020; pp. 9448–9455. [Google Scholar] [CrossRef]
- Walicka, A.; Pfeifer, N. Automatic Segmentation of Individual Grains From a Terrestrial Laser Scanning Point Cloud of a Mountain River Bed. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2022, 15, 1389–1410. [Google Scholar] [CrossRef]
- Luo, H.; Khoshelham, K.; Chen, C.; He, H. Individual tree extraction from urban mobile laser scanning point clouds using deep pointwise direction embedding. ISPRS J. Photogramm. Remote. Sens. 2021, 175, 326–339. [Google Scholar] [CrossRef]
- Djuricic, A.; Dorninger, P.; Nothegger, C.; Harzhauser, M.; Székely, B.; Rasztovits, S.; Mandic, O.; Molnár, G.; Pfeifer, N. High-resolution 3D surface modeling of a fossil oyster reef. Geosphere 2016, 12, 1457–1477. [Google Scholar] [CrossRef] [Green Version]
- Salti, S.; Tombari, F.; Di Stefano, L. SHOT: Unique signatures of histograms for surface and texture description. Comput. Vis. Image Underst. 2014, 125, 251–264. [Google Scholar] [CrossRef]
- Rusu, R.B.; Blodow, N.; Marton, Z.C.; Beetz, M. Aligning point cloud views using persistent feature histograms. In Proceedings of the 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice, France, 22 September 2008; pp. 3384–3391. [Google Scholar] [CrossRef]
- Abbeloos, W.; Goedemé, T. Point Pair Feature Based Object Detection for Random Bin Picking. In Proceedings of the 2016 13th Conference on Computer and Robot Vision (CRV), Victoria, BC, Canada, 1 June 2016; pp. 432–439. [Google Scholar] [CrossRef] [Green Version]
- Hinterstoisser, S.; Lepetit, V.; Rajkumar, N.; Konolige, K. Going Further with Point Pair Features. In Computer Vision—ECCV 2016; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 834–848. [Google Scholar]
- Liu, D.; Arai, S.; Miao, J.; Kinugawa, J.; Wang, Z.; Kosuge, K. Point Pair Feature-Based Pose Estimation with Multiple Edge Appearance Models (PPF-MEAM) for Robotic Bin Picking. Sensors 2018, 18, 2719. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Birdal, T.; Ilic, S. Point Pair Features Based Object Detection and Pose Estimation Revisited. In Proceedings of the 2015 International Conference on 3D Vision, Lyon, France, 19 October 2015; pp. 527–535. [Google Scholar] [CrossRef]
- Xiang, Y.; Schmidt, T.; Narayanan, V.; Fox, D. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. 2018. Available online: https://github.com/yuxng/PoseCNN (accessed on 30 October 2022).
- Wang, C.; Xu, D.; Zhu, Y.; Martín-Martín, R.; Lu, C.; Fei-Fei, L.; Savarese, S. DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15 June 2019; pp. 3338–3347. [Google Scholar] [CrossRef] [Green Version]
- Dong, Z.; Liu, S.; Zhou, T.; Cheng, H.; Zeng, L.; Yu, X.; Liu, H. PPR-Net:Point-wise Pose Regression Network for Instance Segmentation and 6D Pose Estimation in Bin-picking Scenarios. In Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, 3 November 2019; pp. 1773–1780. [Google Scholar] [CrossRef]
- Zeng, L.; Lv, W.J.; Dong, Z.K.; Liu, Y.J. PPR-Net++: Accurate 6-D Pose Estimation in Stacked Scenarios. IEEE Trans. Autom. Sci. Eng. 2021, 19, 3139–3151. [Google Scholar] [CrossRef]
- Deng, X.; Mousavian, A.; Xiang, Y.; Xia, F.; Bretl, T.; Fox, D. PoseRBPF: A Rao–Blackwellized Particle Filter for 6-D Object Pose Tracking. IEEE Trans. Robot. 2021, 37, 1328–1342. [Google Scholar] [CrossRef]
- Deng, X.; Xiang, Y.; Mousavian, A.; Eppner, C.; Bretl, T.; Fox, D. Self-supervised 6D Object Pose Estimation for Robot Manipulation. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May 2020; pp. 3665–3671. [Google Scholar] [CrossRef]
- Yin, Y.; Cai, Y.; Wang, H.; Chen, B. FisherMatch: Semi-Supervised Rotation Regression via Entropy-based Filtering. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022. [Google Scholar] [CrossRef]
- Wang, Y.; Sun, Y.; Liu, Z.; Sarma, S.E.; Bronstein, M.M.; Solomon, J.M. Dynamic Graph CNN for Learning on Point Clouds. ACM Trans. Graph. 2019, 38, 1–12. [Google Scholar] [CrossRef] [Green Version]
- Liu, D.; Arai, S.; Xu, Y.; Tokuda, F.; Kosuge, K. 6D Pose Estimation of Occlusion-Free Objects for Robotic Bin-Picking using PPF-MEAM with 2D Images (Occlusion-Free PPF-MEAM). IEEE Access 2021, 9, 50857–50871. [Google Scholar] [CrossRef]
- Li, G.; Mueller, M.; Qian, G.; Delgadillo Perez, I.C.; Abualshour, A.; Thabet, A.K.; Ghanem, B. DeepGCNs: Making GCNs Go as Deep as CNNs. IEEE Trans. Pattern Anal. Mach. Intell. 2021. [Google Scholar] [CrossRef]
- Luo, W.; Li, Y.; Urtasun, R.; Zemel, R. Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems; Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R., Eds.; Curran Associates, Inc.: Barcelona, Spain, 2016; Volume 29. [Google Scholar]
- Engelmann, F.; Kontogianni, T.; Leibe, B. Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May 2020; pp. 9463–9469. [Google Scholar] [CrossRef]
- Qi, C.R.; Yi, L.; Su, H.; Guibas, L.J. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17), Long Beach, CA, USA, 4 December 2017; Curran Associates Inc.: Red Hook, NY, USA, 2017; pp. 5105–5114. [Google Scholar]
- Li, R.; Wang, S.; Zhu, F.; Huang, J. Adaptive Graph Convolutional Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2 February 2018; Volume 32. [Google Scholar]
- Hu, Q.; Yang, B.; Xie, L.; Rosa, S.; Guo, Y.; Wang, Z.; Trigoni, N.; Markham, A. RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13 June 2020; pp. 11105–11114. [Google Scholar] [CrossRef]
- Yang, B.; Wang, S.; Markham, A.; Trigoni, N. Robust attentional aggregation of deep feature sets for multi-view 3D reconstruction. Int. J. Comput. Vis. 2020, 128, 53–73. [Google Scholar] [CrossRef] [Green Version]
- Tobin, J.; Fong, R.; Ray, A.; Schneider, J.; Zaremba, W.; Abbeel, P. Domain randomization for transferring deep neural networks from simulation to the real world. In Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 24 September 2017; pp. 23–30. [Google Scholar] [CrossRef]
- Planche, B.; Wu, Z.; Ma, K.; Sun, S.; Kluckner, S.; Lehmann, O.; Chen, T.; Hutter, A.; Zakharov, S.; Kosch, H.; et al. DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition. In Proceedings of the 2017 International Conference on 3D Vision (3DV), Qingdao, China, 10 October 2017; pp. 1–10. [Google Scholar] [CrossRef] [Green Version]
- Tremblay, J.; Prakash, A.; Acuna, D.; Brophy, M.; Jampani, V.; Anil, C.; To, T.; Cameracci, E.; Boochoon, S.; Birchfield, S. Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake, UT, USA, 18 June 2018; pp. 1082–10828. [Google Scholar] [CrossRef] [Green Version]
- Katz, S.; Tal, A.; Basri, R. Direct Visibility of Point Sets. In ACM SIGGRAPH 2007 Papers; SIGGRAPH ’07; Association for Computing Machinery: New York, NY, USA, 2007; p. 24. [Google Scholar] [CrossRef]
- Brégier, R.; Devernay, F.; Leyrit, L.; Crowley, J.L. Defining the pose of any 3d rigid object and an associated distance. Int. J. Comput. Vis. 2018, 126, 571–596. [Google Scholar] [CrossRef] [Green Version]
- Johnson, A.; Hebert, M. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 1999, 21, 433–449. [Google Scholar] [CrossRef]
Site | Sawara | Todohokke | Era | |||
---|---|---|---|---|---|---|
Method of measurement | UAV | MBES | UAV | MBES | UAV | MBES |
Size of test region (m) | ||||||
The number of test regions | 14 | 15 | 20 | 10 | 6 | 7 |
Point density (pts/) | 600 | 260 | 370 | 250 | 580 | 800 |
Block type used | A1 | A1 | B1 | B1 + B2 | A2 + A3 | A2 + A3 |
Approx. amount of points in a region | 100,000 120,000 | 30,000 35,000 | 70,000 100,000 | 65,000 80,000 | 400,000 700,000 | 700,000 1,000,000 |
Sawara | Todohokke | Era | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
UAV | MBES | UAV | MBES | UAV | MBES | ||||||
Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall |
80.05 | 55.25 | 90.38 | 69.70 | 82.70 | 74.24 | 72.72 | 75.78 | 88.60 | 49.47 | 86.32 | 68.23 |
Sawara | Todohokke | Era | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
UAV | MBES | UAV | MBES | UAV | MBES | ||||||
Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall |
99.3 | 79.4 | 83.2 | 85.4 | 96.1 | 95.5 | 86.1 | 77.9 | 98.6 | 97.0 | 97.9 | 95.2 |
Sawara | Todohokke | Era | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
UAV | MBES | UAV | MBES | UAV | MBES | ||||||
30 | 0.8 | 31 | 2.3 | 23 | 1.5 | 31 | 2.4 | 29 | 0.5 | 39 | 1.1 |
Sawara | Todohokke | Era | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
UAV | MBES | UAV | MBES | UAV | MBES | ||||||
Fitting Error | Matching Rate | Fitting Error | Matching Rate | Fitting Error | Matching Rate | Fitting Error | Matching Rate | Fitting Error | Matching Rate | Fitting Error | Matching Rate |
33 | 71.32 | 39 | 67.28 | 34 | 66.17 | 38 | 60.54 | 48 | 75.12 | 58 | 71.52 |
Site | Sawara | Todohokke | ||
---|---|---|---|---|
Block type | B1 | B2 | A2 | A3 |
Actual block number | 77 | 58 | 111 | 70 |
Misclassified block number | 4 | 5 | 3 | 3 |
Accuracy | 94% | 96% |
Site | Sawara | Todohokke | Era | |||
---|---|---|---|---|---|---|
Measurement method | UAV | MBES | UAV | MBES | UAV | MBES |
Size of test region (m) | ||||||
The number of generated synthetic point cloud scenes | 500 | 500 | 500 | 500 | 500 | 500 |
Block-type used | A1 | A1 | B1 | B1 + B2 | A2 + A3 | A2 + A3 |
Number of blocks/region | 50 60 | 50 60 | 50 60 | 50 60 | 50 60 | 50 60 |
Epoch | 60 | 60 | 60 | 60 | 60 | 60 |
Training time (h) | 30 | 30 | 30 | 30 | 30 | 30 |
Site | Sawara | Todohokke | Era | |||
---|---|---|---|---|---|---|
Measurement method | UAV | MBES | UAV | MBES | UAV | MBES |
Size of test region (m) | ||||||
Number of test regions | 14 | 15 | 20 | 10 | 6 | 7 |
Number of blocks in a region | 40 50 | 35 50 | 25 35 | 25 30 | 30 50 | 30 40 |
Point density (pts/) | 600 | 260 | 370 | 250 | 580 | 800 |
Block-type used | A1 | A1 | B1 | B1 + B2 | A2 + A3 | A2 + A3 |
Approximate number of points in a region | 100,000 120,000 | 30,000 35,000 | 70,000 100,000 | 65,000 80,000 | 400,000 700,000 | 700,000 1,000,000 |
Time for block instance segmentation (s)/region | 0.8 | 0.4 | 0.7 | 0.5 | 3.6 | 4.2 |
Time for block pose estimation (s)/region | 127 | 72 | 130 | 85 | 175 | 162 |
Time for block-type classification (s)/region | 32 | 24 | 35 | 26 | 160 | 158 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Xu, Y.; Kanai, S.; Date, H.; Sano, T. Deep-Learning-Based Three-Dimensional Detection of Individual Wave-Dissipating Blocks from As-Built Point Clouds Measured by UAV Photogrammetry and Multibeam Echo-Sounder. Remote Sens. 2022, 14, 5575. https://doi.org/10.3390/rs14215575
Xu Y, Kanai S, Date H, Sano T. Deep-Learning-Based Three-Dimensional Detection of Individual Wave-Dissipating Blocks from As-Built Point Clouds Measured by UAV Photogrammetry and Multibeam Echo-Sounder. Remote Sensing. 2022; 14(21):5575. https://doi.org/10.3390/rs14215575
Chicago/Turabian StyleXu, Yajun, Satoshi Kanai, Hiroaki Date, and Tomoaki Sano. 2022. "Deep-Learning-Based Three-Dimensional Detection of Individual Wave-Dissipating Blocks from As-Built Point Clouds Measured by UAV Photogrammetry and Multibeam Echo-Sounder" Remote Sensing 14, no. 21: 5575. https://doi.org/10.3390/rs14215575