Semi-Supervised Training for Positioning of Welding Seams
Abstract
:1. Introduction
2. Related Work
2.1. Machine Learning in Welding
2.2. Keypoint Localization
2.3. Semi-Supervised Learning
3. Proposed Method
3.1. Overview
3.2. Heatmap Generator
Zoom-In Attention Area
3.3. Double Discriminators
3.3.1. Cutout Discriminator
3.3.2. Crop Discriminator
3.4. Overall Semi-Supervised Algorithm
Algorithm 1. Overall Algorithm. |
Require: Labeled data |
Require: Unlabeled data |
Require: Heatmap generator ψ1,2 with θ1,2 |
Require: Discriminator Dcutout with θcutout |
Require: Discriminator Dcrop with θcrop |
Initialize θ1,2 by minimizing Equation (3) on L |
Initialize θcutout and θcrop by minimizing Equation (6) on L |
Threshold |
Threshold |
maximum # steps |
for j = 1 to J do |
Predict on U using ψ1,2, and denote U with its pseudo labels as U’ |
Compute the confidence of each prediction for U’ using Dcutout and Dcrop, respectively |
qualified samples from U’ determined by Dcutout and Dcrop with tcutout and tcrop, respectively. |
Retrainby θ1,2 on by minimizing Equation (3) |
end for |
return Generator ψ1,2 with optimized parameters θ1,2 |
4. Experiments and Results
4.1. Data Description
4.2. Experimental Settings
4.2.1. Training the Heatmap Generator
4.2.2. Training the Two Discriminators
4.2.3. Metric and Evaluation
4.3. Results and Discussion
4.3.1. Comparison with Supervised Learning
4.3.2. Comparison with Different Number of Labeled Data
4.3.3. Comparison with Random Labeled Data
4.3.4. Run-Time
4.3.5. Comparison with the State-of-the-Art in Semi-Supervised Methods
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Fan, J.; Deng, S.; Jing, F.; Zhou, C.; Yang, L.; Long, T.; Tan, M. An Initial Point Alignment and Seam-Tracking System for Narrow Weld. IEEE Trans. Ind. Inform. 2019, 16, 877–886. [Google Scholar] [CrossRef]
- Shirmohammadi, S.; Ferrero, A. Camera as the instrument: The rising trend of vision based measurement. IEEE Instrum. Meas. Mag. 2014, 17, 41–47. [Google Scholar] [CrossRef]
- Maldonado-Ramirez, A.; Rios-Cabrera, R.; Lopez-Juarez, I. A visual path-following learning approach for industrial robots using DRL. Robot.-Comput.-Integr. Manuf. 2021, 71, 102130. [Google Scholar] [CrossRef]
- Xu, D.; Fang, Z.; Chen, H.; Yan, Z.; Tan, M. Compact visual control system for aligning and tracking narrow butt seams with CO2 gas-shielded arc welding. Int. J. Adv. Manuf. Technol. 2012, 62, 1157–1167. [Google Scholar] [CrossRef]
- Li, Y.; Li, Y.F.; Wang, Q.L.; Xu, D.; Tan, M. Measurement and defect detection of the weld bead based on online vision inspection. IEEE Trans. Instrum. Meas. 2009, 59, 1841–1849. [Google Scholar]
- Rout, A.; Deepak, B.; Biswal, B. Advances in weld seam tracking techniques for robotic welding: A review. Robot.-Comput.-Integr. Manuf. 2019, 56, 12–37. [Google Scholar] [CrossRef]
- Xu, Y.; Wang, Z. Visual sensing technologies in robotic welding: Recent research developments and future interests. Sens. Actuators A Phys. 2021, 320, 112551. [Google Scholar] [CrossRef]
- Zou, Y.; Lan, R. An End-to-End Calibration Method for Welding Robot Laser Vision Systems with Deep Reinforcement Learning. IEEE Trans. Instrum. Meas. 2019, 69, 4270–4280. [Google Scholar] [CrossRef]
- Zou, Y.; Chen, J.; Wei, X. A calibration optimization method for a welding robot laser vision system based on generative adversarial network. IEEE Trans. Instrum. Meas. 2021, 70, 1–13. [Google Scholar]
- Yang, L.; Fan, J.; Liu, Y.; Li, E.; Peng, J.; Liang, Z. Automatic Detection and Location of Weld Beads with Deep Convolutional Neural Networks. IEEE Trans. Instrum. Meas. 2020, 70, 1–12. [Google Scholar] [CrossRef]
- Liu, H.; Yan, Y.; Song, K.; Chen, H.; Yu, H. Efficient Optical Measurement of Welding Studs with Normal Maps and Convolutional Neural Network. IEEE Trans. Instrum. Meas. 2020, 70, 1–14. [Google Scholar] [CrossRef]
- Wang, L.; Shen, Q. Visual Inspection of Welding Zone by Boundary-aware Semantic Segmentation Algorithm. IEEE Trans. Instrum. Meas. 2020, 70, 1–9. [Google Scholar] [CrossRef]
- Zhou, Z.H. A brief introduction to weakly supervised learning. Natl. Sci. Rev. 2018, 5, 44–53. [Google Scholar] [CrossRef] [Green Version]
- Moskvyak, O.; Maire, F.; Dayoub, F.; Baktashmotlagh, M. Semi-supervised Keypoint Localization. In Proceedings of the International Conference on Learning Representations, ICLR, Virtual Event, 3–7 May 2021. [Google Scholar]
- Kostopoulos, G.; Karlos, S.; Kotsiantis, S.; Ragos, O. Semi-supervised regression: A recent review. J. Intell. Fuzzy Syst. 2018, 35, 1483–1500. [Google Scholar] [CrossRef]
- Zhang, L.; Ye, Q.; Yang, W.; Jiao, J. Weld line detection and tracking via spatial-temporal cascaded hidden Markov models and cross structured light. IEEE Trans. Instrum. Meas. 2013, 63, 742–753. [Google Scholar] [CrossRef]
- Farhadi, A.; Redmon, J. Yolov3: An incremental improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Newell, A.; Yang, K.; Deng, J. Stacked hourglass networks for human pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 11–14 October 2016; pp. 483–499. [Google Scholar]
- Yang, J.; Liu, Q.; Zhang, K. Stacked hourglass network for robust facial landmark localisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017; pp. 79–87. [Google Scholar]
- Law, H.; Deng, J. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 734–750. [Google Scholar]
- Huang, J.; Zhu, Z.; Huang, G. Multi-Stage HRNet: Multiple Stage High-Resolution Network for Human Pose Estimation. arXiv 2019, arXiv:1910.05901. [Google Scholar]
- Sun, K.; Xiao, B.; Liu, D.; Wang, J. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 5693–5703. [Google Scholar]
- Shi, H.; Wang, Z. Improved Stacked Hourglass Network with Offset Learning for Robust Facial Landmark Detection. In Proceedings of the 2019 9th International Conference on Information Science and Technology (ICIST), Hulunbuir, China, 2–5 August 2019; pp. 58–64. [Google Scholar]
- Munea, T.L.; Jembre, Y.Z.; Weldegebriel, H.T.; Chen, L.; Huang, C.; Yang, C. The progress of human pose estimation: A survey and taxonomy of models applied in 2D human pose estimation. IEEE Access 2020, 8, 133330–133348. [Google Scholar] [CrossRef]
- Chapelle, O.; Schölkopf, B.; Zien, A. Semi-Supervised Learning; MIT Press: Cambridge, MA, USA, 2006. [Google Scholar]
- Lee, D.H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the ICML Workshop on Challenges in Representation Learning, Atlanta, GA, USA, 16–21 June 2013. [Google Scholar]
- Kingma, D.P.; Mohamed, S.; Jimenez Rezende, D.; Welling, M. Semi-supervised learning with deep generative models. Adv. Neural Inf. Process. Syst. 2014, 27, 3581–3589. [Google Scholar]
- Zhang, C.; Zhang, Q.; Hansen, J.H. Semi-supervised learning with generative adversarial networks for arabic dialect identification. In Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 5986–5990. [Google Scholar]
- Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved techniques for training GANs. Adv. Neural Inf. Process. Syst. 2016, 29, 2234–2242. [Google Scholar]
- Dong, X.; Yang, Y. Teacher supervises students how to learn from partially labeled images for facial landmark detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 783–792. [Google Scholar]
- Berthelot, D.; Carlini, N.; Goodfellow, I.; Papernot, N.; Oliver, A.; Raffel, C.A. Mixmatch: A holistic approach to semi-supervised learning. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; pp. 5049–5059. [Google Scholar]
- Xie, Q.; Dai, Z.; Hovy, E.; Luong, M.T.; Le, Q.V. Unsupervised data augmentation for consistency training. In Proceedings of the Advances in Neural Information Processing Systems, Virtual, 6–12 December 2020; Volume 33. [Google Scholar]
- Verma, V.; Lamb, A.; Kannala, J.; Bengio, Y.; Lopez-Paz, D. Interpolation consistency training for semi-supervised learning. In Proceedings of the International Joint Conference on Artificial Intelligence, Macao, China, 10–16 August 2019; pp. 3635–3641. [Google Scholar]
- Zhai, X.; Oliver, A.; Kolesnikov, A.; Beyer, L. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 1476–1485. [Google Scholar]
- Hastie, T.; Friedman, J.H.; Tibshirani, R. The Elements of Statistical Learning: Data Mining, Inference, and Prediction; Springer Series in Statistics. Available online: https://web.stanford.edu/~hastie/ElemStatLearn/ (accessed on 24 October 2021).
- Dong, X.; Yu, S.I.; Weng, X.; Wei, S.E.; Yang, Y.; Sheikh, Y. Supervision-by-registration: An unsupervised approach to improve the precision of facial landmark detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 360–368. [Google Scholar]
- Lucas, B.D.; Kanade, T. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, Vancouver, BC, Canada, 24–28 August 1981. [Google Scholar]
- Wang, L.; Yoon, K.J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Trans. Pattern Anal. Mach. Intell. 2021. [Google Scholar] [CrossRef]
- Honari, S.; Molchanov, P.; Tyree, S.; Vincent, P.; Pal, C.; Kautz, J. Improving landmark localization with semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 1546–1555. [Google Scholar]
- Ukita, N.; Uematsu, Y. Semi-and weakly-supervised human pose estimation. Comput. Vis. Image Underst. 2018, 170, 67–78. [Google Scholar] [CrossRef] [Green Version]
- Yao, Y.; Jafarian, Y.; Park, H.S. Monet: Multiview semi-supervised keypoint detection via epipolar divergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision ICCV, Seoul, Korea, 27 October–2 November 2019; pp. 753–762. [Google Scholar]
- Kumar, A.; Chellappa, R. S2LD: Semi-Supervised Landmark Detection in Low-Resolution Images and Impact on Face Verification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 13–19 June 2020; pp. 758–759. [Google Scholar]
- Cho, H.; Nishimura, K.; Watanabe, K.; Bise, R. Cell Detection in Domain Shift Problem Using Pseudo-Cell-Position Heatmap. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Strasbourg, France, 27 September–1 October 2021; pp. 384–394. [Google Scholar]
- Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico, 2–4 May 2016. [Google Scholar]
- Huang, W.; Yang, C.; Hou, T. Spine Landmark Localization with combining of Heatmap Regression and Direct Coordinate Regression. arXiv 2020, arXiv:2007.05355. [Google Scholar]
- Chandran, P.; Bradley, D.; Gross, M.; Beeler, T. Attention-Driven Cropping for Very High Resolution Facial Landmark Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 5861–5870. [Google Scholar]
- DeVries, T.; Taylor, G.W. Improved regularization of convolutional neural networks with cutout. arXiv 2017, arXiv:1708.04552. [Google Scholar]
- Xiao, B.; Wu, H.; Wei, Y. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 466–481. [Google Scholar]
Network | Ours | Others | ||||||
---|---|---|---|---|---|---|---|---|
Supervised | J = 1 | J = 2 | J = 3 | Supervised | Stacked Hourglass [20] | Simple Baseline [50] | HRNet [24] | |
2 Stage Zoom-in S = 2 | S = 1 | |||||||
Labeled Images | 7415 | 7415 | 7415 | 7415 | 7415 | 7415 | 7415 | 7415 |
Unlabeled Images | 0 | 88,231 | 88,231 | 88,231 | 0 | 0 | 0 | 0 |
Input Size | 224 × 224 | 224 × 224 | 224 × 224 | 224 × 224 | 224 × 224 | 256 × 256 | 256 × 256 | 256 × 256 |
Block Size | 224 × 224 | 224 × 224 | 224 × 224 | 224 × 224 | n/a | 64 × 64 | 64 × 64 | 64 × 64 |
MED | 3.139 | 3.127 | 3.263 | 3.199 | 3.545 | 7.471 | 8.244 | 7.401 |
#Labeled | #Unlabeled | #Testing | Supervised | Semi-Supervised Steps | ||
---|---|---|---|---|---|---|
(#Training/#Validation) | J = 1 | J = 2 | J = 3 | |||
200 (180/20) | 88,231 | 927 | 5.278 | 4.328 | 3.968 | 3.885 |
100 (90/10) | 88,231 | 927 | 6.091 | 4.634 | 5.078 | 3.903 |
50 (45/5) | 88,231 | 927 | 7.718 | 6.213 | 5.256 | 4.245 |
20 (18/2) | 88,231 | 927 | 18.598 | 8.646 | 5.560 | 5.606 |
Set | #Labeled | #Unlabeled | #Testing | Supervised | Semi-Supervised Steps | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
(#Training/#Validation) | J = 1 | J = 2 | J = 3 | J = 4 | J = 5 | J = 6 | J = 7 | J = 8 | ||||
(a) | 15 (10/5) | 88,231 | 927 | 15.252 | 10.401 | 6.101 | 6.063 | 5.283 | 5.732 | 5.040 | 5.067 | 4.890 |
(b) | 15 (10/5) | 88,231 | 927 | 10.756 | 8.387 | 6.360 | 6.164 | 5.518 | 5.282 | 4.713 | 4.874 | 5.574 |
(c) | 15 (10/5) | 88,231 | 927 | 12.853 | 7.401 | 6.464 | 6.392 | 6.098 | 5.952 | 5.748 | 5.988 | 5.754 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, W.; Lang, J. Semi-Supervised Training for Positioning of Welding Seams. Sensors 2021, 21, 7309. https://doi.org/10.3390/s21217309
Zhang W, Lang J. Semi-Supervised Training for Positioning of Welding Seams. Sensors. 2021; 21(21):7309. https://doi.org/10.3390/s21217309
Chicago/Turabian StyleZhang, Wenbin, and Jochen Lang. 2021. "Semi-Supervised Training for Positioning of Welding Seams" Sensors 21, no. 21: 7309. https://doi.org/10.3390/s21217309
APA StyleZhang, W., & Lang, J. (2021). Semi-Supervised Training for Positioning of Welding Seams. Sensors, 21(21), 7309. https://doi.org/10.3390/s21217309