Features Recognition from Piping and Instrumentation Diagrams in Image Format Using a Deep Learning Network
2. Brief Literature Review
3. Training Data Definition for P&ID Recognition
3.1. Training Data Requirement Analysis for P&ID Recognition
3.2. Training Data Structure Definition for P&ID Recognition
4. P&ID Recognition Accuracy Improvement through Preprocessing
4.1. Diagram Alignment
4.2. Removal of Outer Borders and Title Boxes
5. Object Recognition on an Image-Format P&ID
5.1. Symbol Detection
5.2. Character Detection
5.3. Line Detection
5.4. Table Detection
6. Implementation and Experiments
- Limit the symbol types for detection of four types and nine classes. Considering that there are hundreds of symbol types used in P&IDs, there is a need to expand the symbol types intended for detection. In addition to the method that applies a sliding window to the image-classification deep learning model, it is necessary to examine the application of an image-detection deep learning model.
- With regard to character detection, expand the training data for a conventional text-detection deep learning model using text data extracted from P&IDs or improve the conventional deep learning models considering the particularity of the texts included in P&IDs.
- With regard to line detection, develop efficient methods to recognize dotted lines in addition to continued lines, and recognize diagonal lines in addition to horizontal and vertical lines. Furthermore, in the case where several lines intersect, a technique to determine their interconnectedness must be put in place. The proposed method is also prone to line recognition errors when there are noises in the P&ID. Therefore, methods to apply a deep learning technique for line detection should be explored.
- For table detection, develop a method to recognize various table types and shapes, in addition to the most basic form used in this study.
- The training data used in the study was manually extracted from a real P&ID drawing. The absolute amount of training data was, therefore, far from being sufficient. This entailed constraints on the selection of applicable deep learning models. A follow-up study is needed to develop algorithms for automated detection and extraction of training data required for P&ID recognition.
Conflicts of Interest
- Tornbre, K. Technical drawing recognition and understanding: From pixels to semantics. In Proceedings of the Workshop on Machine Vision and Application, Tokyo, Japan, 7–9 December 1992; pp. 393–401. [Google Scholar]
- Fahn, C.-S.; Wang, J.-F.; Lee, J.-Y. A topology-based component extractor for understanding electronic circuit diagrams. Comput. Vision Graph. Image Process. 1988, 43, 279. [Google Scholar] [CrossRef]
- Lee, S.-W.; Kim, J.H.; Groen, F.C. Translation-, Rotation- and Scale- Invariant Recognition of Hand-Drawn Symbols in Schematic Diagrams. Int. J. Pattern Recognit. Artif. Intell. 1990, 4, 1–25. [Google Scholar] [CrossRef]
- Futatsumata, T.; Shichino, G.; Shibayama, J.; Maeda, A. Development of an Automatic Recognition System for Plant Diagrams. In Proceedings of the IAPR Workshop on Machine Vision Applications, Tokyo, Japan, 28–30 November 1990; pp. 207–210. [Google Scholar]
- Benjamin, D.; Forgues, P.; Gulko, E.; Massicotte, J.; Meubus, C. The use of high-level knowledge for enhanced entry of engineering drawings. In Proceedings of the 9th International Conference on Pattern Recognition IEEE, Rome, Italy, 14 May 1988; pp. 119–124. [Google Scholar]
- Kang, S.-O.; Lee, E.-B.; Baek, H.-K. A Digitization and Conversion Tool for Imaged Drawings to Intelligent Piping and Instrumentation Diagrams (P&ID). Energies 2019, 12, 2593. [Google Scholar]
- Kato, H.; Inokuchi, S. The recognition method for roughly hand-drawn logical diagrams based on hybrid utilization of multi-layered knowledge. In Proceedings of the 10th International Conference on Pattern Recognition, Atlantic City, NJ, USA, 16–21 June 1990; pp. 578–582. [Google Scholar]
- Maini, R.; Aggarwal, H. Study and comparison of various image edge detection techniques. Int. J. Image Process. 2008, 3, 1–12. [Google Scholar]
- Xu, L.; Oja, E.; Kultanen, P. A new curve detection method: Randomized Hough transform (RHT). Pattern Recognit. Lett. 1990, 11, 331–338. [Google Scholar] [CrossRef]
- Wang, D.; Haese-Coat, V.; Ronsin, J. Shape decomposition and representation using a recursive morphological operation. Pattern Recognit. 1995, 28, 1783–1792. [Google Scholar] [CrossRef]
- Tian, D.P. A review on image feature extraction and representation techniques. Int. J. Multimed. Ubiquitous Eng. 2013, 8, 385–396. [Google Scholar]
- Tan, W.C.; Chen, I.-M.; Tan, H.K. Automated identification of components in raster piping and instrumentation diagram with minimal pre-processing. In Proceedings of the IEEE International Conference on Automation Science and Engineering (CASE), Fort Worth, TX, USA, 21–25 August 2016; pp. 1301–1306. [Google Scholar]
- Arroyo, E.; Hoernicke, M.; Rodriguez, P.; Fay, A. Automatic derivation of qualitative plant simulation models from legacy piping and instrumentation diagrams. Comput. Chem. Eng. 2016, 92, 112–132. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet Classification with Deep Convolutional Neural Networks. In Proceedings of the NIPS 2012: Neural Information Processing Systems Conference, Lake Tahoe, NV, USA, 3–6 December 2012. [Google Scholar]
- Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J.; Malik, J. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single Shot Multibox Detector. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 21–37. [Google Scholar]
- Sliding Windows for Object Detection with Python and openCV. Available online: https://www.pyimagesearch.com/2015/03/23/sliding-windows-for-object-detection-with-python-and-opencv/ (accessed on 24 September 2019).
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
- TensorFlow. Available online: https://www.tensorflow.org/ (accessed on 24 September 2019).
- Keras. Available online: https://keras.io/ (accessed on 24 September 2019).
- PyTorch. Available online: https://pytorch.org/ (accessed on 24 September 2019).
- Theano. Available online: http://deeplearning.net/software/theano/# (accessed on 24 September 2019).
- LeCun, Y.; Boser, B.; Denker, J.S.; Henderson, D.; Howard, R.E.; Hubbard, W.; Jackel, L.D. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1989, 1, 541–551. [Google Scholar] [CrossRef]
- Fu, L.; Kara, L.B. From engineering diagrams to engineering models: Visual recognition and applications. Comput. Des. 2011, 43, 278–292. [Google Scholar] [CrossRef]
- Rahul, R.; Paliwal, S.; Sharma, M.; Vig, L. Automatic Information Extraction from Piping and Instrumentation Diagrams. In Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods, Prague, Czech Republic, 19–21 February 2019; pp. 163–172. [Google Scholar]
- ISO 10628: Flow Diagrams for Process Plants—General Rules; International Organization for Standardization: Geneva, Switzerland, 1997.
- Everingham, M.; Van Gool, L.; Williams, C.K.; Winn, J.; Zisserman, A. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 2010, 88, 303–338. [Google Scholar] [CrossRef]
- Yu, E.S. Approach of Object Recognition from Image Format Engineering Drawings using Deep Learning. Master’s Thesis, Kyungpook National University, Daegu, Korea, 2019. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86, 2278–2324. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Hinton, G. Learning Multiple Layers of Features from Tiny Images; Technical Report; University of Toronto: Toronto, ON, Canada, 2009; p. 7. [Google Scholar]
- Tian, Z.; Huang, W.; He, T.; He, P.; Qiao, Y. Detecting Text in Natural Image with Connectionist Text Proposal Network. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 56–72. [Google Scholar]
- Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed]
- Karatzas, D.; Shafait, F.; Uchida, S.; Iwamura, M.; i Bigorda, L.G.; Mestre, S.R. ICDAR 2013 Robust Reading Competition. In Proceedings of the 12th International Conference on Document Analysis and Recognition, Washington, DC, USA, 25–28 August 2013; pp. 1484–1493. [Google Scholar]
- Gattullo, M.; Evangelista, A.; Uva, A.E.; Fiorentino, M.; Boccaccio, A.; Manghisi, V.M. Exploiting Augmented Reality to Enhance Piping and Instrumentation Diagrams for Information Retrieval Tasks in Industry 4.0 Maintenance. In International Conference on Virtual Reality and Augmented Reality; Springer Science and Business Media LLC: Cham, Switzerland, 2019; pp. 170–180. [Google Scholar]
|Class (Label)||Instrument North||Instrument West||DR I East||DR I West||DR O East||DR O West||Pump||Valve North||Valve East|
|Prediction score||instrument north||17.05 *||6.36||0.45||−4.54||−4.10||−4.40||0.84||−6.95||−5.3|
|DR I east||1.41||−2.32||19.39||−3.28||2.26||0.35||−4.96||−11.21||1.83|
|DR I west||−6.16||5.35||−8.24||19.82||−8.32||1.68||4.08||−12.69||2.69|
|DR O east||−1.50||−4.81||3.06||−1.88||17.14||0.99||−8.05||−2.15||−6.49|
|DR O west||−6.71||−0.67||−6.97||5.70||2.33||16.15||−7.63||−0.97||−7.26|
|Diagram Name||Recognition Accuracy||Misidentification Rate|
|Test P&ID 1||92.3%||84.2%||93.7%||25%||18.8%||17.9%|
|Test P&ID 2||90.9%||82%||87.5%||22.2%||19.6%||6.25%|
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Yu, E.-S.; Cha, J.-M.; Lee, T.; Kim, J.; Mun, D. Features Recognition from Piping and Instrumentation Diagrams in Image Format Using a Deep Learning Network. Energies 2019, 12, 4425. https://doi.org/10.3390/en12234425
Yu E-S, Cha J-M, Lee T, Kim J, Mun D. Features Recognition from Piping and Instrumentation Diagrams in Image Format Using a Deep Learning Network. Energies. 2019; 12(23):4425. https://doi.org/10.3390/en12234425Chicago/Turabian Style
Yu, Eun-Seop, Jae-Min Cha, Taekyong Lee, Jinil Kim, and Duhwan Mun. 2019. "Features Recognition from Piping and Instrumentation Diagrams in Image Format Using a Deep Learning Network" Energies 12, no. 23: 4425. https://doi.org/10.3390/en12234425