Symbol Detection in Mechanical Engineering Sketches: Experimental Study on Principle Sketches with Synthetic Data Generation and Deep Learning

Abstract

Digital transformation is omnipresent in our daily lives and its impact is noticeable through new technologies, like smart devices, AI-Chatbots or the changing work environment. This digitalization also takes place in product development, with the integration of many technologies, such as Industry 4.0, digital twins or data-driven methods, to improve the quality of new products and to save time and costs during the development process. Therefore, the use of data-driven methods reusing existing data has great potential. However, data from product design are very diverse and strongly depend on the respective development phase. One of the first few product representations are sketches and drawings, which represent the product in a simplified and condensed way. But, to reuse the data, the existing sketches must be found with an automated approach, allowing the contained information to be utilized. One approach to solve this problem is presented in this paper, with the detection of principle sketches in the early phase of the development process. The aim is to recognize the symbols in these sketches automatically with object detection models. Therefore, existing approaches were analyzed and a new procedure developed, which uses synthetic training data generation. In the next step, a total of six different data generation types were analyzed and tested using six different one- and two-stage detection models. The entire procedure was then evaluated on two unknown test datasets, one focusing on different gearbox variants and a second dataset derived from CAD assemblies. In the last sections the findings are discussed and a procedure with high detection accuracy is determined.

1. Introduction

Digitalization and the appearance of data-driven technologies raise a variety of challenges for the development of new products. The goal of implementing these novel methods is to reduce product development time and costs, adapt quickly to changing markets and improve the efficiency of the development process [1]. Moreover, the application of these data-driven methods for reusing existing data and information helps to exploit the potential for product development, since a wide range of virtual and physical data are available throughout the engineering process [2,3].

A few application data sources are drawings and sketches, which are proven ways of communication in various engineering disciplines to quickly exchange specific content. The symbols act like a language of the respective discipline and are only readable by educated personnel. The examples range from technical drawings of mechanical engineering products to computer science flowcharts, circuit diagrams in electrical engineering or architectural plans for buildings, with some examples shown in Figure 1. To support a given design task with existing sketches, the recognition and processing of these images must be automated. For mechanical engineering, the principle sketches are especially appropriate for this purpose, as they are established in the early phases of the product development process and, therefore, have a great influence on the costs and quality of the final product.

Principle Sketches

These sketches are usually one of the first visual representations of the future product and are derived from the principle solution. They are a rough and general representation of the functional solution and form the transition from the abstract–functional to the concrete design step [4], with one example displayed in Figure 2.

These principle sketches are represented in different ways, e.g., standardized symbolic sketches, free line sketches, 3D freehand sketches or unscaled rough drafts [5]. They provide information about the type of components or assemblies, the number of parts and their arrangement. This deliberate abstraction fosters the creation of alternative solutions and is therefore a key factor for successful product development. The exact specification of the components and their geometrical dimensions are determined in the subsequent embodiment phase.

Figure 2. Example of aprinciple sketch and the derived detailed part drawing according to [6].

Figure 2. Example of aprinciple sketch and the derived detailed part drawing according to [6].

Figure 2 depicts the difference between a principle sketch and the resulting technical part. The sketch on the left presents a solution to the problem of converting translational motion into rotational one, while the technical drawing on the right shows the finished component, referred to as part (a) in the principle sketch.

The objective is the automated utilization of this product representation, enabling the analysis of existing data for new developments. Potential applications include finding similar sketches, synthesizing new designs or even improving existing ones. For the realization, the first and very important step is the detection of the symbols in a principle sketch, whereby the focus is on their correct classification instead of their pixel-precise position. This is related to the subsequent contextualization, where a certain deviation of the bounding boxes could be compensated.

Consequently, the next chapter outlines the current state-of-the-art for object detection in engineering drawings through a literature study. Based on the findings, a new approach is presented in the Section 3, which is tested thoroughly on unknown sketches in the Section 4.

2. Related Work

After the introduction and explanation of the principle sketches, this chapter introduces the current state of research in this field. This section is structured in two parts, starting with a comprehensive literature review on the recognition of symbols in technical sketches. Subsequently, the current methods for recognizing objects in images are explained in depth.

2.1. Sketch Detection

The basic motivation for symbol and drawing recognition in engineering was around for a long time, since the first approaches were published in the early 1980s. These methods are mostly based on rules for the recognition of symbols. Recognition involves locating and classifying the various objects in a drawing, with algorithms varying widely depending on the task and scope. In the more recent publication of Moreno et al. [7], the entire digitalization process is structured into two sections, the symbol detection and the contextualization. Contextualization aims to automatically reuse the detected symbols, for example, to find linked documents or to check the drawing for errors.

A general overview of the developed methods is given in Figure 3. The two diagrams show the results of a literature search of over 200 publications in the field of symbol detection and contextualization in engineering, gathered from journal and conference papers from the last 40 years. All methods were classified according to their engineering domain, the use of a learning technique (Machine or Deep Learning) and the dataset type. The engineering domains describe the application scenario, whether it is located in the mechanical, chemical or architectural environment. The learning methods are differentiated according to the type of training or learning on data, whereby all supervised, semi-, weakly-supervised and unsupervised methods are classified as learning methods. The table with all references is provided in

Table A4. Overview of the literature study of symbol digitalization in different engineering domains, sorted by the engineering domain and year of publication.

Source	Year	Title	Engineering Domain	Process Step	Learning Technique	Dataset
[67]	1979	A Threshold Selection Method from Gray-Level Histograms	general	preprocessing	without Learning	own
[68]	1990	A system for interpretation of line drawings	general	detection	without Learning	own
[69]	1994	Precise line detection from an engineering drawing using a figure fitting method based on contours and skeletons	general	preprocessing	without Learning	own
[70]	1994	Skeleton generation of engineering drawings via contour matching	general	preprocessing	without Learning	-
[71]	1994	Finding arrows in utility maps using a neural network	general	detection	with Learning	own
[72]	1996	Automatic learning and recognition of graphical symbols in engineering drawings	general	detection	without Learning	own
[73]	1998	A new algorithm for line image vectorization	general	preprocessing	without Learning	-
[74]	2000	Adaptive document image binarization	general	preprocessing	without Learning	own
[75]	2006	Robust and accurate vectorization of line drawings	general	preprocessing	without Learning	own
[76]	2012	Multi-Level Block Information Extraction in Engineering Drawings Based on Depth-First Algorithm	general	detection + contextualization	without Learning	own
[77]	2017	Adaptive document image skew estimation	general	preprocessing	with Learning	own
[78]	2019	Anchor Point based Hough Transformation for Automatic Line Detection of Engineering Drawings	general	preprocessing	without Learning	own
[79]	2020	Deep Vectorization of Technical Drawings	general	preprocessing	with Learning	open source
[80]	1993	Management of graphical symbols in a CAD environment: A neural network approach	civil	detection	with Learning	own
[81]	1994	Symbol recognition in a CAD enviroment using a neural network	civil	detection	with Learning	own
[82]	1997	A system to understand hand-drawn floor plans using subgraph isomorphism and Hough transform	civil	detection	without Learning	own
[83]	1998	A constraint network for symbol detection in architectural drawings	civil	detection	without Learning	own
[84]	2000	A complete system for the analysis of architectural drawings	civil	detection + contextualization	without Learning	own
[85]	2001	Architectural symbol recognition using a network of constraints	civil	detection	without Learning	own
[86]	2001	Symbol recognition by error-tolerant subgraph matching between region adjacency graphs	civil	detection	without Learning	own
[87]	2002	An object-oriented progressive-simplification-based vectorization system for engineering drawings: model, algorithm, and performance	civil	preprocessing + detection	without Learning	own
[88]	2003	A model for image generation and symbol recognition through the deformation of lineal shapes	civil	detection	without Learning	synthetic
[89]	2005	Using engineering drawing interpretation for automatic detection of version information in CADD engineering drawing	civil	detection	without Learning	own
[90]	2007	Automatic analysis and integration of architectural drawings	civil	detection + contextualization	without Learning	own
[91]	2008	Knowledge Extraction from Structured Engineering Drawings	civil	detection	without Learning	own
[92]	2009	Symbol Detection Using Region Adjacency Graphs and Integer Linear Programming	civil	detection	without Learning	open source
[10]	2010	Generation of synthetic documents for performance evaluation of symbol recognition & spotting systems	civil	detection	with Learning	synthetic
[93]	2010	Symbol spotting in vectorized technical drawings through a lookup table of region strings	civil	detection	without Learning	own
[94]	2011	Statistical Grouping for Segmenting Symbols Parts from Line Drawings, with Application to Symbol Spotting	civil	detection	without Learning	open source
[95]	2012	Object recognition in floor plans by graphs of white connected components	civil	detection	without Learning	own + open source
[96]	2013	Geometric-based symbol spotting and retrieval in technical line drawings	civil	detection	with Learning	own + open source
[97]	2013	Building a Symbol Library from Technical Drawings by Identifying Repeating Patterns	civil	detection	with Learning	open source
[98]	2013	Combining geometric matching with SVM to improve symbol spotting	civil	detection	with Learning	open source
[99]	2013	Efficient symbol retrieval by building a symbol index from a collection of line drawings	civil	detection	with Learning	open source
[100]	2013	A symbol spotting approach in graphical documents by hashing serialized graphs	civil	detection	without Learning	open source
[101]	2014	Data Extraction from DXF File and Visual Display	civil	detection + contextualization	without Learning	own
[102]	2016	Automatic Hyperlinking of Engineering Drawing Documents	civil	detection	without Learning	own
[103]	2017	Graph-Based Deep Learning for Graphics Classification	civil	detection	with Learning	open source
[104]	2018	Extraction of Ancient Map Contents Using Trees of Connected Components	civil	detection	without Learning	own
[105]	2018	Object Detection in Floor Plan Images	civil	detection	with Learning	own
[106]	2019	Graph Neural Network for Symbol Detection on Document Images	civil	detection	with Learning	open source
[18]	2019	Symbol spotting for architectural drawings: state-of-the-art and new industry-driven developments	civil	detection	with Learning	own
[9]	2019	BRIDGE: Building Plan Repository for Image Description Generation, and Evaluation	civil	detection	with Learning	open source
[17]	2020	Symbol Spotting on Digital Architectural Floor Plans Using a Deep Learning-based Framework	civil	detection	with Learning	own / Open Source
[21]	2020	Floor Plan Recognition and Vectorization Using Combination UNet, Faster-RCNN, Statistical Component Analysis and Ramer-Douglas-Peucker	civil	detection	with Learning	own
[107]	2021	Knowledge-driven description synthesis for floor plan interpretation	civil	detection + contextualization	with Learning	open source
[25]	2021	Fine Grained Feature Representation Using Computer Vision Techniques for Understanding Indoor Space	civil	detection	with Learning	open source
[108]	2021	FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting	civil	detection	with Learning	open source
[23]	2021	Towards Robust Object Detection in Floor Plan Images: A Data Augmentation Approach	civil	detection	with Learning	open source
[109]	2021	PU learning-based recognition of structural elements in architectural floor plans	civil	detection	with Learning	own + open source
[110]	2021	3DPlanNet: Generating 3D Models from 2D Floor Plan Images Using Ensemble Methods	civil	detection + contextualization	with Learning	open source
[22]	2022	Automatic Detection and Classification of Symbols in Engineering Drawings	civil	detection	with Learning	own
[111]	2022	CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings	civil	detection	with Learning	open source
[112]	2022	GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings	civil	detection	with Learning	open source
[24]	2022	Mask-Aware Semi-Supervised Object Detection in Floor Plans	civil	detection	with Learning	open source
[113]	2022	Designing a Human-in-the-Loop System for Object Detection in Floor Plans	civil	Erkennung	with Learning	synthetisches Dataset
[114]	2023	Digitalization of 2D Bridge Drawings Using Deep Learning Models	civil	detection + contextualization	with Learning	own + synthetic
[115]	2023	Improving Symbol Detection on Engineering Drawings Using a Keypoint-Based Deep Learning Approach	civil	detection	with Learning	synthetic
[116]	2023	You Only Look for a Symbol Once: An Object Detector for symbols and Regions in Documents.	civil	detection	with Learning	open source
[20]	2023	Leveraging Deep Convolutional Neural Network for Point Symbol Recognition in Scanned Topographic Maps	civil	detection	with Learning	own
[19]	2023	Towards Automatic Digitalization of Railway Engineering Schematics	civil	detection	with Learning	own
[117]	2024	Deep learning-based text detection and recognition on architectural floor plans	civil	detection	with Learning	open source + synthetic
[118]	1997	Adaptive Vectorization of Line Drawing Images	civil + electrical	preprocessing	without Learning	own
[119]	2005	Symbol recognition via statistical integration of pixel-level constraint histograms: a new descriptor	civil + electrical	detection	with Learning	open source
[120]	2006	Symbol recognition with kernel density matching	civil + electrical	detection	with Learning	open source
[121]	2006	Symbol Spotting in Technical Drawings Using Vectorial Signatures	civil + electrical	detection	without Learning	open source
[122]	2007	A Bayesian classifier for symbol recognition	civil + electrical	detection	with Learning	open source
[123]	2007	A New Syntactic Approach to Graphic Symbol Recognition	civil + electrical	detection	without Learning	open source
[124]	2008	On the Combination of Ridgelets Descriptors for Symbol Recognition	civil + electrical	detection	with Learning	open source
[125]	2009	Graphic Symbol Recognition Using Graph Based Signature and Bayesian Network Classifier	civil + electrical	detection	with Learning	own
[126]	2010	A Bayesian network for combining descriptors: application to symbol recognition	civil + electrical	detection	with Learning	open source
[127]	2011	A New Adaptive Structural Signature for Symbol Recognition by Using a Galois Lattice as a Classifier	civil + electrical	detection	with Learning	open source
[128]	2019	GSD-Net: Compact Network for Pixel-Level Graphical Symbol Detection	civil + electrical	detection	with Learning	open source
[129]	2002	TIF2VEC, An Algorithm for Arc Segmentation in Engineering Drawings	civil + electrical + mechanical	preprocessing	without Learning	open source
[130]	2014	BoR: BAG-OF-RELATIONS FOR SYMBOL RETRIEVAL	civil + electrical + P&ID	detection	without Learning	open source
[131]	2013	Img2UML: A System for Extracting UML Models from Images	computer science	detection + contextualization	without Learning	own
[132]	2014	Automatic Classification of UML Class Diagrams from Images	computer science	detection	with Learning	own
[133]	2021	Multiclass Classification of UML Diagrams from Images Using Deep Learning	computer science	detection	with Learning	own
[134]	1982	Automatic Interpretation of Lines and Text in Circuit Diagrams	electrical	detection + contextualization	without Learning	-
[135]	1985	Symbol recognition in electrical diagrams using probabilistic graph matching	electrical	detection	with Learning	own
[136]	1988	An automatic circuit diagram reader with loop-structure-based symbol recognition	electrical	detection	with Learning	open source
[137]	1988	A topology-based component extractor for understanding electronic circuit diagrams	electrical	detection	with Learning	own
[138]	1990	Translation-,rotation- and scale- invariant recognition of hand-drawn symbols in schematic diagrams	electrical	detection	with Learning	own
[139]	1992	Recognizing Hand-Drawn Electrical Circuit Symbols with Attributed Graph Matching	electrical
[140]	1993	Recognition of logic diagrams by identifying loops and rectilinear polylines	electrical	detection	without Learning	own
[141]	1993	A symbol recognition system	electrical	detection	with Learning	own
[142]	1993	A new system for the analysis of schematic diagrams	electrical	detection	without Learning	own
[143]	1995	Automatic understanding of symbol-connected diagrams	electrical	contextu-alization	without Learning	own
[144]	2003	Engineering drawings recognition using a case-based approach	electrical	detection	without Learning	own
[145]	2003	Symbol recognition in electronic diagrams using decision tree	electrical	detection	without Learning	own
[146]	2004	Extracting System-Level Understanding from Wiring Diagram Manuals	electrical	detection	without Learning	own
[147]	2009	A visual approach to sketched symbol recognition	electrical	detection	without Learning	own
[148]	2009	On-line hand-drawn electric circuit diagram recognition using 2D dynamic programming	electrical	detection	with Learning	own
[149]	2011	Recognition of electrical symbols in document images using morphology and geometric analysis	electrical	detection	without Learning	own
[150]	2012	Symbol recognition using spatial relations	electrical	detection	without Learning	own
[151]	1995	Electronic Schematic Recognition	electrical	detection + contextualization	without Learning	own
[152]	2015	Detection and identification of logic gates from document images using mathematical morphology	electrical	detection	with Learning	own
[153]	2016	Hand Drawn Optical Circuit Recognition	electrical	detection	with Learning	own
[154]	2017	Recognizing Electronic Circuits to Enrich Web Documents for Electronic Simulation	electrical	detection + contextualization	with Learning	own
[155]	2018	Analysis of methods for automated symbol recognition in technical drawings	electrical	detection	with Learning	own
[156]	2019	Automatic Abstraction of Combinational Logic Circuit from Scanned Document Page Images	electrical	contextu-alization	with Learning	own
[157]	2020	CIM/G graphics automatic generation in substation primary wiring diagram based on image recognition	electrical	Erkennung	with Learning	own
[35]	2021	Graph-Based Object Detection Enhancement for Symbolic Engineering Drawings	electrical	contextu-alization	with Learning	own
[158]	2021	A public ground-truth dataset for handwritten circuit diagram images	electrical	detection	with Learning	open source
[159]	2022	Substation One-Line Diagram Automatic Generation Based On Image Recongnition	electrical	detection + contextualization	with Learning	own
[160]	2022	Symbol Spotting in Electronic Images Using Morphological Filters and Hough Transform	electrical	detection	without Learning	open source
[161]	2023	Instance segmentation based graph extraction for handwritten circuit diagram images	electrical	detection	with Learning	open source
[162]	2023	ElectroNet: An Enhanced Model for Small-Scale Object Detection in Electrical Schematic Diagrams	electrical	detection + contextualization	with Learning	own
[163]	2023	Single Line Electrical Drawings (SLED): A Multiclass Dataset Benchmarked by Deep Neural Networks	electrical	Erkennung	with Learning	open source
[164]	2023	Intelligent Digitization of Substation One-Line Diagrams Based on Computer Vision	electrical	detection + contextualization	with Learning	synthetic
[36]	2023	Song, Aibo and Kun, Huang and Peng, Bowen and Chen, Rui and Zhao, Kun and Qiu, Jingyi and Wang, Kaixuan	electrical	detection + contextualization	with Learning	own
[165]	2007	An interactive example-driven approach to graphics recognition in engineering drawings	electrical + civil	detection	without Learning	own
[166]	2008	Spotting Symbols in Line Drawing Images Using Graph Representations	electrical + civil	detection	without Learning	own
[167]	1994	Isolating symbols from connection lines in a class of engineering drawings	electrical + P&ID	detection	without Learning	own
[168]	1997	A System for Recognizing a Large Class of Engineering Drawings	electrical + P&ID	detection	without Learning	own
[169]	2014	Accurate junction detection and characterization in line-drawing images	electrical, civil	detection	without Learning	open source
[170]	1996	Vector-based segmentation of text connected to graphics in engineering drawings	electrical, mechanical, civil	detection	without Learning	own
[171]	1989	Processing of engineering line drawings for automatic input to CAD	mechanical	preprocessing	without Learning	-
[172]	1989	Automatic Scanning and Interpretation of Engineering Drawings for CAD-Processes	mechanical	detection	without Learning	own
[173]	1990	Engineering drawing processing and vectorization system	mechanical	preprocessing	without Learning	-
[174]	1990	Interpretation of line drawings with multiple views	mechanical	detection + contextualization	without Learning	own
[175]	1990	Randomized Hough Transform (RHT) in Engineering Drawing Vectorization System	mechanical	detection	without Learning	own
[176]	1991	Detection of dashed lines in engineering drawings and maps	mechanical	detection	without Learning	own
[177]	1992	Celesstin: CAD conversion of mechanical drawings	mechanical	detection + contextualization	without Learning	own
[178]	1992	Dimensioning analysis	mechanical	detection	without Learning	own
[179]	1992	Knowledge-directed interpretation of mechanical engineering drawings	mechanical	detection	without Learning	own
[180]	1993	Recognition of dimensions in engineering drawings based on arrowhead	mechanical	detection	without Learning	own
[181]	1994	Detection of dimension sets in engineering drawings	mechanical	detection	without Learning	own
[182]	1994	Knowledge organization and interpretation process in engineering drawing interpretation	mechanical	detection	without Learning	own
[183]	1994	Syntactic analysis of technical drawing dimensions	mechanical	detection	without Learning	own
[184]	1995	Recognition of dimension sets and integration with vectorized engineering drawings	mechanical	detection + contextualization	without Learning	own
[185]	1995	Vector-based arc segmentation in the machine drawing understanding system environment	mechanical	detection	without Learning	own
[186]	1996	Functional parts detection in engineering drawings: Looking for the screws	mechanical	detection	without Learning	own
[187]	1996	A clustering-based approach to the separation of text strings from mixed text/graphics documents	mechanical	detection	with Learning	own
[188]	1996	Perfecting Vectorized Mechanical Drawings	mechanical	preprocessing	without Learning	own
[189]	1996	Arrowhead recognition during automated data capture	mechanical	detection	without Learning	own
[190]	1997	Orthogonal Zig-Zag: An algorithm for vectorizing engineering drawings compared with Hough Transform	mechanical	detection	without Learning	own
[191]	1998	Detection of text regions from digital engineering drawings	mechanical	detection	without Learning	own
[192]	1998	Generating multiple new designs from a sketch	mechanical	detection + contextualization	without Learning	own
[193]	1998	Segmentation and Recognition of Dimensioning Text from Engineering Drawings	mechanical	detection	without Learning	own
[194]	1998	A system for automatic recognition of engineering drawing entities	mechanical	detection + preprocessing	without Learning	own
[195]	1999	Automated CAD conversion with the Machine Drawing Understanding System: concepts, algorithms, and performance	mechanical	detection	without Learning	own
[196]	1999	Automatic extraction of manufacturable features from CADD models using syntactic pattern recognition techniques	mechanical	detection + contextualization	without Learning	own
[197]	1999	Dimension sets detection in technical drawings	mechanical	detection	without Learning	own
[198]	1999	A complete system for the intelligent interpretation of engineering drawings	mechanical	detection + contextualization	without Learning	own
[199]	2000	Symbol and character recognition: application to engineering drawings	mechanical	detection	with Learning	own
[200]	2000	Engineering Drawing Database Retrieval Using Statistical Pattern Spotting Techniques	mechanical	detection	with Learning	own
[201]	2001	Intelligent system for extraction of product data from CADD models	mechanical	detection + contextualization	with Learning	own
[202]	2004	Strategy for Line Drawing Understanding	mechanical	detection	without Learning	own
[203]	2004	A new way to detect arrows in line drawings	mechanical	detection	without Learning	own
[204]	2009	Information extraction from scanned engineering drawings	mechanical	detection + contextualization	without Learning	own
[205]	2010	An information extraction of title panel in engineering drawings and automatic generation system of three statistical tables	mechanical	detection + contextualization	without Learning	own
[206]	2011	From engineering diagrams to engineering models: Visual recognition and applications	mechanical	detection + contextualization	with Learning	synthetic
[207]	2016	Dimensional Arrow Detection from CAD Drawings	mechanical	detection	without Learning	own
[40]	2017	ConvNet-Based Optical Recognition for Engineering Drawings	mechanical	detection	with Learning	own
[208]	2019	Detection of Primitives in Engineering Drawing using Genetic Algorithm	mechanical	preprocessing	without Learning	open source
[209]	2021	Extraction of dimension requirements from engineering drawings for supporting quality control in production processes	mechanical	detection	with Learning	own
[210]	2021	An Automated Engineering Assistant: Learning Parsers for Technical Drawings	mechanical	detection + contextualization	with Learning	own
[211]	2022	Data Augmentation of Engineering Drawings for Data-Driven Component Segmentation	mechanical	detection	with Learning	synthetic
[37]	2022	AI-Based Engineering and Production Drawing Information Extraction	mechanical	detection	with Learning	synthetic
[212]	2022	Unsupervised and hybrid vectorization techniques for 3D reconstruction of engineering drawings	mechanical	detection + contextualization	with Learning	own
[213]	2023	Graph neural network-enabled manufacturing method classification from engineering drawings	mechanical	detection + contextualization	with Learning	own
[214]	2023	An Approach to Engineering Drawing Organization: Title Block Detection and Processing	mechanical	detection + contextualization	with Learning	own
[39]	2023	AI-Based Method for Frame Detection in Engineering Drawings	mechanical	detection	with Learning	own
[215]	2023	Component segmentation of engineering drawings using Graph Convolutional Networks	mechanical	detection + contextualization	with Learning	own
[38]	2023	Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings	mechanical	detection + contextualization	with Learning	own
[216]	2024	Tolerance Information Extraction for Mechanical Engineering Drawings–A Digital Image Processing and Deep Learning-based Model	mechanical	detection	with Learning	own
[217]	2012	An improved example-driven symbol recognition approach in engineering drawings	mechanical + civil	detection	without Learning	own + open source
[218]	2018	Hand-written and machine-printed text classification in architecture, engineering & construction documents	mechanical + civil	detection	with Learning	own
[219]	2005	An image-based, trainable symbol recognizer for hand-drawn sketches	mechanical + electrical	detection	with Learning	own
[220]	2006	An Efficient Graph-Based Symbol Recognizer	mechanical + electrical	detection	with Learning	own
[221]	2007	An efficient graph-based recognizer for hand-drawn symbols	mechanical + electrical	detection	with Learning	own
[206]	2011	Neural network-based symbol recognition using a few labeled samples	mechanical + electrical	detection	with Learning	synthetic + open source
[222]	1994	Graphic symbol recognition using a signature technique	P&ID	detection	without Learning	own
[223]	1998	Computer interpretation of process and instrumentation drawings	P&ID	detection	without Learning	own
[224]	2006	Using process topology in plant-wide control loop performance assessment	P&ID	contextu-alization	without Learning	-
[225]	2009	Graphic Symbol Recognition Using Auto Associative Neural Network Model	P&ID	detection	with Learning	own
[226]	2015	A 2D Engineering Drawing and 3D Model Matching Algorithm for Process Plant	P&ID	detection + contextualization	without Learning	own
[227]	2016	Automatische Analyse und detection graphischer Inhalte von SVG-basierten Engineering-Dokumenten	P&ID	detection	without Learning	own
[228]	2017	Heuristics-Based Detection to Improve Text/Graphics Segmentation in Complex Engineering Drawings	P&ID	detection	without Learning	own (industry)
[229]	2018	Symbols Classification in Engineering Drawings	P&ID	detection	with Learning	own
[230]	2019	A Digitization and Conversion Tool for Imaged Drawings to Intelligent Piping and Instrumentation Diagrams (P&ID)	P&ID	detection + contextualization	without Learning	own
[231]	2019	Automatic Information Extraction from Piping and Instrumentation Diagrams	P&ID	detection	with Learning	own
[232]	2019	Applying graph matching techniques to enhance reuse of plant design information	P&ID	contextu-alization	without Learning	own
[233]	2019	Features Recognition from Piping and Instrumentation Diagrams in Image Format Using a Deep Learning Network	P&ID	detection	with Learning	own
[26]	2020	Deep learning for symbols detection and classification in engineering drawings	P&ID	detection	with Learning	own (industry)
[234]	2020	Symbols in Engineering Drawings (SiED): An Imbalanced Dataset Benchmarked by Convolutional Neural Networks	P&ID	detection	with Learning	open source
[235]	2020	Object Detection in Design Diagrams with Machine Learning	P&ID	detection	with Learning	synthetic
[32]	2020	Deep Neural Network for Automatic Image Recognition of Engineering Diagrams	P&ID	detection	with Learning	own
[33]	2020	CNN-Based Symbol Recognition in Piping Drawings	P&ID	detection	with Learning	synthetic
[236]	2020	Graph-Based Manipulation Rules for Piping and Instrumentation Diagrams	P&ID	contextu-alization	without Learning	own
[237]	2020	Deep Learning for Text Detection and Recognition in Complex Engineering Diagrams	P&ID	detection	with Learning	own
[238]	2020	Automatic Digitization of Engineering Diagrams using Deep Learning and Graph Search	P&ID	detection + contextualization	with Learning	own
[239]	2020	Reducing human effort in engineering drawing validation	P&ID	detection + contextualization	with Learning	own
[240]	2020	Integrating 2D and 3D Digital Plant Information Towards Automatic Generation of Digital Twins	P&ID	detection + contextualization	without Learning	own
[241]	2020	Component detection in piping and instrumentation diagrams of nuclear power plants based on neural networks	P&ID	detection + contextualization	with Learning	open source
[242]	2021	OSSR-PID: One-Shot Symbol Recognition in P&ID Sheets using Path Sampling and GCN	P&ID	detection	with Learning	synthetic
[243]	2021	Deep Learning-Based Method to Recognize Line Objects and Flow Arrows from Image-Format Piping and Instrumentation Diagrams for Digitization	P&ID	detection	with Learning	own (industry)
[244]	2021	Group of components detection in engineering drawings based on graph matching	P&ID	contextu-alization	without Learning	own
[245]	2021	Automatic Digitization of Engineering Diagrams using Intelligent Algorithms	P&ID	detection	without Learning	own
[246]	2021	Engineering Drawing Validation Based on Graph Convolutional Networks	P&ID	detection	with Learning	own
[247]	2021	Digitize-PID: Automatic Digitization of Piping and Instrumentation Diagrams	P&ID	detection + contextualization	with Learning	own
[248]	2021	Automatic digital twin data model generation of building energy systems from piping and instrumentation diagrams	P&ID	detection	with Learning	own
[30]	2021	Identification of Objects in Oilfield Infrastructure using Engineering Diagram and Machine Learning Methods	P&ID	detection	with Learning	own
[249]	2021	Object detection for P&ID images using various deep learning techniques	P&ID	detection	with Learning	own
[27]	2022	Pattern Recognition Method for Detecting Engineering Errors on Technical Drawings	P&ID	detection	with Learning	own
[250]	2022	Enhanced Symbol Recognition based on Advanced Data Augmentation for Engineering Diagrams	P&ID	detection	with Learning	own + synthetic
[251]	2022	Modern Deep Learning Approaches for Symbol Detection in Complex Engineering Drawings	P&ID	detection	with LearningL	own
[252]	2022	End-to-end digitization of image format piping and instrumentation diagrams at an industrially applicable level	P&ID	detection + contextualization	with Learning	own
[253]	2022	Automated Valve Detection in Piping and Instrumentation (P&ID) Diagrams	P&ID	detection	with Learning	own
[254]	2023	A Symbol Recognition System for Single-Line Diagrams Developed Using a Deep-Learning Approach	P&ID	detection	with Learning	own + synthetic
[239]	2023	Reducing human effort in engineering drawing validation	P&ID	contextu-alization	with Learning	own
[29]	2023	Advancing P&ID Digitization with YOLOv5	P&ID	detection	with Learning	own
[28]	2023	Improved P&ID Symbol Detection Algorithm Based on YOLOv5 Network	P&ID	detection	with Learning	own
[31]	2023	A Complete Piping Identification Solution for Piping and Instrumentation Diagrams	P&ID	detection	with Learning	own
[255]	2023	Automatic anomaly detection in engineering diagrams using machine learning	P&ID	detection + contextualization	with Learning	own
[256]	2023	Digitization of chemical process flow diagrams using deep convolutional neural networks	P&ID	detection	with Learning	own
[257]	2023	Extraction of line objects from piping and instrumentation diagrams using an improved continuous line detection algorithm	P&ID	detection + contextualization	with Learning	own
[258]	2023	Classification of Functional Types of Lines in P&IDs Using a Graph Neural Network	P&ID	detection	with Learning	own
[259]	2023	Demonstrating Automated Generation of Simulation Models from Engineering Diagrams	P&ID	detection + contextualization	with Learning	synthetisches Dataset
[31]	2023	A Complete Piping Identification Solution for Piping and Instrumentation Diagrams	P&ID	detection + contextualization	with Learning	own
[260]	2024	Rule-based continuous line classification using shape and positional relationships between objects in piping and instrumentation diagram	P&ID	detection	ohne ML	own
[261]	2024	Image format pipeline and instrument diagram recognition method based on deep learning	P&ID	detection + contextualization	with Learning	OpenSource Dataset
[262]	2024	Semi-supervised symbol detection for piping and instrumentation drawings	P&ID	detection + contextualization	with Learning	own
[34]	2024	Auto-Routing Systems (ARSs) with 3D Piping for Sustainable Plant Projects Based on Artificial Intelligence (AI) and Digitalization of 2D Drawings and Specifications	P&ID	detection + contextualization	with Learning	own

in Appendix C, sorted by the engineering domain and year.

The first diagram (Figure 3a) displays the distribution of developed methods per engineering domain. The bar chart demonstrates that many procedures were created for mechanical, civil, electrical and chemical engineering. In computer science and for the general methods, significantly fewer contributions were found. The reason is that only general methods in the context of the engineering domain are considered, e.g., the recognition of arrows or text in symbols. General methods for digitalization of drawings such as vectorization or Optical Character Recognition (OCR) are not included.

In addition to the distribution among the disciplines, the number of publications over time is shown in Figure 3b in two-year increments. A differentiation was drawn between methods with and without a learning technique, which includes Machine and Deep Learning. For many years most approaches applied no learning techniques and focused on the use of conventional algorithms and heuristics. However, an increase in approaches with Machine or Deep learning is observed from the middle of the 2010s onwards. The expansion in publications over time can be explained by the development of new learning procedures (e.g., CNN or Visual Transformer) and growing computational performance.

In addition to the chronological observation, another comparison of the literature is provided in the sunburst diagram in Figure 4. This shows the number of publications for the different categories of sketch digitalization. The analysis reveals that most of the methods focus on symbol detection and less on contextualization or preprocessing. Furthermore, the proportion of procedures with and without learning methods is almost similarly distributed across the entire study.

Learning methods have the disadvantage of requiring a large amount of labelled training data, which is why Figure 4 shows a breakdown of the individual solutions in the outermost circle. A closer look at the methods with learning reveals that many rely on their own dataset. This can be provided by industry partners or is self-generated through databases or scraping methods. Afterwards, these datasets must be manually labelled to be used for training the model.

Another possibility is to utilize open-source datasets, e.g., the GREC’03 [8] for electrical engineering or BRIDGE  [9] and SESYD [10] for architecture. These labelled and prepared datasets help to compare the results of different approaches and trained models. But, for principle sketches in mechanical engineering, there is currently a lack of publicly accessible datasets. A small proportion of the analyzed paper employs a synthetic dataset to train the model. This approach allows to control the data generation and thus adapts the model to a given task. In addition to the high effort for the initial development of a synthetic dataset, the verification of the trained model is challenging. This requires a small dataset that is independent of the synthetic generation and enables to verify the transferability of the trained model.

Regardless of the dataset, Deep Learning-based object detection models are currently state of the art for symbol detection. Accordingly, the next paragraph will briefly discuss the existing methods.

2.2. Object Detection Models

Deep Learning models for object detection are divided into one- and two-stage models. One of the popular single-stage procedures is the YOLO (You Only Look Once) model by [11]. As the name implies, the image is viewed only once and based on that, multiple bounding boxes and classes are proposed. The whole procedure is described as a regression problem, which makes the whole approach fast to train and run.

One of the first two-stage methods was the region-based CNN (RCNN) model of [12]. This approach avoids the problem of having many possible regions for an object’s position, as with the rigid sliding window approach, by using a search algorithm to narrow down the possible bounding box regions to around 2000 suggestions. The AlexNet from [13] is applied as a CNN architecture for the feature generation of different image regions, which are subsequently classified. The disadvantage of the method is the high training effort, the slow detection speed and the missing ability to improve the search algorithm via training. Therefore, the Fast RCNN [14] and later the Faster RCNN [15] were developed to improve the disadvantages of the first model. The Faster RCNN applies a learning search algorithm, the Region Proposal Network (RPN), and applies the enhanced detector from the Fast RCNN. With the release of the MASK RCNN, object segmentation was integrated into the task spectrum [16]. This is again based on the Faster RCNN, but with the implementation of a backbone network, which is responsible for the generation of the feature map.

These object detection methods were employed for the recognition of symbols in sketches, with a variety of approaches applied in the field of civil engineering, some with YOLO [17,18,19,20], RCNN [21,22,23,24] or both models [9,25]. Both detection types were adopted for chemistry diagrams, examples of YOLO models are found in [26,27,28,29,30,31] and for the RCNN in [32,33,34]. There are fewer methods for electrical and mechanical engineering, mainly due to the lack of available datasets. Examples of RCNN-based detection in electrical engineering are presented in [35,36]. For the mechanical engineering domain different approaches for each detection type can be found, one with YOLO [37,38,39] and one with RCNN [40].

2.3. Summary and Conclusion

In summary, symbol detection in engineering domains is a long-standing research area that is continuously adopting the latest technology to accomplish the task. Currently, Deep Learning models are mainly used for recognizing symbols in technical sketches or drawings. However, the availability of data is a major challenge, as the Deep Learning based models require a lot of labelled data for training. The problem becomes even more apparent when it comes to domain-specific tasks like the detection of principle sketches, where open-source datasets are not available and scraping cannot be used. Thus, the following research question emerges: how can the detection of symbols in principle sketches from the mechanical engineering domain be achieved and has the data generation or model architecture a greater influence on the detection?

In order to answer this question in detail, a general framework for the solution of symbol recognition is presented first. The individual steps are then explained in more detail. Finally, the procedure is analyzed and evaluated with unknown datasets.

3. Method

The presented framework for solving the symbol detection of principle sketches is commonly divided into two parts: data generation and models. The detailed steps are shown in Figure 5, starting with the synthetic data generation.

Two different implementation approaches are investigated, whereby both should enable the training of object detection models through the synthetic generation of training images. The transferability of the model to real sketches is especially relevant for the given application domain. The first approach for the image generation is based on the preliminary work of Bickel et al. [41,42]. The second type is newly developed and employs the SketchGraph package from Seff et al. [43] in combination with PTC Onshape [44] for the sketch generation. Subsequently, various object detection models are trained and evaluated on the different training datasets. The aim is to determine the impact of the two steps on the detection of unknown sketches. The varying types of data generation and detection models are presented in the following subsections.

3.1. Data Generation

The objective of the synthetic data generation is to reproduce the symbols of principle sketches as accurately as possible so they are recognised with a high degree of accuracy on real sketches. However, as there is not enough existing data for this sketch type, the symbols must be generated randomly. For this purpose, two procedures were developed with different python packages (Pillow [45] and SketchGraph [43]), which are presented and examined in detail afterwards. The schematic process of the image generation is given in the lower part of Figure 5.

The same constraints apply to both generation methods, starting with a randomly chosen symbol shape within defined limits. Then, the number and variation of the symbols is freely selectable. The total number of training images is also eligible and is primarily dependent on the computing power and required time.

3.1.1. Pillow

The first approach is based on the widely used python library “Pillow”, which was primarily developed for image processing and manipulation. The ease of operation and the ability to quickly generate images with new content were the reasons for selecting the package. In addition, the procedure is capable of multiprocessing, which allows for quick data generation. The general approach is presented in Algorithm 1 and has one main parameter, the activation of a random background. The idea is to start with the creation of a white image in the given size, then a random background consisting of rectangles, lines and ellipses may be generated, if the option is selected. After that, a random number of symbols are drawn on the input image. For this purpose, first a random position of the new symbol is determined and checked on whether this collides with an existing bounding box, if not, the position is passed to the next function.

Algorithm 1: Overview of the data generation algorithm with the Pillow package

This decisive function in the algorithm is “drawSymbol”, in which the selected symbol is drawn through the Pillow package according to previously defined rules. The symbol variables like rotation, line thickness, general size, height and width are defined randomly for each symbol. In addition, a variation factor is defined, which generates the deviations for each symbol shape. Afterwards, depending on the symbol type, the relevant points are generated and then drawn via the “ImageDraw” class. The background generation is used as the main parameter to study the different types of data generation. Therefore, two datasets are considered for training, one with and one without the random background. In theory, the aim is to mimic possible connection lines and, thus, train a more robust detection model by using a random generated background. The result of the different Pillow variants are shown in Figure 6 outlined in blue.

3.1.2. SketchGraph

Unlike the Pillow approach, the SketchGraph uses another way of creating symbols, similar to a sketcher in CAD programs. At first, a command sequence is generated that contains all the relevant information for creating the sketch. SketchGraph distinguishes between node and edge operations and their associated properties. The sequence is then converted into a sketch object, which interacts with Onshape’s API. This link is necessary because Onshape works as a geometric constraint solver and generates the final sketch. The solved sketch can then be rendered with various settings. There is an option to select a “hand-drawn” mode, which alters the lines of the sketch to mimic a hand-drawn sketch. The general program flow is again listed in Algorithm 2.

Algorithm 2: Overview of the data generation algorithm with the SketchGraph package and the Onshape API

Another difference is the possibility to improve the imitation of sketches. Based on the real sketches, symbol pairs are initially formed using previously defined rules, e.g., two gear symbols are directly placed next to each other. In addition, connecting lines between similar domain symbols are also generated, for example, between a rotary joint and a fixed bearing. The selection of contact partners and associated symbols is randomized, taking into account the location of existing symbols for the connecting lines to avoid overlap. As in the previous approach, symbol selection, positioning and variation are randomised, with collision checking performed in each object class. The individual symbol classes employ general rules for the structure of the respective symbol. A special rule was added for the planetary gear, since this symbol requires a lot of space compared to the others. During the creation of the sketch, the number of planetary gears is checked and reduced to one symbol per sketch in order to leave enough space for the other symbols. In addition to the many minor factors, two major elements can be set in the dataset generation: hand-drawn symbols and more realistic sketch representation. Therefore, all possible combinations are used for the training, which results in four different training datasets (SketchGraph 1: no additions, SketchGraph 2: hand-drawn, SketchGraph 3: realistic and SketchGraph 4: hand-drawn + realisitic). The gray framed example images in Figure 6 illustrate the results for the four combinations.

The disadvantage of this method is that it significantly increases the amount of time it takes to generate the data. Although multiprocessing and multiple Onshape documents are used to generate the sketches, the required time is still high. On average, the creation of a sketch with the SketchGraph method takes about 0.7–1 s, whereas the Pillow approach only needs 0.001 s.

4. Experiments

In order to answer the research question, a number of studies are carried out to determine whether the data or the detection model have the larger influence on the recognition accuracy of unknown sketches. For this purpose, a total of six training datasets were generated, two with Pillow and four with the SketchGraph approach. For a better comparison, one-stage and two-stage detection models are investigated, according to the github stars [46] and citation numbers (see

Table A1. Overview of citation numbers of popular object detection algorithms according to Google Scholar on 23 April 2024. Algorithm collection derived from object detection roadmap from [61].

Name	Reference	Citation Number
Faster RCNN	[15]	73,707
YOLO: You only look once	[11]	46,768
Ssd: Single shot multibox detector	[62]	37,166
RCNN	[12]	36,358
MASK RCNN	[16]	34,912
Fast RCNN	[14]	33,608
RetinaNet	[63]	28,943
FPN: Feature pyramid	[64]	25,464
DETR	[65]	11,003
Cornernet	[66]	4241

in Appendix A), popular representatives are the YOLOv5 and the MASK RCNN and Faster RCNN.

4.1. Training Datasets and Evaluation Metrics

The procedure for creating the training datasets was presented in detail in Section 3.1. To ensure that the results are comparable, both methods generated approximately 100,000 training images per dataset with a resolution of 600 × 600 pixel. The SketchGraph method appears to be more prone to error in the generation of images, with the connection to Onshape sometimes failing. This results in up to 100 fewer images per dataset. Both methods produced the same 14 symbol types: fixed bearing, floating bearing, swivel joint, ball joint, straight gear, helical gear, bevel gear, planetary gear, and bearing. Examples of all symbols are shown in Figure 7. The last six symbols are created in two variants; with and without a hollow shaft. The blueprints for the general shapes of the symbols were found in [6,47,48,49,50,51,52]. Subsequently, the trained models will be tested and evaluated on two unknown test datasets, which are presented in detail in the following chapters.

The COCO dataset [53] evaluation metrics consisting of the mean average precision for an IoU of 0.50 (mAP50) and the mean average precision (mAP) are applied to compare the detection capabilities. Both metrics are used for the evaluation, although the mAP50 is more relevant for the present use case, as it is more important for the later contextualization to recognize the symbol correctly than to determine the exact position.

4.2. Implementation

For the realization of the study, the following implementation of the detection models were used: MASK RCNN [54], YOLOv5 [55], Faster RCNN [56]. All of the models were trained using transfer learning with the weights of the COCO dataset as the initial values. The YOLO and Faster RCNN models were trained on a computational server with two AMD EPYC 7643 processors (Advanced Micro Devices (AMD), Santa Clara, CA, USA), 256 GB of RAM and two Nvidia A40 (46 GB), while the MASK RCNN model was trained on a workstation PC with an Intel Xeon W-2125 (Intel, Santa Clara, CA, USA), 32 GB of RAM and an Nvidia Titan V (NVIDIA, Santa Clara, CA, USA).

There are different architectures available for the YOLO model, which vary the width and depth of the model. In this study, the S, M and L models were tested to analyze the influence of the model structure in more detail. For the MASK RCNN models, the backbone networks resnet50 and resnet101 were used and for the Faster RCNN a resnet50 was used.

4.3. Results—Gear Stages Dataset

A collection of principle sketches of gearboxes was used as the first unknown dataset. A total of 30 transmission variants were drawn, which can be roughly grouped according to their gear stage. The sketches consisted of the symbols bearing, straight-, helical- and bevel-gear as well as the housing of each gear unit, with the exact instances counted in

Table A2. Overview of symbol instances in the unknown gear stages dataset without the scaling applied.

Category	Instances	Category	Instances
bevel gear	36	general bearing	216
bevel gear hollow shaft	0	general bearing hollow shaft	0
ball joint	0	swivel joint	0
planetary gear	0	straight gear	82
planetary gear hollow shaft	0	straight gear hollow shaft	0
fixed bearing	0	helical gear	54
floating bearing	0	helical gear hollow shaft	0

in Appendix B. The dataset was created and labelled manually, with the simplest drawing containing six and the most complex twenty-two symbols. Subsequently, the dataset was scaled in three steps, resulting in a total of 90 sketches. The largest sketch was defined as the maximum and then drawings with 80% and 66% scaled sketches were generated. The symbols in the training data varied independently in size, so scaling of the training images was not conducted. The different scales were intended to provide an additional challenge for the models and test their abilities in detection of symbols with unknown sizes. An excerpt of the dataset is shown in Figure 8, with an example for each gear stage illustrated.

The results of the recognition are listed in

Table 1. Evaluation metrics for all models and training datasets on unknown data with best results per model highlighted.

Unknown Gear Stages
Dataset	SketchGraph 1		SketchGraph 2		SketchGraph 3		SketchGraph 4		Pillow 1		Pillow 2
Metric	mAP50	mAP	mAP50	mAP	mAP50	mAP	mAP50	mAP	mAP50	mAP	mAP50	mAP
YOLOv5 S	0.563	0.303	0.180	0.073	0.993	0.694	0.992	0.684	0.388	0.138	0.716	0.245
YOLOv5 M	0.805	0.455	0.076	0.033	0.994	0.689	0.994	0.685	0.625	0.253	0.868	0.309
YOLOv5 L	0.742	0.444	0.084	0.050	0.993	0.686	0.994	0.686	0.817	0.397	0.832	0.315
MASK RCNN r50	0.149	0.056	0.080	0.031	0.741	0.389	0.834	0.440	0.439	0.268	0.235	0.154
MASK RCNN r101	0.105	0.048	0.135	0.057	0.949	0.522	0.785	0.407	0.127	0.079	0.351	0.224
Faster RCNN r50	0.286	0.117	0.241	0.110	0.736	0.526	0.757	0.535	0.042	0.013	0.197	0.105

and indicate that a good detection of relevant symbols is achievable. In addition, the metrics demonstrated that different datasets lead to significantly varying results, with the lowest mAP50 at 0.042 and highest 0.994. In general, the models trained on the SketchGraph dataset performed better, especially when a more realistic data generation was chosen. The best results were achieved with the SketchGraph 3 and 4 dataset and the YOLO v5 M model, each with an mAP50 of 0.994. The exception is the SketchGraph 2 dataset, which did not work with any model type, neither RCNNs nor YOLO. A comparison of the models revealed a clear tendency in favor of the YOLO models, which achieved better results than those of the RCNN family. The average difference between YOLO and RCNN models was 0.302 (mAP50) and 0.171 (mAP), respectively.

4.4. Results—Unknown Assemblies

As a final test, a more challenging unknown dataset was generated for the models. This was derived from the “3D Assembly Repository” of Lupinetti et al. [57] and consisted of several step assemblies, sorted into different product categories. Complex drawings were derived from these step assemblies, with sketches created for six assembly categories. Again, an example sketch for each category is displayed in Figure 9. The assemblies applied were hydraulic reduction, hydraulic rotor, double rotor turbine, wind turbine, differential and landing gear. In contrast to the first unknown dataset, considerably more symbols and many different variants were included in the individual sketches. The entire dataset consisted of 24 images and a detailed list of the occurring symbols is provided in Appendix B in

Table A3. Overview of symbol instances in the unknown assembly dataset.

Category	Instances	Category	Instances
bevel gear	3	general bearing	117
bevel gear hollow shaft	1	general bearing hollow shaft	17
ball joint	0	swivel joint	21
planetary gear	28	straight gear	45
planetary gear hollow shaft	0	straight gear hollow shaft	0
fixed bearing	5	helical gear	5
floating bearing	4	helical gear hollow shaft	0

.

In general, the result scores, which are shown in

Table 2. Evaluation metrics for all models and training datasets on unknown data with best results per model highlighted.

Unknown Assemblies
Dataset	SketchGraph 1		SketchGraph 2		SketchGraph 3		SketchGraph 4		Pillow 1		Pillow 2
Metric	mAP50	mAP	mAP50	mAP	mAP50	mAP	mAP50	mAP	mAP50	mAP	mAP50	mAP
YOLOv5 S	0.659	0.419	0.274	0.180	0.904	0.554	0.893	0.537	0.492	0.170	0.559	0.239
YOLOv5 M	0.717	0.432	0.193	0.149	0.904	0.567	0.868	0.537	0.485	0.188	0.561	0.248
YOLOv5 L	0.688	0.450	0.200	0.142	0.906	0.562	0.891	0.565	0.597	0.226	0.723	0.283
MASK RCNN r50	0.111	0.050	0.080	0.030	0.501	0.184	0.451	0.131	0.189	0.124	0.011	0.007
MASK RCNN r101	0.110	0.037	0.140	0.055	0.642	0.238	0.575	0.195	0.142	0.062	0.061	0.044
Faster RCNN r50	0.352	0.223	0.365	0.241	0.628	0.376	0.664	0.430	0.144	0.026	0.392	0.217

, reveal that the recognition level drops in comparison to the first dataset. This is mainly due to the more complex drawings and increased variety of symbols. Furthermore, finer differences between symbols must be detected in this dataset, e.g., the bevel gear with and without hollow shaft. The findings from the first study are evident in this dataset as well, since the YOLO models performed better than the RCNN architecture and the SketchGraph 2 dataset produced the worst results for all models. The combination of Pillow data generation with the YOLO models also achieves good mAP50 values, although the YOLOv5 L provides better recognition results than the other YOLO derivatives. The highest accuracy is provided by the YOLOv5 L with SketchGraph 3 dataset, resulting in a mAP50 of 0.906 and mAP of 0.562.

4.5. Discussion

The evaluation of the different models and datasets helps in answering the stated research question (see Section 2.3). The first part of the question was successfully answered by the developed procedure, especially through the synthetic generation of the training data. For the second question part, the obvious answer is that the data are more essential for the object detection, since without them no training would be possible. But, this answer is quite simplistic and does not reflect the whole truth.

In general, the results show that the dataset has a higher influence on recognition accuracy than the model, because with the third and fourth SketchGraph dataset it was possible for the YOLO and RCNN models to detect the unknown sketches with high accuracy, as displayed in the top row in Figure 10. Similarly, the second SketchGraph dataset highlights that a less representative dataset leads to problems for all models, as incorrect features are learned from the synthetic sketches. A visual example of the detection results with the SketchGraph 2 dataset and the YOLOv5 S model is depicted in the lower right corner of Figure 10. The influence of the models on the detection results becomes apparent in the Pillow datasets, where the one-stage methods achieved better results than the two-stage ones. A closer look at the RCNN models indicates that the results vary greatly and, thus, no model can be clearly identified as the best for the given task. In contrast, the YOLO models show that the M and L versions are best suited for recognition, as they also delivered higher detection accuracies for the Pillow dataset than the S version.

The structure of the test dataset also influences the evaluation, particularly since the focus was more on realistic drawings and less on quantity. Consequently, symbols were included in the training dataset, but were not present in the test dataset, e.g., the ball joint. However, this problem was tolerated, because the goal of the recognition model was to be as broadly applicable as possible, rather than being a specific solution for only three or four symbols. This particular requirement also leads to detection problems. In this study, the confusion between swivel joints and general bearings occurred frequently, which is also visible in the lower left result image in Figure 10. Since the bearing symbol virtually consists of two swivel symbols, the confusion is quite understandable. One possible solution is to apply a rule-based approach and to ignore the bounding boxes of the swivel joints when they overlap with those of the general bearing. The data generation of the swivel joint could likewise be improved through the integration of connection links, to make the symbols easier to distinguish from each other.

Another recognition challenge was the scaling of the symbols in relation to the drawing. It was noticeable that the RCNN model had significantly larger errors in recognition, especially for smaller symbols. An adjustment of the data generation with a larger variance of the symbol size might lead to a general improvement. For the RCNN models, a smaller anchor size could eventually result in a better detection of smaller symbols. The selection of object recognition models concentrated on the established methods, although other models could be included in the next analyses, e.g., approaches with visual transformers or graph neural networks.

In addition, the synthetic data generation approach offers advantages over generative AI approaches such as generative adversarial networks (GAN) [58] or variational autoencoders (VAE) [59,60]. These techniques cannot be used if no or very little data are available, and it may also be difficult for the automatically trained models to map small details of the symbols, such as a bearing with or without a hollow shaft. Furthermore, GAN and VAE are sometimes very expensive and difficult to train, which leads to a high dependency of the results on hyperparameters and greater training time consumption.

5. Conclusions

In summary, this paper first provided a general overview of current approaches and solutions for the detection and contextualization of principle sketches in engineering, supported by a literature review. The question was raised whether the data or the model architecture has more influence on the recognition accuracy. For this purpose, several datasets were synthetically generated and common one- and two-stage models were trained. Two unknown datasets were produced to test the recognition accuracy, whereby in the first one the symbols were additionally scaled in three steps. The results revealed that the data generation has more influence, since well-prepared data performed overall better with different models. In the opposite case, poorly created data led to almost no detection of the symbols at all.

In the next step, the hyperparameters of the models could be optimized and other architectures be used for training. Recognition of hand-drawn principle sketches is an additional challenge and the SketchGraph approach may be a good starting point.