A Two-Stage Deep Learning Method for Auxiliary Diagnosis of Upper Limb Fractures Based on ResNet-50 and Enhanced YOLO

Abstract

Aiming at the problem that the existing auxiliary diagnosis methods for fractures are mostly limited to specific body parts and lack generality and robustness when applied to multi-part diagnoses, this study proposes a two-stage upper limb fracture auxiliary diagnosis method based on deep learning and develops a corresponding auxiliary diagnosis system. In the first stage, this study employs an improved ResNet-50 model combined with transfer learning and a Squeeze-and-Excitation (SE) attention mechanism for fracture image localization. In the second stage, an improved You Only Look Once (YOLO) model based on Scale Sequence Feature Fusion (SSFF) and Triple Feature Encoder (TFE) modules is used for fracture diagnoses in different body parts. Contrary to the traditional methods that are tailored to specific body parts, the integrated design approach presented in this paper is better suited to meeting the diagnostic needs of multiple body parts, demonstrating better generality and clinical application potential.

1. Introduction

Fractures are common and frequent injuries that occur regardless of age and may occur in any part of the body. If a fracture in a specific part of the body is not treated in time, it may have a serious impact on the patient’s health [1]. Therefore, the timely and accurate diagnosis and treatment of fractures are crucial. Currently, medical imaging techniques have become a key tool for clinicians to assist in fracture diagnoses [2], with X-rays being the method of choice for assessing the condition of patients with fractures [3]. However, accurate interpretation of these images requires extensive clinical experience and expertise on the part of the physician. In clinical diagnosis, the omission or misdiagnosis of fractures occurs from time to time due to factors such as limited resources for senior doctors, a lack of experience in diagnostic imaging, and high clinical workloads [4]. Therefore, the development of auxiliary diagnosis systems for fractures based on medical image analysis technology to improve the accuracy and efficiency of diagnoses has become a focus of attention in the fields of both academic and clinical practice.

At present, deep learning technology has achieved significant results in the field of medical image analysis [5,6]. This study mainly utilizes deep learning technology to achieve two tasks: image classification and object detection. In the field of fracture image classification, by analyzing medical images, the presence of fractures is determined, and the specific types of fractures are identified and classified. Kim et al. [7] used the pre-trained Inception v3 model to identify fractures in the wrist. Urakawa et al. [8] used the VGG16 model to detect fractures in the intertrochanteric femur. Studies [9,10,11] on fracture recognition in other parts of the body have also achieved good results. Object detection models identify the lesion area and mark the possible fracture locations. Yahalomi et al. [12] used Faster R-CNN to detect and localize distal radius fractures. Li et al. [13] used the YOLO v3 model to automatically identify vertebral fractures and evaluated the diagnostic performance of the model through accuracy, sensitivity, and specificity analyses.

Currently, auxiliary diagnosis systems for fractures developed using deep learning technology have achieved certain results. Tomita et al. [14] developed an automated fracture assessment tool capable of assisting in detecting the presence of osteoporotic fractures in patients during CT scans of the chest, abdomen, and pelvis. Arpitha et al. [15] proposed a computer-aided diagnosis platform specifically for detecting, labeling, and segmenting the lumbar region in imaging data and further distinguishing the nature of spinal compression fractures, including whether they are benign or malignant.

In summary, the existing research mostly focuses on the diagnosis of fractures in a single part and fails to cover a wider range of fracture locations. Moreover, most studies do not involve the classification of fractured body parts. However, in actual medical applications, orthopedic surgeons need to make professional divisions based on skeletal parts and image types to help them make quick and accurate judgments on images from any part. Therefore, this paper proposes a two-stage upper limb fracture auxiliary diagnosis method that integrates body part classification and fracture diagnosis for multiple body parts and develops a corresponding fracture auxiliary diagnosis system to improve the diagnostic ability and diagnostic efficiency of orthopedic surgeons for fractures in different parts.

2. Materials and Methods

This study proposes an innovative two-stage upper limb fracture auxiliary diagnosis method, adopting a “classification-first, then diagnosis” strategy. As shown in Figure 1, fracture images are first classified by location, and then specific diagnostic models for each location are applied to distinguishing abnormal types. Compared with single-stage diagnostic methods, this two-stage approach significantly improves the accuracy and applicability of auxiliary diagnoses of fractures in specific body parts. The method is designed in a modular fashion, including four parts: a pre-preparation module, a classification and recognition module, a diagnostic module, and a system-development module. The functions of each module are described in detail below.

The pre-preparation module: This module involves collecting or creating datasets according to the research objectives and preprocessing the datasets for subsequent research. In this study, multiple datasets, including MURA, GRAZPEDWRI-DX, and FracAtlas, were integrated into an upper limb fracture dataset comprising the hand, elbow, and shoulder.

The classification and recognition module: The core task of this module is to construct an upper limb fracture location recognition model based on the dataset. This study employs an improved ResNet-50 model incorporating transfer learning and the Squeeze-and-Excitation (SE) attention mechanism to enhance the recognition ability for upper limb fractures.

The diagnostic module: This module constructs specific fracture diagnostic models for each part, implementing the corresponding diagnosis models based on the classification results from the previous module. An improved YOLO model incorporating Scale Sequence Feature Fusion (SSFF) and Triple Feature Encoder (TFE) modules is used to build diagnosis models for the hand, elbow, and shoulder.

The system-development module: This module involves analyzing the user requirements, designing a user-friendly system interface, conducting system trials, and ensuring the system can complete the intended tasks and achieve all preset functions.

In the application of this method, researchers can select the appropriate models and optimization algorithms according to the actual research situation to develop more suitable auxiliary diagnosis systems. This study focuses on research into auxiliary diagnosis of upper limb fractures.

2.1. Data Collection and Preprocessing

By integrating real, publicly available patient fracture image data from multiple hospitals, this study constructs an upper limb fracture X-ray image dataset comprising the hand, elbow, and shoulder. The main datasets are briefly introduced as follows.

The MURA [16] dataset provides 12,173 patients, 14,863 studies, and a total of 40,561 multi-angle X-ray images from Stanford Medical Center from between 2001 and 2012, covering seven areas of the upper extremities—the elbow, fingers, forearm, hand, humerus, shoulder, and wrist—classified as normal or abnormal.

The GRAZPEDWRI-DX [17] dataset provides a total of 20,327 X-ray images of traumatic wrist injuries in children from 6091 patients who were treated at the University Hospital of Graz between 2008 and 2018, labeled with a variety of diagnostics by placing a bounding box, such as fractures, periosteal reactions, and other information.

The FracAtlas dataset [18] is a collection of 4083 X-ray images from three major hospitals in Bangladesh with masks and bounding boxes and classification labels for each fracture instance.

The dataset constructed using this method contains the three categories of hand, elbow and shoulder, with 2000 cases of fracture images in each category, covering five types of diagnostic information, including fractures, periosteal reactions, pronator signs, soft tissues, and text. In addition, the following preprocessing operations were used in this study:

• Data augmentation. To enhance the diversity of the medical image dataset for specific body regions, random cropping was used for data augmentatio;
• Size transformation. The classification task dataset was resized to 224 × 224, and the object detection task dataset was resized to 640 × 640 to meet the training requirements of the corresponding models.

2.2. The Classification Recognition Stage—The Upper Limb Fracture Location Recognition Model

ResNet [19], as a deep convolutional neural network model, introduces residual blocks, allowing the network to have a significantly increased number of layers while maintaining a good performance and solving the gradient vanishing problem often encountered in neural network training. This study employs an improved ResNet-50 model that integrated transfer learning and the Squeeze-and-Excitation (SE) attention mechanism to construct a more accurate upper limb fracture location recognition model.

The transfer learning strategy used in this study uses the weights of the ResNet-50 model pre-trained on ImageNet-1k [20] as the initialization parameters, without freezing any layer’s parameters to adapt to the new upper limb fracture dataset. This approach allows all of the network parameters to participate in gradient descent and be updated, aiming to enhance the model’s recognition ability for new categories. By training on the new dataset, the model can more accurately identify different body parts.

SENet [21] is integrated into the residual block of the ResNet-50 model to form an enhanced SENet-ResNet residual block. Applied to the fracture image classification recognition module, SENet adaptively adjusts the channel activations, enhancing the model’s discriminative power for image features without adding extra spatial dimensions. This enables the model to focus more precisely on key areas during fracture image location recognition, thereby improving the accuracy of fracture location identification. The structural comparison between the improved and original ResNet-50 models is shown in Figure 2, where the numbers after ‘fc’ denote the output dimensions of the two fully connected layers in the SE module.

2.3. The Diagnosis Stage—A Fracture Diagnosis Model Based on Improved YOLOv7

Compared with traditional two-stage object detection models, YOLO series models have the advantages of a fast detection speed and ease of implementation and have been widely used in practical applications. To further enhance the model performance, inspired by Kang et al. [22], this study introduces the Scale Sequence Feature Fusion (SSFF) and Triple Feature Encoder (TFE) modules into YOLOv7 and incorporates the Channel and Position Attention Mechanism (CPAM) [22] to integrate the features from these two modules, improving the diagnostic accuracy of the method for multi-annotated types of fracture images during the diagnosis stage.

The introduction of the SSFF module, applied to the diagnosis process for fracture images, decomposes the fracture images into five levels of features, each capturing different details or scales of image information. After aligning the resolutions of all feature maps with the lowest-level feature map containing key information for small object detection, effective extraction of the scale sequence features is achieved. The introduction of the SSFF module enables the integration of the high-level information in deep feature maps with the detailed information in shallow feature maps, enhancing the model’s detection capability for small targets.

The latest Triple Feature Encoder (TFE) module [22] can simultaneously focus on large-, medium-, and small-sized features, with an emphasis on amplifying features in large-sized feature maps to capture the image details better. After introducing the TFE module into the diagnostic model, the output feature maps of fracture images combine large-, medium-, and small-sized feature maps, maintaining the original resolution but increasing the number of channels threefold.

To integrate the detailed and multi-scale information introduced by the SSFF and TFE modules, this study introduces the Channel and Position Attention Mechanism (CPAM) [22]. It includes a channel attention network and a position attention network. The channel attention network receives the detailed feature output image from the TFE module, while the position attention network processes the output from the SSFF module. The final combination enhances the focus on spatial positions.

The improved YOLO network structure integrates the SSFF module, the TFE module, and the CPAM dual-attention combination module, with the rest of the architecture remaining the same as the original YOLO structure. Through these integrations, the improved YOLOv7 model, while retaining the advantages of the original architecture, significantly enhances the detection performance of the diagnostic model for the subtle abnormalities in fracture images, as shown in Figure 3.

2.4. Development of the Fracture Auxiliary Diagnosis System

2.4.1. Analysis of the System Function and Workflow

The two-stage system was implemented and tested in a simulated clinical environment. The system not only improved the diagnostic accuracy but also increased the efficiency of doctors in diagnosing fractures, demonstrating its potential for practical application. Based on the user requirements and the completeness of the auxiliary diagnosis system for fractures, the user experience interface designed in this study should include three main functions: upload, prediction, and display. Brief descriptions of these functions are as follows:

• The upload function: This includes the functions of uploading fracture images and selecting the model weights. The upload function allows users to select images for detection from their computers. According to the design of this study, after an image is uploaded, it will first be classified by the location classification system, and then the corresponding object detection model weights will be called based on the classification results. Subsequent users can also upload their own newly trained target detection model weights based on this function.
• The prediction function: This function is the core of the auxiliary diagnosis system, including body part classification detection in the fracture images and object detection of fracture regions. Users can easily trigger the prediction process through the buttons on the UI interface, and the system will automatically connect and call the corresponding prediction or detection functions.
• The display function:This function is responsible for displaying the original images and diagnostic results. On the one hand, it shows the original fracture images before auxiliary diagnosis; on the other hand, it displays the images processed by the auxiliary diagnosis system, including the classification results and bounding boxes annotated by the target detection model.

Combining the above functional analysis, the fracture image auxiliary diagnosis system works by first dividing the input fracture images into parts and then calling the object detection models trained for the corresponding parts to annotate the fracture images, highlighting the suspected fracture areas and further categorizing abnormal types. A diagram of the specific workflow is shown in Figure 4.

2.4.2. The Design of System Interfaces

The structure of the auxiliary diagnosis system designed in this study mainly consists of four layers: the user interface layer, the logic layer, the core functionality layer, and the data layer. The following is a brief description of each layer:

• The user interface layer:A simple and practical UI interface was designed using Qt Designer to facilitate user operations.
• The logic layer: This layer was mainly implemented through Python (version 3.11.5) programming of the backend of the UI interface. The buttons in the UI interface are bound to various functions in the program through the backend logic. These functions are responsible for calling certain models, such as classification recognition or target detection, thereby enabling the execution of the corresponding functions by clicking buttons.
• The core functionality layer:In the fracture image auxiliary diagnosis system designed in this study, the weights of the pre-trained optimal fracture image body part classification model and the fracture image target detection model are saved for easy calling through the classification model’s predict function and the target detection model’s detect function to achieve the corresponding functions.
• The data layer: This mainly includes the datasets used during model training. This layer is generally not involved in the normal operation of the auxiliary diagnosis system. However, if the system needs to be maintained or adjusted, or if a hospital wishes to add new images to the dataset for training based on its clinical data, this layer will be involved.

Based on the above functions and the four-layer structure of the user interface, the user interface was designed using Qt Designer software. Modules such as QPushButton, QLabel, and QFrame can be used to complete functions such as creating buttons and text, image display, and final beautification of the user interface. After the basic design is completed, the interface’s construction is finished. The final running interface of the auxiliary diagnosis system is shown in Figure 5.

3. Results

3.1. The Evaluation Metrics

The upper limb fracture location recognition model in this study uses a confusion matrix and evaluation metrics such as accuracy, precision, and sensitivity [23] to assess the model performance. The binary confusion matrix is shown in

Table 1. The binary confusion matrix.

True	Predicted Labels
Labels	Positive	Negative
Positive	TP	FN
Negative	FP	TN

.

Here, TP, FP, FN, and TN represent the number of true positives, false positives, false negatives, and true negatives, respectively. The formulas for accuracy, precision, and sensitivity are as follows:

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(1)

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(2)

$$\mathrm{Sensitivity}={\displaystyle \frac{TP}{TP+FN}}$$

(3)

Additionally, the fracture diagnosis models for each part use the mean average precision (mAP) [24] as the evaluation metric, with mAP0.5 and mAP0.5:0.95 being the primary indicators used in this study. mAP0.5 refers to the average precision calculated at an IoU threshold of 0.5.

$${\mathrm{mAP}}_{0.5}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathrm{AP}}_i,IoU=0.5$$

(4)

mAP0.5:0.95 is the average of the calculated mean average precision over multiple IoU thresholds (from 0.5 to 0.95 in steps of 0.05).

$${\mathrm{mAP}}_{0.5:0.95}={\displaystyle \frac{1}{10}}\sum _{k=1}^{10}{\mathrm{mAP}}_k$$

(5)

3.2. The Experimental Results of the Recognition Model

Both transfer learning and the introduction of the SE attention mechanism contribute to an improvement in the model’s classification performance. To further analyze how transfer learning and SE attention mechanisms affect the model performance, the accuracy during the training process was plotted every 10 epochs for the hand category, as shown in Figure 6.

The observation results indicate that after the introduction of the SE attention mechanism, the model’s accuracy increased slightly every 10 epochs, recognizing fracture images of different parts better. With transfer learning, the model’s accuracy significantly improved in the early stages of training (the first 10 epochs), thanks to pre-training. These data demonstrate that the introduction of transfer learning and the SE attention mechanism effectively improved the ResNet-50 model, with a performance significantly better than that of the original model.

Meanwhile, this study compared several models, including AlexNet, VGG, Inception V3, ResNet-34, the original ResNet-50, pre-trained ResNet-50, SE-ResNet-50, and the improved ResNet-50 used in this study, on the upper limb fracture dataset. The dataset was divided into a training set, a validation set, and a test set at a ratio of 6:2:2. Other parameters (such as a BatchSize of 64 and an initial learning rate of 0.0001) were kept consistent to ensure the accuracy of the experimental results. The accuracy and loss values of the models on the training set and the validation set are shown in

Table 2. The training metrics of the various models.

Model	Loss	Accuracy	Val_Loss	Val_Accuracy
AlexNet	0.113	0.931	0.121	0.922
VGG	0.121	0.930	0.124	0.928
Inception v3	0.189	0.926	0.202	0.916
ResNet-34	0.149	0.924	0.151	0.914
ResNet-50	0.138	0.942	0.136	0.938
ResNet-50 + SE	0.115	0.949	0.127	0.942
ResNet-50 + Transfer	0.045	0.975	0.048	0.970
ResNet-50 + SE + Transfer	0.039	0.981	0.043	0.976

.

A comparison of the results on the classification performance on the test set is shown in

Table 3. Comparison of classification results of different models.

Model	Accuracy	Sensitivity			Precision
		Elbow	Hand	Shoulder	Elbow	Hand	Shoulder
AlexNet	0.919	0.933	0.924	0.987	0.973	0.964	0.954
VGG	0.923	0.929	0.931	0.996	0.960	0.969	0.948
Inception v3	0.913	0.834	0.925	0.991	0.994	0.944	0.929
ResNet-34	0.910	0.916	0.921	0.973	0.966	0.982	0.939
ResNet-50	0.930	0.923	0.933	0.992	0.982	0.954	0.939
ResNet-50 + SE	0.932	0.930	0.931	0.993	0.986	0.963	0.962
ResNet-50 + Transfer	0.976	0.952	0.989	0.996	0.989	0.970	0.967
ResNet-50 + SE + Transfer	0.977	0.965	0.995	0.998	0.995	0.972	0.968

below.

The results show that the improved model used in this study outperforms the other mainstream models in terms of its performance.

3.3. The Experimental Results of the Diagnosis Model

To verify the improvement effects of the modules mentioned in this paper on the baseline model, the following ablation experiments were designed: the baseline model, the YOLO model with the SSFF module, the YOLO model with the TFE module, the YOLO model with both the SSFF and TFE modules, and the final model with the SSFF, TFE, and CPAM modules. These models were trained under the same experimental parameters and hardware conditions, with the primary evaluation metrics being mAP0.5 and mAP0.5:0.95. The results are shown in

Table 4. The results of the ablation experiments.

Model	mAP0.5	mAP0.5:0.95
YOLO	0.759	0.397
YOLO + SSFF	0.772	0.414
YOLO + TFE	0.768	0.410
YOLO + SSFF + TFE	0.778	0.422
YOLO + SSFF + TFE + CPAM	0.791	0.438

.

The results showed that the SSFF module improved mAP0.5 by 0.013 and mAP0.5:0.95 by 0.017, primarily by enhancing the detection of small-scale fractures. The TFE module further improved mAP0.5 by 0.006 and mAP0.5:0.95 by 0.008, demonstrating its effectiveness in capturing multi-scale features. Finally, the CPAM module contributed an additional 0.013 to mAP0.5 and 0.016 to mAP0.5:0.95 by refining the feature maps through channel and position attention. These modules effectively improved the model’s performance and enhanced the ability to diagnose small-region targets in fracture imaging diagnosis.

In addition to this, this study compared YOLOv5 [25], YOLOv8 [26], the YOLOv8-SENet improved model [27], and the YOLOv8-CBAM improved model [28]. All of the models were trained with the same default parameters (such as BatchSize and the IoU) and on the same device and cloud server for 100 epochs to minimize external influences. The YOLO improved model used in this study achieved the following indicators after training on the homemade dataset. The confusion matrix is obtained as shown in Figure 7 below.

Observation of the confusion matrix shows that the improved YOLO model performs well in recognizing the categories of soft tissue, fractures, and text. Below are the bar charts (Figure 8) for the mAP values measured in the fracture category for each model (Model 1: YOLOv5 [25]; Model 2: YOLOv8 [26]; Model 3: YOLOv8-SENet [27]; Model 4: YOLOv8-CBAM [28], and the model used in this study), with the main evaluation metrics of mAP0.5 and mAP0.5:0.95.

Based on the above analysis, the YOLO improved model used in this study demonstrated the best performance for fracture type detection and diagnosis. Compared with other basic or existing improved YOLO models, this improved model can more effectively identify and locate subtle abnormalities in fracture images, enhancing the performance in fracture image diagnosis tasks.

3.4. Application Effects of the System

National Taiwan University first applied the YOLOv9 algorithm model to the GRAZPEDWRI-DX dataset, achieving a state-of-the-art (SOTA) performance, with mAP0.5:0.95 increasing to 43.73% [29]. To verify the application effect of the system in diagnosing different parts, the following experiments were designed. First, 50 fracture images of the hand, elbow, and shoulder that were not in the training set were selected. Second, two methods were used for diagnosis. Method 1: Without prior part recognition and classification, the hand target detection model trained directly using the YOLOv9 algorithm model was used for diagnosis to simulate the optimal performance of a single-part auxiliary diagnosis system in current multi-part fracture cases. Method 2: The recognition model was first used to determine the part of the image, and then the corresponding part-specific object detection model was called for diagnosis, simulating the multi-part auxiliary diagnosis system designed in this study. The mAP0.5 value of the diagnosis model in the category of ‘fracture’ is used as the evaluation criterion, and the experimental results for the two scenarios are shown in

Table 5. The experimental results of the two methods.

mAP0.5 (Fracture)	Hand	Elbow	Shoulder	Average
Method 1	0.898	0.706	0.695	0.766
Method 2	0.903	0.875	0.862	0.880

.

As shown in the table, Method 2 outperformed Method 1 in its diagnostic performance across three body parts and demonstrated a superior average performance. Although the model used demonstrated an excellent performance in this field, its performance may be reduced when facing multi-part diagnoses due to the complexity of part classification. It is only suitable as a single model for diagnosing a single part and does not show robustness. Method 2 achieved zero misclassifications in the initial classification stage and employed dedicated object detection models tailored to the hand, elbow, and shoulder. This targeted diagnostic strategy enabled significant improvements in diagnostic accuracy for the elbow and shoulder compared to that of Method 1, with the mAP0.5 metric increasing by 23.9% for the elbow and 24.0% for the shoulder.

This demonstrates that single-part fracture auxiliary diagnosis systems perform poorly in multi-part clinical applications. In contrast, the “classification-first, then detection” system proposed in this study significantly improves the accuracy, practicality, and scalability compared to those of traditional single-part auxiliary diagnosis systems when facing fractures in multiple body parts in a clinical environment.

4. Discussion

This study aims to address the limitations of the existing fracture image auxiliary diagnosis systems when facing fractures in various body parts and to improve their practicality and effectiveness in clinical settings. The main innovations and contributions of this study are as follows:

• A novel “classification-first, then detection” method was proposed, and a corresponding auxiliary diagnosis system was developed. The system first classifies the fracture images by location and then calls the corresponding models for precise annotation and diagnosis, effectively enhancing its practicality and effectiveness in clinical applications. The system also features a user-friendly interface to provide auxiliary decision-making for clinicians.
• Inspired by Kang et al. [22], we enhanced our model by adopting their network architecture and innovatively applied it to fracture imaging diagnosis for the first time. This improvement boosted the model’s ability to detect subtle fracture types, offering strong support for subsequent research.

Simultaneously, there are still some limitations and imperfections in the research work of this paper.

• The possibility that the input relationship of the TFE module in the diagnostic phase model can be inverted or independent remains in the theoretical investigation stage, and there is still a lack of substantive experiments to verify its invertibility or independence. Subsequently, we will design additional experiments to validate the feasibility of this idea.
• In the proposed two-stage auxiliary diagnosis method, the models used in the classification and diagnosis phases have achieved promising results but have nott reached the state-of-the-art level or the optimal performance for their respective tasks. Future work will involve further research to identify higher-performing models to integrate into the proposed method.

For future developments in auxiliary diagnosis methods for fractures based on deep learning, the following directions are proposed based on the knowledge gained in this study: Firstly, the accuracy and efficiency of the auxiliary diagnosis system should be improved further by expanding the high-quality dataset, and then the application of the system should be promoted. Second, we wish to intelligently upgrade the future test result interface to not only provide diagnosis results but also automatically retrieve and recommend treatment plans from medical databases, helping physicians develop more accurate treatment plans for patients.