A Framework of Faster CRNN and VGG16-Enhanced Region Proposal Network for Detection and Grade Classification of Knee RA

We developed a framework to detect and grade knee RA using digital X-radiation images and used it to demonstrate the ability of deep learning approaches to detect knee RA using a consensus-based decision (CBD) grading system. The study aimed to evaluate the efficiency with which a deep learning approach based on artificial intelligence (AI) can find and determine the severity of knee RA in digital X-radiation images. The study comprised people over 50 years with RA symptoms, such as knee joint pain, stiffness, crepitus, and functional impairments. The digitized X-radiation images of the people were obtained from the BioGPS database repository. We used 3172 digital X-radiation images of the knee joint from an anterior–posterior perspective. The trained Faster-CRNN architecture was used to identify the knee joint space narrowing (JSN) area in digital X-radiation images and extract the features using ResNet-101 with domain adaptation. In addition, we employed another well-trained model (VGG16 with domain adaptation) for knee RA severity classification. Medical experts graded the X-radiation images of the knee joint using a consensus-based decision score. We trained the enhanced-region proposal network (ERPN) using this manually extracted knee area as the test dataset image. An X-radiation image was fed into the final model, and a consensus decision was used to grade the outcome. The presented model correctly identified the marginal knee JSN region with 98.97% of accuracy, with a total knee RA intensity classification accuracy of 99.10%, with a sensitivity of 97.3%, a specificity of 98.2%, a precision of 98.1%, and a dice score of 90.1% compared with other conventional models.


Introduction
Cancer starts when healthy cells in the breast change and grow out of control, forming a mass or sheet of cells called a tumor. A tumor may be benign or cancerous. Malignant means a cancerous tumor can grow and spread to other body parts. If a tumor is benign, it can enlarge but not spread. Radiologists must use a computer-aided detection validation accuracy. In the future, it will be possible to combine the proposed method with other models to find diseases above and below the knee more complexly. Feature fusion can detect and classify different diseases [20]. The model contained both a joint-detection step and a joint-evaluation step. There were 216 radiographs taken from 108 RA patients, with 186 assigned to the training/validation dataset and 30 to the test dataset. In the training/validation dataset, pictures of the PIP joints, the thumb's IP joint, and the MCP joints were manually cropped, evaluated by clinicians for joint space narrowing (JSN), and then augmented. To train and test a deep convolutional neural network for joint evaluation, 11,160 images were used. The joint detection machine learning system was trained using 3720 carefully selected images. Putting these two methods together made a model for figuring out how badly radiographic finger joints damage it. With a sensitivity of 95.3%, the model estimated JSN and erosion for the PIP, thumb IP, and MCP joints. JSN's accuracy range was 49.6-65.4 percent, while its erosion range was 70-74.1 percent. The correlation between model and clinician scores per image was 0.72 to 0.88 for JSN and 0.54 to 0.56 for erosion [21].
The proposed system architecture was made up of a CNN layer and a multilayerbased metadata learning layer. This was conducted so that the information was reliable. Sparse coding estimates and metadata-based vector encoding were used for the additional dimension. To keep the geometric format of supervised data [22], a well-structured kneighbored network was used to build nearby limitation atoms. This study proposes SVM-based detection of finger joints and mTS score estimation. Using X-radiations of 45 RA patients, the suggested approach recognized finger joints with 81.4% accuracy and evaluated erosion and the JSN score with 50.9% and 64.4% accuracy, respectively [23]. The proposed model scored JSN and erosion for PIP, thumb IP, and MCP joints with 95.3% sensitivity. JSN had an accuracy range of 49.6-65.4% and an erosion range of 70-74.1%. They correlated the model and clinician scores per image at 0.72-0.88 for JSN and 0.54-0.75 for erosion [24]. The accuracy of the modified pre-trained GoogleNet model was 89%, whereas that of the proposed custom model was 95%. Google Net had a sensitivity of 84% and a specificity of 90%. Model number three was 95% sensitive and 94% specific. When extracted features from customized models (SIFT + CNN) are compared with those from ML classifiers, the custom3 model performed better [25]. The suggested method was compared with other fuzzy clustering methods that are already used to show how well it works. We compared the support vector machine (SVM), Decision Tree (DT), rough set data analysis (RSDA), and fuzzy-SVM classification algorithms to find the best way to group things [26]. The authors aim to create an AI-based computer-aided diagnosis tool that can classify abnormalities by reading chest X-radiations and help doctors make an accurate diagnosis quickly. We used a Google-created convolutional neural network (CNN) called XceptionNet to find those pathologies in the ChestX-radiation-14 data. Additionally, the same data are being used to run other CNN-ResNet algorithms [27]. At the 100th training iteration, the mean square error and the false recognition rate dropped below 1.1%, suggesting that the LPRNN was trained correctly. Edge preservation index values were above the experimental threshold of 0.48, signal-to-noise ratios (SNRs) were greater than 65 dB, peak SNR ratios were greater than 70 dB, and destruction times were faster [28]. Principal Component Analysis improved characteristics, while the Co-Active Adaptive Neuro-Fuzzy Expert System sorted images of the brain into glioma or non-glioma groups. The PCA and CANFES classification techniques had a sensitivity (Se) of 97.6%, a specificity of 98.56%, an accuracy of 98.73%, a precision of 98.85%, a false positive rate of 98.11%, and a false negative rate of 98.185 [29]. Table 1 illustrates the various state-of-the-art methods for knee RA classification.

•
The proposed system predicts the minimal joint space narrow region and knee RA severity grade value. • The proposed system's experimental analysis was carried out using various criteria. The RA severity classification parameters such as sensitivity, specificity, accuracy, precision, and dice score. • Our proposed system classification paradigm outcomes perform better than traditional techniques.

Organization of Work
Section 1: this paper also discusses the techniques for dataset validation and inflammatory mediator ground truth production and discusses the different state-of-the-art method performances. Section 2: following the data collection step, pre-processing and segmentation of the thermograms take place. Section 3: in the last step, the algorithm differentiates between abnormal and normal knee thermograms and then divides aberrant knee thermograms into three distinct categories. Section 4: provides the RA classification results from various parameters and compares the results of various existing techniques.

Materials and Methods
The primary goal of this study is to examine whether or not a deep learning strategy is effective for RA categorization. In our presented system, we use two approaches like; (i) feature extraction for ROI localization using a deep learning model (active F-CRNN + Hybrid ResNet101 with domain adaption); and (ii) feature selection via a supervised learning technique (marginal joint space narrowing region). To classify the severity of RA in the knee, AlexNet was used. The following procedures were carried out on our system.

Materials
The study encompassed patients with RA symptoms who were older than 55. (knee pain, rigidity, palpitation, and impaired functioning). The BioGPS database repository provided the digitized X-radiation images of the patients (805 men and 1207 women), which are a publicly accessible dataset; hence, the total number of patients is 2012. We discarded 181 radiographic images out of a total of 3353 records for reasons such as postoperative assessment, injury, and infection. Thus, only 3172 X-radiation images were acquired for analysis. Knee X-radiation digital images start in the DICOM format, but are easily converted to the universal JPG format for further use [30]. The digital X-radiation of the knee joint had a resolution of around 3000 by 1500 pixels. Before conducting the analysis, the image brightness was standardized. Three medical domain specialists reviewed each digital X-radiation set (from the Dindigul scan center, Dindigul). Medical domain experts manually examined each digital X-radiation image in order to obtain two ground truth data points (minimum joint space narrowing area and RA classification using CBD grading criteria). Table 2 displays the numerous consensus-based decision grades used in analyzing rheumatoid arthritis.  Table 3 depicts the total number of digitized knee X-radiation images and CBD grading evaluations by three clinical professionals. We used 80% of the data for training, and we further split the training data into training (70%) and validation (10%). The remaining data (20%) were used for testing. The split-up of the dataset is displayed in Table 4.

Methods
The initial step involved reducing an image to a size of 227 × 227 × 3. We used two convolutional neural networks for RA detection and classification in the second phase (Hybrid ResNet101 and VGG16). The image characteristics must be sufficient for accurate CBD grade determination and effective RA classification. We identified marginal joint space narrowing and categorized RA using these convolutional neural networks to extract instructive visual characteristics. To complete the RA classification process, two CNNs were used. First, we used ResNet101 with a domain adaptation strategy to identify marginal joint space contraction. Second, we used VGG16, which was trained using a domain adaptation technique, to classify RA. Finally, we evaluated the method's effectiveness and contrasted our findings with those of other similar techniques already in use. Figure 1 shows a flowchart of the recommended process. The above Equations (1) and (2) represent the overall loss value ( Loss local ) of an enhanced region proposal network along with an association of the classification loss. ( smooth f n Loss local1 , Loss local1 is robust loss, b i is aground-truth regression targets, and t a i 1 is a predicted tuple. p is computed by a softmax.

Determination of Joint Space Narrowing
In deep learning, the F-CRNN is just one of the methods. The faster CRNN architecture is now the standard object identification method because of its ability to anticipate and score single or numerous items in an image. The enhanced-region proposal network and F-CRNN are two integral parts of the F-CRNN network. To ensure that the Quick R-CNN module receives only the best region suggestions, the ERPN generates them. To identify areas of interest in digital X-radiation pictures, we trained the ERPN. The end-to-end convolutional network (ERPN) can accurately anticipate the boundaries and scores of objects of interest at any coordinate. The ResNet101 network was used for F-CRNN feature extraction.
Each convolutional layer in ResNet101 was followed by a batch normalization and activation layer (ReLU). By avoiding parallel connections to the typical layers, this architecture facilitated more efficient training of deep neural networks. Features were extracted, and convolutional feature maps were generated using a combination of convolutional and max-pooling layers. Image characteristics were fed into ERPN, and region suggestions were generated as outputs. The ROI pooling layer took the feature vectors from the function maps. Each vector function was linked to the underlying layer. We individually trained the ROI detection model for the AP view's medial and lateral compartments. When the algorithm produced several ROI detections, we chose the ROI with the highest prediction accuracy for each knee joint. To evaluate the proposed model, we counted the narrow regions of the marginal joint space that achieved IoU ≥ 0.70. As a result of the detection, we saved the predicted bounding boxes. We used weights that had already been trained on ResNet-101, and then used the domain adaptation method to fine-tune them. Figure 2 shows how modified ResNet-101 can find approaches with a narrow joint space in the knee. The most important part of the Faster R-CNN architecture is ERPN. ERPN predicts the scores of objects and their locations. The algorithm compares the narrow areas of the knee joint space in the medial and lateral compartments to find the narrow area in the middle. The best thing about this method is that it can find even the smallest changes in knee joint space.

RA Classification
For knee RA severity classification, we conducted this research using a modified version of the VGG16 architecture and a domain adaptation technique, as shown in Figure 3. The VGG16 model was made up of five convolutional layers, three max-pooling layers, and three fully connected layers-all the digital X-radiation images needed to be resized to (227 × 227 × 3). In our implementation, X-radiation image information for training purposes accounted for eighty percent of the total, while X-radiation image information for evaluating purposes accounted for twenty percent. Although there are sixteen layers in VGG16, only a subset of those layers is required for feature extraction. In order to shorten the amount of time needed for training and establish more control over the fitting process, we assigned a dropout ratio value of 0.5 to the completely connected layer (fcl6) and the fully connected layer (fcl7). The characteristics were taken from the fully connected layers designated fcl6 and fcl7, respectively. To categorize the retrieved features into 1000 categories, VGG16's architecture used a fully connected layer (fcl8). Then, we conducted one last round of tuning on the pre-trained VGG16 model's ability to classify RA by changing parameters in the model's last three layers. The model's last three layers were swapped out for a fully linked layer, a softmax layer, and a classification layer. In addition, a newly linked layer was assigned to five groups of RA grades for the dataset: Grade 0, Grade 1, Grade 2, Grade 3, and Grade 4. We trained the proposed network by using digital knee X-radiations, a small-batch test dataset, gradient descent, and maximal epochs. Our proposed network learning strategy used stochastic gradient descent, and we compared its performance to previous efforts. The proportion of knee X-radiation images from the test set for which the network correctly predicted the RA grade was used to calculate proposed work accuracy. The proposed approach achieved an overall accuracy of 99.10% in classifying knee RA cases. Table 5 illustrates the Visual Geometry Group (VGG16) CNN operation for RA grade classification. Figure 4, depicts the RA classification using VGG16 architecture.

Experimental Parameter Settings
For our investigation, we utilized a machine with 8 GB of RAM and 256 GB of SSD, an Intel i3-core CPU, and Radeon R2 graphics. For image processing, we selected MATLAB 2020-a. Each stochastic gradient descent iteration used a batch size of 256 for both the F-CRNN and the enhanced-region proposal networks, and a learning rate of 4e-3 was applied. The presented approach method corresponds to around 4 h of model training, and the maximum number of iterations was 0.6 k.

Detection of Marginal Knee Joint Space Narrowing
Sample images of the marginal joint space narrowing region of interest can be seen in Figure 5. The IoU (Intersection of Union) metrics were used to evaluate our region of interest detection system. This metric was the size of the intersection between the area of the actual bounding box and the area of the predicted bounding box divided by the size of the area of both boxes added together. When the IoU was 0.70, the narrow marginal joint space was found in 99.72% of the knee joints using our presented model. Additionally, Figure 6 depicts the ROC curve for marginal joint space narrow detection. The results of the presented marginal joint space narrow detection model obtained a sensitivity rate of 98.67%, a Dice score of 98.58%, a precision rate of 98.46%, a specificity rate of 98.50%, a false positive rate of 0.0100, a false negative rate of 0.0197, and an overall accuracy rate of 98.97%, as shown in Table 6, and the graphical illustration of Table 6 values is depicted in Figure 7. Table 7 demonstrates the metric performance outcomes of the proposed ResNet101 and VGG16 model to classify the RA. From Table 7, the outcome of the VGG16 outperforms the well-pre-trained ResNet101 model in classifying RA.

Parameter Metrics for Performance Computation
In our presented systems, classification accuracy analysis was computed by five different performance metrics: sensitivity, specificity, precision, accuracy, and Dice score.
Here, β-represents the true positive values, ø-represents the true negative values, µ-represents the false negative values, and γ-represents the false positive values.

Intensity Classification of Rheumatoid Arthritis
The presented model achieved 99.10% accuracy on the whole test set. The confusion matrix of the presented method is shown in Figure 8, and its performance is compared in detail to that of current methods in Table 5. In Figure 8, we examine the training and learning procedure as a whole to assess the planned activity's success. Table 6 demonstrates the highest accuracy rate for classifying CBD grades zero-three-four knee joints. The knee joints with a CBD grade of one or two are the toughest to categorize. As can be seen in Figure 8, there is only a marginal amount of room for error when classifying knee joints as CBD Grades zero, three, or four. Knee joints that are classified as CBD Grades one or two have a small number of marginal misclassifications. In several circumstances, the proposed approach incorrectly estimated CBD Grade two as Grade one and vice versa. Joint space narrowing and bony spur development are significantly different in CBD Grade four knee joints. However, CBD-grade one knee joints show little change in JSN or osteophyte growth compared with the other classes. Types of knee RA and their intensity levels are shown in Figure 5. Table 8 and Figure 9 illustrate the JSN accuracy of the proposed and other state-of-the-art methods comparison.   In this study, we developed a deep learning model to automatically grade the severity of knee RA using a consensus-based approach. We compared the proposed work to prior strategies and found that it outperformed the competition. At the elementary level, notably in Grade one and Grade two, we found that our method differed from that of the medical professionals. We evaluated the presented work by comparing its results with similar existing studies. Compared with previously existing models, the presented work (a knee joint space narrowing diagnosis and class label) fares very well. It takes about 7 h of training to reach 0.6 k iterations. The outcomes of the presented methodology are shown in Table 9, which includes the outcomes of each CBD grade individually. Multiple metrics were employed to estimate the model's performance, as indicated in Table 10. Figures 10 and 11 depict the ROI curve for RA severity classification for both knees. work (a knee joint space narrowing diagnosis and class label) fares very well. It takes about 7 hours of training to reach 0.6 k iterations. The outcomes of the presented methodology are shown in Table 9, which includes the outcomes of each CBD grade individually. Multiple metrics were employed to estimate the model's performance, as indicated in Table 10. Figures 10 and 11 depict the ROI curve for RA severity classification for both knees.   work (a knee joint space narrowing diagnosis and class label) fares very well. It takes about 7 hours of training to reach 0.6 k iterations. The outcomes of the presented methodology are shown in Table 9, which includes the outcomes of each CBD grade individually. Multiple metrics were employed to estimate the model's performance, as indicated in Table 10. Figures 10 and 11 depict the ROI curve for RA severity classification for both knees.    Figure 12 shows that the presented system outperformed other methods in terms of sensitivity (Se), specificity (Sp), precision (Pr), accuracy (Acc), and dice score (Ds), demonstrating deep learning's capability. Figure 13 depicts the CBD grade outcome doughnut chart. In this research, the presented model increased overall ROI detection accuracy by up to 0.5 percent and improved classification accuracy by up to 1.18 percent. The proposed model is more dependable as a result of the detailed knee JSN characteristics. The improvement was satisfactory, and we agree with the observation that the AP view has a significant portion of the information necessary to assess the severity of knee RA with the CBD grading system. The CBD score is often examined using the AP view alone. Table 6  Our active deep CNN model acquired a knee joint identification accuracy of 98.97% and a knee RA severity classification accuracy of 99.10% using the presented methodology. This model also gives superior performance to handmade features. The active deep CNN model that we have presented and the pre-trained domain adaptation models that are employed in our system produce improved prediction accuracy outcomes for the five classes of knee RA that were experimentally determined.

Conclusions and Future Work
In this study, we propose a way to find and classify rheumatoid knee arthritis by using a deep convolutional neural network (CNN). We use the domain adaptation strategy to use already-trained models such as ResNet101 and VGG16. We evaluate the results against standard methods. The results of our experiments show that our proposed method is better at diagnosing rheumatoid knee arthritis than the current best practices. The presented approach achieved an classification accuracy of 98.97% and 99.10%. We used active deep CNN to predict and grade knee RA and then compared our results to work performed in MATLAB 2020a before. In this study, we provide a deep learning method for the automated detection and classification of RA in the knee. The presented methodology would analyze digital X-radiation pictures of the knee to identify the ROI (minimum knee joint space narrow area) and determine the degree of rheumatoid arthritis. Soon, we want to use this method to assign grades to MRI scans of knees affected by rheumatoid arthritis.
A potential direction for future research might be developing a system to assist medical professionals in identifying the location and cause of knee inflammation using thermogram images as a secondary diagnostic tool. The size of the dataset will also be increased so that temperature-flow patterns specific to arthritis can be made for better classification. Additionally, the presented method can be used with other models to find diseases other than knee problems in a hybrid and flexible way. Even more, this may be combined with feature fusion techniques for diagnosing and categorizing a wide range of additional disorders.