A Unified Framework for Automatic Detection of Wound Infection with Artificial Intelligence

Abstract

Background: The surgical wound is a unique problem requiring continuous postoperative care, and mobile health technology is implemented to bridge the care gap. Our study aim was to design an integrated framework to support the diagnosis of wound infection. Methods: We used a computer-vision approach based on supervised learning techniques and machine learning algorithms, to help detect the wound region of interest (ROI) and classify wound infection features. The intersection-union test (IUT) was used to evaluate the accuracy of the detection of color card and wound ROI. The area under the receiver operating characteristic curve (AUC) of our model was adopted in comparison with different machine learning approaches. Results: 480 wound photographs were taken from 100 patients for analysis. The average value of IUT on the validation set with fivefold stratification to detect wound ROI was 0.775. For prediction of wound infection, our model achieved a significantly higher AUC score (83.3%) than the other three methods (kernel support vector machines, 44.4%; random forest, 67.1%; gradient boosting classifier, 66.9%). Conclusions: Our evaluation of a prospectively collected wound database demonstrates the effectiveness and reliability of the proposed system, which has been developed for automatic detection of wound infections in patients undergoing surgical procedures.

1. Introduction

Surgical site infection (SSI) is one of the most common surgical infectious complications and can increase the length of hospital stay, costs, the number of readmissions, and the need for further wound debridement [1,2]. The risk factors for SSI are multifactorial and include wound cleanliness and microbial, patient-related, and procedure-related factors [3]. With advances in surgical skills and perioperative care, the rate of SSI may be expected to decrease. However, the rate of SSI after hospital discharge is increasing, as postoperative lengths of stay have dramatically decreased because of enhanced recovery after surgery programs in developed countries [4]. Furthermore, more elderly people are undergoing surgery with worldwide aging. Surgical patients require novel solutions to enable them to take a more active role in daily self-care of wounds. If the early signs of SSI are not detected by patients or caregivers, delays in visits to the clinic can result in severe wound complications that require hospital readmission, with or without surgical intervention [5].

Traditionally, the surgical staff would make a call to inquire the patients of the recovery of the general condition and the wound, whereas the condition of the wound is difficult to be remotely evaluated by verbal communications only. To reduce the gap between home and hospital, mobile health (mHealth) has been created to help patients upload wound images and obtain immediate responses regarding their wound condition and has proved feasible in several studies [6,7,8]. Furthermore, the patients receive feelings of reassurance because the medical staff continuously and seamlessly monitor their recovery. However, these novel programs depend on professional staff to review the wound pictures and to provide responses, which may add extra workload to already burdened staff [5].

Objectives

To reduce the workload on professional staff, we developed an automatic monitoring protocol for surgical wounds, using smartphone images in patients undergoing general surgery at an Asian academic center. These patients receive diverse surgical procedures, such as open laparotomy, hernia repair, and minimal invasive surgery. They account for a high proportion of surgical patients and are a challenge for artificial-intelligence-supported wound monitoring. At present, medical professionals diagnose wound infections based on the early signs and symptoms of skin redness, swelling, pain, and warmth; the first two components may be accessed on visual examination. With enhancement of the quality of digital images by smartphones, previous work has proved that these images are a reliable tool and may play an integral role in making diagnostic and therapeutic decisions comparable to those made by in-person evaluation [9]. The aim of this study was to develop an expert system for automatic identification of wound types and detection of wound infection.

2. Materials and Methods

2.1. Application for Collection of Wound Images

In order to collect wound images during the perioperative period, we developed a smartphone Android-compatible application (APP) for surgeons to upload these pictures. The study protocol was approved by the Institutional Review Board of the National Taiwan University Hospital (201705076RINC). The programming language to create this APP was Java, and the integrated development environment was Android Studio (ver. 3.0.1) with Java Develop Kit (ver. 1.8.0_152). Just after completing the operation, during hospitalization, and in outpatient clinic follow-up, the surgeon photographed the wound area (from subcostal area to inguinal area), accompanied with a reference of CameraTrax^® color card by the same smartphone under a non-flash model, and the images were transmitted to the server, for further storage and analysis.

2.2. Interface of Wound Annotation

The surgical professionals annotated the images at the website and classified them according to type of procedure (laparotomy, minimal invasive surgery, or hernia repair), interest of wound area, and abnormal wound features. We developed a drawing tool implemented with a tablet personal computer and stylus to precisely mark the wound area and wound features with corresponding colors. The annotation process was as follows:

First, the surgical professional chose the type of procedure and ranked the interpretation according to the degree of difficulty. If some pictures were transmitted in error, they were classified as exceptions.

Second, the actual wound was circled in purple, the peri-wound area was circled in red, the region of interest (ROI; any area of abnormal features near the wound) was circled in orange, the color card was drawn in green, and the drainage tube was drawn in black (Figure 1). These features train the system to identify the location of the wound and focus on the significant wound area.

Third, the features of the wound infection were marked by painting them in corresponding colors. Nine colors were used to describe the characteristics of the wound infection, which were bleeding, erythema, necrosis, pus, bruise, granulation, swelling, allergy, and pigmentation (Figure 2). Next, these patterns were stored in the database, in the form of string of the Base64 encoding, for analysis.

2.3. Model Development

We adopted the deep convolutional neural networks (CNN), pretrained 50-layer residual networks (ResNets) [10], which was modified to our algorithm by replacement of both Global Average Pooling and Fully Connected layers with Conv6 (kernel size 1 × 1 and 21 predicting classes) and Bilinear Up-sampling 2D layers (resizing the feature fit to the input shape, 320 × 320) (Figure 3). This new algorithm was pretrained on PASCAL VOC 2012 database to have network weights [11].

2.4. Detection of Color Card and ROI

The function of a color card is to adjust the color and to describe the characteristics of the wound infection for each image. To identify the color card and the ROI, the predicting class in Conv6 layer was set at 2 (with or without this feature). We adopted the Speed-Up Robust Feature [12] to extract the four key points of the photographed color card and enhanced the match of the four key points between the actual and the photographed color card by the Fast Library for Approximate Nearest Neighbors [13].

Next, we used the intersection-union test (IUT) as a metric to evaluate the accuracy of the detection of color card and wound ROI [14]. The IUT is equal to the area of overlap between the color card image detected by the deep learning algorithm and the actual color card in the wound photograph divided by the area of union encompassed by the two color cards.

2.5. Image Processing and Color Correction

To correct for uneven brightness of the wound images for data analysis, we applied the log transform ($s=c·lo{g}_v\left(1+v·r\right)$, where s is the output, and r is the input value), followed by gamma transform ($s=c·r^{\gamma }$), over all the RGB color of pixels in the image for brightness correction. After the adjustment, we normalized the value ranging from 0 to 255, to convert the result to the regular range of intensity in RGB color. With the identification of the color card in the pictures, we used eight pairs of colors between the actual and the photographed color card, to adjust the pixels by the correction matrix, which is described below.

$$\left[x_1{x}_2{x}_3{x}_4{y}_1{y}_2{y}_3{y}_4{z}_1{z}_2{z}_3{z}_4\right]\left[rgb1\right]=\left[r^{\prime }{g}^{\prime }{b}^{\prime }\right]$$

The wound length was calculated with reference to the length in the color card.

2.6. Detection of Wound Infection

Using the data from ROI images with corresponding diagnosis of abnormal wound features, we used Xception (one deep CNN) [15] to predict whether the images had the features of wound infection (binary classification task). Wound infection is defined as the presence of redness, pus, or necrosis. Kernel support vector machines classifiers [16], random forest [17], and gradient boosting classifiers [18] were trained to predict wound infection. The classification accuracy, recall, precision, F1 score, and area under the receiver operating characteristic curve (AUC) of these four methods were calculated. The discriminatory power of the four models was analyzed, using the AUC to assess the ability of the model in identifying images with wound infection. Furthermore, we also trained and tested the models by performing fivefold cross-validation.

2.7. Experimental Setup

This proposed framework was executed with MATLAB R2018b (MathWorks, Inc., Natick, MA, USA) on a Windows 10 operating system. All the experiments were performed on a server computer with Intel^® Xeon central processing unit (CPU) E5-2650, 16 GB random access memory (RAM), and a NVIDIA (Santa Clara, CA, USA) GeForce GTX 1080 Ti GPU. With the graphics card, it offered the equivalent processing capability for both the training and the validation models.

3. Results

During the study period, 480 wound photographs were taken from 100 patients for further analysis. The numbers of cases of hernia, minimal invasive surgery, and laparotomy were 54, 28, and 18, respectively.

3.1. Detection of Color Card and Wound ROI

Of the 480 images, 384 were used as a training set, and 96 were used as a validation set. We applied the fivefold stratification as cross-validation with the same division between the training and validation sets [19]. The average value of IUT on the validation set with fivefold stratification to predict the area of the color card was 0.978 (standard deviation, 0.003).

With regard to the results of wound ROI, the average value of IUT on the validation set with fivefold stratification was 0.775 (standard deviation, 0.020). Figure 4 shows the process and autonomic output of our model.

3.2. Detection of Wound Infection

For the detection of wound infection, 380 images (including 108 images of wound infection) were used for training, while we validated the performance on 100 images (including 28 images of wound infection). We also adopted fivefold cross-validation to assess the outcomes of the four models.

Table 1. Comparison of performance of our model, kernel support vector machines, random forest, and gradient-boosting classifier for detection of wound infection.

Model	Accuracy	Recall	Precision	F1 Score	Specificity	False Positive	False Negative	AUC
Our model	79.5% ± 4.2%	77.1% ± 6.1%	82.7% ± 7.8%	79.4% ± 3.9%	82.2% ± 3.5%	17.8% ± 6.5%	22.9% ± 6.0%	83.3% ± 2.8%
Kernel support vector machines	41.1% ± 2.1%	40.8% ± 8.7%	48.2% ± 3.3%	38.3% ± 3.5%	56.2% ± 2.6%	43.8% ± 7.3%	59.2% ± 8.8%	44.4% ± 4.5%
Random forest	58.6% ± 5.9%	64.8% ± 12.4%	44.5% ± 13.6%	43.1% ± 11.4%	19.2% ± 7.5%	80.8% ± 2.5%	35.2% ± 12.3%	67.1% ± 7.3%
Gradient boosting classifier	63.4% ± 4.2%	65.2% ± 7.4%	54.5% ± 6.4%	58.7% ± 3.3%	65.2% ± 5.3%	34.8% ± 4.6%	34.8% ± 7.4%	66.9% ± 4.9%

AUC: area under the receiver operating characteristic curve.

shows the classification accuracy, recall, precision, F1 score, and AUC for each method. Our algorithm achieved an accuracy of 79.5%, recall of 77.1%, precision of 82.7%, F1 score of 79.4%, and AUC score of 83.3%, whereas the other three methods achieved lower AUC scores (kernel support vector machines, 44.4%; random forest, 67.1%; gradient boosting classifier, 66.9%). For detailed comparisons among the four methods, we show the receiver operating characteristic (ROC) curves in Figure 5. The ROC curve is a probability curve, and AUC characterizes the degree of separability. One model with a higher score is better at detecting the aimed outcome.

4. Discussion

We have designed a novel wound image analysis system for surgical patients undergoing elective procedures, including laparotomy, minimal invasive surgery, and hernia. This artificial-intelligence-implemented algorithm recognizes not only wound borders, but also potential features of the wound infection, and calculates the length of the wound by comparison with a reference color card. The reason why we chose CNN as a model is that it has become dominant in various computer vision tasks with pre-training models and good performances in feature extractors (inception model or ResNets). With the pre-training model, you can extract useful attributes with its trained weights and tune your CNN model by feeding data on each level for the new task. This study demonstrates the feasibility of using expert-based machine learning methods to predict wound infection, with acceptable results. To the best of our knowledge, this is one of the innovative systems focusing on the automatic detection of wound area and diagnosis of wound infection.

Traditionally, patients discharged from surgery receive little care, until they report for a routine clinic follow-up [5]. Self-care is a unique problem for surgical patients rather than medical patients during this period. Among surgical patients under the early recovery and discharge program, wound complications are more likely to occur after discharge than before discharge [20]. Without appropriate management and early detection of wound complications, patients may be particularly vulnerable after discharge, given their lack of clinical experience, their increased medical costs, and the likelihood of hospital readmission.

In summary, mHealth is a scalable and flexible platform supporting the practice of medical and perioperative care with the provision of mobile devices [8,21,22,23], which improve the quality of perioperative care and have proliferated [24,25]. Although some academic centers have developed the mHealth system for postoperative wound care with positive results, it places an additional strain and workload on their provider teams, even if they are enthusiastic about it [5,7,25,26,27]. The advantage of our method is that it provides automatic detection of wound infection, which may alleviate the providers’ workload to a certain degree.

In this study, we used expert-based annotation of wound features to improve the performance because of the limited number of wound pictures. Traditionally, inpatient wounds are assessed periodically, but photographs are not taken or documented in the medical records unless complications occur. Furthermore, almost no data on wounds after discharge are available for analysis. It is necessary for surgical professionals to systemically collect perioperative wound photographs. Because of the shortage of available wound photographs, our initial task was to construct a cloud-based server to store wound data obtained by mobile phones. Next, we performed expert-based annotation of wound features to improve the prediction outcomes of deep learning. With this infrastructure, we can continue to collect wound data, which may further improve the accuracy of detection of wound complications [28,29].

One recent report questioned the accuracy of detection of wound complications by using wound photographs and clinical information from surgical professionals [25]. With such a simulation, the use of wound images increased the specificity of detection of wound complications and surgeon confidence, but worsened the sensitivity of detection of wound complications. Given the importance of image quality and experience with remote evaluation, some strategies should be implemented, such as participant training, standardization of patient photographic techniques, and synchronization of definitions of wound features [30]. In our study, the photographs were taken by surgeons during direct outpatient visits, which eliminated misdiagnosis and provided accurate data for further analysis.

Our next goal is to design one application that will provide knowledge about self-care and real-time feedback from surgical professionals. Every uploaded wound photograph will be evaluated by the algorithms implemented in the server, and the output with abnormal wound features will be assessed by surgical professionals before sending alarm messages to the patient or caregiver. Although previous studies utilized CPU or GPU in mobile phones to perform wound analysis, the image resolution should be compromised to fit the limited performance of mobile phones [31,32]. Moreover, it is difficult for mobile phones to run deep-learning algorithms. Therefore, to allow for a more complex and computationally demanding analysis of high-resolution photographs taken by modern smartphone cameras, we chose to do data mining in the server because the mobile internet is becoming popular, with fast internet speed.

Although the recent improvements of deep learning have been significant, there still exist challenges to its application to medical imaging/wound images. A well-annotated large medical dataset is needed and considered as an important key for clinical application, because most of the notable achievements of deep learning are majorly dependent on large datasets. In the future, surgical academic communities should continuously collect the wound images during the perioperative periods that would foster research on the detection of wound infections.

This study has some limitations. The study was performed on Asian subjects, which may limit its generalizability to patients of different nationalities or ethnicities. In addition, more patients participating in this study provide diverse quality of wound photographs, which contain inter-person variation because different mobile phone cameras have different color space characteristics or noise levels. This may cause a potential concern interfering with our algorithm. To overcome these limitations, color calibration is routinely adjusted by referenced color card to minimize these confounding factors. However, it may cause extra work for the patients or caregivers to take photographs with a color card. Finally, a “test” set in conjunction with a validation dataset should be considered to evaluate the performance of the final model. We will collect more wound images to reflect it.

5. Conclusions

We introduced a unified framework for automatic analysis of wound condition with the use of supervised machine learning, to overcome the shortage of data on wound images. Our system provides acceptable performance and is efficient enough to process the wound image within five seconds. The methodology and results presented in this study may have important implications for the field of surgical wound evaluation and diagnosis. With our proposed model, this system provides complementary information to partially relieve the workload of medical professionals in charge of telecare. Furthermore, this methodology can be directly extrapolated to other similar surgical procedures, such as cardiovascular and orthopedic operations.