Healthcare AI for Physician-Centered Decision-Making: Case Study of Applying Deep Learning to Aid Medical Professionals

Abstract

This paper aims to leverage artificial intelligence (AI) to assist physicians in utilizing advanced deep learning techniques integrated into developed models within electronic health records (EHRs) in medical information systems (MISes), which have been in use for over 15 years in health centers across the Republic of Serbia. This paper presents a human-centered AI approach that emphasizes physician decision-making supported by AI models. This study presents two developed and implemented deep neural network (DNN) models in the EHR. Both models were based on data that were collected during the COVID-19 outbreak. The models were evaluated using five-fold cross-validation. The convolutional neural network (CNN), based on the pre-trained VGG19 architecture for classifying chest X-ray images, was trained on a publicly available smaller dataset containing 196 entries, and achieved an average classification accuracy of 91.83 ± 2.82%. The DNN model for optimizing patient appointment scheduling was trained on a large dataset (341,569 entries) and a rich feature design extracted from the MIS, which is daily used in Serbia, achieving an average classification accuracy of 77.51 ± 0.70%. Both models have consistent performance and good generalization. The architecture of a realized MIS, incorporating the positioning of developed AI tools that encompass both developed models, is also presented in this study.

1. Introduction

AI in medical diagnosis represents a powerful tool that assists physicians in their daily work. With the help of AI systems, physicians receive valuable support that enables them to quickly identify potential problems and notice details that the “human eye” might miss. AI can analyze large amounts of data in record time, saving valuable time while also providing additional accuracy in analyzing images or other medical data.

However, the final decision still needs to be made by the physicians. Based on the suggestions provided by the AI system, the physician considers all relevant information from the patient’s medical history (from the EHR) and decides on treatment. AI can clearly show the physician which aspects of the data it has highlighted, such as parts of an image that are considered suspicious, ensuring transparency in the decision-making process [1].

The greatest advantage of this approach is that responsibility for the patient’s treatment remains entrusted to the physician. AI systems are not autonomous; they simply assist the physician in making more informed decisions by providing additional information and recommendations, while the physician retains a pivotal role in the diagnostic process.

The benefits of this integrated approach are manifold. First, it enables faster and more accurate diagnostics, thereby reducing patient waiting times and enhancing the quality of healthcare. AI provides greater access to medical expertise, particularly in rural areas where specialists may not be readily available. It also reduces the number of errors that can occur due to fatigue or stress. Meanwhile, the AI system, through continuous learning from new cases, gradually becomes more accurate, always under the supervision of experts.

Today, modern MIS and EHR systems should incorporate AI-based tools to enhance the quality of healthcare services, improve overall performance, and streamline daily operations. Deep learning [2] is a branch of AI and machine learning (ML). Deep learning algorithms are based on artificial neural network (ANN) models. Deep learning techniques include DNNs, recurrent neural networks, CNNs, and deep belief networks. These techniques are used to solve complex problems such as natural language processing (NLP), computer vision, bioinformatics, audio recognition, medical image analysis, and speech recognition. Deep learning provides valuable solutions for medical image analysis problems and is a key method for future applications [3]. A DNN is an ANN with more hidden layers between the input and output layers. A CNN [4] is a type of DNN model that enables the extraction of higher representations of image content. CNNs are a state-of-the-art image classification technique [5], which uses some features of the visual cortex. In this paper, a state-of-the-art technique for the classification of chest X-ray images is presented. The DNN model is used to optimize scheduling time slots. Both models are created in order to combat the COVID-19 pandemic.

The most efficient and reliable way to diagnose patients infected by COVID-19 during the pandemic and monitor their recovery is the chest X-ray and CT image [6]. By using the CNN, the model presented in this paper was developed, which rapidly classifies patients according to the chest X-ray image into two categories: one category is patients with COVID-19, while the other is those without COVID-19. The automatic classification assists the physicians in establishing the COVID-19 diagnosis.

Due to social distancing and other measures that limited the movement of patients during the COVID-19 pandemic, a considerable percentage of patients did not appear for their scheduled appointments. The appointments for an examination with a physician and/or diagnostic device remained reserved; however, the patients did not appear for them. Those are also often scheduled examinations on expensive diagnostic devices, for which patients must wait for an extended period of time, sometimes even 6 months. In this regard, there was a need to develop a tool to optimize patient scheduling in available slots. During the COVID-19 pandemic, primary healthcare institutions operated at reduced capacity (with fewer available slots for scheduling appointments), making it essential to optimize and increase the occupancy rate of these slots. The tool presented in this paper utilizes the developed DNN to classify patients with scheduled appointments as either arriving or not arriving at their scheduled time slots. Patients classified as those who were “potentially not coming” to their scheduled appointments should have been contacted by phone to confirm their arrival. An option was also supported for scheduling a higher number of patients classified as “potentially not coming” in the same time slots, which was modeled according to airline companies’ overbooking of seat procedures [7].

The objective of this paper is to assist physicians in making final decisions by utilizing deep learning techniques incorporated into AI tools in MIS and EHRs. This study showcases the effective application of AI in addressing critical situations, such as the COVID-19 pandemic, by prioritizing AI-assisted Physician-Centered Decision-Making. The model for rapidly classifying chest X-ray images into two classes, non-COVID-19 and COVID-19, is designed using a CNN. The model for the classification of appointments of patients in two classes during the pandemic: 0—patient is predicted to arrive at the scheduled time slot, 1—patient is predicted not to arrive, is created using the DNN.

This study presents the need for physician-centered decision-making in fast-reacting situations, utilizing AI tools that encompass both developed DNN models. The primary goal of this research is to integrate AI into existing MISes and enhance EHR with AI, rather than competing with sophisticated models in the field of study. Overall, the developed models presented in this paper have achieved remarkable accuracy and performance. Outbreaks, such as COVID-19, are exact situations where rapid physician-centered decision-making, assisted by AI tools, is crucial.

After the Introduction section, the Background and the Materials and Methods sections follow, respectively. The authors then present the results in the Results section and discuss them in the Discussion section. After the Discussion, the Conclusion section and the Reference list follow.

2. Background

The coronavirus SARS-CoV-2 appeared by the end of 2019 [8]. It was discovered in China in the city of Wuhan, the province of Hubei [9]. The disease it causes is called COVID-19. For COVID-19, the World Health Organization (WHO) assigned the urgent ICD-10 diagnosis U07.1 [10]. The newly emerged virus was connected to the virus SARS-CoV since 2013 [11]. The basic symptoms caused by COVID-19 are described in [9]. Figure 1 shows the exponential rise in the number of infected and deceased people worldwide during the COVID-19 pandemic [12]. The main crisis centers became Europe and North America, after the emergence of the virus in China. Due to the characteristics of COVID-19, the WHO decided to mark the epidemic of the virus SARS-CoV-2 as a pandemic on 11 March 2020.

Deep learning techniques for biomedical and health informatics are described in depth in [13]. Automated diagnosis of tuberculosis from chest X-rays has been tackled with either hand-crafted algorithms or ML approaches such as support vector machines and CNNs [14]. The detailed descriptions of deep learning for medical image analysis and the principles and methods of neural networks and deep learning concepts are given in [3]. The authors show COVID-NET [15], a tailored deep convolutional neural network design for detecting COVID-19 cases from chest X-ray images with an accuracy of 92.6%. Markus Schmitt shows nine ways of machine learning that help to combat the viral pandemic [16]. The approach that enables the automatic detection of the coronavirus disease using X-ray images and deep convolutional networks is shown in [17]. The authors used the pre-trained models ResNET50, Inception V3, and Inception-ResNetV2. They achieved the best performance with the pre-trained model ResNET50. Models with a location-attention mechanism could more accurately diagnose COVID-19 at the chest radiography with an overall accuracy rate of 86.7% and could be a promising supplementary diagnostic method for frontline clinical physicians [18]. The symptoms and establishing the COVID-19 diagnosis based on chest X-ray images are shown in the article [19]. Developing an intelligent healthcare system [20] is a crucial factor for further improvements from the perspective of patients, healthcare employees, and medical science researchers. To support early detection efforts, researchers proposed a Genetic Deep Learning Convolutional Neural Network model enhanced with Huddle Particle Swarm Optimization instead of traditional gradient descent. Using public chest X-ray datasets, the model effectively identifies COVID-19 and other pneumonia types, achieving 97.23% accuracy, 98.62% sensitivity, 97.0% specificity, and 93.0% precision. Authors emphasize that future work should involve expanding to larger datasets and broader lung disease prediction [21]. Many institutions improved their IT infrastructure to prepare for the exponential rise of patients suffering from COVID-19 [22]. The prediction about the visits of patients can be helpful for stakeholders in hospitals or other health centers to make decisions in managing human resources and materials [23].

The survey paper [24] comprehensively examines various interpretation and visualization techniques applied to deep learning models in medical imaging. This paper reviews methodologies, discusses their applications, and evaluates their effectiveness in enhancing the interpretability, reliability, and clinical relevance of deep learning models in medical image analysis. While significant progress has been made, challenges related to scalability, clinical integration, robustness, standardization, and ethical considerations persist.

This paper [25] presents a comprehensive review of ChatGPT’s applications in healthcare, focusing on its role in enhancing patient engagement, symptom assessment, and decision support to improve diagnostic accuracy. It evaluates ChatGPT’s usefulness across medical specialties and highlights its benefits in clinical, educational, and administrative settings. While the model seems promising, the review also addresses key challenges, including potential inaccuracies, ethical issues, and regulatory concerns. The goal is to provide a balanced perspective for healthcare professionals, researchers, and policymakers, emphasizing the need for continual updates as medical knowledge evolves.

3. Materials and Methods

This research uses two datasets and two different deep learning techniques for binary classifications: one for X-ray image classification using CNN and another for optimizing time slot scheduling using DNN. The Graphics Card NVIDIA GTX 1080 Ti 11 GB GDDR5X, Intel Core i7-9700K, 32 GB RAM, and 512 GB SSD are used to train proposed models. As software tools during research, we used TensorFlow 2.18.0 [26], Keras 3.7.0 [27], NumPy 2.0.2 [28], pandas 2.2.3 [29], matplotlib 3.10.0 [30], OpenCV 4.12.0 [31], scikit-learn 1.6.0 [32], and PyCharm 2024.3.1.1 for Python 3.12.8 [33].

3.1. Model for the Chest X-Ray Image Classification

This research uses a dataset, shared by Dr. Joseph Cohen [34], to classify chest X-ray images. The dataset contains chest X-ray/CT images with COVID-19, acute respiratory distress syndrome (ARDS) [35], MERS [36], SARS [37], and pneumonia. The authors used 98 X-ray images that were labeled with COVID-19. Also, 98 non-COVID-19 chest X-ray images were selected from the dataset Chest X-Ray Images (Pneumonia) [38]. Our research is based on 98 COVID-19 and 98 patients with Non-COVID-19 disease (a total of 196 images). All images are preprocessed: adjusted contrast, adjusted saturation, removed black bars, and saved in 224 × 224 image format.

Deep CNN [39] models may take longer to train on large datasets. Transfer learning [40] involves using models trained on one problem but used on a related similar problem. Transfer learning decreases the time necessary for training deep neural network models and provides a better reduction of the generalization error. A way to shorten this process is to reuse the model weights from pre-trained models developed for standard computer vision benchmark datasets, such as the ImageNet [41] image recognition tasks [42]. We created the CNN model based on the VGG19 [43] pre-trained model to classify chest X-ray images into two classes: Non-COVID-19 and COVID-19 (Figure 2).

All images are randomly split into two separate datasets with a ratio of 80% (for training) and 20% (for testing). The k-fold cross-validation [44] is used to evaluate the CNN model. The value for k is set to 5 (Figure 3). The batch size is set to 32, the number of epochs to 300, and the learning rate to 0.001 (hyperparameters were determined experimentally).

3.2. Model for the Optimization of Time Slots Scheduling

The City of Nis is the third-largest city in the Republic of Serbia (RS) by area and population and became the second-largest COVID-19 hotspot in the country. It is served by a single health center with 63 branches, making the Health Center Nis (HCN) the largest primary healthcare provider in Serbia and the Balkans, serving around half a million patients. In RS, several nationwide licensed MISes are used, one of which is MEDIS.NET [45]. Developed at the Faculty of Electronic Engineering in Nis, it is used in 25 primary healthcare facilities across southeastern RS and has been in operation at HCN for over 15 years.

The dataset used for training the DNN model is extracted from the database of HCN. It contains features presented in

Table 1. List of features in the dataset.

Features	Features Tag	Descriptions
Doctor	D	Physician’s ID
Gender	G	1—Male, 0—Female
Chronic diagnosis	Cd	0—No, 1—Yes
Day of week	dow	[1–7], 1—the first day of the week (Monday)
Quartal	Q	[1–4], annual quarter
Week number	Wn	Week number of the year
Family feast	Ff	The scheduled examination is on the date of the top 10 family feasts in Serbia
New Year’s Day	nyd	30th, 31st December and 1st, 2nd,3rd January
Christmas	Ch	6th, 7th, 8th January
Type of health insurance	Thi	1—employed (codes: 10 **) 2—unemployed (codes: 9999, 11 **) 3—pensioners (codes: 12 **) 4—other types of insurance (codes: 15 **) 5—sensitive groups (codes: 1820, 421) 6—foreign citizens (codes: 17 **) 7—state health care users (codes: 18 **) 8—special regulations (codes: 19 **) 9—soldiers (codes: 90 **, 434, 1012, 1913) 10—conventions (codes: 20 **) 11—pregnant women (codes: 412, 1811)
Patient age	Pa	1—[less than 1] years 2—[1–6] years 3—[7–18] years 4—[19–34] years 5—[35–49] years 6—[50–64] years 7—[65–74] years 8—[75+] years
Seasons	S	1—Spring 2—Summer 3—Autumn 4—Winter
Specialty for scheduling	Ss	1—neuropsychiatry 2—dentist 3—psychiatry 4—medical doctor 5—dermatovenerology 6—otorhinolaryngology 7—physical medicine and rehabilitation 8—internal medicine 9—pediatrics 10—ophthalmology 11—gynecology and obstetrics 12—general medicine
Number of days until the scheduled date	ndusd	Number of days from the moment of scheduling an examination slot until the actual examination
Percent of non-attendance	pna	Percentage of missed scheduled appointments by patients
Status	stat	0—patient is predicted to arrive at the scheduled time slot 1—patient is predicted not to arrive at the scheduled time slot

**—Represent combinations of other available types of health insurance codes in the Republic of Serbia.

, which are related to the time slot scheduling of patients. The data in the dataset are extracted from the database for a period of 5 years (from 2015 to 2019).

The relationship between different input features in a dataset is presented in Figure 4 and indicates how much they tend to change together. The distribution of data within a dataset is shown in Figure 5.

The DNN model is created to binary classify scheduled time slots by feature status (0—patient is predicted to arrive at the scheduled time slot, 1—patient is predicted not to arrive). The dataset initially contained 388,194 rows. After processing (removing rows with nullable values, all values are normalized), the data dataset contains 341,569 (88%) rows. The dataset is very well balanced by feature status (Figure 5), with almost 50% for classes 0 and 1. It is randomly split into two divided datasets with a ratio of 80% (for training) and 20% (for testing). The k-fold cross-validation is used to evaluate the DNN model. The value for k is set to 5. The batch size is set to 512, the number of epochs to 100, and the learning rate to 0.01 (hyperparameters were determined experimentally). The model for binary classification is shown in Figure 6.

3.3. Performance Metrics

In order to measure the performance of proposed models, a Classification report [46] is used. Variables TP, FP, TN, and FN are True Positive, False Positive, True Negative, and False Negative, respectively. The accuracy, sensitivity, specificity, precision, and F1-score [47] were used to calculate the model’s performance.

$$\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}}}$$

(1)

$$\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(2)

$$\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(3)

$$\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2·{\displaystyle \frac{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}·\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(4)

The accuracy determines how correctly the values are predicted. The precision determines how many of the predictions are correct. The recall determines how many of the correct results are discovered. The F1-score uses precision and recall calculations. It is a measure of balanced average results.

$$\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(5)

Sensitivity is called the true positive rate (TPR), recall, or probability of detection. This is a measure of actual positives that are correctly identified (e.g., the ability of a test to identify patients with a disease correctly).

$$\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{N}}{\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}}}$$

(6)

Specificity is called the true negative rate (TNR). This is a measure of actual negatives that are correctly identified (e.g., the ability of a test to identify a patient without the disease correctly).

4. Results

This section presents the achieved results and performance of the two developed DNN models, as well as their application in MIS.

4.1. Model for the Chest X-Ray Image Classification

Figure 7 presents the performance of the model for chest X-ray image classification. The confusion matrix and classification report are shown in Figure 8. The maximum accuracy, precision, sensitivity, specificity, and F1-score are 94.87%, 94.74%, 94.74%, 95.00% and 94.74%, respectively, for fold number 3. The average accuracy, precision, sensitivity, specificity, and F1-score across all folds are 91.83 ± 2.82%, 92.78 ± 4.36%, 90.83 ± 2.22%, 92.90 ± 4.42% and 91.76 ± 2.78%.

Table 2. Metrics for a model for chest X-ray image classification.

Fold Number	Accuracy	Precision	Sensitivity	Specificity	F1-score
1	0.8718	0.8500	0.8947	0.8500	0.8718
2	0.9231	0.9474	0.9000	0.9474	0.9231
3	0.9487	0.9474	0.9474	0.9500	0.9474
4	0.9231	0.9444	0.8947	0.9500	0.9189
5	0.9250	0.9500	0.9048	0.9474	0.9268
Mean:	0.9183	0.9278	0.9083	0.9290	0.9176
95% CI:	[0.8834, 0.9533]	[0.8738, 0.9819]	[0.8807, 0.9359]	[0.8741, 0.9838]	[0.8830, 0.9522]
Standard Deviation:	0.9183 ± 0.0282	0.9278 ± 0.0436	0.9083 ± 0.0222	0.9290 ± 0.0442	0.9176 ± 0.0278

shows metrics for a model for chest X-ray image classification.

TP is the number of patients (94.7%) that the model classified as having COVID-19, and they are patients with that diagnosis. TN is the number of patients (95%) predicted by the model not to have the disease COVID-19, and who actually do not have the disease. Patients without and with the disease can be identified with an accuracy of 95% using the proposed model. FP is the number of patients (5%) predicted by the model to be positive for COVID-19, but actually they are not. This is where the problem lies because such patients would most likely be sent to a temporary COVID-19 hospital and thus stay in a highly infectious environment. There is a high chance that such patients will become infected with COVID-19 after additional exposure to the viral environment. FN is the number of patients (5.3%) predicted by the model to not be positive for COVID-19 but are actually positive. Such patients would be discharged without an established diagnosis of COVID-19 and could infect a larger population of people with whom they have social contact and would not receive adequate healthcare. Nonetheless, chest X-ray imaging is performed for patients who have a distinct picture of the disease of COVID-19, so they would certainly not be immediately discharged home without additional testing (RT-PCR testing). In order to increase the performance of the model, it is necessary to increase the percentage of TP and TN and decrease FP and FN.

4.2. Model for the Optimization of Time Slots Scheduling

Figure 9 presents the model’s performance for optimizing time slot scheduling. The confusion matrix and classification report are shown in Figure 10. The maximum accuracy, precision, sensitivity, specificity, and F1-score calculated on the confusion matrix are 78.23%, 82.82%, 72.59%, 84.16%, and 77.37%, respectively, for fold number 4. The average accuracy, precision, sensitivity, specificity, and F1-score across all folds are 77.51 ± 0.7%, 82.53 ± 0.46%, 71.57 ± 1.08%, 83.86 ± 0.38% and 76.66 ± 0.70%.

Table 3. Metrics for a model for the optimization of time slot scheduling.

Fold Number	Accuracy	Precision	Sensitivity	Specificity	F1-score
1	0.7697	0.8231	0.7074	0.8367	0.7609
2	0.7819	0.8234	0.7284	0.8376	0.7730
3	0.7748	0.8318	0.7119	0.8433	0.7672
4	0.7823	0.8282	0.7259	0.8416	0.7737
5	0.7669	0.8202	0.7047	0.8338	0.7581
Mean:	0.7751	0.8253	0.7157	0.8386	0.7666
95% CI:	[0.7665, 0.7838]	[0.8196, 0.8311]	[0.7022, 0.7291]	[0.8338, 0.8434]	[0.7579, 0.7753]
Standard Deviation:	0.7751 ± 0.0070	0.8253 ± 0.0046	0.7157 ± 0.0108	0.8386 ± 0.0038	0.7666 ± 0.0070

shows metrics for a model for the optimization of time slot scheduling.

TP is the number of patients (72.59%) the model predicted would arrive at the scheduled time. Those patients would definitely come at their scheduled time. This has no positive or negative effects on the model. TN is the number of patients (84.16%) that the model classified as not coming to the scheduled examination, and they really did not come. It is possible to double appointments in those appointments to ensure a higher degree of occupancy of the available appointments. FP is the number of patients (15.84%) for whom the system predicted they would come for an examination but who did not come for the scheduled examination. Therefore, those appointments remain unfilled. To increase the occupancy of this time slot, two patients can be scheduled at the same time. FN is the number of patients (27.41%) for whom the model predicted that they would not come for examination, but they did come. Such patients are generally a potential problem for crowding and overbooking appointment slots. In real situations, a little space is always left between two adjacent slots to schedule an examination to provide medical services to such patients. The duration of those breaks between two appointment slots is defined depending on the specialization and the type of service in the health center. For the system to have a better performance, it is necessary to increase the percentages of TP and TN and decrease the percentages for FP and FN, in order to increase accuracy, sensitivity (recall), specificity, precision, and F1-score.

4.3. AI Tools in MIS That Use Developed Models

Figure 11 shows two modules in MEDIS.NET that use described and developed models to classify chest X-ray images and optimize time slot scheduling. The physician can activate the tool for the chest X-ray to diagnose COVID-19 automatically from the patient’s health record. The tool relies on the developed CNN model for classifying chest X-ray images.

During the examination appointment for each selected patient, an estimate can be obtained as to whether the patient will appear at the scheduled time. That functionality relies on the developed DNN model.

Figure 12 and Figure 13 show how both developed models are integrated into MIS MEDIS.NET and EHR.

5. Discussion

In this research, chest X-ray images are used to predict COVID-19, and an average accuracy of 91.83 ± 2.82% is accomplished by using the pre-trained VGG19 model for the classification of non-COVID-19 and COVID-19 cases. Accuracy values are increased through the training process of 300 epochs, while loss values are decreased (Figure 7). The performance metrics of the model for X-ray image classification are presented in Figure 7. In [48], the authors proposed a detection of COVID-19 by using X-ray images based on deep features and SVM. They obtained an accuracy of 95.38% for ResNet50 and SVM. The authors in [18] proposed a prediction model for distinguishing COVID-19 from influenza-A viral pneumonia by using CT images. They used deep learning techniques and achieved an accuracy of 86.7%. The Inception transfer learning model is used by the authors in [49] to predict COVID-19 cases by CT images. They accomplished an accuracy of 89.5%. In [17], the author compares three pre-trained models for X-ray image classification: InceptionV3, ResNet50, and Inception ResNetV2, and obtained the accuracy of 97%, 98%, and 97%, respectively. In their research, the best result was obtained by using a pre-trained ResNet50 model with an accuracy of 98%.

Our trained model for the classification of chest X-ray images can be used at the Department of Radiology at the HCN as a supporting tool for physicians during and after the COVID-19 pandemic (Figure 11, left side). Radiologists in the HCN can use this new tool within the already existing radiology module. This tool can be well-accepted and more appealing to young physicians and physicians at the Department of Radiology in HCN, during the COVID-19 pandemic. Despite the satisfactory classification average accuracy of 91.83 ± 2.82%, the radiologists are those who make the final decision to establish the diagnosis of COVID-19 and to decide whether to hospitalize the patient.

The developed DNN model for the optimization of time slot scheduling is used for the implementation of the subsystem of MEDIS.NET for the intelligent identification and assessment of patients who will not come to the appointment with the chosen physician or/and to expensive diagnostic examinations for which a patient needs to wait, sometimes even for several months. The subsystem is part of the Scheduling Module of MEDIS.NET, shown in Figure 11 on the right side. The developed DNN model is used for the classification of scheduled patient examinations into two classes: 0—patient is predicted to arrive at the scheduled time slot, 1—patient is predicted not to arrive at the scheduled time slot. For each created patient appointment, the physician receives a message that informs him/her about the possibility of the patient’s non-arrival in the scheduled time slot. During the COVID-19 pandemic, patients were able to schedule an examination appointment directly with the chosen physician by calling the Call Centre of the HCN (Figure 13). At the moment of scheduling, using the developed DNN model, a Call Center operator can receive an estimate, the classification of the scheduled appointment is determined into the two previously mentioned classes, and information is provided about the potential non-arrival of a patient to the scheduled appointment. The operator can mark the slot in that way, allowing the other patient to be scheduled simultaneously. Usually, in those overlapping slots, those patients are scheduled for whom the system issues a warning during the scheduling process that they will not arrive at the scheduled appointment. Airline companies worldwide have used similar principles for overbooking seats for many years. The paper [50] shows optimal overbooking strategies in airlines by using a dynamic programming approach in continuous time. By using the developed DNN model, upon the request of the health center employees, each patient who had a scheduled examination appointment could check whether he/she would arrive at the time of the appointment, and a corresponding report is generated. After the system check, daily reports were created about the patients who were not going to arrive at the scheduled appointments as per the assessment of the system. After the check, the patients were contacted by phone to confirm their arrival at the scheduled appointments. It was required to contact patients with scheduled appointments on expensive diagnostic devices, like CT and MRI, by phone and who were classified as patients who were potentially not arriving at the examination (class 1). During the COVID-19 pandemic, all physicians at the health center worked according to the changed, reduced schedule only a few days a week. The number of possible time slots for scheduling appointments was significantly reduced (72%), making it more difficult for patients to book their desired appointments. Due to the reduction of possible slots for the scheduling of appointments, a check about the possible non-arrival of patients at their appointments was necessary. Even beyond the work reduction, the number of scheduled appointments on a daily basis was immense. The average number of scheduled slots daily was 1738. Because of such a huge number of appointments, Call Center operators were not able to call each patient separately to confirm their arrival at the scheduled appointment. The average percent of non-arrivals of patients at the scheduled slots during the COVID-19 pandemic was 62%. If more patients arrived for one scheduled slot, the patients would then be examined by the first available physicians. Such a case is unlikely to happen for scheduled slots on expensive diagnostic devices (CT, MRI) since the patients were contacted about their arrival at the appointment in advance.

The satisfaction of end users, who directly or indirectly use the software, is always a key goal during the development of large and complex information systems, especially those such as medical information systems. During the COVID-19 pandemic, we were unable to interview doctors and patients who were both direct and indirect users of the system. In regular circumstances, overbooking patients for one appointment slot is unfeasible. This does not mean that today this model is unusable, but that it can be used for resource planning (appointments are made with the chosen physicians, but also with diagnostic devices). According to the regulations of the Republic of Serbia, it is possible to schedule only one patient per time slot. In Serbia, a central registry for scheduling examinations was developed, and all MISes were integrated with it.

The model’s average accuracy for optimizing time slot scheduling is 77.51 ± 0.7%. Accuracy values are increased through a training process of 100 epochs, while loss values are decreased (Figure 9). In order to increase the efficacy of the classification of time slot scheduling, the authors of this paper have developed two more models by using Random Forest [51] and Logistic Regression [52] for the same dataset created for the training of the neural network model. After training the models, the accuracy for the developed models that used Random Forest (Figure 14) and logistic regression is 74% and 73%, respectively. With these three models, the voting system is created. The voting system is used to more precisely determine whether the patient is arriving at the scheduled appointment. The soft voting method is applied, treating all classifiers equally. The final output is generated by averaging the outputs of each individual classifier [53] as shown in Figure 15. Using the voting system, the number of non-arrivals is reduced to below 10%.

5.1. Architecture of MIS MEDIS.NET and Position of AI Tools

The architecture of MIS MEDIS.NET is shown in Figure 16. The MIS MEDIS.NET is composed of several interconnected modules designed to support healthcare professionals in performing their tasks more efficiently, while also ensuring comprehensive storage and analysis of medical data. These modules significantly contribute to the efficient and coordinated operation of healthcare institutions. At the heart of the MIS is the EHR, which serves as its foundational component. Beyond the EHR, the system also incorporates modules for financial management, administration, and user privileges, static and dynamic reporting, as well as mHealth and TeleHealth functionalities. In addition, it includes Laboratory and Radiological Information Systems, along with a set of artificial intelligence tools, which contain the two previously described modules. Both realized modules (Module for the Department of Radiology, Module for Scheduling Patient Appointments) are directly dependent on artificial intelligence tools, electronic health records, and the Radiological Information System. The models presented in this paper have been developed and integrated into artificial intelligence tools.

Figure 16 shows the collaboration between MIS and other available services in the Republic of Serbia. The MIS MEDIS.NET is equipped with services that enable communication and data exchange with external systems. In the RS, a Vertical Integration has been established, enabling the secure exchange of key patient information between different healthcare providers and MISes. This system allows physicians to access medical data from other institutions upon obtaining patient consent, which is particularly important for providing more personalized and continuous care. Moreover, centralized data exchange on a national level supports monitoring of rare and chronic diseases, electronic prescription management, school enrollment procedures, appointment scheduling, preventive care, public health surveillance, along with other functionalities aimed at improving the accessibility and quality of care.

This paper [54] identifies key challenges and solutions for the implementation of modern MIS MEDIS.NET in a Serbian healthcare organization, including issues such as poor infrastructure, complex user interfaces, and low IT literacy. It outlines strategies for software architecture, development, and deployment to improve system usability and adoption. In the paper [55], the use and acceptance of the MIS MEDIS.NET in Serbian primary healthcare are analyzed. The study evaluates the success rate of entering various medical records and assesses user acceptance through the Technology Acceptance Model, focusing on perceived ease of use and usefulness. Results show high acceptance and accuracy for frequently used features, such as service registration. In contrast, specialized functionalities (e.g., for adult physical exams) are used less, indicating areas for further system improvement.

5.2. Physician-Centered Decision-Making Supported by AI

Table 4. Pros and cons of decision-making using AI vs. physician-centered decision-making vs. physician-centered decision-making supported by AI in MIS and EHR.

Decision-Making Using AI	Physician-Centered Decision-Making	Physician-Centered Decision-Making Supported by AI
Pros:	Pros:	Pros:
High accuracy in identifying patterns in large datasets.	Leverages experience, expertise, and patient history.	Combines AI’s data analysis with the physician’s clinical judgment, improving decision quality.
Reduces human error, especially in complex diagnoses (e.g., cancer detection).	Personalized care based on the physician’s knowledge and understanding of the patient.	AI enhances decision-making by providing additional insights and predictions.
Speeds up the diagnostic process, leading to quicker treatments.	Physicians can consider a wide range of factors, including non-quantifiable elements like intuition.	Reduces workload by automating routine tasks, allowing physicians to focus on critical decisions.
Cons:	Cons:	Cons:
Requires large, high-quality datasets to work effectively.	Can be influenced by bias, fatigue, or external factors affecting judgment.	AI recommendations might be misunderstood or misinterpreted without a proper understanding from the physician.
Potential for over-reliance on AI, leading to reduced clinical skills over time.	Time-consuming process, especially for complex diagnoses or rare diseases.	AI may not always be trained on specific patient data, leading to inaccurate suggestions or bias in recommendations.
Lack of transparency in AI models (black-box problem) can hinder trust and interpretation of results.	May not always have access to the latest research or technological advancements, leading to suboptimal decisions.	Over-reliance on AI could reduce physician autonomy or lead to an “AI-driven” mindset, potentially sidelining human judgment.
Ethical concerns, especially with privacy, data usage, and algorithmic bias.	Risk of misdiagnosis or treatment errors due to human limitations or misinterpretation of symptoms.	AI models are only as good as the data they are trained on, and poor-quality data can affect the quality of support it offers.

provides a comparative analysis of three possible approaches to decision-making in MIS: AI-assisted decision-making, traditional physician decision-making, and the AI-supported combined approach. Each option has its advantages and disadvantages, including accuracy, speed, risk of errors, and the level of physician intervention. This analysis facilitates the understanding of how AI can enhance MISes while simultaneously emphasizing the importance of the physician’s role in final decision-making. This study, presented in this paper, is primarily focused on physician-centered decision-making supported by AI.

5.3. Protection of Personal Data

MIS MEDIS.NET was implemented in accordance with the requirements of the Ministry of Health of the Republic of Serbia, which are based on the EuroRec standard. After implementation, it received a certificate from the Ministry of Health of the Republic of Serbia for use in health institutions in the Republic of Serbia.

We are committed to complying with national data protection laws and the EU General Data Protection Regulation (GDPR). To ensure data privacy and security, we implemented strict safeguards throughout all stages of data management. Data are collected securely, with explicit consent from individuals and clear privacy notices. All data are encrypted during storage and transmission, with access restricted to authorized personnel who use multi-factor authentication. Anonymization or pseudonymization is used where possible (e.g., personal patient and physician data, demographic data, and sensitive information that requires special attention under the Law on Personal Data Protection in Serbia). Anonymized and pseudonymized data are securely destroyed when no longer needed and retained only as long as necessary. Staff in health institutions received extensive lessons on data protection during the deployment of MIS MEDIS.NET. These measures ensure compliance with data protection laws and maintain high standards of privacy. For scientific research purposes and to improve MIS MEDIS.NET, which is used daily, the Faculty of Electronics in Nis and the Laboratory for Medical Informatics have been granted permission to use anonymized data from HCN.

Final decisions about treatment rest solely with the physician, who bears full responsibility and signs the patient’s treatment documents. AI tools serve primarily as a support for physicians, streamlining their work routine and clinical workflow.

6. Conclusions

We use deep learning techniques to create a model for the classification of chest X-ray images and a model for optimization of the time slot scheduling. With these models, we conduct a study demonstrating the significant impact of deep learning in healthcare, especially during the COVID-19 pandemic. The CNN model efficiently classifies chest X-rays, aiding rapid diagnosis, while the DNN model optimizes appointment scheduling, enhancing resource utilization.

A CNN with a pre-trained VGG19 model classifies chest X-ray images into non-COVID-19 and COVID-19 categories, achieving an average accuracy of 91.83 ± 2.82%. A DNN model with four dense layers classifies patients who miss scheduled appointments, achieving an average accuracy of 77.51 ± 0.7%. Both models are integrated into the MEDIS.NET system and have significantly contributed to combating the COVID-19 pandemic.

Future research should focus on expanding datasets, conducting external validation, refining models, conducting real-world deployment studies, and integrating feedback to support model refinement. This includes creating new DNN models for clustering patients by therapy and risk factors, predicting diagnoses, selecting laboratory tests, and forecasting disease spread. Chest X-ray models could be trained with larger datasets to classify images into categories like COVID-19, SARS, MERS, and pneumonia. Models for time slot scheduling could incorporate additional features like weather, traffic, and chronic therapy expiration. The deep learning techniques used for COVID-19 can also be applied to other epidemics and daily healthcare operations, laying the foundation for improved efficiency and outcomes in healthcare systems.

This study emphasizes the importance of physician-centered decision-making in urgent situations like pandemics, leveraging AI tools that incorporate advanced DNN models. The main objective of this research is to integrate AI into current MISes and improve EHRs with AI, rather than to compete with already established models in the field. In general, these developed models have demonstrated outstanding accuracy and performance. Events like the COVID-19 outbreak highlight scenarios where fast, physician-centered decision-making, supported by AI tools, is essential. The authors acknowledge that AI presents a major challenge for both healthcare professionals and engineers in the field of medical informatics, highlighting the importance of automating physicians’ processes and decision-making in daily practice, while ensuring that the physician remains at the center of final decision-making (physician-centered decision-making).

The method presented in the paper can also be applied to other diseases, such as ‘coin syndrome’ or lung tumors. Firstly, it is essential to collect high-quality data to build a robust dataset. The data from the dataset needs to be well-annotated by experienced physicians in their respective fields of practice. After that, it is necessary to carry out quality preprocessing of the collected data and select those that can be used as a final dataset for training the created model (this process must be carried out under the constant supervision of experts—physicians of the appropriate specialization). After creating the model, training it, and fine-tuning the hyperparameters, it is necessary to validate the model using new data that was not included in the previously created dataset. After that, it is necessary to implement the developed AI model in the existing electronic patient record within the medical information system. Ultimately, experts in the field (physicians of the relevant specialty) must validate the AI model.

The methodologies and results presented here serve as a foundation for integrating deep learning into healthcare systems for improved efficiency and patient outcomes.