Multi-Modal Data Analysis for Pneumonia Status Prediction Using Deep Learning (MDA-PSP)

Evaluating several vital signs and chest X-ray (CXR) reports regularly to determine the recovery of the pneumonia patients at general wards is a challenge for doctors. A recent study shows the identification of pneumonia by the history of symptoms and signs including vital signs, CXR, and other clinical parameters, but they lack predicting the recovery status after starting treatment. The goal of this paper is to provide a pneumonia status prediction system for the early affected patient’s discharge from the hospital within 7 days or late discharge more than 7 days. This paper aims to design a multimodal data analysis for pneumonia status prediction using deep learning classification (MDA-PSP). We have developed a system that takes an input of vital signs and CXR images of the affected patient with pneumonia from admission day 1 to day 3. The deep learning then classifies the health status improvement or deterioration for predicting the possible discharge state. Therefore, the scope is to provide a highly accurate prediction of the pneumonia recovery on the 7th day after 3-day treatment by the SHAP (SHapley Additive exPlanation), imputation, adaptive imputation-based preprocessing of the vital signs, and CXR image feature extraction using deep learning based on dense layers-batch normalization (BN) with class weights for the first 7 days’ general ward patient in MDA-PSP. A total of 3972 patients with pneumonia were enrolled by de-identification with an adult age of 71 mean ± 17 sd and 64% of them were male. After analyzing the data behavior, appropriate improvement measures are taken by data preprocessing and feature vectorization algorithm. The deep learning method of Dense-BN with SHAP features has an accuracy of 0.77 for vital signs, 0.92 for CXR, and 0.75 for the combined model with class weights. The MDA-PSP hybrid method-based experiments are proven to demonstrate higher prediction accuracy of 0.75 for pneumonia patient status. Henceforth, the hybrid methods of machine and deep learning for pneumonia patient discharge are concluded to be a better approach.


Introduction
Pneumonia was ranked the third cause of death in Taiwan and the mortality rate was 64.4/100,000 in 2019 [1]. The old age (>65 y/o) and severity of pneumonia were associated with higher mortality and longer hospital stays [2][3][4][5]. According to Taiwan's guidelines, community-acquired pneumonia (CAP) is defined as a pulmonary parenchymal acute infection in patients who acquire the condition in the community [6]. The Taiwan guideline suggested treating CAP by empirical, the severity of pneumonia (CURB-65), and risk of A machine learning approach to risk stratifying patients for ARDS. A. Yamagata et al. 2020 [24] To investigate the prognostic factors related to 30-day mortality in patients.
Nursing and healthcare-associated pneumonia Student's t-test and chi-squared test Univariate analysis and multivariate analysis using stepwise logistic regression.
p value J. Zhang et al. 2020 [25] A confidence aware anomaly detection (CAAD) model, with a shared feature extractor, an anomaly detection module, and a confidence prediction module for viral pneumonia.

MDA-PSP
Pneumonia status prediction for patient discharge within 7 days or not.

MDA-PSP Objectives
i. Scrutinizing the digital data for the pneumonia prediction using artificial intelligence: Artificial intelligence is considered to be an important factor for the pneumonia prediction in patients. Therefore, the MDA-PSP uses combined vital signs and CXR of patients to predict pneumonia status using a trained model and classifying them using deep learning. ii.
Leverage medical doctor's experience for the pneumonia status: MDA-PSP provides an effective solution for pneumonia prediction based on different parameters of 19 vital signs and five different sections in CXR multi-modal data analysis with their combinations. Thus, combining the patient's physiological data and chest X-ray image's within first 3 days and the training model to achieve better accuracy by the predictive model interpretability benefits the physicians analysis. iii.
Mean time analysis of the patient's pneumonia recovery within the next 7 days: The analysis is evaluated by a trained model consisting of SHAP and dense layers with class weights having the input readings from the initial to 2nd/3rd day of vital signs and CXR that can effectively predict the recovery status of the patient on the 7th day.

Literature Survey
Recently, the use of artificial intelligence has been found to be beneficial within medical hospitals for improving the quality of disease diagnosis, recovery prediction, and decreasing the analysis related errors in vital signs as well as radiology. At present, there Diagnostics 2022, 12, 1706 4 of 31 is no combination of clinical data and CXR data to establish a machine learning model to simulate doctors' decision-making to assist in the care of community-type pneumonia patients. Therefore, this paper approaches the MDA-PSP multi-modal data analysis to achieve high accuracy for the pneumonia prediction process by using vital signs and CXR images using pre-processing and classification, respectively. The recent studies structure shown in Figure 1 is used to define the pneumonia symptoms-based prediction analysis, which include machine learning, deep learning classifiers, and hybrid models.

Literature Survey
Recently, the use of artificial intelligence has been found to be beneficial within medical hospitals for improving the quality of disease diagnosis, recovery prediction, and decreasing the analysis related errors in vital signs as well as radiology. At present, there is no combination of clinical data and CXR data to establish a machine learning model to simulate doctors' decision-making to assist in the care of community-type pneumonia patients. Therefore, this paper approaches the MDA-PSP multi-modal data analysis to achieve high accuracy for the pneumonia prediction process by using vital signs and CXR images using pre-processing and classification, respectively. The recent studies structure shown in Figure 1 is used to define the pneumonia symptoms-based prediction analysis, which include machine learning, deep learning classifiers, and hybrid models. Several machine learning (ML) based models for pneumonia prediction include support vector mechanism (SVM), L2-logitic regression, random forest, gradient boosted classifier (GBC), XGBoost, etc. Acute respiratory disease syndrome (ARDS) detection by label uncertainty accounting using machine learning is presented by N. Reamaroon et al. [26]. Grading by confidence weights is assigned to labels for training input to SVM for good confidence in prediction. Additionally, a time-series method analyses for patient's clinical data inter-correlation by limiting overfitting is an improvement over classical SVM.
Stratification of patient risk for ARDS using machine learning is demonstrated by D. Zeiberg et al. [22]. The model uses EHR as features input to L2-logitic regression, successfully identifying patients observed for at least the first 6 h with hypoxia. Moreover, the detail model validation with multi-fold cross validation with optimal hyper-parameter tuning is performed. A machine learning predictive model for ARDS event analysis in ICU patients is demonstrated by X.-F. Ding et al. [27]. A random forest model is used with parameters including the decision tree numbers and random subset feature size. Eventually, a best split and k feature selection approach provides a better ROC curve. Diagnosis of ICU patients with their ventilation management by using machine learning and IoT is presented by G. Rehm et al. [28]. The data used is only of approximately 35 patients with k-fold cross validation, synthetic minority oversampling technique (SMOTE) for extremely random trees classifier (ERTC), GBC, and multi-layer perceptron (MLP) as ML algorithms. The data processing is done as micro-batch processing on the Amazon Web Services (AWS) cloud platform of 5 min sampling. ARDS identification from readily available clinical data by ML classifier models is demonstrated by P. Sinha et al. [29]. The dataset includes demographic, respiratory-based parameters, vital signs, and laboratory data to be processed by the 10-fold cross validation, which is evaluated by a gradient Several machine learning (ML) based models for pneumonia prediction include support vector mechanism (SVM), L2-logitic regression, random forest, gradient boosted classifier (GBC), XGBoost, etc. Acute respiratory disease syndrome (ARDS) detection by label uncertainty accounting using machine learning is presented by N. Reamaroon et al. [26]. Grading by confidence weights is assigned to labels for training input to SVM for good confidence in prediction. Additionally, a time-series method analyses for patient's clinical data inter-correlation by limiting overfitting is an improvement over classical SVM.
Stratification of patient risk for ARDS using machine learning is demonstrated by D. Zeiberg et al. [22]. The model uses EHR as features input to L2-logitic regression, successfully identifying patients observed for at least the first 6 h with hypoxia. Moreover, the detail model validation with multi-fold cross validation with optimal hyper-parameter tuning is performed. A machine learning predictive model for ARDS event analysis in ICU patients is demonstrated by X.-F. Ding et al. [27]. A random forest model is used with parameters including the decision tree numbers and random subset feature size. Eventually, a best split and k feature selection approach provides a better ROC curve. Diagnosis of ICU patients with their ventilation management by using machine learning and IoT is presented by G. Rehm et al. [28]. The data used is only of approximately 35 patients with k-fold cross validation, synthetic minority oversampling technique (SMOTE) for extremely random trees classifier (ERTC), GBC, and multi-layer perceptron (MLP) as ML algorithms. The data processing is done as micro-batch processing on the Amazon Web Services (AWS) cloud platform of 5 min sampling. ARDS identification from readily available clinical data by ML classifier models is demonstrated by P. Sinha et al. [29]. The dataset includes demographic, respiratory-based parameters, vital signs, and laboratory data to be processed by the 10-fold cross validation, which is evaluated by a gradient boosting machine (GBM) and XGBoost ML models in R. The evaluation is based on mortality at day 90 in different phenotypes by applying classifiers. The early prediction of ARDS using supervised ML is presented by S. Le et al. [30]. The MIMIC-III dataset is used similarly to the previous reference approach of 10-fold cross validation and XGBoost for early prediction of ARDS. The evaluation is performed on parameters of age, ICU ward admission, MEWS severity patients at admission, and median length-of-stay at 12 h, 24 h, and 48 h demographics for 1.90 k patients yielding good prediction. Some hybrid methods of machine learning, Diagnostics 2022, 12, 1706 5 of 31 artificial neural network, genetic algorithm, and natural language are found to be also effective in ARDS [31][32][33][34][35][36].
For CXR, symptom detection possesses multiple image processing and computer vision techniques that are discussed below as CNN, auto-encoder (AE), convolutional neural filter (CNF), etc. Chest X-rays (CheXNet) source-based pneumonia detection by radiology level analysis with deep learning is demonstrated by P. Rajpurkar et al. [37]. A convolutional model of 121 layers is designed to detect 14 different types of diseases by using heat maps, class activation mappings (CAMs), and feature maps. One hundred thousand frontal X-rays are used for analysis and detection resulting in higher F1 score. CXR images used for rib suppression by ICA method is presented by HX Nguyen et al. [38]. The use of rib suppression helps in early lung nodule detection by histogram equalization and frangi filter with independent component analysis (ICA). The Japanese Society of Radiological Technology (JSRT) database is used with images of 154 nodules and 93 non nodules having high resolution beneficial for training and testing. Bone suppression in chest radiography by deep learning is demonstrated by M. Gusarev et al. [39]. A stacked denoising autoencoder (AE) is used with a multi-layer convolutional neural model with reduced mean squared error (MSE) and maximized multi-scale structural similarity (MS-SSIM). An educational dataset is used with contrast-limited adaptive histogram equalization (CLAHE) for local contrast enhancement. Effectively performing bone suppression by CNF in chest X-ray is presented by N. Matsubara et al. [40]. Overlapping of bones in the lung fields can be affected by bone suppression using CNF based on CNN having a spatial filter configured by convolution, max pool, and fully connected layers. The dataset uses CT volume of cancer imaging by the National Institute of Health (JSRT), claiming higher experiments accuracy without losing soft-tissue information. Recently, several researchers used CXR as a medium to detect respiratory infection by deep learning in many studies [41][42][43][44]. The plan to organize and present research contents can be stated as Section 3 materials and methods containing architecture, its functioning, system model, algorithms, flowcharts and their respective description. In Section 4, results of the system configuration, dataset details, experiments, and results are presented. Finally, Section 5 has a discussion followed by conclusion for the research, acknowledgments, references, and Appendix.

Materials and Methods
Hospital Research Project and Approach: The MDA-PSP project is performed in Taichung Veterans General Hospital (TCVGH), which consists of a total 1500 beds for the patients in the central region of Taiwan.

Pre-Requisite and Hospitalization Criteria of the Patients
This work consists of patients with the criteria of admittance of the patients in the general ward. The patient should be an adult with an age 18 or higher and diagnosed with community acquired pneumonia < 48 h after admission. Whereas the patients excluded are due to the criteria of having one or multiple of the following: (a) A patient is directly admitted in the ICU, (b) The patients who have died in the hospital, (c) The patient is admitted for less than 3 days, (d) Acute respiratory failure and in need of a ventilator, (e) Admission to other hospital >72 h due to pneumonia, and (f) Women who are pregnant or need to provide breast-feeding. The vital-sign dataset given as input to the model is presented in the form of detailed statistics in Table 2. The dataset consists of various parameters, which have a specific range and is calculated for 3 consecutive days within the hospital. Every patient's data is presented in the form of mean, median, and mode values for the detailed statistics of day 1, day 2, and day 3.  Table 3 presents the comorbidity data details, which is categorized as per the patient's affected by the disease and the percentage of the same patients discharged within 7 days and after 7 days. Table 4 summarizes the various vital signs with their respective categories. All the vital signs within the dataset are categorized as lab data, comorbidity data, basic vital signs, and scores based on the 3 days' patients collected data. The comorbidity data shows the scope of complications that may affect the discharge of the patient's status, which can be more or less severe. Table 5 presents the vital signs checked by the physician as the basic factor for any treatment and its overcoming analysis. All the patients are adults, so the age is considered only for those greater than or equal to 20 years. The pulse rate also known as heart rate determines the number of times the heart beats per minute. The amount of oxygen traveling through a human body with his/her red blood cells is known as oxygen saturation or "O2 sats". The healthy adult's oxygen saturation is usually found within the range 95% to 100%. The respiratory rate is the breaths taken by the adult per minute. Figure 2 presents the MDA-PSP system model, which can be explained in four parts as multi-modal input data, data pre-processing with feature importance, dense layer architecture, and evaluation of the final results.

The Decision for Discharge within 7 Days
The MDA-PSP is designed to predict the status of the pneumonia infection. As all the patients considered are affected by pneumonia, the prediction of the pneumonia status after the treatment is made available. Usually, the patients with pneumonia are given one course of antibiotics for 5 to 7 days. The training data consists of outcome after the 3 days of the patient's status which is based on the doctor's decision. Multiple factors involved in the doctor's decision for discharge are the X-ray status, vital sign status, and the basic vital signs given in Table 5. The prediction determines whether the patient is discharged within (less than or equal to) 7 days or more than 7 days. Eventually, if the overall prediction is worse, then the discharge is given probably after 7 days. Whereas if the prediction is improving, then discharge is given within 7 days to the patient.

Input Data for Multi-Modal Analysis
The input given to this system is CXR images and vital signs data as shown on the left most section of Figure 2 for the multi-modal data analysis. Every CXR image has 5 sections of top-left, bottom-left, top-right, bottom-right, and center. The first four sections are used to check infiltrates, whereas the center section is used to check for cardiomegaly. In the case of vital sign data, the details are presented by the identifiers/labels and statistics as shown in Table 2.

Data Preprocessing
To achieve accurate attribution values and consistency, SHAP preprocessing is utilized for balanced weights distribution in the feature model [45]. The additive feature attribution methods possess local accuracy, missingness, and consistency. The feature attribution sum is equal to the function output stated as local accuracy. A missing feature is credited with no importance and is given as missingness (zi' = 0). Whereas a large impact feature is obtained even after the model change and has no decrease in the attribution, which is known as consistency. The attribution method for additive features is known to have g as an explanation model, which uses binary variable linear function: Here, the z {0, 1} M , ∅ i R, and M is the number of input features. The observed feature is usually represented by (z ; i = 1) or by (z ; i = 0) as unknown and the attribution values for features is given as ∅ i 's. There are multiple important techniques for data pre-processing that include data cleaning, data rule definition, and data supplement.
An MDA-PSP data flow diagram presents the plan and implementation details as shown in Figure 3. At the start, the MDA-PSP system takes the input of multi-modal data in the form of several patient's CXR and vital signs data. Initially, all the patients who are affected by pneumonia and recovered are only considered, whereas the patients who have died during the arrival/treatment/multiple disease disorder are excluded from this work. In the data pre-processing stage, the data is normalized and standardized to avoid any miscalculations in the predictions. A separate algorithm is applied for the processing of CXR and vital signs, so as to provide the best results in their respective category. In the case of vital signs data, a feature importance is first analyzed to capture the top set of features for better predictions by using SHAP. The SHAP features are then given as input to the dense layer classifier with hyper-parameter tuning [46,47] for best parameter orchestration. In a similar way, the CXR images are processed by using the gray scale conversion, resized to a standard values, and inference rules-based system is approved by the medical doctors. Later, a dense layer classifier with hyper-parameter tuning is implemented.
which is known as consistency. The attribution method for additive features is known to have as an explanation model, which uses binary variable linear function: Here, the 0,1 , ∅ ϵR, and M is the number of input features. The observed feature is usually represented by ( ; = 1) or by ( ; = 0) as unknown and the attribution values for features is given as ∅ 's. There are multiple important techniques for data preprocessing that include data cleaning, data rule definition, and data supplement.
An MDA-PSP data flow diagram presents the plan and implementation details as shown in Figure 3. At the start, the MDA-PSP system takes the input of multi-modal data in the form of several patient's CXR and vital signs data. Initially, all the patients who are affected by pneumonia and recovered are only considered, whereas the patients who have died during the arrival/treatment/multiple disease disorder are excluded from this work. In the data pre-processing stage, the data is normalized and standardized to avoid any miscalculations in the predictions. A separate algorithm is applied for the processing of CXR and vital signs, so as to provide the best results in their respective category. In the case of vital signs data, a feature importance is first analyzed to capture the top set of features for better predictions by using SHAP. The SHAP features are then given as input to the dense layer classifier with hyper-parameter tuning [46,47] for best parameter orchestration. In a similar way, the CXR images are processed by using the gray scale conversion, resized to a standard values, and inference rules-based system is approved by the medical doctors. Later, a dense layer classifier with hyper-parameter tuning is implemented. Successively, the concatenation of feature vectors from both dense layers of multimodal data is combined and re-evaluated with dense layers having new hyper-parameters consisting of 2 / 3 and 1 / 3 ratios for training and testing, respectively.

Functional Model
The detailed implementation of the MDA-PSP is shown in the form of a functional model in Figure 4. In the beginning, the doctor initiates the process by providing the input data to the MDA-PSP system. The vital sign data is then preprocessed by using the imputation, i.e., substitute missing values by using series mean, median, average, etc. The variables are then reduced by using categorical grouping. The input multi-modal data is then improved using the hospital's standard operating procedure (SOP)-based adaptive imputation, i.e., advance data pre-processing of univariate for statistical, multi-variate for regressions, and time series/interpolation for algebraic operations. The labeling of the raw data first needs to be performed by the domain experts/medical doctors to possess high quality training data. The labeling of data is performed on the trained first 3 days (72 h) of the patients records after the treatment that can be evaluated by the outcome for 0 (No Discharge) or 1 (Discharge) to the patient. Successively, the concatenation of feature vectors from both dense layers of multimodal data is combined and re-evaluated with dense layers having new hyper-parameters consisting of ⅔ and ⅓ ratios for training and testing, respectively.

Functional Model
The detailed implementation of the MDA-PSP is shown in the form of a functional model in Figure 4. In the beginning, the doctor initiates the process by providing the input data to the MDA-PSP system. The vital sign data is then preprocessed by using the imputation, i.e., substitute missing values by using series mean, median, average, etc. The variables are then reduced by using categorical grouping. The input multi-modal data is then improved using the hospital's standard operating procedure (SOP)-based adaptive imputation, i.e., advance data pre-processing of univariate for statistical, multi-variate for regressions, and time series/interpolation for algebraic operations. The labeling of the raw data first needs to be performed by the domain experts/medical doctors to possess high quality training data. The labeling of data is performed on the trained first 3 days (72 h) of the patients records after the treatment that can be evaluated by the outcome for 0 (No Discharge) or 1 (Discharge) to the patient. Similarly, the image CXR data then can be pre-processed by using computer vision operations on the greyscale, resize, etc. Later, the symptom model construction is performed by the DNN algorithm by producing a symptom vector as the output. Ultimately, the final hybrid model of DNN-BN, which takes a combined symptom vector from the vital signs and CXR data, is then evaluated to predict the outcome as either 0 or 1. This outcome is then utilized by the medical doctors by applying inference rule-based knowledge systems. It is specified as checking the outcome, age, pulse rate, SaO2, respiratory rate, and comorbidity disease stage. Therefore, the decision is then given for the final discharge to the patient.

Algorithms
This subsection is used to present the various MDA-PSP algorithms with pseudocode for the detail working. The pre-processing is performed by using the imputation and Similarly, the image CXR data then can be pre-processed by using computer vision operations on the greyscale, resize, etc. Later, the symptom model construction is performed by the DNN algorithm by producing a symptom vector as the output. Ultimately, the final hybrid model of DNN-BN, which takes a combined symptom vector from the vital signs and CXR data, is then evaluated to predict the outcome as either 0 or 1. This outcome is then utilized by the medical doctors by applying inference rule-based knowledge systems. It is specified as checking the outcome, age, pulse rate, SaO2, respiratory rate, and comorbidity disease stage. Therefore, the decision is then given for the final discharge to the patient.

Algorithms
This subsection is used to present the various MDA-PSP algorithms with pseudocode for the detail working. The pre-processing is performed by using the imputation and adaptive imputation for the patient's vital signs data. Basically, the purpose of preprocessing is avoiding inconsistency within the dataset and its limitations. The experiments section demonstrates the results obtained by using this method.
Algorithm 1 given above is used to present the data preprocessing performed on the patient's vital signs data, CXR images, and has obtained feature vectors from it. Later, the feature vectors of the vital sign data and CXR images are used as the multi-modal data analysis, which is used to predict the pneumonia status of the patient. In step 1, vital signs record of the patient is taken as input in the form of raw data (DRaw, IRaw). In step 2, the vital sign data and CXR images are being labeled by the doctors for the exceptional cases in the combined training and test set. In step 3, the inference rules (RInference) have the conditions specified by the doctors. In step 4, the threshold determines the cut off limit for the decision for the patient's discharge. In step 5, the output is given in the form of NN confidence (NN Conf ) and in step 6, the MDA-PSP system alerts the prediction of the patient's discharge within the 7 days or not. In step 7, the candidate set 1, candidate set 2, the NN Conf , and the candidate final (candidate Final ) used for storing the feature vectors are initialized to NULL. In step 8, the vital sign raw data (DRaw) is pre-processed by using the imputation (DProcessed IM ) by the mean, median, or by k-nearest neighbor algorithm. The preprocessing is performed on the input (IRaw) by the imputation and stored in the CXR image processed (IProcessed IM ). Later in step 9, the DProcessed IM is used to map the multiple data by reducing it to a single category (DProcessed CG ) for identification. The CXR image (IProcessed IM ) is divided into four sections as the upper-left, lower-left, upperright, and lower-right to capture the various patterns for the pneumonia by identifying the infected area and is stored as a categorized image (IProcessed CG ). In step 10, the vital signs preprocessed data (DProcessed VS ) is obtained by combining the adaptive imputation data for univariate, multivariate, and time series/interpolation with the labeling performed by the medical doctor/domain expert (DProcessed CG ). In step 11, the CXR image is set to standard size and converted to gray scale for better image processing, which is later labeled by the doctor to provide detailed identification patterns (IProcessed IP ) on the current dataset. In step 12, the inference rules (RInference) are set to be checked for preliminary conditions including the patient affected by pneumonia and admitted in the general ward of the hospital. Whereas the clinical checkup parameters as specified in Table 5 includes age, pulse rate, oxygenation (SaO2), respiratory rate and comorbidity for multiple diseases with each consisting of 20% points are rated based on doctor's observations. In step 13, if the DProcessed VS is found to be consistent and if the score counted from the inference rules (RInference) is greater than or equal to that specified by the doctor, which is considered to be ready to be processed further, then in step 14, by using the dense layer, the feature vectors are captured from the DProcessed VS, and the processed image (IProcessed IP ) and is stored in candidate Set 1 and candidate Set 2, respectively, instep 15. In step 16, the data is considered to be inconsistent or the score is less than the threshold, then an error message is printed by the system in step 17 as inconsistent data. In step 18, as the inconsistent data is found to be insufficient for processing, then the algorithm terminates. In step 19, the candidate finally stores the combined feature vectors of the vital signs and CXR image as the candidate Set 1 and candidate Set 2, respectively. In step 20, the NN confidence (NN Conf ) is the dense layer with batch normalization (BN) prediction for the input with candidate final (candidate Final ). In step 21, if the condition checks whether NN confidence (NN Conf ) is greater than or equal to the threshold, then the MDA-PSP system raises an alert to allow discharge for the recovered patient in step 22. Training this model will help doctors determine whether a patient is discharged from the hospital within 7 days. In addition to strengthening treatment, the mobility of the bed becomes higher and medical resources can be used more effectively. In step 23, if NN confidence (NN Conf ) is less than the threshold value then an alert is raised for no discharge to the patient as the recovery is not satisfactory in step 24. Finally, in step 25, the NN confidence (NN Conf ) and alert is returned by the system. Input: DRaw and IRaw, Vital signs record and CXR images of the patients.

3.
RInference , Inference rules with threshold specified by the doctors.
Alert, MDA-PSP system alert for discharge or no discharge within 7 days. 7.

Results
In this section, all the experiments carried out for the model implementation, comparisons, and results will be presented. The experiments and testing performed on the hardware for this model is presented in Table 6. A series of experiments will be presented in the following subsections that will be evaluated by using a detailed analysis and its respective implementation. In this work, we simulate the doctor's decision-making process, establish a predictive model, and predict whether a patient can be discharged on the seventh day by the third day of hospitalization. Ultimately, this may improve the quality of medical care and reduce medical expenses. The focus here will be to understand the behavior of data with different preprocessing and its suitable machine and deep learning methods to achieve a better combination for the evaluations. Nevertheless, the problem needs to be clearly defined as the incorrect information makes the experiment invalid. The dataset used in this work belongs to a TCVGH, which has the observations and statistics as presented in Figure 5. The length of hospital days presents a maximum number of days of hospitalization (including emergency unit) as high as 104 days for the affected patients, whereas Figure 5 only presents for the 24 days, as it is found to be more effective having the x-axis as days and the y axis as number of cases. The features used within this experiment can be referred to in Table 4, whereas Table 7 provides detailed machine and deep learning results with cutoffs. In Figure 6, a new set of features are constructed by the addition of the new 12 h of features by the doctor's recommendation to analyze the feature importance, SHAP scoring (Figure 6a), and applying it on machine and deep learning-based methods to have better prediction results (Figure 6b). The 12 h features include RR, Pulse, SBP, Pao2, and BT_metric for the evaluation. Figure 6c predicts the class 0 frequency as the no discharge prediction for the patients with high confidence, whereas Figure 6d shows the discharge prediction with less confidence. Therefore, it can be inferred that patients discharged within the hospital around 7 days may not have obvious clinical features within 3 days.

Vital Signs for Data Observations, Feature Scoring and Evaluation
The dataset used in this work belongs to a TCVGH, which has the observatio statistics as presented in Figure 5. The length of hospital days presents a maximum ber of days of hospitalization (including emergency unit) as high as 104 days for fected patients, whereas Figure 5 only presents for the 24 days, as it is found to b effective having the x-axis as days and the y axis as number of cases. The feature within this experiment can be referred to in Table 4, whereas Table 7 provides d machine and deep learning results with cutoffs. In Figure 6, a new set of features a structed by the addition of the new 12 h of features by the doctor's recommenda analyze the feature importance, SHAP scoring (Figure 6a), and applying it on m and deep learning-based methods to have better prediction results (Figure 6b). T features include RR, Pulse, SBP, Pao2, and BT_metric for the evaluation. Figure 6c p the class 0 frequency as the no discharge prediction for the patients with high conf whereas Figure 6d shows the discharge prediction with less confidence. Therefore be inferred that patients discharged within the hospital around 7 days may not ha vious clinical features within 3 days.

CXR Imaging Sections for Symptom Categorization
The CXR image can be categorized based on the symptoms of infiltrate and cardiomegaly. Furthermore, the symptoms are also judged by the doctors based on the quality of the CXR, the locations of the symptoms observed, and the partitions into four equal CXR sections with the label severity. As given in Table 8, the labels were set to normal, slight, medium, and severe. Later, during the experiments in Table 9, the demonstration of the prediction accuracy was noticed quite higher for the normal and severe symptoms/labels. The infiltrate is given by four equal sections of the CXR and cardiomegaly as only independent CXR. The CXR images are processed in the following steps: (a) Establish an inference model for automatic grading of chest X-ray image infiltration, which is based on 600 doctors' interpretation, (b) Image infiltration extraction feature by vector technology, and (c) Data and image integration for machine learning inference model establishment.

CXR Imaging Sections for Symptom Categorization
The CXR image can be categorized based on the symptoms of infiltrate and cardiomegaly. Furthermore, the symptoms are also judged by the doctors based on the quality of the CXR, the locations of the symptoms observed, and the partitions into four equal CXR sections with the label severity. As given in Table 8, the labels were set to normal, slight, medium, and severe. Later, during the experiments in Table 9, the demonstration of the prediction accuracy was noticed quite higher for the normal and severe symptoms/labels. The infiltrate is given by four equal sections of the CXR and cardiomegaly as only independent CXR. The CXR images are processed in the following steps: (a) Establish an inference model for automatic grading of chest X-ray image infiltration, which is based on 600 doctors' interpretation, (b) Image infiltration extraction feature by vector technology, and (c) Data and image integration for machine learning inference model establishment.

Symptom Feature Extraction and CXR Combination with Vital Signs
The severity discrimination model of each affected area to extract feature vectors from the average of first and last available patients CXR images is used to predict discharge in this scenario. The combination of four parts of the infiltrate and the symptoms of cardiomegaly is to check for a possible improvement within the results. The interpretable layers added within the dense CNN are used to check which layer size performs best for the prediction improvement. Adding cardiac enlargement/cardiomegaly feature results helps to significantly improve the identification of the patients who can be discharged from the hospital within 7 days. These results are considered to be slightly better than using vital signs only, in comparison to the symptom feature extraction combined with the vital sign, i.e., average values of two X-ray from the first 3 days of the admittance in the hospital. Similarly combining the features of the infiltrate, cardiomegaly, and vital signs indicates that slight improvement with the constructed CNN layer can also provide a better output. Adding the vital signs and cardiac enlargement characteristics at the same time has limited the improvement in the identification of the patients, who can be discharged from the hospital within 7 days, whereas the significance of adding the heart enlargement feature and adding the vital sign feature may be very similar to those who can be discharged within 7 days. The data balancing methods are necessary for the unbalanced data to make the model training well. In Figure 7, the performance statistics for class weights is shown, which indicates the down sampling has a significant increase in label 0 in the original class weight experiment, and the label 1 is found to be declining. Even if having balanced data in both the discharge/no discharge categories, there is no improvement in the results. The details of the naïve, up sampling, down sampling, and class weight-based evaluation can be referred to in the Appendix A Section from Figures A1-A10   Thus, an optimal cut point is selected based on the balance within the positive and negative predictions by avoiding the false alarm simultaneously. Therefore, the cut point is considered to be of high importance as it is critical to the MDA-PSP model success factor.
In the case of no class-weights used during the experiments, the F1 score for label 0 has improved but no significant improvement is noticed in label 1. Table 10 shows that in the case of dense layers with class weights, a slight decrease in label 0 is noticed but a significant improvement to 0.47 from 0.38 in comparison to no class weights. Figure 8 provides a calibration plot for MDA-PSP machines and deep learning algorithms for detailed analysis to evaluate how much the classifier is calibrated, i.e., every class label has differing probabilities that are measured. The linear straight line is the ideal calibrated model curve.  Thus, an optimal cut point is selected based on the balance within the positive and negative predictions by avoiding the false alarm simultaneously. Therefore, the cut point is considered to be of high importance as it is critical to the MDA-PSP model success factor.
In the case of no class-weights used during the experiments, the F1 score for label 0 has improved but no significant improvement is noticed in label 1. Table 10 shows that in the case of dense layers with class weights, a slight decrease in label 0 is noticed but a significant improvement to 0.47 from 0.38 in comparison to no class weights. Figure 8 provides a calibration plot for MDA-PSP machines and deep learning algorithms for detailed analysis to evaluate how much the classifier is calibrated, i.e., every class label has differing probabilities that are measured. The linear straight line is the ideal calibrated model curve.

Discussion
The MDA-PSP overall findings are summarized as the combined CXR symptom vector and vital sign to build a model for discharge prediction, which have training accuracy of 75%, accuracy rate of not being discharged as 81%, and accuracy rate of discharge as

Discussion
The MDA-PSP overall findings are summarized as the combined CXR symptom vector and vital sign to build a model for discharge prediction, which have training accuracy of 75%, accuracy rate of not being discharged as 81%, and accuracy rate of discharge as 50%. In the discussion section, a detailed discussion about the various factors directly and indirectly affecting the patient status for the discharge is presented. Later, the Venn diagram will be presented to study logical relations between the sets for similarities and differences.

I.
Insurance: In Taiwan, the government provides national health insurance (NHI) cards to the citizens. The NHI card requires a once a year payment of nominal amount, which then provides health insurance for any major natural or accidental treatment within any Taiwanese major hospital. Even though it provides insurance for any case, the time period for the patient to be admitted should be no more than 7 days for the refund. So, this forms the need for one of our motivations to design an XAI model for prediction of the patient's discharge on the 7th day. Due to insurance benefit conditions, both the doctor and patient prefer to have discharge within 7 days of the hospital admittance. II.
Suitable Discharge Time: The suitable discharge time for the patient is considered to be less than or equal to 7 days in the treatment. Nevertheless, less time is always favorable as it can save hospital resources as well as of the doctors, patients, family, hospital staff, etc. Whereas in the case of unfavorable cases, the patient is required to stay for a longer time duration that may lead to losing the insurance claim for the refund. The suitable discharge time can also refer to the minimum treatment time required based on the patient's health severity and may be the trainee doctor's decision. The pneumonia is known to happen in any age range from one day born infant to any adult patient, so the decision also considers the medicine treatment effects. In the case of diet based on certain regional, lifestyle factors, the discharge time may vary, too. III.
Doctor's Recommendation: In rare cases, the doctors usually recommend the patient to take a delayed discharge. Complexity of the case may depend on age, lifestyle for the slower recovery, or adapting to the normal health status. In addition, there are foreign patients who need to be treated exclusively and by experienced doctors, as the treatment approach and the recovery may vary depending on different continents. In some cases, the treatment by some medicine may react in the medical report. Therefore, the doctor needs more time to change the treatment and to have patience for the recovery process. IV.
Patient's Mental Status: The discharge time also depends on the patient's willingness or feeling energetic to confirm complete recovery. The doctors usually check for vital signs and medical reports for the discharge approval. In rare cases, if the patient is not mentally ready to discharge and possess good financial background, then the doctor may allow to continue stay depending upon beds availability. In some cases, if a patient is addicted to some habits, then he prefers to stay until complete recovery. It can also depend on how much the patient needs medical facilities to be received in a special exclusive room for the admittance. As soon as the patient is recovered, the doctor recommends discharge and continues to monitor vital signs remotely by using sensor watch, video camera-based consulting sessions, etc. The patient then later on can stay connected with the specialist doctor to report the symptoms, if any, as the follow-up. V.
Re-admission Issue: In some exceptional circumstances [48,49], the patient has to go through the re-admission in the hospital. In the case of mid-size hospitals, there may be an undetected issue or the specialist doctor and medical condition predicting machines are not available in the emergency situation. Moreover, if the patient is in the transfer period because of his work, business, family shifting, or other factors, then the patient has to request for transfer options with the hospital by co-operation process. In some insurance cases, the patient can only get refund advantage of the hospital charges with referring to some specific hospital.
Possibly, there can be some recommendations by the family or doctor to shift to a specialty hospital for fast recovery and experienced approach towards serious health conditions. VI.
Family Care/Support: In the case of some single people residing remotely [50] from the hometown region, the doctor may advise the patient to stay a couple of days more for the complete health recovery convenience. Even if there are some foreign patients working in the company, they may need special treatment and consulting for the health recovery. When the family and financial support is good, then the services can be shifted to the patient's home by a visiting doctor. Whereas in the case of serious health conditions, the doctor may advise the patient to shift to a exclusive room, reserved for the special treatment facility. VII.
Multiple Disease Disorder (Comorbidity) of a Patient: In special cases, a patient is admitted in the hospital with a chronic disease [51]. Later on, a past or new disease is diagnosed, which is required to be treated carefully. In such cases, extra time is required for the complete health recovery of the patient. There are some cases when an addicted patient needs psychological counseling for controlling habits and adopting a healthy lifestyle. A complex case can be stated as when there is a dependency between the diseases, which may require specialist treatment. Even though it is a rare case to extend discharge time for the secondary disease, it needs to be cured. When a patient may have to stay for the secondary disease for more time than expected initially, the hospital must provide necessary support for such extension. Ultimately, it is recommended by the doctor to cure completely rather than schedule follow-up for the diagnosis and treatments. Pneumonia is the third of the top ten causes of death in Taiwan as well as in the world. The longer the hospital stay, the more likely it is to cause complications, hypoxemia, anemia, hypoalbuminemia, etc. VIII.
Deploying AI System in the Hospital: Considering most of the application domain, AI models help to predict a patient's conditions but not to diagnose the patient's outcomes. Doctors are responsible for diagnosis and taking actions with treatments, and AI helps to provide decision choice and recommendations. AI models provide high-quality recommendations for junior doctors. Hands-on experiences are one of the major parts of doctor training. Young doctors will have a high-quality baseline of patient diagnosis with the help of AI models. Moreover, it improves the patientcare quality, even saving patient's lives. In most hospitals, doctors are a critical resource and almost overrun. It is not possible to take care of all patients at any time. Automatic patient's data collection for AI models will strongly help the patients care for 24 h/day. If the sensitivity and specificity of AI model predictions are good enough, it will greatly relieve the loading of doctors, and provide persistent health-care service during patient's stay period in the hospital.

Venn Diagram Presentation for the Detail Analysis of the Prediction Results
The intersection diagram is used to observe the feature intersection of the vital signs and CXR for the prediction analysis. The four Venn diagrams are produced as given in Figure 9 and Table 11: i.
Cannot be discharged within seven days, the prediction is correct (true positive). ii.
Unable to be discharged within seven days, the prediction was wrong (false positive). iii.
Able to be discharged within seven days, the prediction was wrong (false negative). iv.
Able to be discharged within seven days, the prediction is correct (true negative). Diagnostics 2022, 12, x FOR PEER REVIEW 20 of 31  The patient numbers are used in each set to calculate the average scores of the symptoms. Each set will be combined to calculate the scores that include four symptoms (four sections of lung infiltration) and seven symptoms (four types of lung infiltrate, cardiac hypertrophy, and two types of pulmonary hydrops). The label 0 outcome means hospitalization for more than seven days and label 1 outcome is the patient can be discharged within 7 days. Therefore, it is proved that for Figure 9a, vital signs are more accurate and for Figure 9d, CXR is more accurate. The Venn diagram proved that different data (CXR and Vital Sign) as shown in Figure 10 and detailed in Table 12 have high reproducibility for patients who can be discharged from the hospital (label 0), which means that these patients have characteristics leading to the success of both models. For patients who cannot be discharged from the hospital (label 1), the repeatability is significantly reduced. On the contrary, the part of mis-guessing label 1 is significantly improved. The uncertainty of the data representing label 1 is higher, which also confirms our past experiments; the predictive ability of label 1 is also relatively low.  The patient numbers are used in each set to calculate the average scores of the symptoms. Each set will be combined to calculate the scores that include four symptoms (four sections of lung infiltration) and seven symptoms (four types of lung infiltrate, cardiac hypertrophy, and two types of pulmonary hydrops). The label 0 outcome means hospitalization for more than seven days and label 1 outcome is the patient can be discharged within 7 days. Therefore, it is proved that for Figure 9a, vital signs are more accurate and for Figure 9d, CXR is more accurate. The Venn diagram proved that different data (CXR and Vital Sign) as shown in Figure 10 and detailed in Table 12 have high reproducibility for patients who can be discharged from the hospital (label 0), which means that these patients have characteristics leading to the success of both models. For patients who cannot be discharged from the hospital (label 1), the repeatability is significantly reduced. On the contrary, the part of mis-guessing label 1 is significantly improved. The uncertainty of the data representing label 1 is higher, which also confirms our past experiments; the predictive ability of label 1 is also relatively low.

Limitations of the Prediction System
The following are the limitations of the MDA-PSP system: a. The medical doctors need to be trained for interpreting classification results and identifying false positive cases. b. The classification model must adapt after appending new data for training and evaluation. c. The hidden layers in the dense do not provide complete information for the doctor's detail analysis. In the future, the system can be made more transparent at every step using explainable AI (XAI).

Conclusions
Pneumonia is the third of the top ten causes of mortality in the world including Taiwan. To overcome such serious respiratory disease, MDA-PSP provides effective data preprocessing operations by SHAP feature analysis, imputation, adaptive imputation for vital signs, and CXR by using classification to achieve better outcome. The data prepro-

Limitations of the Prediction System
The following are the limitations of the MDA-PSP system: a.
The medical doctors need to be trained for interpreting classification results and identifying false positive cases. b.
The classification model must adapt after appending new data for training and evaluation. c.
The hidden layers in the dense do not provide complete information for the doctor's detail analysis. In the future, the system can be made more transparent at every step using explainable AI (XAI).

Conclusions
Pneumonia is the third of the top ten causes of mortality in the world including Taiwan. To overcome such serious respiratory disease, MDA-PSP provides effective data preprocessing operations by SHAP feature analysis, imputation, adaptive imputation for vital signs, and CXR by using classification to achieve better outcome. The data preprocessing and class weights-based classification is the most prominent process for the evaluation. Therefore, for the patients who are admitted in the general ward because of pneumonia, their symptoms are evaluated to have prediction of discharge within 7 days. The dense-BN with class weights has provided accuracy of 75% by the multi-modal data analysis. Various methods from machine and deep learning is applied to have an overall analysis and their performance comparison prediction on the patient's data. In the future, we plan to provide clinical care suggestions, assist doctors in decision-making, and control the number of beds, which can be extended to medical institutions with insufficient equipment. Additionally, we plan to have cross-academic model effectiveness verification and apply for Software as a Medical Device (SaMD). Also, need to design a plan on introducing clinical auxiliary care and observing the effectiveness of clinical application, such as hospitalization days, medical expenses, etc.

Informed Consent Statement:
The approval by the committee has led to the waiving of the patient's consent, who are part of this work. The patient's personal identification information has been not included to assure anonymity within this work.

Data Availability Statement:
The private data is not allowed to be disclosed due to hospital policy. Table A1 shows raw vital signs data used without any preprocessing to be evaluated with the machine learning algorithm. It can be noticed that the F1 score is quite suffering in the evaluation. In the case of preprocessing used for data clean up as presented in Table A2 for hybrid dataset evaluation with different machine and deep learning-based methods, there is a significant improvement in the F1 score. As discussed earlier, in the results as given below about the four-class and two-class based evaluation, the two-class evaluation emerges to be more accurate, and its detail output is shared in Tables A3 and A4. The accuracy can be compared for Tables A3 and A4 based on the same features by different class results. The confusion matrix and performance statistics for different method analysis is presented by Figures A1-A10. It shows that double up sampling in comparison to the null class weights has decline in label 0 but no significant improvement is noticed in label 1. Similarly, in the case of triple up sampling in comparison to the null class weights, it has slight decline in label 0 but no significant improvement is noticed in label 1.                         Nevertheless, quadruple up sampling in comparison to the null class weights has decline in label 0 but a slight improvement is noticed in label 1. Ultimately, down sampling with no class weights is not considered as a good method for this experiment. Similarly, double and triple up sampling is not recommended. Whereas for quadruple up sampling, although label 1 has improved, the sacrifice is not good to practice in actual situations and test concentration.

Appendix B.1. Introduction (Extension)
AI-based status prediction is considered to be a better alternative [21], for the case of pneumonia as well as hospital planning for general ward/ICU space. In hospital management, the quality measure can be ensured by the MDA-PSP system as a part of product prototype development, preliminary verifications, and later used in real environment.
The background for MDA-PSP can be presented as standard scalar, synthetic minority oversampling technique (SMOTE), random forest, machine learning, deep learning, etc. [22]. Standard scalar is used to scale the data to a standard range. Each input variable is scaled separately by subtracting the mean then dividing by standard deviation for shifting the distribution to get the mean as zero and the standard deviation as one for dataset preprocessing. SMOTE is one of the statistical techniques to balance the number of cases in the dataset. So, the minority cases are handled by generation of new instances within the dataset, whereas the majority cases are unchanged, thus solving the imbalance issue within the dataset. The random forest classifier is a well-known meta estimator that is used to fit multiple decision tree classifiers on dataset subsamples. Averaging is used for Nevertheless, quadruple up sampling in comparison to the null class weights has decline in label 0 but a slight improvement is noticed in label 1. Ultimately, down sampling with no class weights is not considered as a good method for this experiment. Similarly, double and triple up sampling is not recommended. Whereas for quadruple up sampling, although label 1 has improved, the sacrifice is not good to practice in actual situations and test concentration.

Appendix B
Appendix B.1. Introduction (Extension) AI-based status prediction is considered to be a better alternative [21], for the case of pneumonia as well as hospital planning for general ward/ICU space. In hospital management, the quality measure can be ensured by the MDA-PSP system as a part of product prototype development, preliminary verifications, and later used in real environment.
The background for MDA-PSP can be presented as standard scalar, synthetic minority oversampling technique (SMOTE), random forest, machine learning, deep learning, etc. [22]. Standard scalar is used to scale the data to a standard range. Each input variable is scaled separately by subtracting the mean then dividing by standard deviation for shifting the distribution to get the mean as zero and the standard deviation as one for dataset preprocessing. SMOTE is one of the statistical techniques to balance the number of cases in the dataset. So, the minority cases are handled by generation of new instances within the dataset, whereas the majority cases are unchanged, thus solving the imbalance issue within the dataset. The random forest classifier is a well-known meta estimator that is used to fit multiple decision tree classifiers on dataset subsamples. Averaging is used for the prediction accuracy improvement and over-fitting is kept in control. Machine learning is a subset of AI, which is used to make predictions based on training data. Most of the machine learning algorithms are based on statistical process analysis. It can be further categorized as supervised, unsupervised, and reinforcement learning. Deep learning is a well-known machine learning algorithm class that helps to construct and utilize multiple layers for different feature extraction from the input data. It is widely used in the image processing field for its application and classification.
In short, MDA-PSP uses a hybrid approach for predicting the status of pneumonia and improving the classification process. Ultimately, a good prediction and complete process is accomplished. In MDA-PSP, the research gap is concerned about multi-modal data analysis, which is quite scarce. This research is exclusively designed for demonstration of prediction on the 7th day of discharge status. The MDA-PSP research is rational as it provides prediction on the 7th day of discharge of patients affected by pneumonia using a hybrid approach. This approach is approved by the medical practitioners as the complete process for the assessment of the pneumonia disease as presented in the results of Section 4.