Validity and Reliability of Self-Assessment Tool for Risk Prioritization Following Exposure to Tuberculosis in a Hospital Setting

The Modified Self-Assessment (MSA) and Present Self-Assessment (PSA) forms are questionnaires used to prioritize the risk of infection of health workers exposed to tuberculosis (TB) in Srinagarind Hospital in Thailand. As MSA was developed from PSA, the validity and reliability of MSA need to be assessed. The research aim is to examine the content validity of MSA and to assess the respective reliability of MSA and PSA vis-à-vis expert opinion. Seven experts determined the content validity index (CVI) of MSA. MSA and PSA were used to prioritize the TB contact of 108 subjects, and we compared the result with the risk assessed by the experts. The respective Kappa agreements between MSA and PSA and the experts were used to assess reliability. The result of the content validity index revealed that MSA had I-CVI > 0.83 for all questions and an S-CVI/Ave above 0.90 for all factors. The Kappa agreement of contact priority between MSA and the experts was 0.80; it was 0.58 between PSA and the experts. MSA can, thus, be used to prioritize contact with tuberculosis in Srinagarind Hospital. MSA is a valid risk communication tool for aerosol-generating procedures. Further study should be conducted in other hospitals, and the number of participants should be increased in order to come to a concrete result.


Introduction
Tuberculosis (TB) infection is considered an urgent global health issue [1]. The World Health Organization (WHO) has devised a three-pronged strategy to end TB globally: (1) Reach 90% of all people who need TB treatment, (2) include 90% of all people in key populations, and (3) achieve at least 90% treatment success [2]. Health workers (HWs) are a key population because they have a 2-4 times increased risk of TB infection compared to the general population [3]. Occupational latent TB infection (LTBI) can lead to active TB (aTB), which can be a source of transmission to others and a complicated infection in and of itself [4,5]. LTBI screening and LTBI management in HWs are necessary in order to achieve WHO's "End TB Strategy" [6].
There are three screening models for LTBI surveillance in HWs: preplacement screening, periodic screening, and postexposure screening. The Center of Disease Control and Prevention (CDC) recommends all HWs should be screened for LTBI (i.e., preplacement and postexposure) [7]. Postexposure screening and treatment of LTBI should be considered a strong recommendation for screening HWs for LTBI [6][7][8][9].
Thailand has a high burden of TB; thus, HWs encounter complicated cases of TB with comorbidities. As a consequence, TB infection among HWs is often due to occupational exposure [3,10]. Contact investigations play an integral role in the strategy for ending TB, and these activities were introduced in the 2005 CDC Guideline and the 2007 Maryland Guideline [11,12]. These guidelines are the protocol of contact investigations, involving complex decisions with respect to the investigation priority of HWs after exposure to TB using epidemiology triad components: hosts (HWs), agents (TB patients), and environment (contact activity, contact duration, and room size).
The Occupational Health and Safety Office (OH&S) at the Faculty of Medicine, Khon Kaen University Thailand (KKU) created a Present Self-Assessment (PSA) form-a tool for prioritizing contact investigations after occupational TB exposure [10]. The PSA content was adopted from the two guidelines mentioned [11,12]. HWs exposed to TB have their contact prioritized using PSA. High and medium priority contact are followed up for LTBI/aTB investigation. Panthong et al. [10] studied the benefit of prioritization using PSA and found that the results were not consistent with the interferon-gamma release assay test. HWs' confusion regarding some of the PSA questions led to inappropriate contact prioritization. The validity and reliability of PSA have not been assessed as some PSA content that is essential to determining contact priority is lacking. The present study aims to (1) examine the content validity of the Modified Self-Assessment (MSA) form and (2) assess the respective reliability of MSA and PSA with respect to expert opinion.

Study Design
A three-step descriptive study was conducted. First, the MSA form was created by improving the content of the PSA form. Second, MSA was examined for content validity by seven experts. Third, its reliability was assessed using agreement measurements between both PSA and MSA and the experts.

Study Population for Examining Content Validity
Seven eligible content validity experts were asked to examine the content validity of MSA. The seven included 1 chest medicine doctor, 3 occupational medicine doctors, 1 infectious control manager, 1 safety officer, and 1 epidemiologist.

The Study Population of Reliability Assessment
The reliability expert was an occupational medicine doctor with two years of experience in contact TB history prioritization using the two guidelines. The reliability expert prioritized the TB contact of subjects with regard to their TB contact history through telephone interviews.
Srinagarind Hospital has a TB contact investigation procedure. All HWs are to report the contact event to OH&S in order to follow the procedure. HWs who reported the contact event between 1 January 2020 and 31 March 2020 were asked to participate in this study. The sample size (n = 108) was calculated using Cohen's kappa sample size calculation using the following formula [13]: where n = sample size Z α = 95% CI = 1.96 Z β = 80% of power testing = 0.84 K 0 = lower border of Kappa = 0.60 Q 0 = coefficient of K 0 = 0.64 K 1 = upper border of Kappa = 0.80 Q 1 = coefficient of K 1 = 0.36 One hundred and eight subjects, who had given informed consent, were asked to complete both MSA and PSA within 72 h after contact. They also provided their contact history to the reliability expert through a telephone interview within 5 days of completing MSA and PSA.

MSA and PSA
Both MSA and PSA comprise questions relating to the aTB and LTBI risk factors: (1) infectivity of the index case, (2) personal factors of the HWs (underlying disease, health status, and medication), (3) contact activity, and (4) environmental factors (contact duration, size of the room). MSA represents an improvement on PSA as it has more comprehensive questions regarding these factors [14][15][16][17][18]. MSA and PSA are included in Tables A1 and A2  of Appendix A. A TB patient will be categorized into one of three levels based on infectivity. When HWs have contact with each TB patient category, the priority of contact is determined stepby-step: personal factors (risk of aTB), contact activity (risk of LTBI), and environmental factors (risk of LTBI). The steps of the contact prioritization process are presented in Figures 1-3, according to the category type [10].

Study Tool
A content validity examination form (CVEF; shown in Table A3 of Appendix A) was used by seven experts. Each question of CVEF is rated on a 4-scale (strongly agree, agree, disagree, and strongly disagree). For reliability assessment, a data record sheet was completed by the reliability expert. The contact history collected during the telephone interviews with each of the 108 subjects was recorded on 108 separate datasheets. Since the data of MSA were facts, the content validity examination was proper. We used the Item Content Validity Index (I-CVI) and the Scale Content Validity Index per Average (S-CVI/Ave) for data analysis of content validity. I-CVI was calculated from the proportion of experts who "strongly agreed" or "agreed" to each question item of CVEF. S-CVI/Ave refers to the content validity of the factor, calculated from the average value of all I-CVI scores. The cutoff values of I-CVI and S-CVI/Ave depend on the number of experts. Polit and Beck [19] suggested the acceptant of I-CVI and S-CVI/Ave, in terms of 6-8 experts, should ≥0.83 and ≥0.90, respectively. The I-CVI and S-CVI/Ave formulas are presented in Table A3 of Appendix A.

Reliability of MSA and PSA
Reliability was assessed using Cohen's kappa (Kappa) agreement between MSA and expert opinion and PSA and expert opinion using the following formula: where P 0 = proportion of observed agreement P E = proportion of expected agreement IBM SPSS Statistics version 19 (with Khon Kaen university's license) was used for Kappa analysis; the process of calculation is demonstrated in Tables A4 and A5 of Appendix A. An acceptable Kappa value for a moderate level of reliability is considered to be ≥0.60 [20]. Kappa agreement analysis is a critical aspect in determining contact priority. There are three levels of contact priority-high, medium, and low. Personal factors, contact activity factors, and environmental factors were also included in the Kappa agreement analysis, albeit these have only minor agreement.
Personal factors have two levels of priority, the same as the contact activity factors. The level is defined as high risk and low risk. For example, HWs with items in personal factors are marked as high-risk for aTB; HWs who performed "suction" (a contact activity factor) are marked as high-risk for LTBI. On the other hand, HWs who are healthy or have other conditions that are not present in personal factors are marked as low-risk for aTB; HWs who perform "oral medication" (a contact activity factor) with TB patients are marked as low-risk for LTBI.
Kappa agreement was calculated based on the comparison between each level of priority between MSA and expert opinion and PSA and expert opinion.

Ethics
The primary ethical concern was the confidentiality of personal data. This report was certified by the Institutional Review Board (Number IRB00001189) and the Khon Kaen University Research Ethics Committee (HE621446).

Demographic Data
The 108 subjects who participated in the reliability study had been exposed to TB patients. The subjects comprised nurses (50.9%), nurse assistants (20.4%), and technicians (14.8%). The contact workplace included inpatient departments (45%), intensive care units (23%), and laboratory units (19%). The demographic data are shown in Table 1.

Content Validity Assessment of MSA
As for personal factors, the items that had an I-CVI value of 0.86 were "bronchiectasis" and "BMI < 18.50 kg/m 2 " because one of seven experts evaluated these items as low-risk for aTB. By contrast, the rest of the personal factors had an I-ICVI value of 1 as all seven experts agreed that these carried a high risk of aTB. The S-CVI/Ave value for personal factors was 0.99.
The contact activity factor for MSA was divided into high risk for LTBI and low risk for LTBI. As to high risk, the items had an I-CVI value of 0.86, including "autopsy" and "carry sputum". As to low risk, the items had an I-CVI value of 0.86, including "cleaning the room", "bathing in bed", "transferring the patient", "vital sign assessment", and "talking to the patient". Thus, the S-CVI/Ave value for contact activity factors was 0.96.
Room size is part of the environmental factors. The priority levels for room size are small, medium, and large. The I-CVI value for a small room, which includes examination rooms, individual rooms, operative rooms, bronchoscopy rooms, and X-ray rooms, was 0.86. The I-CVI value for medium rooms, which include wards, emergency rooms, OPD stations, and recovery rooms, was 0.86. The remaining items, vis-à-vis room size, had an I-CVI value of 1 because the seven experts strongly agreed/agreed with the content. Thus, the S-CVI/Ave value for environment factors was 0.91. The resulting content validity assessment is presented in Table 2.  (1)

Discussion
The purpose of the current study is to develop the Modified Self-Assessment form, a tool for prioritizing HWs who have been in contact with TB patients. The MSA content was improved from PSA by adding items to the personal factors, contact activity factors, and environmental factors categories, which are risks for aTB and LTBI [14][15][16][17][18]. Content validity was then re-examined. The reliability of MSA was assessed by determining the Kappa agreement between MSA and the experts. Additionally, the reliability of PSA was assessed in order to compare MSA and PSA.

Validity Assessment
The content validity of MSA passed the I-CVI and S-CVI/Ave criteria. To pass the I-CVI criteria (≥0.83), an item should have agreement from at least 6 experts, resulting in an I-CVI value of 0.86. The items in the three factors of MSA comprise both clinical and epidemiological content. Seven experts were selected because they were clinicians (a chest specialist, three occupational medicine specialists, and three epidemiologists). The differences in expert subject areas might account for disagreements on some items. For example, regarding contact activity factors, one of the epidemiologists disagreed that "bathing in bed" and "talking to the patient" were low-risk for LTBI because HWs might come into contact with TB through aerosol droplets in the air or on surfaces during these activities. By contrast, clinicians suggested that these items were low-risk because they are not aerosol-generating procedures (AGPs).
The literature rates the most common risks for aTB as those related to personal factors [14][15][16][17][18]. As such, the resulting content validity confirmed that personal factors had the highest number of single points of I-CVI, contributing to the highest S-CVI/Ave. On the other hand, according to the experts, contact activity factors and environmental factors had a lower S-CVI/Ave value than personal factors.

Discussion of Contact Priority Results
We conducted a reliability assessment of the application of MSA to HWs who have come in contact with TB patients. Agreement between MSA and the experts was assessed because MSA represents the experts who make decisions on contact priority based on the contact history of the subjects. MSA had strong agreement with the contact priority of the experts (Kappa 0.80 (95% CI: 0.70, 0.90)). MSA could, thus, be applied to prioritize HWs' contact exposure to TB. By contrast, the agreement between PSA and the experts was weak (Kappa 0.58 (95% CI: 0.44, 0.72))-PSA should not, therefore, be used to prioritize contact [20].

Discussion of Personal Factor Results
Personal factors can also be used to evaluate the fitness for work of HWs who have come into contact with TB. The respective Kappa agreement for personal factors for MSA was 0.87 (95% CI: 0.77, 0.96), 0.64 (95% CI: 0.37, 0.91), and 1 for underlying diseases, health status, and medication. The results reveal that personal factors have a high Kappa agreement because these are the subject's individual data, which would not be affected by recall bias. The Kappa agreement for underlying disease for MSA was higher than that for PSA (Kappa 0.87 (95% CI: 0.77, 0.96) vs. Kappa 0.59 (95% CI: 0.43, 0.76)) because MSA has more items covering personal factors than PSA. While PSA could not detect a subject at risk of aTB, MSA and the experts could. Moreover, PSA did not include items for health status and medication in the personal factors category, while MSA did.

Discussion of Contact Activity Factor Results
Contact activity results in LTBI. AGPs play a major role in respiratory infection [21,22]. Chanpho et al. [23] postulated that the HWs who perform AGPs had a significant association with LTBI (OR = 2.04, with 95% CI: 1.20, 3.48, p = 0.007). Although the result of contact activity factors had a Kappa agreement for MSA than PSA (Kappa 0.75 vs. Kappa 0.84), both MSA and PSA passed the agreement criteria (Kappa ≥ 0.60). PSA has five contact activity items (i.e., bronchoscopy, ET-tube intubation, suction, CPR, and sputum induction), while MSA has 22. The most common contact activity in the current study was suction, which was present in both MSA and PSA. Agreement in MSA would be raised if the number of subjects was to be increased. Based on the expert interviews, it was found that most of the subjects were not aware of the risk of LTBI when performing AGPs. It was also noted that when subjects were completing the MSA form, the risks for communication and awareness of LTBI were confirmed as factors, whereas in PSA, they were not.

Discussion of Environment Factor Results
Environmental factors had the lowest Kappa agreement, especially for contact duration. MSA had a weak agreement for contact duration (Kappa 0.50) and minimal agreement for PSA (Kappa 0.23). The result can be explained in two ways. First, the subjects could not recall the contact duration. Second, the six-level priority of contact duration might result in low agreement. Notwithstanding, the contact duration agreement for MSA was better than PSA due to differences between the MSA and PSA forms: subjects had to complete real contact duration for MSA, while PSA was more general and less accurate (Tables A1 and A2 of Appendix A).
Even though less than 12 air ventilation changes per hour is a risk for LTBI [8,24], HWs were unable to comment on ventilation during their contact activity. Studies have shown that room size can predict the risk of LTBI [12,25,26]. A room size of less than 20 m 2 is a factor associated with LTBI for HWs. Therefore, room size is an environmental factor in the current study. As for the Kappa agreement with respect to room size, MSA was better than PSA (0.70 vs. 0.56), probably because MSA had more items on room size than PSA.

Application
Job title and workplace are risk factors for TB infection. Previous studies have shown that the most commonly infected persons were nurses, physicians, nurse assistants, and medical technicians because of frequent contact with TB patients and the performance of AGPs [14,15,27,28]. The current study did not include physicians because of their recall bias. By contrast, nurses, nurse assistants, and technicians have assigned workplaces and could recall their in-place contact history more clearly. Using MSA may not be appropriate for jobs that have highly variable workplaces, as with physicians.
The hierarchy of hazard control is the key to successfully controlling TB infection among HWs [4]. Reduction of sources of infection is the best way to control exposure, for example, limiting the number of TB patients, early diagnosis, the effectiveness of TB treatment, and decreasing the duration of hospital stay [8]. AGPs should also be better controlled, i.e., by using in-line suction, site ventilation control, and warning signs. Environmental controls should be used when appropriate, e.g., room ventilation, airflow management, availability of negative pressure rooms, ultraviolet germicidal irradiation, and disinfectant [29]. Finally, personal protective equipment should be adequate for HWs. All HWs should be evaluated as fit-for-duty before working with TB patients. Additionally, they should be trained so as to have adequate knowledge for best safe work practices.

Limitation and Suggestion
The information from the participants might not be factual. For example, if the participants informed the experts (or MSA) of a history of diabetes mullites, this study did not test their fasting blood sugar to prove if they had diabetes mullites. This study was conducted in one hospital in Thailand. A limited number of participants might result in a wide range of 95% CI of Kappa. MSA should be tested in various hospitals with a large number of participants. Recall bias is inevitable in such a study as ours, especially vis-à-vis environmental factors. In order to reduce recall bias, environmental factors should be obtained from official hospital records instead of memory. Additionally, we should address the contact priority of MSA, as compared to the LTBI test, in order to evaluate the usefulness of MSA.

Conclusions
The MSA form has improved the content for prioritizing HWs after contact with TB. The content of MSA comprises three categories associated with LTBI and aTB (viz., personal factors, contact activity factors, and environmental factors). The content validity of MSA was affirmed by seven experts. The reliability of MSA was confirmed using the respective agreement between PSA and MSA and the experts. Agreement of contact priority between MSA and the experts was strong (Kappa = 0.80 (95% CI: 0.70, 0.90)), whereas agreement between PSA and the experts was weak (Kappa = 0.58 (95% CI: 0.44, 0.72)). MSA is, thus, a tool that can be used for prioritizing HWs after contact with TB. MSA can also be used for risk communication to increase awareness of AGPs. Additionally, MSA can be used as a prototype for TB contact priority in high-burden countries, where there is limited LTBI screening. Further studies should be conducted in the other hospitals, and the number of participants should be increased in order to come to a concrete result.

Informed Consent Statement:
Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the patient(s) to publish this paper.

Acknowledgments:
The authors thank (1) the health personnel who participated in the study and (2) Bryan Roderick Hamman for assistance with the English-language presentation of the manuscript.

Conflicts of Interest:
The authors declare no conflict of interest.   Autopsy pulmonary TB patient 1 2 3 4 5 6 7

Environment factors
These rooms are of small size, which carries a high risk of LTBI.

Expert T I High
Medium Low