Validation of the EFFECT questionnaire for competence-based clinical teaching in residency training in Lithuania

Abstract

Background and aim: In 2013, all residency programs at the Lithuanian University of Health Sciences were renewed into a competency-based medical education curriculum. To assess the quality of clinical teaching in residency training, we chose the EFFECT (evaluation and feedback for effective clinical teaching) questionnaire designed and validated at the Radboud University Medical Centre in the Netherlands. The aim of this study was to validate the EFFECT questionnaire for quality assessment of clinical teaching in residency training. Materials and methods: The research was conducted as an online survey using the questionnaire containing 58 items in 7 domains. The questionnaire was double-translated into Lithuanian. It was sent to 182 residents of 7 residency programs (anesthesiology reanimathology, cardiology, dermatovenerology, emergency medicine, neurology, obstetrics and gynecology, physical medicine and rehabilitation). Overall, 333 questionnaires about 146 clinical teachers were filled in. To determine the item characteristics and internal consistency (Cronbach’s α), the item and reliability analyses were performed. Furthermore, confirmatory factor analysis (CFI) was performed using a model for maximum-likelihood estimation. Results: Cronbach’s α within different domains ranged between 0.91 and 0.97 and was comparable with the original version of the questionnaire. Confirmatory factor analysis demonstrated satisfactory model-fit with CFI of 0.841 and absolute model-fit RMSEA of 0.098. Conclusions: The results suggest that the Lithuanian version of the EFFECT maintains its original validity and may serve as a valid instrument for quality assessment of clinical teaching in competency-based residency training in Lithuania.

1. Introduction

For the delivery of high-quality patient care, high-quality clinical teaching of residents is essential [1,2]. Clinical teaching is accomplished in real-life situations through health care services for patients under strict control of a residents’ teacher. The quality of this process is crucially important to train young physicians who are able to provide evidence-based health care services, and to acquire necessary clinical skills, knowledge, and competencies [3,4,5].

Following the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG 2015), universities have to review their programs on a regular basis ensuring their compliance with international aims meeting learners’ and social needs, especially on quality assurance [6]. After the Lithuanian University of Health Sciences renewed its residency programs according to the methodologies based on the intended learning outcomes and competencies (CBME), urgent need emerged to implement a quality evaluation system for clinical teaching based on scientific evidence [7]. There are many instruments available for the evaluation of clinical teaching [5,8,9]. One of these instruments is the EFFECT questionnaire (evaluation and feedback for effective clinical teaching), a theory-based, reliable, and valid instrument designed and validated by Fluit et al. from the Radboudumc Health Academy in the Netherlands [9].

This study aimed at validating the EFFECT questionnaire for quality assessment of clinical teaching in residency training.

2. Materials and Methods

The EFFECT questionnaire is based on theories of workplace learning and clinical teaching, and incorporates the Canadian Medical Education directives for Specialists (CanMEDS) competences [9]. The authors have validated the questionnaire following five sources of validity described by Downing [10,11]. Although EFFECT relies on an international literature study and is based on the theory that is internationally recognized as highly relevant to medical education, the authors claim the caution that is warranted in extrapolating their findings to other countries with different residency training programs and different feedback cultures as one of its possible limitations [12]. The aim of our study was to assess the validity of the Lithuanian version of EFFECT.

The EFFECT questionnaire consists of 58 items in 7 domains of clinical teaching: role modeling, task allocation, planning, providing feedback, teaching methodology, assessment, and personal support. The role modeling domain contains 4 subdomains: clinical skills, scholarship, general competencies, and professionalism. The items can be scored using a 6-point Likert scale (1, very poor; 2, poor; 3, intermediate; 4, satisfactory; 5, good; 6, excellent; and “not (yet) able to evaluate”). The option “not (yet) able to evaluate” was chosen if a specific item did not (yet) occur during clinical teaching. Having obtained the authors’ agreement to use the questionnaire, we made its double translation from Dutch to Lithuanian by 2 professional translators. In addition to the original items, information on gender, residency program, and the year of training was included.

To determine item characteristics, item means and standard deviation were calculated. For the assessment of internal consistency and reliability, the Cronbach’s alpha was calculated. Finally, structural equation modeling was applied to determine the amount of interdependency between items and constructs using the existing factorial solution as a model for maximum-likelihood estimation. In addition, common incremental measures of the scale fit in structural and equation modeling – the Comparative Fit Index (CFI) and Root Mean Square Error of Approximation (RMSEA) – were calculated [13,14]. Correlations between the dimensions were determined by correlation coefficients from the estimated covariance matrix. Correlation coefficients with a magnitude of 0.7–1.0 indicated interdependency of the factors. All the calculations were run with SPSS 20 and AMOS 20.

The study was approved by Bioethics Centre of the Lithuanian University of Health Sciences. The study was performed as an online survey.

3. Results

The survey data were collected during 2015–2016. A total of 182 residents (48 men and 134 women) were asked to fill in the EFFECT questionnaire about the teachers who were their supervisors within a residency program. The residents could decide how many teachers they wanted to evaluate not necessarily filling in the questionnaire for every teacher they worked with. We received a total of 333 questionnaires: 67.9% (n = 226) were completed by women and 32.1% (n = 107) by men. Description of the study population and the number of questionnaires filled in per residency program are presented in Table 1.

Table 1. Description of the study population and number of questionnaires filled-in per residency program.

Residency program	No. of residents in the program	No. of teachers in the program	No. of teachers evaluated in the program	No. of questionnaires filled-in within the program
Anesthesiology reanimathology	53	70	21	57
Cardiology	30	53	22	45
Dermatovenerology	19	10	8	37
Emergency medicine	17	74	56	115
Neurology	11	22	12	20
Obstetrics and gynecology	34	37	16	30
Physical medicine and rehabilitation	18	18	11	29
Total	182	284	146	333

The largest proportion (36.9%) of the questionnaires was filled in by first-year residents, followed by third-year (25.2%), second-year (24.3%), and fourth-year (13.5%) residents.

The results of the item characteristics are provided in Table 2. The items were rated on a 6-point Likert scale. The mean scores ranged from 4.58 (item 29, “reminds me of previously given feedback”, and item 50, “helps and advises me on how to maintain a good work-home balance”) up to 5.40 (the item 9, “applies to guidelines and protocols”). More than 20% of the answers in item 12 “have a bad news conversation”, item 40 “reviews my reports”, item 50 “helps and advises me on how to maintain a good work-home balance” were scored as “not (yet) able to evaluate”, while this proportion was over 70% for all the assessment domain items (51–58). Factor loadings varied from 0.788 (item 30) to 0.957 (item 74).

Table 2. Item characteristics.

Domain	Mean	SD	Factor loading	NAE, n	NAE, %
Role modeling
Role modeling clinical skills
1. Ask for a patient history	5.21	1.029	0.900	31	9.3c
2. Examine a patient	5.23	1.027	0.932	31	9.3
3. Perform clinical actions	5.31	.981	0.878	21	6.3
Role modeling general CanMEDS roles
4. Cooperate with other health professionals while providing care to patients and relatives	5.25	1.047	0.855	9	2.7
5. Communicate with patients	5.20	1.110	0.923	5	1.5
6. Cooperate with colleagues	5.02	1.249	0.850	1	.3
7. Organize his or hers own work adequate	5.20	1.079	0.795	5	1.5
8. Apply guidelines and protocols	5.40	.990	0.804	6	1.8
9. Treat patients respectfully	5.31	1.080	0.869	5	1.5
10. Handle complaints and incidents	5.24	1.185	0.850	63	18.9
11. Have a bad news conversation	5.26	1.071	0.860	77	23.1
Role modeling scholarship
12. Apply academic research results	5.28	1.054	–	12	3.6
Role modeling professionalism
13. Indicates when he/she himself/herself does not know something	4.92	1.287	0.834	18	5.4
14. Reflects on his/her own actions	4.99	1.273	0.887	6	1.8
15. Is a leading example of how I want to perform as a specialist	5.13	1.291	0.937	4	1.2
Task Allocation
16. Gives me enough freedom to perform tasks on my own that suit my current knowledge/skills	5.25	1.056	0.856	4	1.2
17. Gives me tasks that suit my current level of training	5.20	1.110	0.856	3	.9
18. Stimulates me to take responsibility	5.22	1.021	0.805	7	2.1
19. Gives me the opportunity to discuss mistakes and incidents	4.99	1.372	0.884	18	5.4
20. Teaches me how to organize and plan my work	4.82	1.321	0.861	26	7.8
21. Prevents me from having to perform too many tasks irrelevant to my learning	4.73	1.483	0.835	28	8.4
Planning
22. Reserves time to supervise/counsel me	4.94	1.309	0.932	5	1.5
23. Is available when I need him/her during my shift	5.08	1.353	0.910	59	17.7
24. Sets aside time when I need him/her	5.11	1.177	0.942	13	3.9
Giving feedback
Quality of the feedback
25. Bases feedback on concrete observations of my actions	4.91	1.233	0.883	33	9.9
26. Indicates what I am doing correctly	4.92	1.305	0.904	14	4.2
27. Discusses what I can improve	4.76	1.255	0.915	22	6.6
28. Allows me think about strengths and weaknesses	4.72	1.307	0.914	35	10.5
29. Reminds me of previously given feedback	4.58	1.368	0.881	65	19.5
30. Formulates feedback in a way that is not condescending or insulting	4.95	1.402	0.788	28	8.4
Content of the feedback
31. My clinical and technical skills	5.06	1.131	0.916	42	12.6
32. How I monitor the boundaries of my clinical work	4.97	1.204	0.942	53	15.9
33. How I collaborate with my colleagues in patient care	4.98	1.121	0.948	56	16.8
34. How I apply evidence-based medicine to my daily work	4.99	1.151	0.922	46	13.8
35. How I make ethical considerations explicit	4.98	1.183	0.942	55	16.5
36. How I communicate with patients	5.00	1.167	0.954	54	16.2
Teaching abilities (methodology)
37. Reviews the learning objectives	4.69	1.331	0.802	49	14.7
38. Asks me to explain my choice for a particular approach (status, resign form, etc.).	4.99	1.111	0.806	43	12.9
39. Discusses the possible clinical courses and/or complications	5.06	1.060	0.855	26	7.8
40. Reviews my reports	4.93	1.141	0.795	75	22.5
41. Stimulates me to find out things for myself	5.09	1.081	0.810	17	5.1
42. Stimulates me to ask questions	5.24	1.201	0.856	3	0.9
43. Stimulates me to actively participate in discussions	5.02	1.282	0.908	22	6.6
44. Explains complex medical issues clearly	5.10	1.205	0.890	17	5.1
Personal support
45. Treats me respectfully	5.20	1.243	0.890	1	.3
46. Is an enthusiastic instructor/supervisor	5.16	1.147	0.838	1	.3
47. Allows me know I can count on him/her	5.05	1.347	0.957	2	.6
48. Supports me in difficult situations (e.g. during morning report)	5.03	1.412	0.945	24	7.2
49. Is open to personal questions/problems	4.95	1.402	0.917	47	14.1
50. Helps and advises me on how to maintain a good work-home balance	4.58	1.617	0.889	96	28.7
Assessment
51. Prepares progress reviews	4.89	1.036	–	251	75.1
52. Makes a clear link with previously set learning objectives during these reviews	5.06	1.021	–	246	73.7
53. Gives me the opportunity to raise issues of my own	5.35	1.124	–	240	71.9
54. Formulates next-term learning objectives during these reviews with me	5.13	1.124	–	243	72.8
55. Explains how staff was involved in the assessment	4.89	1.255	–	253	75.7
56. Reviews my portfolio during the assessment	5.07	1.153	–	246	73.3
57. Pays attention to my self-reflection	4.97	1.276	–	244	73.1
58. Gives a clear and exhaustive assessment	5.16	1.209	–	241	72.2
Mean scores (scale 1 = very unsatisfactory, 6 = excellent) with corresponding standard deviation and factor loadings per item. Columns 4 and 5 contain frequencies (number of questionnaires and percentage) per item that residents indicated “not (yet) able to evaluate” (NAE) of the EFFECT questionnaire. For deleted items no factor loadings were calculated.

The Cronbach’s alpha coefficients ranged from 0.91 to 0.97 indicating a high internal consistency of all subdomains (Table 3).

Table 3. Cronbach’s alpha of the domains.

Domain	No. of items	Cronbach’s alpha
Role modeling clinical skills	3	0.93
Role modeling CanMEDS roles	8	0.95
Role modeling scholarship	1	–
Role modeling professionalism	3	0.91
Task allocation	6	0.94
Planning	3	0.94
Quality of the feedback	6	0.95
Content of the feedback	6	0.97
Teaching methodology	8	0.95
Personal support	6	0.96
Assessment	8	0.97

The “role modeling scholarship” subdomain was not included into the confirmatory factorial analysis as it has only one item – “apply academic research results.” The items of the “assessment” domain were not included due to high proportion of “not (yet) able to evaluate” answers. Therefore, only 9 subdomains of the questionnaire were used in analysis.

The examination of the factorial structure of the questionnaire using confirmatory factor analysis resulted in a satisfactory model-fit. The comparative fit index CFI (0.841) means reached the area of a permissible model-fit. The absolute model-fit RMSEA of 0.098 revealed a moderate matching of the postulated factorial structure with the empirical data. The correlations between the factors varied from 0.699 to 0.916 (Table 4).

Table 4. Correlations between factors of the questionnaire.

Factors	1	2	3	4	5	6	7	8	9
1	1.0
2	0.894	1.0
3	0.886	0.916	1.0
4	0.715	0.813	0.870	1.0
5	0.750	0.829	0.902	0.909	1.0
6	0.677	0.764	0.795	0.856	0.819	1.0
7	0.724	0.791	0.774	0.813	0.760	0.904	1.0
8	0.798	0.850	0.908	0.900	0.892	0.882	0.857	1.0
9	0.708	0.818	0.881	0.877	0.915	0.789	0.755	0.890	1.0
1, role modeling clinical skills; 2, role modeling general CanMEDS roles; 3, role modeling professionalism; 4, task allocation; 5, planning; 6, feedback (quality); 7, feedback (content); 8, teaching methodology; 9, personal support.

Confirmatory factor analysis demonstrated that the 9 subscales of the EFFECT Lithuanian version corresponded to 9 different factors, which strongly correlated between themselves.

4. Discussion

The results of this study indicate that the Lithuanian version of the EFFECT questionnaire has acceptable psychometric properties and can be used for the evaluation of clinical teaching within residency training.

All the domains demonstrated a satisfactory reliability coefficient. However, residents indicated that part of the items could not be judged. This especially holds true for the “assessment” domain and some items of the role modeling domain (item 11 and 12). As portfolio assessment is not implemented at the moment, this explains why residents could not fill in these items.

For the “role modeling” items 11 and 12, it is possible that residents do not have much possibilities to observe their supervisors, for instance, when they bring bad news to their patients, or handle complaints and incidents. The same results were also found in the original studies conducted in the Netherlands, it could mean that residents just do these complex tasks, without having good examples in mind [9].

Compared with the original research, there are some differences in our survey that could influence the results [9].

First, the present replication study received fewer questionnaires (333 vs. 407). Nevertheless, the sample size exceeded the lower bound of 5 residents per item and, therefore, was regarded as big enough for a confirmatory factor analysis [15,16].

The original sample recruited its participants based on a department where they at the moment of the survey. In total, 6 departments in 4 different hospitals representing 3 specialties (pediatrics, pulmonary diseases, and surgery) were involved. This approach allowed the residents to evaluate all teachers of a specific department, thereby minimizing memory bias. The residents of our study were recruited based on the residency program, i.e., anesthesiology reanimathology, cardiology, dermatovenerology, emergency medicine, etc. They possibly had to evaluate clinical teachers from different clinical departments whom they met at a different timetable; therefore, this could result in memory bias due to different time of their encounter.

Another difference was the smaller number of domains included into confirmatory factorial analysis. As mentioned in Section 3, the “role modeling scholarship” domain was not included into confirmatory factorial analysis as it has only 1 item. However, we decided to leave it within the questionnaire as application of research results in training is one of the most important requirements [17].

The limitation of our study is related to the exclusion of the “assessment” domain from confirmatory factorial analysis (due to high proportion of “not (yet) able to evaluate” answer, ranging from 71.9% to 75.7% for different items) as it did not allow us to validate the full factorial structure of the EFFECT questionnaire. However, we need to keep this domain within the questionnaire as this is the weakest part of current clinical teaching, which has to be improved. It should be taken into consideration that CBME requires multifaceted assessment that embraces formative and summative approaches [18], the processes which are continuous and frequent [19]. It should also be noted that systematic training of clinical teachers on how to supervise residents in a competency-based curriculum has been started at the Lithuanian University of Health Sciences only recently. Therefore, in order to assess quality of clinical teaching using EFFECT, we will need to revalidate the questionnaire in its full structure once the “assessment” domain will become the daily practice in residency studies.

5. Conclusions

The results of this study indicate that the Lithuanian version of EFFECT has acceptable psychometric properties for evaluation of clinical teachers. Further research should be undertaken to examine the full factorial structure of EFFECT.