Student and Tutor Satisfaction with Problem-Based Learning in Azerbaijan

This article examines tutors’ and students’ satisfaction with the implementation of problem-based learning (PBL) at the Azerbaijan University of Architecture and Construction. A total of 28 pilot academic staff members and their students participated in PBL during one semester and received a questionnaire about their experiences at the end of the semester. In total, 649 students were involved in the intervention. Descriptive statistics and factor analyses were used to analyze the data. In total, the response rate among students was 61.7%, and 82.1% among tutors. More than 83% of the students thought that the PBL should be kept as part of the module. A total of 91.3% of tutors agreed that PBL is a great tool for student learning. According to the factor analysis, tutors believed that PBL can develop students’ ability for group/team work. Tutors also identified some barriers in applying PBL. For example, they mentioned a lack of relevant skills to apply PBL in higher education. Both students and tutors found the PBL to be a suitable learning tool for their curriculum.


Introduction
PBL can help to change teaching from simply reproducing what was learned to critical thinking and self-development. During the 1970s, PBL was viewed as a pedagogical innovation [1]. Three variants of PBL are distinguished in the literature: the Canadian (Mc-Master), the Dutch (Maastricht), and the Danish approaches [2]. The PBL-based curriculum was first launched at McMaster University in Hamilton in 1969. It attracted international attention, especially from the newly established university in Maastricht in the Netherlands. In parallel to the development of PBL at McMaster and Maastricht, a version of PBL was developed in Denmark. The PBL approach in Denmark was specific since it was the first PBL application in engineering education at the University of Alborg in Denmark. In the Danish and Maastricht models, students first analyze and define the problem within a given domain or interdisciplinary context. A focus on the community is typical in the Danish model. Although there are some minor differences between the three universities, they all emphasize three common elements: group work (teamwork), problem-oriented learning, and a community-oriented attitude. The Danish approach differs because of its emphasis on an interdisciplinary approach [3]. It means that the problem (task) is not analyzed from the view of one disciple (subject) only. In the Danish and Maastricht models, students first analyze and define the problem within a given domain or interdisciplinary context. For example, in the Danish model (and in some studies at Maastricht University also), the degree of self-direction of students is higher, and there is a greater focus on skills (e.g., planning, monitoring) and the application of knowledge to real-life situations. In general, PBL supports students to obtain academic knowledge through real life cases. At the same time, it introduces them to research, on how to use and understand research findings, but also on how to understand and use data [4].
The Azerbaijan University of Architecture and Construction is one of the leading state universities in Azerbaijan. It started the Bologna Process in 2005 [5]. Before that, education at the university was conducted according to the traditional Soviet teaching system. This system, inherited from the ex-Soviet Union, had several typical features such as an emphasis on academic lectures in auditorium, up to 10 non-core subjects, significantly more hours of lectures than practical/laboratory/project lessons, the absence of internship programs, the absence of lecturers with a practical background, a short period of time before exams, lack of departments/centers for career planning, an education process that is immune from developments in business and economics, and the absence of a system of credit points [5]. One of the main problems was a huge gap between theoretical knowledge and real practical competencies and skills, which are necessary for occupation and career development. A typical feature was the strong theoretical emphasis and little attention being paid to practical application and to the acquisition of skills. A major reason for this was the lack of laboratories and techno centers for teaching practical skills. Furthermore, there was an absence of university-industry collaboration and a lack of interest in businesses and entrepreneurial activities among university faculty and management. Although universities in the former Soviet states have recognized the need for change in the educational process, the implementation of the Bologna Process was not sufficiently successful as most decisions to change the system faced huge barriers in implementation and were opposed and blocked by educational traditions [5]. The higher education system in Azerbaijan does not seem to be able to prepare students well for the labor market. One of the main indicators demonstrating this is the high unemployment rate of university graduates. The number of unemployed people who have completed secondary education is almost equal to the number of unemployed people with a higher education degree [6]. Thus, a university level education and a recognized diploma do not provide many benefits. Higher education appears not to provide the necessary skills for the labor market or to prepare graduates for start-up firms and entrepreneurial activities [7]. As indicated above, there is a necessity to change and modernize the content and format of the educational process in Azerbaijan. One way of doing this is by applying more modern educational approaches such as PBL [8,9].
Moreover, the level of satisfaction with the current higher education system is low. Therefore, it was decided that the PBL approach would be introduced in "pilot" groups of students. A pilot group of tutors agreed to support this. PBL provides students not only with theoretical knowledge but also with practical skills. It is built on an education process that is more democratic than the classic Soviet traditional way of education [10,11]. PBL helps students to develop soft skills by activating learning and discussing problems and working together in tutorial groups.
There are some difficulties in applying PBL in post-Soviet countries, as for example the fact that university-industry collaboration is not part of the university culture. This limits the ability of tutors to develop real-life cases based on the state of the art in the industry. Furthermore, reporting findings from the literature and an analytical approach to the literature are not part of the higher education curriculum. The number of hours spent on a subject/module depends on available staff. Teaching is mainly based on lectures, which only consist of lecturers explaining theory. Practically oriented lessons based on laboratory experiments, project-based lessons with research and analytical work, using real problems and cases, are only a minor part of the curriculum. Interactive lessons are not accepted as traditional lectures where the lecturer addresses the students. The average age of lecturers is around 65 years. This poses difficulties in the application of more modern teaching methods, as these frequently require change, adaptation, and special training, as well as a change in culture for tutors and lecturers.
To the best of our knowledge, there are no studies on the application of PBL in countries such as Azerbaijan that so heavily depend on traditional forms of teaching in higher education. To modernize higher education in Azerbaijan, we chose a selected group of academic staff members of AzUAC (14.02.2.19, www.azmiu.edu.az, accessed on 4 April 2020) to participate in a pilot application of PBL during the second semester of the 2020 academic year. The group was selected according to the interest they had shown in innovation in teaching in their previous work. Most of them were already involved in projects supported by Europe (such as Erasmus, Tempus, Backis), government financed education/scientific grants, or university internal activities/projects. They already had experience with different didactic approaches and motivation to participate in this pilot application of PBL.
First, a paper-based survey about the current knowledge of tutors was conducted to understand the current situation. In this paper, we do not report results from this initial survey. Second, during an info day, the pilot tutors of AzUAC were familiarized with the main concept of PBL in teaching by presentations based on Maastricht University (www.maastrichtuniversity.nl/, accessed on 4 April 2020) materials about PBL. At the same time, incentives such as financial support for personal development were suggested for the best tutors in the group. This was done to motivate staff members. Third, the next winter semester was selected as a probation period where the PBL intervention was applied. After the info day, three two-day-long special workshops were organized for tutors during the winter semester to support the application of PBL. During these events and sessions, practical tips and cases were provided and discussed [12].
The PBL group of academic staff members, consisting of 26 tutors, was formalized. A WhatsApp group was created to share experiences among the tutors during the experimental semester and to encourage each other. Therein, tutors were enabled to make short videos from their classes and shared these videos (students were informed and appropriate permission was received). At the same time, it helped to better manage and share information. After the pilot PBL semester, tutors filled in a questionnaire about their experiences with PBL.
Furthermore, at the beginning of the winter semester, students also received information about PBL. Tutors started to teach their courses by the PBL method. At the end of the course, students also received questionnaires about their experiences with PBL.
Because of the COVID-19 pandemic, lessons and some of the training sessions were conducted by Zoom, and because of some technical issues with distance learning, a few days were wasted. Furthermore, the pandemic and the complete lockdown influenced individuals and presented additional difficulties and opportunities in teaching.
The aim of this paper was to examine and to report on the experiences with PBL of tutors' and students' communities at Azerbaijan University of Architecture and Construction. For this purpose, we used the data from surveys collected in the spring semester of the 2019/2020 academic year (from February to May).
In this paper, we focus on satisfaction with PBL among students and tutors. The reason for doing so is that satisfaction is a prerequisite for an effective implementation of PBL [13]. A reform of the traditional curriculum is only possible if students and tutors support and are satisfied by the alternative PBL method. There have been a number of previous studies on the satisfaction with PBL among students and tutors. Student satisfaction with PBL is evidently positive [13]. Previous studies have found that PBL has positive effects on student performance and interest, exam scores, effort, the ability of working in groups, improvements in soft skills, and student satisfaction [13,14]. Athene complexity of the use of PBL in traditional teaching contexts has also been previously documented in the literature [7,15].

Methods
After the pilot semester with PBL, the surveys about students' and tutors' satisfaction were conducted. As PBL was newly introduced, it is relevant to know how satisfied students and tutors are with this, for them, new teaching method. Furthermore, the wider implementation of new teaching methods such as PBL is likely to be more successful and more supported if students and tutors are satisfied with it.
The surveys, on which findings we report here, were conducted in the last weeks of the semester, i.e., in May and June, among students and tutors. The design of the PBL intervention in AzUAC is described in Figures A1 and A2 in the Appendix A. descriptive statistics such as means, standard deviations, and minimum and maximum values for each question were calculated. The descriptive table were performed with the statistical software SPSS.
Student survey. In order to measure student satisfaction, we included in the questionnaire several questions on the satisfaction with the organization of the subject, the instructiveness of the subject, the link of the subject with your prior knowledge, the productivity of the tutorial group, the link between the subject and the assessment, the quality of tasks/problems/cases, and the quality of lectures. In total, the questionnaire consisted of 26 questions (see Supplementary Materials S#3). Similar questions were applied in the previous research (ref.). In total, the response rate was 61.7%, with 401 completed questionnaires. The survey was conducted in electronic form only. We were also interested in students' ideas and personal comments. Therefore, we included some open questions as well.
To analyze the data, Descriptive statistics were calculated, factor analysis was performed, and Cronbach's alphas were calculated to assess the reliability of the factors.
Tutor survey. The questionnaire for tutors was based upon an existing questionnaire [16] (Supplementary Materials S#4). In total, the survey consisted of 65 questions. We carried out a pilot survey among 5 randomly selected tutors before presenting the survey to all tutors. There was a pilot group of tutors who agreed to fill in the main questionnaire and to give their comments to improve the survey. Some corrections in the questionnaire were made according to the valuable comments of the tutors. By their suggestion, the survey was prepared in Google Forms. This enabled us share the link easily in the WhatsApp group and to export the data to SPSS.
We included questions on the satisfaction with the pedagogical method, supervising problem processing in tutorial groups, potential barriers to student learning in PBL, the tutor's role in the tutorial group, and the relationship between theory and practice in PBL.
We also included questions on socio-demographic characteristics such as gender, education, and number of years worked in academia. The response rate was 82.1%, with 23 completed questionnaires from tutors. The descriptive statistics consisted of frequency distributions and other descriptive characteristics for each question (Table 4). To measure the internal consistency, we applied factor analysis and the Cronbach's alpha (Tables 5 and 6). An open question was also included in the survey. We asked tutors 'What additional problems have you had?'

Student Survey
Below are the results of the descriptive statistics of the students' survey with the frequency distribution for each question. No one indicated that the organization or instructiveness was very poor. Students reported that PBL helped them to make a link between practical cases and theoretical materials. Moreover, more than 84% of respondents agreed that it supported them in group working.
Students in general said that PBL helped them to formulate clear learning goals themselves. Around 77% of students who participated in the intervention said that PBL lessons were well organized and effective. More than 83% of students who participated in the intervention said that PBL should be part of this subject/module. Almost 73% of students mentioned that this approach was helpful in understanding lectures. However, there were 11% of students think thought that tutors did not motivate them to summarize what they learned in their own words enough. Moreover, almost 9% of students believed that tutors (very) poorly helped them to apply what they had learned to the task, situation, or problem. As is evident from Table 1, the mean was around four for almost all questions, indicating that, on average, the 'good' answer was selected by respondents. The standard deviation was low and between 0.64-1.16, indicating that most responses were close to the average. Reliability statistics are shown in Table 2. The Cronbach's alpha was above 0.9, showing high internal consistency and a good reliability of the items included. The factor analysis in SPSS suggested initially that there were five underlying factors. However as three of them only had only one or two items that belonged to these factors, we decided for better interpretation to limit it to two factors. The results of the factor analysis of students' survey are shown below. Table 3 describes the factor analysis of students' main survey. The variance and component transformation matrix are also presented (Appendix A, Tables A1 and A2).  Factor 1 included questions Q16, Q17, Q18, Q19, Q20, Q21, Q22, Q23, Q25, while  factor 2 included Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q11, Q12, Q13, and Q15. Factor 1 refers to the quality of teaching and shows that students in general had a positive attitude towards PBL, while factor 2 refers to the learning outcomes. Thus, to the questions Q11 and Q12 on comprehension outcome, 66% and 76% of the students answered agree/fully agree, respectively. As for Q13, which supports their ability for evaluation, more than 81% of respondents thought that the tutor stimulated this. As for the questions, Q14 and Q15 showed that more than 75% of students agreed/fully agreed on the positive realization from the tutors' side as well. Thus, students evaluated learning outcomes in the 'PBL semester' as quite positive.
In the open question (Supplementary Materials S#3 question 11) 'Tips to improve this subject', students mentioned some barriers in their learning such as weak library facilities, absence of the possibility to see videos in the auditorium, lack of practical and case-related tasks in the syllabuses, lack of tradition to invite guest speakers from business or industry sectors, and issues in communication with the tutor because of the pandemic. As for the second open question, 'Tips to improve this subject for the tutor' (Supplementary Materials S#3 question 26), students advised lectures be more oriented to real problems. At the same time, they advised making all lessons (not only experimental ones) more interactive, providing more possibilities to work in tutorial groups, trying to make lessons fun and not boring, applying a more interactive way of teaching, and improving the means of explanation in distance mode.

Tutor Survey
Below are the results of the descriptive statistics of the staff survey. As is evident from the descriptive table, academic staff members agreed that PBL as a didactic approach is very useful. With regards to the supervising problem processing by tutors, the opinions were positive too. According to the tutors' opinion, PBL is based on real-life cases and helps to develop practical skills. The link between lectures and practically oriented lessons is more evident with PBL. Although all tutors agree that group discussion facilitates problem processing and helps students share experiences with each other, three tutors said that this might be stressful for students.
Tutors also identified some barriers in applying PBL. For example, they mentioned a lack of relevant skills to apply PBL in higher education. Moreover, they were motivated to obtain more detailed training in order to acquire more knowledge about PBL. Furthermore, 19 respondents out of 23 agreed that they need more time and resources in order to become familiar with PBL. They also expressed the need for more specific materials for teaching PBL. No one fully disagreed about this matter, and 19 agreed or fully agreed. Moreover, 20 tutors said they were interested in training, and only 2 were doubtful about this. All 23 tutors had plans to apply PBL in their future teaching. Furthermore, there were five individuals that doubted if they were really well prepared for the PBL semester. The majority of the tutors were sure that this method has many advantages compared with the traditional way of teaching. As is evident from Table 4, the mean was around three to four almost in all questions, indicating that on average the 'good' was selected among respondents. The standard deviation was low, indicating that most answers were close to the average.   Reliability statistics are shown in Table 5. As is clear from this table, Cronbach's Alpha was above 0.8 points. Thus, the questionnaire had a good reliability. The factor analysis for the tutors' survey suggested five factors, but as there were very few questions belonging to each factor, we decided to restrict it to only two factors. The results of the factor analysis of tutors' survey are shown below. Table 6 describes the factor analysis of tutors' main survey. Total variance and component transformation matrix are also shown below (Tables A1 and A2).  Factor 1 includes questions Q8-Q13, Q15, Q16, Q23, Q24, Q29, Q32, and Q38-Q41, while factor 2 includes Q14, Q17-Q22, Q26, Q27, Q31, Q33, Q34, Q35, Q37, and Q42. Factor 1 refers to tutors' satisfaction level with PBL, which is high. They believe that PBL can develop students' ability in group/team working. Factor 2 refers to tutors' perception of PBL as a factor to improve their skills during the educational process. This refers to learning outcomes and tutors' skills improvement during the educational process and supervising.
In the open question (Supplementary Materials S#4 question 16), tutors mentioned some difficulties and problems such as the absence of necessary equipment in the auditorium, lack of connection to worldwide well-known virtual platforms, and lack of video and presentations from previous tutors. Except for some technical issues, they also mentioned barriers such as problems with student engagement and motivation, as well as lack of student experience with PBL. At the same time, a few tutors mentioned the absence of their own experience with PBL and that it does not enable them to be sufficiently effective and professional. They felt a need to visit other universities where PBL is applied. Besides these, tutors noticed the additional challenges related to the pandemic.

Discussion
It is evident from the previous literature that the implementation of the Bologna Process was not successful as most efforts to modernize the curriculum and teaching methods faced huge obstacles and resistance because of educational traditions [17][18][19]. Classical traditional approaches in education have become rooted in the system, which made reforms difficult. The reason is that mainly fundamental/theoretical subjects and specialties are useful in a plan-oriented approach but are not in a market-oriented economy. To the best of our knowledge, few articles have addressed this topic [4]. An ageing academic staff, a non-competitive culture, and the unattractiveness of being employed in higher education for younger generations, among others, explain this reluctance to change. According to a study in Kyrgyzstan, whose historical and mental background is quite similar the lecture-centered learning, is still prevalent, and the 'jump' to a new system with a more modern approach is not easy for academic staff [20], especially if we take into account that the majority of staff including administrative staff also received their education from kindergarten up to university in the Soviet tradition. Walz et al. [20] describes numerous challenges in the post-Soviet country Kyrgyzstan and the difficulties with the modernization of higher education. They conclude that "Taking the first steps to explore and implement open pedagogy may be a challenge for instructors, particularly with its strong emphasis on student agency, active, engaged student learning, the instructor as a 'coach,' and interaction with curation and creating in the 'real world' rather than assignments that mainly the teacher sees".
During our PBL pilot semester, we also encountered a number of challenges. There were auditoriums with broken or absent necessary equipment, lack of internet connection or the inability to use well-known virtual platforms, and a lack of a video database with podcasts or multimedia files from previous tutors. Aside from these technical issues, there were other barriers such as problems with student engagement and motivation. In particular, this was evident during the distant learning phase because of the pandemic. At the same time, there was not enough knowledge and experience among students with PBL. There was an issue with a few tutors who clearly were not confident with PBL and who claimed that it did not enable them to be sufficiently effective and professional. They expressed a need to visit other universities where PBL is applied. Besides these issues, some tutors mentioned additional difficulties related to the unexpected fully distance mode because of COVID-19.
Reform of the curriculum in Azerbaijan is necessary as the current way of teaching does not properly prepare students for the needs of the labor market. Practical and applied subjects and specialties do not exist and are not developed. Thus, the curriculum is fully theoretical, and practical knowledge was very limited. Necessary competencies, skills, and learning outcomes are not part of the higher education system. Difficult and deep theoretical knowledge is the major pillar of high education [5]. Finally, the inheritance of the culture of socialism plays a role as well. As a result, Azerbaijan has a very traditional culture of teaching in higher education, one that is mainly oriented around the lecturer and focused on theoretical knowledge. However, many initiatives for renewal have been initiated in Azerbaijan.
The PBL pilot intervention that we conducted is one of these renewals. We found that according to students and tutors' opinions, this helped students develop necessary skills. Moreover, such initiatives trigger the process of modernization of higher education. At the same time, the application of PBL in engineering-based specialties helps both tutors and students to be more focused on real cases and the challenges of industry. Tutors' tips and creativity helps their colleagues from the department also. We experienced this during different sessions as other tutors participate at events through their colleagues. Tutors asked permission for participation of their colleagues as they were interested as well. The abovementioned results show that PBL has the potential to become accepted in Azerbaijan. This fact is important not only for this country, but for the region itself and other countries with traditional teaching methods. However, despite the problems and similarities in the outdated teaching methods in post-Soviet countries, students and staff [17,21] from countries with a similar background and history with a similar culture of curriculum seem to agree on the fruitfulness of the modernization of higher education. The authors mention the positive impact on learning effectiveness and motivation if students feel themselves leading in the process of their own education [17]. Moreover, this directly influences their self-development and ability to work in tutorial groups [21]. This approach supports students in increasing necessary competences and skills for the right 'track' in their future career development [18]. Another study in the Russian Federation also argued that the PBL method triggers students to learn new things more quickly; helps students to adapt to new real cases more easily; and helps them to work independently and to demonstrate skills in computer technologies and communication, teamwork, critical-thinking, creativity, etc. [19].
According to government programs of the Azerbaijan Republic, thousands of young people have the opportunity to study abroad. We expect that after their study and coming back to Azerbaijan and that their active involvements in the higher education process both as tutors or as administrative staff members will have its advantages. Curriculum development is ongoing in many universities of the country. Erasmus+ projects provide a positive impact as well. Government and business are also interested in internship programs and volunteering work. Furthermore, a young, modern, and open-minded generation outside academia will influence the academic community, as well as university top management. The same holds for new generations within the university.
Moreover, the market economy has a positive impact upon competition. Universities are interested in recruiting more students and to be attractive for the best students. Success in employability and career of graduates' influence on high interest from the enrollees' side directly. Thus, universities are motivated to modernize [5]. At the same time, applying PBL inadequately is full of risks [22,23]. This was shown by our findings as well. The unskillful application of PBL might confuse tutors and disrupt the learning process of students. There are studies that describe such negative outcomes [23] and show a low level of satisfaction with PBL because of a lack of preparation and a need for adequate regulation of didactical innovations.
This we recognize in our PBL pilot as well. Because of the lack of previous experience with PBL during education (primary and high schools are also traditional), there is a problem in understanding the basic pillars of PBL such as interactive learning grounded on evidence-based research. The focus on traditional education with its repetition of tutors' lecture and learning by heart theoretical information is questioned. Nevertheless, the positive thoughts and attitudes of both of students and tutors suggest that innovation in education should be continued and might over time become more fruitful.

Limitations
This study used a cross-sectional design. This implies some limitations; for example, it does not warrant causal inferences. Moreover, this study did not evaluate the effects of the application of PBL in AzUAC. The evaluation of the effectiveness of PBL needs additional study and analysis of learning outcomes. We leave it to further research to compare the outcomes of traditional teaching methods with outcomes of the PBL approach. This can be done by using experimental design in analyzing the learning outcomes.

Conclusions
During the one-semester-long intervention period, tutors were using PBL instead of the traditional way of teaching. We were interested in the satisfaction with the use of PBL. Tutors had many challenges to overcome, including a lack of experience in working with PBL, and this was exacerbated by the COVID-19 pandemic and the switch to online learning. In total, 424 (401 students and 23 tutors) people successfully finished the PBL semester and participated in the surveys. The majority of the students, 395 persons, expressed satisfaction with one of the main pillars of PBL, the tutorial groups. As for subject content, more than 44% of students thought that it was difficult or even too difficult, while 13% thought that it was too easy. Thus, around 50% of students were not quite satisfied with the subject content. This is a positive finding, taking into account the fact that there was no practical experience with PBL before from both sides of education and that the pilot tutors were front-runners among the staff community. It is also clear that more than 82% of the students thought that the PBL sessions should be kept as part of this module, while 4.7% of them disagreed with this. This is in accordance with tutors, among who 91.3% agreed that PBL is a great tool for student learning.