An Assessment Tool Predicts Learning Effectiveness for Project-Based Learning in Enhancing Education of Sustainability

Abstract

Project-based learning (PBL) has been widely applied to stimulate learning motivation, cultivate implementation capability, and improve learning effectiveness. The purpose of this study was to establish an assessment tool for the learning effectiveness of PBL in scientific and technological education. The methods include literature review, focus groups, and survey research. After the literature review, a draft of the scale was established, and the content validity of the scale was verified by 8 experts before being pre-tested. A total of 232 copies of valid official scales were issued and recovered. The research obtained produced subscales for “Flow experience perception”, “Self-efficacy”, “Product evaluation” and “Learning motivation”, which can be used to evaluate the learning effectiveness of the PBL in curricula of science and technology.

1. Introduction

It is one of the important topics to construct “Apply what you’ve learned” from the curricula of science and technology in higher education [1]. Project-based learning (PBL) is considered as an important way to effectively develop university students’ integration and application abilities in the field of industrial education [2,3]. Many studies have showed that the use of PBL can indeed cultivate independent thinking, problem discovery and resolution, multiple ways of thinking, and meaningful learning among university students [4,5,6,7]. The effectiveness of PBL not only improves students’ problem-solving ability, but also enhances the motivation and attitude of their learning [8], thus enhancing their employability [1]. Warin et al. [9] point out that PBL has several characteristics, which are usually multidisciplinary and possibly long-term effect (i.e., in weeks or months). To create a new product or performance, participate in a real task, and have general routines followed by flexible steps, are all factors that are integrated in project-based application curricula.

In the past, research on this topic and PBL focused on the two main aspects of the capstone course [10,11] and the capstone project [12]. The former of the capstone course is the final and peak learning experience of university education. The former method enables students to integrate and deepen what they have learned in the university, so that learning can be completed steadily. It is a very important experience for senior students [10]. The latter capstone project is also known as the capstone experience, the culminating project, and the senior exhibition, which can be regarded as a multifaceted assignment designed for those students who have high academic and intellectual skill; this usually is implemented in the last year of their high school or junior high school, or at the end of an academic course or learning pathway [12]. For example, ChanLin [13] applies technology to PBL in an integrated manner and teaches students to use various Web material sources, and the use of different software to record, analyze and present information. Li [14] takes the teaching experiment method, carrying on four years of project-based design and experiment-oriented curricula to verify the effectiveness of the project making curricula.

The definition of project-based “Learning” (PBL) in this study is based on the philosophy of constructivism in order to eliminate the phenomenon of knowledge fossilization after learning. The item of project making is to lay emphasis on the writing of application software systems of solar cell modules, photovoltaic technologies, Internet of Things (IoT), Information Technology (IT), and so on. The definition of the project-based “Curricula” is that the curricula design is to arrange complex and real tasks as the core of the project. It focuses on the integration of knowledge learning in different technology areas. Learners need to go through a series of exploratory actions and situations of cooperative learning, in order to learn the knowledge of problem solving [15], and skills of flexible use for the knowledge. The implementation of the curricula has the characteristics of teachers’ interdepartmental support and introduction of enterprise teachers’ guidance. Among the learning units in the curricula, emphasis is placed on working in tandem with the project-making of technological applications and providing structured features with highly transverse linkage and longitudinal integration for enhancing education of sustainability. The structure of the paper begins with the learning theory of “Learning by doing” to explore the assessment tool of learning effectiveness based on project-based curricula. Finally, the feasibility of the tool was confirmed by the study through empirical survey.

2. Literature Review

2.1. Learning Theory

Project-based curricula planning and implementation is based on the theory of “Learning by doing” put forward by John Dewey. Dewey was a pragmatist whose central idea was that the truth of theory, the justice of action, or the value of a common practice and custom are all determined by its effectiveness. He emphasized durability, portability, and practicality [16]. “Learning by doing” is one of Dewey’s most important educational ideas, which can be regarded as the product of empiricism, behaviorism and progress theory. In other words, creating sufficient conditions for learners to “Experience” is the key to education [16]. The significance of Dewey’s experience is that it contains the dual semantics of both the content and result of experience (i.e., what men do) and the activity and process of experience (i.e., how men act) [17]. Dewey’s experience-based philosophy emphasizes not only “Something to do”, but also “Something to learn”. He advocated letting learners in the experiential learning situation realize” Learning by doing”, through the learners self-reflective thinking to understand the relationship between things, so that the learning becomes “A matter of course” [17].

Dewey advocated that the crucial premise of education should be closely related to the learner. He believed that the approach to truth is through observation and experimentation and that knowledge should be the product from learning by doing. Unless it is generated from the experiment results, there is no true knowledge and understanding of things [18]. Dewey also believed that through the process of reflective thinking, learners will be able to transform the original experience, without introspection, into a deeper experience of self-examination. The learned meaning of this kind of autonomous learning is formed by the students themselves from the familiar experience of the individual, rather than acquired by the teachers’ extrinsic indoctrination, while the experience gained by the students’ own learning will internalize the learning result, and be profound and long lasting. Therefore, it is vital to provide the necessary auxiliary environment for learners to promote their learning experience.

2.2. Evaluation of the Learning Effectiveness of Project-Based Curricula

PBL, an effective way of learning, can be described as follows [2,7,19]:

(1)

To integrate different disciplines through the special project of knowledge and skills;

(2)

To arrange complicated homework, design situations that can promote learning motivation, develop individual meta-cognitive strategies, and utilize cooperative learning, so that learners can not only acquire knowledge, but also skills to solve problems;

(3)

To learn how to apply knowledge and solve the problems.

The adoption of PBL enables the implementation of project-making curricula to be realized with the learning effect of integration and application. Students will be able to improve their learning effectiveness of professional skills in their own professional course.

The evaluation of the effectiveness of PBL is usually based on formative and summative evaluations. For example, the research of Chien and Liu [2] in quantitative data collection is to use learning motivation scales, game-making flow meters, and Game Maker learning self-efficacy scales. In the collection of qualitative data, the researchers’ observation record, teaching interviews, digital game project production and learning process files, etc. are adopted [20]. After the literature review in this study, four subscales for project making such as flow experience perception, self-efficacy, product evaluation, and learning motivation are developed as a tool for evaluating the learning effectiveness. The structure of this assessment tool can be regarded as Figure 1.

2.2.1. “Flow Experience Perception” Subscale

According to the definition from Qiu [21], the so-called “Flow” is fully concentrated to and engaged in the immediate activity, as a result that anything else is insignificant. It is in a state of loss of self-consciousness and regardless of the passage of time. Every behavior, action, and thought, with the previous initiatives, is issued naturally and inevitably, as if it were of playing Jazz music. Experience itself is so pleasant that people are willing to do it, even if it costs too much. Not for anything else, just for the sake of the activity itself, and doing it for the sake of doing [22,23,24].

Measuring the degree of perceived flow from students after completing the product generated by the project making can help us understand the learning effectiveness of the project making. Chien and Liu [2] designed the scale of the “Digital game production flow meter”. The scale was composed of two subscales; they include the “Heart flow experience perception scale” and the “Flow experience self-perception scale”. Csikszentmyhalyi [25] uses Experience Sampling Method for the study of flow; four indexes are used as the basis. These are:

• Challenges and Skills: these two scoring ratios are used to determine whether personal experience has entered a flow state.
• Experience Quality: the perception of the activity by the tester and the distinction between the seven grades, such as obsession, concentration, devotion, selflessness and sense of achievement.
• Activity: asks the tester to describe the activities they are engaged in.
• Motivation: asks the tester to evaluate individual motivation.

Based on the above increase of “Clear target” indexes, the first draft of the items is shown in

Table 1. “Flow Experience Perception” subscale of the first draft of the items.

(1)	I have a clear objective when I am making a project, and I know exactly what to do next.
(2)	In the process of making a project, I often unconsciously place myself in a situation how to complete the project.
(3)	Every time I finish an action, I feel happy immediately.
(4)	The project has some challenges for me; so long as I believe I have the ability to complete it, I can do it well.
(5)	When the project was made, work items to be completed were challenging, and I had to concentrate on doing my best and had no time to deal with irrelevant information.
(6)	When the project was made, I did my best to work in front of me. I did not think about the past or the future, and the sadness, fear and frustration in my life were all left behind.
(7)	I was very focused on the project and did not care about anything unrelated to a job assignment.
(8)	I often feel that time passes very quickly when I am making a project. It feels like just a while, but it takes me an hour.
(9)	I am very familiar with the tasks of the project, and if I concentrate on doing my best, the chances of success are one hundred percent.
(10)	Making a project is very interesting. I get pleasure from doing it.
(11)	Making a project sometimes takes a lot of time, but I still enjoy it.

Source: revised from Qiu [21]; Csikszentmyhalyi [25].

below.

2.2.2. “Self-Efficacy” Subscale

(1) Definition of Self-Efficacy

Students learning self-efficacy has a strong predictive ability for academic achievement [3,26,27,28,29]. Self-efficacy is defined as: a person believes that he can successfully achieve a particular result [30]. Students with high self-efficacy can more quickly participate in learning activities, have a more active learning attitude, and are willing to work hard spontaneously. In the face of difficulties, there is a higher persistence and less anxiety. They tend to accept challenging tasks, are willing to spend more time pursuing the goal, experience in-depth understanding before handling things, and are able to self-regulate and adopt better strategies [4,5,27,31,32]. Dellingera et al. [33] pointed out that efficacy belief has both situation and task specificity. The efficacy belief will be different depending on strength, level and generality. The “Strength” refers to how strong the learner’s belief is to have the ability to accomplish a task. The “Level” refers to the efficacy belief being different because of the difficulty of tasks, or tasks with different levels. “Generality” refers to the scope of a task which can be generalized into another similar activity.

(2) Assessment Indexes

Bandura [26] uses “Multi-level self-efficacy scale” to measure students’ learning self-efficacy, which can be divided into four dimensions: ability, effort, persistence, and learning method. On the basis of the four dimensions, Hong, Huang, and Qiu [34] simplified the indexes of learner’s self-efficacy to three dimensions: ability, effort and persistence, and learning method. This study adopted the previous mentioned four dimensions: ability, effort, persistence, and learning methods, and the first draft of the items is shown in

Table 2. Project-based “Self-efficacy” subscale of the first draft of the items.

A. Ability
(1)	I think I have good ability which is enough to complete the end-product of the project.
(2)	I have the analytical techniques required for making a project.
(3)	I think I have the required techniques to perform a project.
(4)	I think I can do very well in learning project curricula.
(5)	I am able to accept challenging learning assignments in each curriculum.
B. Effort
(6)	I think I can do my best to study hard.
(7)	I can spend considerable time and effort to learn the content of teaching material.
(8)	I am willing to spontaneous effort hard to learn the various subjects in the project curricula.
(9)	I think I have much positive attitude towards learning.
C. Persistence
(10)	Even if the work pressure is a lot, I think I can stick to it to the last minute and do my best to learn more.
(11)	In the face of difficult curricula content, I insist on putting my mind into it and learning it.
(12)	I believe I can successfully accomplish every particular task in the process of making a project.
(13)	I have a strong belief that there is no sense of anxiety that cannot be done about the task of the project.
(14)	I insist on completing a project specific task, even if it takes more time.
D. Learning Method
(15)	When I make a project, I will learn the relevant knowledge and recognize it before I start the operation.
(16)	I believe I can quickly join the learning activities in the process of making a project.
(17)	To complete the project is my goal; I would like to spend more time to pursue the achievement of this goal.
(18)	I can deeply understand and know each process for handling things.
(19)	In order to complete the end-products of the project and school assignments, I can self-adjust and adopt better learning strategies.
(20)	Even if the learning environment is not good enough, I can find learning methods suitable for my own to learn school assignments.

Source: revised from Bandura [26]; Hong, Huang and Qiu [34].

.

2.2.3. “Product Evaluation” Subscale

The planning and implementation of higher education curricula emphasizes the acquisition of students’ broad skills, knowledge, and attitude so as to successfully guide their future career development in the job market [35,36]. PBL is an integrated learning approach and an important way of achieving this goal [8]. In particular, it is important for learners to acquire integrated knowledge, skills, and attitudes instead of focusing on competition in learning grades among peers.

Filho, Shiel, and Paco [8] point out that integrated project learning should pay attention to: commitment, knowledge, diversification, dynamism, and communication. On the assessment of project outcomes, Zhang et al. [37] use six dimensions to assess the effectiveness of project making. They are: good teaching, clear goals, appropriate workload, generic skills, motivation, and information technology (IT) usage. Warin et al. [9] explore project making, adopting new PBL methods, and applying them to STEM (Science, Technology, Engineering, and Mathematics) subjects. They also evaluate the project outcomes by using following two dimensions: “Application method” and “Software skill development”. The first draft of the items is shown in

Table 3. “Product Evaluation” subscale of the first draft of the items.

A. Clear Goal
(1)	I have a clear understanding of the project subjects and content to be completed.
(2)	I understand what highlights of this curriculum I can learn from the project.
(3)	I know what analytical method should be adopted for the project’s end-product to be finished.
(4)	I adhere to the working procedures that should be followed for the completion of a project’s end-product.
B. Flexible Subject Design
(5)	The content design of project assignments is from easy to hard. It is a little hard but not too hard for my level.
(6)	Learning the project-making curricula gives me more confidence to solve similar problems.
(7)	The content design of the project is very interesting and let me have an impulse to finish it as soon as possible.
(8)	After completing the project, I also know what the content of the teaching material is provided and how it is actually applied.
(9)	Generally speaking, as long as I am willing to devote enough time, I can learn the curricula content related to the project.
C. Application Method
(10)	Project curricula have clear rules to guide team interaction and how the whole project making should proceed.
(11)	I know what I am doing for the task assignments in the team, and the work schedule I should complete.
(12)	I follow the methods or techniques taught in class and apply them to project making.
(13)	I will try to solve the problems I have encountered by using the methods I have learned in the project related curricula.
(14)	I will use the method of project management to complete the tasks assigned by the team within a given period of time.
D. Value Based Skill Development
(15)	I think the achievement of project making can solve practical problems and has application value.
(16)	I often communicate with other team members, and I feel that my interpersonal relationship skills are getting better.
(17)	This curriculum can help me develop my problem solving skills.
(18)	In order to complete the project assignments, I have to apply knowledge of different curricula to improve my integration competence.
(19)	The project assignments will require me to try to break through what is taught from the content of the teaching material and complete the project end-product with originality.

Source: revised from Filho, Shiel, and Paco [8]; Zhang et al. [37]; Warin et al. [9].

below.

2.2.4. “Learning Motivation” Subscale

The main types of motivation are: extrinsic motivation, which relies on external things, such as reward temptation to motivate learners’ learning behavior; and intrinsic motivation, which relies on the learner’s fondness for the activity itself to initiate learning action [21,24,38].

Chienand Liu [2] designed five indexes for the measurement of learning motivation. (1). Value: intrinsic motivation orientation. (2). Value: extrinsic motivation orientation. (3). Value: the value of homework. (4). Expectation: the control belief of learning. (5). Expectation: the self-efficacy belief in learning and achievement. The motivation scale used by Huang, et al. [39] includes four variables, namely, concentration, relationship of immediate concern to oneself, self-confidence and sense of satisfaction. Among them, “Concentration” refers to whether it can arouse curiosity or interest in students so that they pay attention to it. “Relationship of immediate concern to oneself” refers to whether students perceive if the teacher’s teaching can meet the learning needs, andifthe knowledge they are learning is useful. “Self-confidence” concerns whether a student can perceive that he or she will succeed in completing the task of learning through many efforts. “Sense of satisfaction” is the internal and external rewards that are received from the teaching.

This study uses the two dimensions of value and expectation, and the first draft of the items is shown in

Table 4. “Learning Motivation” subscale of the first draft of the items.

A. Intrinsic Goal Orientation (Value)
(1)	I want to study challenging project curricula so that I can learn useful knowledge and skills.
(2)	I want to study interdisciplinary project curricula, even though the content of these curricula is difficult.
(3)	Studying the content of learning materials in a thematic way can help me learn more efficiently.
(4)	I want to learn hands-on project assignments.
B. Extrinsic Goal Orientation (Value)
(5)	My greatest wish is that I can get good grades in every curriculum.
(6)	I like to do the project by myself very much under the guidance of the teacher and learning among the students to finish the project.
(7)	To get the curricula credit, I think I can display the competence of doing practical work in my family, friends and future job.
(8)	I think it will be very helpful for my future career to get the credits of the project curricula.
C. The Value of Project Making (Value)
(9)	The practical work I have learned from the project making will allow me to apply it to the real workplace.
(10)	I am interested in various curricula that use project-based learning.
(11)	I think the content of teaching material I learned from the project making will help me learn other subjects more confidently.
(12)	I think the curricula content applied by the project making will be of practical value.
D. The Control Belief of Learning (Expectation)
(13)	If the learning method is correct, I can learn the curricula content of the project.
(14)	If I study hard, I can understand the curricula content of each teaching subject material in the project making applications.
(15)	If I insist on doing my project homework and completing it by deadline, I can learn the content of my homework.
(16)	If I do my homework with my whole efforts, I will be confident in the application of the content of the teaching materials.
E. Self-Efficacy Belief in Learning and Achievement (Expectation)
(17)	I believe I can get excellent results in project making.
(18)	I believe I can learn the basic concepts taught in different curricula.
(19)	I believe I can learn the most complex teaching materials taught in different curricula.
(20)	I believe I can finish the end-product of the project making by the appointed time.
(21)	I believe I can apply the method taught by the project making to complete the project’s end-product.

Source: revised from Chien and Liu [2]; Huang et al. [39].

below.

3. Methodology

Literature Review: The exploration and analysis of the literature is to establish four drafts of the subscale to form a draft scale, to serve as a basis for discussion and revision by the experts of focus group.

Focus group method: Focus group discussion is one of the most popular qualitative research methods in the social sciences, with a particular emphasis and application in the developmental program evaluation [40]. The focus group meeting was designed to establish the content validity of the four subscales required for the evaluation of learning effectiveness. Eight experts were invited in the fields of the curricula, IT, Internet of Things (IoT), and solar energy nanomaterials. A draft of the scale was sent out one month before the meeting, and two focus group meetings were held to jointly discuss and revise the draft of the total scale. Based on the literature review and expert consultation, flow experience perception, self-efficacy, product evaluation and learning motivation were confirmed as the 4 assessment subscales.

Survey Research Method

● Development of research instruments: The assessment tool consists of four subscales, flow experience perception (11 items), self-efficacy (17 items), product evaluation (19 items), learning motivation (21 items), and demographic information (i.e., gender and identification). Each subscale item was compiled in accordance with the operational definitions, as well as behavioral patterns displaying the learning effectiveness for PBL. The items were designed on the basis of a 5-point Likert scale (1 = not important, 5 = very important).

A draft scale was pretested after it is revised and confirmed by the focus group experts. The pre-test stage was conducted by the third-grade undergraduate students majoring in the fields of the IT, IoT, and solar energy who are performing the project-making curricula in a university of science and technology. Approximately 37 copies of valid questionnaires were recovered. After analysis, the reliability of the scale was established by taking Cronbach’s Alpha value greater than 0.7 as the item.

● Statistical analysis: The official test was conducted two months after the implementation of the pre-test in all the third-grade undergraduate students of 4universities of science and technology. The official test was performed at the end of the semester for the students to complete the software of the project making.

In total, 232 returned copies, or 77.33% of the questionnaires, were valid. In terms of students’ sampling, an example of PBL on Nanotechnology curricula, included the solar raw materials industry in the upstream end (e.g., pure metals, iron alloys and titanium metal, etc. and other metal materials, silicon chip production); the solar manufacturing industry in the midstream end (e.g., amorphous silicon thin film solar cells and modules; mono-crystalline silicon, polycrystalline silicon solar cells and modules; silicon solar cells and modules; poly-silicon solar cells and modules; silicon, thin film solar cells), and the solar module and integration industry in the downstream end (e.g., small-scale solar power systems, solar electric power converter and solar power system).

In statistical analysis, in addition to presenting the scale pre-test reliability, the Independent Samples Test was used to examine whether there were differences of views between male and female students in the subscales. By using Multiple Regression Analysis, take the four subscales of the product evaluation (X₁), self-efficacy (X₂), flow experience perception (X₃), and learning motivation (X₄) as the “Predictor” and project making learning effectiveness (Y) as the “Criterion”. By using independent variables (i.e., subscales) and an estimated multiple regression equation, a prediction of the dependent variable (i.e., learning effectiveness) can be made. Then, the following multiple regression equation can be established:

Y = b₀ + b₁ X₁ + b₂ X₂ + b₃ X₃ + b₄ X₄

4. Results

4.1. Reliability of the Scale in the Pre-Test

The overall reliability of the total scale reached an excellent level by setting the Cronbach’s Alpha value at 0.976. The results of the pre-test reliability analysis are shown in

Table 5. Results of pre-test reliability analysis of the subscales.

Subscale	Overall Reliability Cronbach’s Alpha	Cronbach’s AlphaDistribution	Analysis Result
a. Flow experience perception	0.819	0.790~0.813	To a good level, all the items are retained.
b. Self-efficacy	0.940	0.931~0.939	To an excellent level, all the items are retained.
c. Product evaluation	0.939	0.932~0.940	To an excellent level, all the items are retained.
d. Learning motivation	0.928	0.919~0.929	To an excellent level, all the items are retained.

.

4.2. Formal Test Results Analysis

The subjects under test were undergraduate junior students. The test was conducted at the end of the semester during the same week for finishing implementation of the project-making curricula of the solar nanotechnology. Among them, 143persons were male students (61.6%) and 89 persons were female students (38.4%). The Cronbach’s Alpha value was 0.987, so that the overall reliability of the scale reached an excellent level.

4.3. Difference Test in the Average Number of Male and Female Students in the Project Making

In order to understand the PBL of the professional applications, the Independent-Samples T Test was used to see if there were any differences of views between male and female students on the scales. The results are shown in

Table 6. Independent Samples Test.

		Levene’s Test for Equality of Variance		t-Test for Equality of Means
		F	Sig.	t	df	Sig. (2-Tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the Difference
									Lower	Upper
a9	Equal variances assumed	0.343	0.559	2.020 *	230	0.045	0.223	0.111	0.005	0.441
	Equal variances not assumed			1.990	177.715	0.048	0.223	0.112	0.002	0.445
b13	Equal variances assumed	0.620	0.432	2.104 *	230	0.036	0.252	0.120	0.016	0.487
	Equal variances not assumed			2.117	190.552	0.036	0.252	0.119	0.017	0.486
b14	Equal variances assumed	1.546	0.215	2.957 **	230	0.003	0.329	0.111	0.110	0.548
	Equal variances not assumed			2.980	191.627	0.003	0.329	0.110	0.111	0.546
b24	Equal variances assumed	7.229	0.008	2.064 *	230	0.040	0.220	0.107	0.010	0.431
	Equal variances not assumed			1.972	159.888	0.050	0.220	0.112	0.000	0.441
b28	Equal variances assumed	0.238	0.626	2.222 *	230	0.027	0.246	0.111	0.028	0.465
	Equal variances not assumed			2.297	206.508	0.023	0.246	0.107	0.035	0.458
b29	Equal variances assumed	1.721	0.191	3.972 ***	230	0.000	0.426	0.107	0.215	0.637
	Equal variances not assumed			3.963	185.426	0.000	0.426	0.107	0.214	0.638
c30	Equal variances assumed	0.123	0.726	2.899 **	230	0.004	0.317	0.109	0.101	0.532
	Equal variances not assumed			3.037	213.581	0.003	0.317	0.104	0.111	0.522
c31	Equal variances assumed	0.076	0.782	2.545 *	230	0.012	0.295	0.116	0.067	0.524
	Equal variances not assumed			2.599	199.343	0.010	0.295	0.114	0.071	0.519
c39	Equal variances assumed	0.072	0.789	2.525 *	230	0.012	0.267	0.106	0.059	0.476
	Equal variances not assumed			2.601	204.234	0.010	0.267	0.103	0.065	0.470
c48	Equal variances assumed	3.746	0.054	3.918 ***	230	0.000	0.409	0.104	0.203	0.614
	Equal variances not assumed			3.809	169.891	0.000	0.409	0.107	0.197	0.620
d52	Equal variances assumed	0.011	0.918	2.038 *	230	0.043	0.220	0.108	0.007	0.433
	Equal variances not assumed			2.012	179.094	0.046	0.220	0.109	0.004	0.436
d54	Equal variances assumed	0.312	0.577	2.292 *	230	0.023	0.237	0.103	0.033	0.441
	Equal variances not assumed			2.279	183.509	0.024	0.237	0.104	0.032	0.442
d63	Equal variances assumed	7.214	0.008	2.139 *	230	0.033	0.243	0.114	0.019	0.467
	Equal variances not assumed			2.020	153.434	0.045	0.243	0.120	0.005	0.481

* p < 0.05; ** p < 0.01; *** p < 0.001.

.

As a result, the items have significant differences of views between male and female students on the overall scale, shown in

Table 7. The items for male and female students having different views on the overall scale.

Subscale	Item No.	Item
a. Flow experience perception	a9	I am very familiar with the tasks of the project, and if I concentrate on doing my best, the chances of success are one hundred percent.
b. Self-efficacy	b13	I have the analytical techniques required for making a project.
	b14	I think I have the required techniques to perform a project.
	b24	I insist on completing a project specific task, even if it takes more time.
	b28	In order to complete the end-products of the project and school assignments, I can self-adjust and adopt better learning strategies.
	b29	Even if the learning environment is not good enough, I can find learning methods suitable for my own to learn school assignments.
c. Product evaluation	c30	I know the main points of the curricula that I can learn from the project.
	c31	I know the analytical method to be used for the project end-product to be finished.
	c39	Project curricula have clear rules to guide team interaction and how the whole project making should proceed.
	c48	The project assignments will require me to try to break through what is taught from the content of the teaching material and complete the project end-product with originality.
d. Learning motivation	d52	My greatest wish is that I can get good grades in every curriculum.
	d54	Getting the credits of the project is very helpful for me to find a good job in the future.
	d63	If I insist on doing my project homework and completing it by deadline, I can learn the content of my homework.

.

4.4. Regression Analysis

The purpose of the regression analysis is to understand the extent to which the subscales can explain the learning effectiveness of the total scale.

Table 8. Model summary.

Model	R	R Squared	Adjusted R Squared	Std. Error of the Estimate	Change Statistics					Durbin-Watson
					R Squared Change	F Change	df1	df2	Sig. F Change
1	0.952 a	0.907	0.907	21.445	0.907	2228.925	1	228	0.000
2	0.991 b	0.981	0.981	9.678	0.074	892.484	1	227	0.000
3	0.997 c	0.994	0.994	5.511	0.013	474.142	1	226	0.000
4	1.000 d	1.000	1.000	0.000	0.006	2,051,608,053,956,100.000	1	225	0.000	1.851

^a. Predictors: (Constant), Eva; ^b. Predictors: (Constant), Eva, Eff; ^c. Predictors: (Constant), Eva, Eff, Flo; ^d. Predictors: (Constant), Eva, Eff, Flo, Mov.

provides the relevant data for the stepwise regression of the following four modes.

Mode 2: The multivariate correlation coefficient (R) of the independent variables Eva (i.e., product evaluation), Eff (i.e., Self-efficacy) with respect to the dependent variable is equal to 0.991. The R squared is equal to 0.981 and the R squared after the adjustment is equal to 0.981. The standard error is 9.678, and the R squared change quantity is 0.074. According to the coefficient of determination, R squared (i.e., Multiple Determination Coefficient), the “Self-efficacy” in model 1 can explain 90.7% of the total variance correlated to the dependent variable of the “Project making learning effectiveness”. In model 2, the Eva and Eff accounted for 98.1% explanation of the total variance, and 7.4% explanation of the total variance can be improved by the addition of the independent variable Eff.

Mode 3: variables that include in the regression model are Eva, Eff, and Flo. The multivariate correlation coefficient of the project making overall score Tal with respect to these variables is 0.997. The combined explanation variance is 99.4%, and the Flo could improve 1.3% of the total explanation variance. Similarly, in model 4, the multivariate correlation coefficient of the variables Eva, Eff, Flo, Mov with respect to the project making overall score Tal is 1. The combined explanation variance is 100%, and the Mov could improve 0.6% of the total explanation variance.

The results of Collinearity Diagnostics are shown in

Table 9. Collinearity Diagnostics.

Model	Dimension	Eigenvalue	Condition Index	Variance Proportions
				(Constant)	Eva	Eff	Flo	Mov
1	1	1.987	1.000	0.01	0.01
	2	0.013	12.583	0.99	0.99
2	1	2.980	1.000	0.00	0.00	0.00
	2	0.016	13.625	0.99	0.06	0.09
	3	0.004	27.307	0.01	0.94	0.91
3	1	3.972	1.000	0.00	0.00	0.00	0.00
	2	0.019	14.336	0.90	0.01	0.02	0.06
	3	0.005	29.527	0.05	0.00	0.72	0.71
	4	0.004	32.154	0.04	0.99	0.26	0.23
4	1	4.967	1.000	0.00	0.00	0.00	0.00	0.00
	2	0.020	15.647	0.92	0.00	0.01	0.04	0.01
	3	0.005	30.627	0.03	0.00	0.12	0.78	0.24
	4	0.004	35.650	0.02	0.61	0.52	0.07	0.04
	5	0.003	40.052	0.03	0.39	0.35	0.12	0.71

. The maximum Condition Index (CI) value shown in Mode 4 is 40.052, which is greater than 30, indicating that there is a multi-collinearity problem among independent variables. However, the Variance Proportions of the variables Eva (0.39) and Eff (0.35) shown in the Table are not high enough, which indicates that the multi-collinearity problem is not serious and still within the acceptable range.

Based on the above relevant data, the four subscales of learning motivation, product evaluation, self-efficacy, and flow experience perception, are used to predict the learning effectiveness of the project making. The summary of the stepwise multiple regression analysis is shown in

Table 10. A summary of the Stepwise Multiple Regression Analysis by using the subscales for predicting the learning effectiveness.

Selected Variable Order	R	${\mathbf{R}}^2$	$\mathbf{Improved} \mathbf{Explanation} \mathbf{Variance} \Delta \mathbf{R}$	F Value	Net F Value	Unstandardized Coefficients	$\mathbf{Standardized} \mathbf{Coefficients} \mathsf{\beta }$
Intercept Constant						4.603
1. Product evaluation	0.952	0.907	0.907	2228.925	2228.925	2.000	0.332
2. Self-efficacy	0.991	0.981	0.074	5918.273	892.484	2.000	0.324
3. Flow experience	0.997	0.994	0.013	12,327.298	474.142	2.000	0.215
4. Learning motivation	1.000	1.000	0.006	-	2,051,608,053,956,100.000	1.000	0.188

. The results indicate that the multivariate correlation coefficient of the learning effectiveness of the project making predicted by the four predictable variables of the four subscales is 1.000. The combined explanation variance is 1.000. That is to say, the four predictable variables can predict the students’ learning effectiveness of the project making up to 100%.

From the observation of the explanation variance among individual variables, the prediction power of “Product evaluation” reaching 90.7% is the best. The rest orders are: “Self-efficacy” 7.4%, “Flow experience” 1.3%, and “Learning motivation” 0.6%. From the standardized regression coefficient observation, the value of $\mathsf{\beta }$ coefficients of the four variables, namely, product evaluation (X₁), self-efficacy (X₂), flow experience perception (X₃), and learning motivation (X₄) are all positives. This means that the effect of the four variables on the project making learning effectiveness (Y) a positive relationship. The four predictable variables with respect to the project making, namely, learning motivation, product evaluation, self-efficacy, and flow experience perception, can be used as the criterion variables to completely and positively predict the “Learning effectiveness” of the students.

Therefore, the standardized regression equation obtained is as follows:

$${\mathrm{Z}}_{\mathrm{y}} =0.332{\mathrm{Z}}_{\mathrm{X}1} +0.324{\mathrm{Z}}_{\mathrm{X}2} +0.215{\mathrm{Z}}_{\mathrm{X}3} +0.188{\mathrm{Z}}_{\mathrm{X}4}$$

The original regression equation is as follows:

$$\mathrm{Y}=4.603+2{\mathrm{X}}_1 +2{\mathrm{X}}_2+2{\mathrm{X}}_3 +{\mathrm{X}}_4$$

5. Discussion

5.1. Flow Experience Perception Positively Predicts the PBL Effectiveness

The results of this study are consistent with those of Csikszentmyhalyi [22,23] and Qiu [21]. The results show that the project making “Flow experience perception” subscale contains indexes, such as clear goals, challenges and skills, empirical quality, activity, motivation, and so on. It is enough to fully perceive the flow experience of the learners. From the differences of views between male and female students, the main objections should be on two indexes: “Clear goals” and “Empirical quality”. In view of the implementation of the PBL, the male and female students should pay attention to how to operate the next action to achieve the current goal, and the step-by-step actions to accomplish the goal. The project making activities should focus on the learning effect of the activity itself, and try to make learners feel engaged with the activities, and achieve different levels of effects such as obsession, concentration, devotion, selflessness and sense of achievement.

5.2. Self-Efficacy Positively Predicts the PBL Effectiveness

The results of this study are consistent with those of Bandura [26], Pajares and Kranzler [29], and Pajares [41]. The results show that student learning self-efficacy has a strong ability to predict the consistent results of academic achievement. The different views between male and female students are mainly reflected in the two indexes of “Ability” and “Learning method”. In view of the implementation of the PBL, male and female students should pay attention to the reinforcement of these two indexes in the learning guidance of self-efficacy.

5.3. Product Evaluation Positively Predicts the PBL Effectiveness

The results of this study are consistent with those of Filho, Shiel, and Paco [8], as well as Sterling and Thomas [42]. The results show that it is important for individual learners to acquire integrated knowledge, skills and attitudes in the project making curricula, rather than academic grades in competition between peers. The following four indexes, namely, clear goals, flexible subjects design, application methods, and value skills development, can be used as the four evaluation indexes of the project product. The different views between male and female students are mainly reflected in a single index of “Clear goal”. In view of the implementation of the PBL, male and female students should pay attention to the desired goal that the subject wants to achieve, which is particularly important for project making.

5.4. Learning Motivation Positively Predicts the PBL Effectiveness

The results of this study are consistent with those of Qiu [21], Csikszentmyhalyi [24], and Chien and Liu [2]. The results show that in the course of PBL activities, learners should be encouraged to enjoy the project making activities themselves, so as to arouse learning action. The different views between male and female students are mainly reflected in the two indexes of “External goal orientation (value)” and “Learning control beliefs (expectation)”. In view of the implementation of the PBL, male and female students’ should pay attention to the value of external goals and the reinforcement of expectations.

5.5. Subscales Can Positively Predict the Learning Effectiveness

The significance of the original regression equation obtained in this study is:

$$\mathrm{PBL}\mathrm{Effectiveness}=4.603+2\times \mathrm{Learning}\mathrm{Motivation}+2\times \mathrm{Product}\mathrm{Evaluation}+2\times \mathrm{Self}-\mathrm{efficacy}+\mathrm{Flow}\mathrm{Experience}\mathrm{Perception}$$

It means that, if students have high scores on the four scales, namely, strong learning motivation, good product evaluation results, high self-efficacy, and positive flow experience perception, it is predictable that these students “Learning effectiveness” in the PBL must be excellent.

6. Conclusions

This study is based on Dewey’s theory of “Learning by doing” to develop the integration and application abilities and cultivate independent thinking among university students, to achieve the educational philosophy of finding and solving problems, diversification of thinking, and meaningful learning, thereby conducting the effectiveness evaluation on the PBL for enhancing education of sustainability. The study through the process of literature analysis, focus group expert discussion, and survey research, establishes four project making subscales, namely, flow experience perception, self-efficacy, product evaluation, and learning motivation, suitable for forming a learning effectiveness assessment tool on the project making curricula of scientific and technological education.

The multiple regression equation obtained from this study shows that the four subscales of the students’ learning motivation, product evaluation, self-efficacy and flow experience perception, can positively and comprehensively predict students’ “Learning effectiveness” on project making. As a result, this scale can be applied practically to the evaluation of the learning effectiveness on the project making curricula.

However, in this study, the objects to be tested with these subscales are the students who are performing the project making. Formal questionnaires will be filled out after the students have completed the project curricula. There is no experimental study on teaching differences before and after learning. In addition, the selection of core competencies and curriculum group contents in the PBL curricula is not in the scope of this study. It is suggested that future study can further strengthen of the establishment of the core competencies and the planning and design of the project making curricula in the PBL curricula, using the tool obtained from this study for teaching experiments to evaluate the learning effectiveness of PBL.