Comparative Evaluation of Sustainable Framework in STEM Intensive Programs for Secondary and Tertiary Education

: Sustainability of learning environments is a key pillar of all societal development frameworks. A variety of research address the development of education as a ﬁne balanced relation between ﬂexibility, adaptability, innovation, and efﬁcient resource allocation. The main limitation of current approaches is the lack of correlation between various efﬁciency analyses and budget expenditure of learning environments. The current research aims at undertaking a comparative evaluation of a sustainable framework in STEM intensive programs for secondary and tertiary education. This was done using several established methods like the Plan, Do, Check, Act cycle for the development main framework, the Analytic Hierarchy Process for efﬁciency evaluation and Value Analysis for budget expenditure allocations and improvement identiﬁcation. The main framework is based on learning objectives deﬁned in accordance with Blooms’ revised taxonomy and student feedback was collected through surveys and group feedback. The main results of the study show that the framework had overall efﬁciencies over the 80% threshold in both secondary and tertiary education, whilst some of the components scored under 65%, identifying immediate improvement features. Further research involves the transition to an online and mixed teaching environment, by adapting the content and framework structure with the aid of smart learning environments. of of a and reproductible


Introduction
Sustainable education development is on the current agenda of all major international organizations like the European Commission, OECD and UNESCO. This is mainly due to the fact that a future sustainable society is reliant on the development and implementation of a sustainable education system. Additionally, The United Nations Education Scientific and Cultural Organization [1] is the main promoter of the Education for Sustainable Development (ESD) and the Global Action Program (GAP), investing in the development of new skills, values, and attitudes with the aim of creating a sustainable future in education.
Sustainable education development is often referred to as Sustainability Education or Education for Sustainable Development (ESD). UNESCO gives a comprehensive definition, stating that "ESD allows every human being to acquire the knowledge, skills, attitudes and values necessary to shape a sustainable future. ESD empowers learners to take informed decisions and responsible actions for environmental integrity, economic viability and a just society, for present and future generations, while respecting cultural diversity" [1]. In this context, it is important to evaluate the societal and economic impact of current educational frameworks and propose educational shifts that could lead to reliable ESD systems. Worldwide, there are several initiatives which address these impending issues, with additional requirements coming from the current SARS-CoV-2 epidemic, which is impacting education systems at an unprecedented level and with yet unknown outcomes [2].
The Organization for Economic Co-operation and Development (OECD) offers a variety of changes in principles for future education systems under the aegis of the Learning map, whilst using Shapiro-Wilk Normality tests and Mann-Whitney U tests to evaluate the normal distribution and significant differences. Results showed the correlation between an objective oriented curriculum and acquired sustainability competences. Although inclusive, this method is limited to a tertiary education environment and cannot offer reproducibility criteria outside the studied environment. Moreover, their findings suggest that current learning frameworks do not show any significant improvement in student learning and selected subjects are failing to reach their ESD learning objectives.
Within the current research, learning objectives are constructed based on Bloom's revised taxonomy [10,11], for both secondary and tertiary STEM intensive programs. Bloom's revised taxonomy was chosen due to the potential gains in structuring curriculum described in his work by Irvine [12]. Due to the simplicity of application, the method is highly appropriate and recommended by a plethora of researchers in both secondary and tertiary curriculum development.
Lysenko et al. [13] analyze innovations, business, education development and competitiveness, as indicators of the effectiveness of regional innovation clusters in the higher education system. The research team has established a close relationship between the level of development of regional innovation clusters, indicators of business and innovations development, and the level of competitiveness, with direct impact on the development of higher education system (HEI). An important implication of their study is that in order to be sustainable, learning frameworks need to address real needs within the regional economical environment of the area in which the learning frameworks are implemented. Limitations of the study relate to a lack of curriculum specific criteria, which could greatly influence the outcome of the study depending on the region specifics. Additionally, there is no clear correlation between expenditure on HEIs and the proposed effectiveness indicators. This aspect is of particular importance when analyzing the efficiency of STEM education programs, which are known to be more resource intensive, compared to other educational programs [14].
STEM education addresses all levels of education, from elementary to tertiary, integrating content into a cohesive learning paradigm based on real-world applications. What separates STEM from the traditional science and math education is the blended learning environment and showing students how the scientific method can be applied to everyday life. STEM pillars are in accordance with the EC development strategy for 2020. Sustainable STEM education integrates STEM fields of knowledge and experience with social and emotional learning and civic engagement, by promoting inquiry-based learning and scientific thinking and practice and by encouraging interactive, learner-centered teaching that enables exploratory, action oriented, reflective, and transformative learning [15]. In this paper, the authors considered the secondary and tertiary STEM intensive programs as learning programs in which intensive courses (one/two-week programs) are taught focused on the development of practical skills in STEM fields. Within the current study, blended learning is used to define both the used multidisciplinary approaches and the blend of teaching techniques.
In a recent study, conducted by Christopoulos et al. [16] the research team explored the development of a learning analytics theoretical framework for STEM education for virtual reality applications, concluding on both the advantages and the limitations of their study. A key finding identified specific features which address the realistic needs of STEM students. Flexibility is promoted as a mandatory characteristic for successful curriculum development in STEM education. Lee et al. [17] analyses the implications of pedagogical flexibility in curriculum use and how their findings can promote mathematical flexibility in students. Based on these findings, authors developed a modular framework with interdisciplinary curriculum content, applicable to both secondary and tertiary learning environments. In this approach, flexibility can be expressed by either the curriculum developer, to cover specific needs of the target group, but also by the program participants who can chose modular alternative learning routs, depending on their personal preferences. Furthermore, Roblyer [18] identifies three basic types of curriculum design, namely: subject-centered, learner-centered, and problem-centered design. Higher education curriculum development is typically based on subject-centered approaches, which focuses on a particular subject or discipline, the model being less concerned with individual learning styles compared to other forms of curriculum design. One major drawback of this approach is that students who are not responsive to this model tend to fall behind, because of decreased levels of engagement and motivation [19]. Thus, another key feature which was considered in the development of the STEM framework was the learning styles, as defined by Gardner [20].
Considering the abovementioned, the main limitations of current approaches can be summarized: lack of correlation between various efficiency analyses and budget expenditure of learning environments; absence of an efficiency evaluation on an integrated curriculum approach, which considers multiple design variables; gap in STEM interdisciplinary approaches which consider work experience exposure in correlation with future integration of participants into the workforce market.
Based on the EC conclusion that "efficiency is usually referred to as the relation between the resources put into the production process and the output obtained" [5] the authors propose an efficiency evaluation method for the sustainable framework in STEM intensive programs for secondary and tertiary education. To correlate spending decisions and customer value of the framework, the current study deploys Value Analysis (VA), aiming at identification of targeted improvement areas. Value Analysis (VA) has been proven to be very efficient when measuring value creation within the design process, weather it is targeted at physical products [21] or at system processes.
The main objective of the study is to present the development and implementation of a sustainable framework for STEM intensive programs within secondary and tertiary education, based on four central dimensions, such as: efficiency, effectiveness, growth and control. It follows both the similarities and differences within the framework, between the two levels of education, in terms of pedagogical approach, content structure, learning objectives, learning styles, delivering formats, instructional strategies and assessment methods. The framework addresses the limitations of current initiatives, while providing a comprehensive comparison between secondary and tertiary education.

Materials and Methods
The current study proposes the development and implementation of a sustainable framework for two STEM intensive programs within secondary and tertiary education.
The two intensive programs are currently being implemented at University PO-LITEHNICA of Bucharest (UPB) through external funding grants, awarded for three years, each. Both projects are currently in their second year of implementation. The funding grant calls targeted specifically students and pupils with learning difficulties and from disadvantaged areas, focusing on closing the gap between Romania and other EU countries on the percentage of early leavers from education and training, and the percentage on tertiary educational attainment. Thus, the proposed sustainable framework in STEM intensive programs for secondary and tertiary education was designed to address specific learning environments. The two projects, MOTIV and TecHUB 4.0, are described further on.
MOTIV (Mobilization, Organization and Objectives for Future University Education-REASON for the future!) is a STEM focused project aimed at improving the results, access and participation of disadvantaged students from secondary level education, into the higher education system. With an overall target group of 350 national high school students, MOTIV implemented in July-August 2019 its' first intensive STEM program with 100 eligible participants and 8 extra volunteer participants. The objective of the program was to orient the target group participants in terms of further study options in technical higher education, facilitating the transition from secondary to tertiary education, by providing an integrated package of STEM courses and other relevant activities. Designed as a twoweek intensive program the curricula was constructed based on STEM topics integrated Sustainability 2021, 13, 978 5 of 33 with career counselling, social and cultural activities. Seven modules constituted the core curricula components, namely: Mathematics, Physics, Informatics, Computer Aided Design (CAD), 3D Printing, Mechatronics and Robotics. Apart from the intensive program implemented in 2019, two more similar educational programs will be funded through the MOTIV project. The scope is to implement and test the structural and content related changes revealed during feedback. TecHUB 4.0 (TecHnology and EntrepreneUrship Education-Bridging the Gap for Smart Product Development) is an Erasmus+ funded project which aims at promoting Additive Manufacturing (AM) entrepreneurship towards young students, who wish to design and develop smart products in an interactive environment, through the involvement of professors, industry specialists, researchers, managers and entrepreneurs. It is currently being implemented with one national partner and two international partners, as follows: "Lucian Blaga" University of Sibiu (ULBS) from Romania; Czestochowa University of Technology (PCz) from Poland; Lancaster University (LU) from the United Kingdom of Great Britain. Amongst other activities, TecHUB 4.0 targets the implementation of two intensive summer school programs, one of which was deployed in early July 2019. Designed as a one-week intensive STEM focused program the educational event had a target group of 100 national and international students and 4 additional volunteers. In both secondary and tertiary intensive educational programs, volunteers were considered active participants in the study and a part of the target groups. Apart from the level of knowledge, the main difference between the two programs was that TecHUB 4.0 promoted a multidisciplinary approach to module delivery. Although the content of the modules included knowledge and development of competencies in mathematics, physics, informatics, 3D printing, CAD and mechatronics and robotics, the programs' structure was built on the broader idea of teams developing a technical Start-up [19]. This was done considering that eligible participants were accepted from both bachelor and masters' programs, thus their level of competences [19] entailed a more complex learning environment.
For both programs, gamification elements were used to implement all activities. A complex reward system was developed and used to deploy all activities, substituting the classic grading and scoring systems [22,23]. For the secondary education STEM intensive program, the content was gamified using several important elements, such as: guest, levels, achievements, reward system and time tracking. The reward system included badges, tokens, privileges, incentives, passes, virtual and physical goods [23]. The reward system was comprised of the following elements: financial incentives-scholarships; "survival kit"-incentives; level maps and activity plans; diplomas; individual and team trophies; public recognition; tokens and badges; privileges; passes for participation at social and recreational activities; competitive advantage at social and recreational activities based on Level and Achievement performance [23]. When designing the gamified learning content for the tertiary education STEM intensive program, a basic three-tiered framework [24] was considered: goal-focused activities, reward mechanisms and progress tracking. To address these, some of the most used elements in gamifications [25] were considered: credit points (CP), progress and ranking graphs and levels [22]. Infrastructure was made available by UPB within four of its' faculties: Faculty of Industrial Engineering and Robotics (FIIR), Faculty of Applied Sciences (FSA), Faculty of Biotechnical Systems Engineering (FISB) and Faculty of Mechanical Engineering and Mechatronics (FIMM). Infrastructure requirements were similar for both programs due to their STEM focused curriculum and to maintain unity throughout the framework and entailed the usage of: four main technical laboratory rooms with work benches and storage areas for 30 participants at a time; one large learn and lounge space for both learning experiences and recreational activities; one amphitheater with 300 places for large group activities; 150 personal computers; 6 video projectors (one in each room, one in the learn and lounge space and one in the amphitheater); 60 FDM and SLA 3D printers; 20 complex Arduino kits; 20 pre-programmed robots; topic specific instruments (e.g., vacuum chamber, calipers, electronic scale etc.) and software (for CAD; 3D printing; Programming; Sustainability 2021, 13, 978 6 of 33 Web design etc.). Catering, accommodation, travel and adequate office supplies for all participants were funded through the two abovementioned projects.
Learning objectives (LO) were comprehensively defined, for both intensive programs, using Bloom's revised taxonomy [10,11], considering a 3-tiered approach of each program: Methods and Structure, Professional Development, Overall Performance. Learning levels were considered throughout the development process. Learning objective structure was kept as clear and concise as possible by following these three steps: (1) Who will be able to perform a certain learning activity? (2) What is the activity to be performed? (verb + goal); (3) How and under what circumstances will the activity be performed? (conditional).
Instructional strategies, as defined by White and Braddy [26], were evaluated and selected to be used in the two STEM intensive programs, primarily due to the studentcentered advantages that they bring.
Step-by-step learning with innovative approaches, both guided and independent practice and tackling real-life situations using case studies, were mostly used in MOTIV educational program implementation. Due to the specificity of the target group and the content structure, TecHUB 4.0 instructional strategies aimed at the advantages brought by project-based learning, team-based and blended learning approaches. Most often, cooperative learning, interactive instruction, reciprocal teaching and group activities were used. The most appropriate instructional strategies were selected and particularized for each target group and module, allowing participants to display their skills, ideas and their existing knowledge on a particular subject. They were encouraged throughout the learning experience to self-monitor and assess their learning, regardless of their knowledge levels in different modules. Selection of the most appropriate instructional strategies was made via a pilot course implemented at UPB before the design of the sustainable framework. The pilot course was deployed in three sessions and targeted different modules of the curriculum content. Results were gathered using ongoing feedback collected by the trainer, as well as a survey completed at the end of each session of the pilot study by the participants. Participants were selected randomly amongst eligible UPB students. Results were processed by authors and brainstorming and nominal group technique were used to select the final instructional strategies, corelated with module content.
Pedagogical approaches often pass as elaborated instructional strategies [27]. Defined as the broadest structured environment for theory and practice of learning, pedagogical approaches assess how this process influences, and is influenced by, the social, political and psychological development of learners. In this context, six main pedagogical approaches were used during the implementation of both intensive programs: secondary-inquiry learning, differentiated instruction; experiential learning; tertiary-open-ended instruction, integrated learning, locus of control. Peer teaching, cooperative learning and case studies were used in both programs [28].
To accurately calculate efficiency of the implemented intensive programs, the authors proposed a quantitative approach, by designing 20 components (Section 3.2) which define the core characteristics of the programs. A targeted survey was constructed and deployed amongst all participants, in order to evaluate the overall performance of these 20 components. Individual efficiencies were calculated for each of the components and an importance coefficient was assigned by using the Analytic Hierarchy Process (AHP) tools [29]. Value analysis (VA) was undertaken [30] to establish the cost-value benefits of the two programs and identify which components need improvement in terms of either cost reduction or increase in added value [31].
Effectiveness of the programs was ensured by dividing the LO into tasks and making sure the trainers, speakers and tutors completed them by using written guides and progress charts for each discipline and module within both programs.
Control and growth were implemented through a multitude of complementary tools all throughout the third and the fourth stages of the proposed PDCA cycle (Figure 1). Control meant following highly standardized quality procedures imposed by national regulatory structures [32,33], as well as European guidelines [34]. Control also entailed the accurate planning and implementation of ongoing feedback tools and final student surveys. The main ongoing feedback tools were shared ideas and stories, questions, quizzes, polls, volunteers' "suggestion box" and direct dialog. Generally, ongoing feedback was deployed for individual tasks and used to improve activities on the spot. Final feedback was deployed as a survey and targeted the efficiency calculations of all program components, enabling structural improvements for future programs.
Control and growth were implemented through a multitude of complementary tools all throughout the third and the fourth stages of the proposed PDCA cycle (Figure 1). Control meant following highly standardized quality procedures imposed by national regulatory structures [32,33], as well as European guidelines [34]. Control also entailed the accurate planning and implementation of ongoing feedback tools and final student surveys. The main ongoing feedback tools were shared ideas and stories, questions, quizzes, polls, volunteers' "suggestion box" and direct dialog. Generally, ongoing feedback was deployed for individual tasks and used to improve activities on the spot. Final feedback was deployed as a survey and targeted the efficiency calculations of all program components, enabling structural improvements for future programs.
Considering the abovementioned, the sustainable framework was designed based on four central dimensions, namely: efficiency, effectiveness, growth and control. This was achieved by integrating the four dimensions with a continuous development PDCA (Plan, Do, Check, Act) cycle ( Figure 1).

Results
Design and implementation of such complex educational programs, which target highly rated results, came with several challenges.
Firstly, the developers of the programs aimed at designing more than a complex curriculum, but rather a highly efficient and rewarding learning experience for all participants involved. In this regard, the authors thoroughly analyzed the general objective of each learning module within the intensive program and proposed an extensive set of LO, aimed at ensuring high rates of content retention, amassment of positive learning experience, specific competence development and expression of student's beliefs of self-awareness, worth and value within their own professional growth environment (Appendix A-Tables A1-A3). LOs were defined based on Bloom's revised taxonomy, in accordance with the procedure described in the previous section and considering the 3-tiered approach of each program, namely: Methods and Structure, Professional Development, Overall Performance.
Secondly, the designed framework was aimed at being flexible in terms of adaptability to other domains and subjects and in terms of repeatability of obtained results. This meant the usage of theoretical and scientifical approaches in all PDCA cycle phases. Thus, based on the LO and module structure of both programs, a correlation matrix (Table 1) has been developed between all structural learning environment elements, namely: learn- Considering the abovementioned, the sustainable framework was designed based on four central dimensions, namely: efficiency, effectiveness, growth and control. This was achieved by integrating the four dimensions with a continuous development PDCA (Plan, Do, Check, Act) cycle ( Figure 1).

Results
Design and implementation of such complex educational programs, which target highly rated results, came with several challenges.
Firstly, the developers of the programs aimed at designing more than a complex curriculum, but rather a highly efficient and rewarding learning experience for all participants involved. In this regard, the authors thoroughly analyzed the general objective of each learning module within the intensive program and proposed an extensive set of LO, aimed at ensuring high rates of content retention, amassment of positive learning experience, specific competence development and expression of student's beliefs of self-awareness, worth and value within their own professional growth environment (Appendix A-Tables A1-A3). LOs were defined based on Bloom's revised taxonomy, in accordance with the procedure described in the previous section and considering the 3-tiered approach of each program, namely: Methods and Structure, Professional Development, Overall Performance.
Secondly, the designed framework was aimed at being flexible in terms of adaptability to other domains and subjects and in terms of repeatability of obtained results. This meant the usage of theoretical and scientifical approaches in all PDCA cycle phases. Thus, based on the LO and module structure of both programs, a correlation matrix (Table 1) has been developed between all structural learning environment elements, namely: learning objectives, content structure, pedagogical approach, learning styles, delivering formats, instructional strategies and assessment methods. To calculate the efficiency of the learning experiences based on the proposed matrix structure and validate the sustainability of the framework, authors used targeted surveys, AHP and VA. Thirdly, resource management (financial, material, human) was an intensive and particularly demanding process, which required specialized personnel and software tools in order to be accurately deployed. Resource management was done two months prior to the start date of both programs and it took into consideration all aspects of the programs, without being stringent to the target group. For example, high school students had 24-h tutoring and monitoring, using a shift schedule for volunteer tutors. The activity was presented to students as a part of the rewarding system, each group of students earning through their learning activities certain access codes to a variety of activities performed together with their group tutor.

Structural Development of Intensive Programs
Structural development of the STEM intensive programs was based on an eleven-step process, as follows: (1) Set the background and the rationale of the program; (2) State a clear mission for the program in the context of the prior established background; (3) Design the story and identify how all activities are going to unfold; Put in place an appropriate reward system, corelated with the story of the program; (4) Identify the modules which need to be introduced in the story so as to fulfill the stated mission; (5) Establish the role of each module within the programs' structure and amongst each other; (6) Design the learning objectives of each module using Bloom's revised taxonomy; (7) Develop the learning content which will allow students to reach the set objectives; (8) Select the most appropriate pedagogical approaches and instructional strategies in relation to the content and LO's; (9) Set learning style of each module or a mix of learning styles that will best ensure the accomplishment of the LO's; (10) Establish the most appropriate delivering format or a mix of them; (11) Put in place formative and summative assessment methods.
Based on the abovementioned process, before the implementation stage begun, the authors engaged in a proper resource management process and risk analysis.
A comprehensive list of learning objectives was defined in correspondence with the general goals of the framework and with the content and structure of each intensive program. Bloom's revised taxonomy [10,11] was the basis for learning objective structure. Learning objectives have been divided in two main categories, overall learning objectives (OLO) and content specific learning objectives. OLO are the same for both intensive programs, as they target the development of student competences in relation to the applied methods, content structure and overall learning framework performance (Appendix A- Table A1). Content specific learning objectives are defined and analyzed for each one of the two programs, as they address competences focused on learning modules, targeting specific knowledge areas and levels of improvement. For the secondary education program, a list of 31 content specific learning objectives (LO S ) were defined and analyzed based on 7-module curricula (Appendix A- Table A2). The tertiary education program was implemented based on 24 content specific learning objectives (LO T ), which were defined considering a 6-module curricula structure (Appendix A- Table A3). Due to the transdisciplinary characteristics of the tertiary modules, the content specific learning objectives have been defined in a non-hierarchical fashion, but rather in the order of the complementary activities within each module.
Based on the previous research conducted by the authors [6,7,19,20,35] and their professional experience, curriculum design considered the correlation of content module with the following: learning objective, pedagogic approach/instructional strategy, learning style, delivering format, assessment. The importance of considering the abovementioned elements when designing curriculum is also emphasized by results outlined by a variety of studies [3,17,36,37].
The proposed STEM framework focuses on blended learning, including learnercentered, and problem-centered design of curriculum content, thus, learning styles are of particular importance in these hypotheses. Furthermore, the design process of the framework appealed to eight learning styles based on psychologist Gardner's [38] theory of multiple intelligences, namely: visual; verbal; logical; auditory; social; intrapersonal; physical and naturalistic [20]. For maximum learning efficiency, these learning styles were paired with learning objectives, delivery formats, instructional strategies/pedagogical approaches, assessment methods and module content.
Considering the abovementioned, a correlation matrix was put together to show the general designed structure of the intensive programs (Table 1). The matrix was constructed using brainstorming and nominal group technique (NGT), corelated with resource (human, material, financial) requirements and availability of UPB. The brainstorming and NGT sessions were moderated by the authors and ideas were prioritized into hierarchical solutions. The results of the sessions were correlated with the pilot course results obtained in a previous step. Additionally, recommendations from teachers and trainers who previously implemented standard STEM subjects were gathered using interview sessions, either face-to-face, either using various media devices. Answers were recorded on interview data sheets and duplicates were eliminated. Ideas were ranked with the most important receiving a rank of 5, and the least important receiving a rank of 1. These methods were overlapped in order to minimize the bias that could have been generated throughout the process.
If deployed together for a similar program, the proposed eleven-step process ( Figure 1) and correlation matrix (Table 1) will ensure the repeatability of the learning experience, up to subjective reasoning and personal preference of individual participants. It is important to mention that the proposed matrix can be adapted to any STEM curriculum or any complementary learning subjects.

Efficiency Evaluation
Final feedback of both programs entailed the design and implementation of a custom survey, which facilitated efficiency evaluation in correlation to the overall and content specific LO. The unprocessed survey results are presented in Appendix B. Figure A1 shows the survey results after implementation of the STEM intensive program for secondary education, whilst Figure A2 refers to the results of the tertiary intensive education program. Structured for the two aforementioned LO categories, the survey focused on 20 main components (C i , i = 1, 20) which define the implemented activities ( Table 2). The components were defined by the authors based on the learning objectives, thus the divided structure in overall and content specific components. Additionally, components considered the content of each curriculum module, based on research conducted and validated by the authors in their own work [6,7,19,20,22,23]. The custom survey was piloted by the authors in both STEM intensive programs, and applied in the last day of implementation, before the final award ceremonies. The questions strictly followed the components' structure; thus, each survey was comprised of 20 items. In the last day of implementation, a QR code was provided to all STEM intensive programs participants, which opened on their mobile phones a Google Forms survey. Items with single and multiple answers were used. Authors received 104 answers for the tertiary educational program and 108 answers for the secondary intensive program.
Generally, each component was evaluated by active student participants on a 1 to 5 Likert-scale scale, 5 being Strongly agree/Very satisfied and 1 being Strongly disagree/Very dissatisfied [39]. Fifteen items used a 1 to 5 scale, four components used a 1 to 4 scale and one component used a Yes/No answer. In this last case, No got assimilated with value 1 and Yes with value 2.
Efficiency for each of the 20 components was calculated using the general Formula (1).
Resource investment was made for a maximum outcome; thus, the effort of each component (Effort i ) was considered reached if all students scored with the upper limit of the given scales.
where, J is the upper limit of the used scale for each i component; N-number of active participating students in the STEM intensive programs. Effect was calculated based on the students' feedback on each of the components, using a formula for each of the three scales, as follows: j·n j , i = 1, 4 ; 10, 12 ; 14, 20 ( 3) where, j is the used scale for each i component; j can take values within 1, 5, 1, 4 or 1, 2; n j -number of students which evaluated the i component within the j given scale, with a specific score. For example, the efficiency for component C1 in the tertiary intensive program was calculated using relation (6), considering the following (based on Appendix B- Figure A2):

•
The scale of C1 was from set from 1 to 5, thus J = 5; • There were 104 active participants in the tertiary STEM intensive program, thus N = 104; • The scale of C1 was from set from 1 to 5, thus j = 1, 5; • Number of students which gave a specific score within the j scale: 51 students scored with 5; 32 students scored with 4; 14 students scored with 3; 2 students scored with 2; 5 students scored the C1 component with 1.
In a similar way and using Formulae (1), efficiency was calculated for each component of the program, as presented in Table 2.
Given the complexity of the framework, each component efficiency was considered to have a different importance level within the program structure. Thus, a rigorous AHP methodology was applied to define the importance of each one of the 20 components, comparing one pair of components at a time. Authors considered AHP as the optimum method, as it reduces the predisposition to biases in the outcome of the decision-making process [40,41], the entire process requiring 190 comparisons between pairs of components.
Inconsistencies within AHP are considered acceptable if the consistency ratio has a value below 10% [42], which is the standard applied in the current framework. Goepel's AHP Online System was used to deploy the proposed tool [43]. The resulting weights of the evaluated components were based on the principal eigenvector of the decision matrix (Figure 2a). Component importance coefficient (c i ) was given by the Priority in the AHP Ranking (Figure 2b) which was done based on the 190 pairwise comparisons.
AHP consolidated results show a consistency ratio of 1.0% (Figure 3), which is well below the imposed limit, thus the inconsistencies within the model are acceptable and the calculated importance coefficients can be used in further calculations.
AHP Online System was used to deploy the proposed tool [43]. The resulting weights of the evaluated components were based on the principal eigenvector of the decision matrix (Figure 2a). Component importance coefficient (ci) was given by the Priority in the AHP Ranking (Figure 2b) which was done based on the 190 pairwise comparisons.
AHP consolidated results show a consistency ratio of 1.0% (Figure 3), which is well below the imposed limit, thus the inconsistencies within the model are acceptable and the calculated importance coefficients can be used in further calculations.   The overall efficiency of each STEM intensive program was composed by using the formulae below: where, EO-Overall efficiency of the program; EC1 ÷ EC20-Individual efficiencies of each component of the program; c1 ÷ c20-importance coefficients of individual efficiencies calculated with AHP. Using relation (7) and the results obtained in Table 2 and Figure 1, the overall efficiencies for the secondary (EOS) and tertiary (EOT) programs are given below: An efficiency value of over 80% is considered highly desirable, with only 20% possibility of improvement. As the tertiary program was implemented first, an increase in efficiency can be observed due to the amendments made based on the received feedback. There is still room for improvement, thus value analysis is undertaken to identify the specific program components which can be further amended.

Value Analysis
Value analysis (VA) is undertaken in order to address the relationship between highquality educational outcomes and expenditure. Thus, the value of each program component is analyzed in relation to its' correspondent cost. In this case the value weightings of The overall efficiency of each STEM intensive program was composed by using the formulae below: where, E O -Overall efficiency of the program; E C1 ÷ E C20 -Individual efficiencies of each component of the program; c 1 ÷ c 20 -importance coefficients of individual efficiencies calculated with AHP. Using relation (7) and the results obtained in Table 2 and Figure 1, the overall efficiencies for the secondary (E OS ) and tertiary (E OT ) programs are given below: An efficiency value of over 80% is considered highly desirable, with only 20% possibility of improvement. As the tertiary program was implemented first, an increase in efficiency can be observed due to the amendments made based on the received feedback.
There is still room for improvement, thus value analysis is undertaken to identify the specific program components which can be further amended.

Value Analysis
Value analysis (VA) is undertaken in order to address the relationship between highquality educational outcomes and expenditure. Thus, the value of each program component is analyzed in relation to its' correspondent cost. In this case the value weightings of the components are assimilated with their individual importance coefficients established previously using AHP and are further on noted with x i . Costs are given by the financial plan of each project and the co-financing plan of UPB and their weighting are further on noted with y i . VA uses the smallest squares method to accurately analyze the relation between components costs and value [31]. The smallest squares method leads with the hypothesis that an estimator S should tend to a minimum value, while an S' dispersion should tend to zero. In order to do so, the components value and costs must be in perfect balance. in other words, value and costs weightings of all 20 designed components should tend to be perfectly aligned along a regression line. VA aims to reevaluate and redesign components that are above the regression line, by either increasing their value or decreasing their costs.
Further on VA is applied to both the secondary (Table 3, Figure 4) and the tertiary (Table 4, Figure 5) STEM intensive programs. As the two intensive programs were designed based on the same framework, the regression parameter a and angle α are very close in value, but components are distributed differently alongside the regression line (Figures 4 and 5).  VA parameters have been obtained after processing the information in Table 4: Based on the above computational elements, the regression line = 0.917364 is plotted in Figure 5, alongside all 20 components and their importance in value and costs.   VA parameters have been obtained after processing the information in Table 3:

Discussion and Conclusions
Research results show that designing a sustainable learning framework for STEM intensive programs requires extensive use of high-quality technology, online interactive tools and face-to-face guidance. These findings are in accordance with previous conducted research, which show that amongst the most significant predictors of blended learning effectiveness are content design features, technology quality, online tools and trainer support [18,44,45].
The developed framework includes project-based, team-based and blended learning approaches, with the aim of targeting the proven benefits of each one [46][47][48][49]. The sustainable framework in STEM intensive programs for secondary and tertiary is constructed based on a PDCA cycle with eleven main steps: 1. Background and the rationale of the STEM intensive programs should be defined in correlation with the existing requirements; 2. The mission of the STEM intensive programs must be clearly stated considering the previously established background; 3. A Gantt chart should be set up defining all activities and necessary resources, accompanied with the story of the programs; An appropriate reward system, corelated with the story of the program must be detailed; 4. The most appropriate modules which can fulfill the stated mission should be identified and puzzled together within the storyline of the curriculum; 5. Each module should have a specific role within the programs' structure and amongst each other; 6. Module LO's should be designed based on Bloom's revised taxonomy; 7. Learning content is developed in close correlation with the LOs, which will allow participants to reach the set objectives; 8. Pedagogical approaches and instructional strategies are selected in relation to the content and LO's; 9. Learning style is selected for each module or a mix of learning styles that will best ensure the accomplishment of the LO's. 10. The most appropriate delivering format or a mix is established; 11. Formative and summative assessment methods are selected and prepared in correlation with module content and LO's. The eleven-step framework is divided in four tiers, as shown in Figure 1.
In order to validate the framework structure and content, alongside with its benefits, efficiency evaluation was constructed and deployed. Both of the obtained overall efficiency values are acceptable but show that there are still components which need improvement. Both programs registered their highest component efficiency values for C13-Recommend program to a colleague. This shows that the overall learning experience of both target groups were positive within the designed framework. The lowest registered component efficiency value in the secondary program was 68.7% for C19-Enhancement Based on the above computational elements, the regression line y = 0.91138x is plotted in Figure 4, alongside all 20 components and their importance in value and costs.
VA parameters have been obtained after processing the information in Table 4: Based on the above computational elements, the regression line y = 0.917364x is plotted in Figure 5, alongside all 20 components and their importance in value and costs.

Discussion and Conclusions
Research results show that designing a sustainable learning framework for STEM intensive programs requires extensive use of high-quality technology, online interactive tools and face-to-face guidance. These findings are in accordance with previous conducted research, which show that amongst the most significant predictors of blended learning effectiveness are content design features, technology quality, online tools and trainer support [18,44,45].
The developed framework includes project-based, team-based and blended learning approaches, with the aim of targeting the proven benefits of each one [46][47][48][49]. The sustainable framework in STEM intensive programs for secondary and tertiary is constructed based on a PDCA cycle with eleven main steps: 1. Background and the rationale of the STEM intensive programs should be defined in correlation with the existing requirements; 2. The mission of the STEM intensive programs must be clearly stated considering the previously established background; 3. A Gantt chart should be set up defining all activities and necessary resources, accompanied with the story of the programs; An appropriate reward system, corelated with the story of the program must be detailed; 4. The most appropriate modules which can fulfill the stated mission should be identified and puzzled together within the storyline of the curriculum; 5. Each module should have a specific role within the programs' structure and amongst each other; 6. Module LO's should be designed based on Bloom's revised taxonomy; 7. Learning content is developed in close correlation with the LOs, which will allow participants to reach the set objectives; 8. Pedagogical approaches and instructional strategies are selected in relation to the content and LO's; 9. Learning style is selected for each module or a mix of learning styles that will best ensure the accomplishment of the LO's. 10. The most appropriate delivering format or a mix is established; 11. Formative and summative assessment methods are selected and prepared in correlation with module content and LO's. The eleven-step framework is divided in four tiers, as shown in Figure 1.
In order to validate the framework structure and content, alongside with its benefits, efficiency evaluation was constructed and deployed. Both of the obtained overall efficiency values are acceptable but show that there are still components which need improvement. Both programs registered their highest component efficiency values for C13-Recommend program to a colleague. This shows that the overall learning experience of both target groups were positive within the designed framework. The lowest registered component efficiency value in the secondary program was 68.7% for C19-Enhancement in physics competences. This result was discussed during the final feedback session and it was concluded that the variety of problem types, problem formulation and scenario development must be improved. The tertiary program recorded its' lowest component efficiency value (80.76%) for C17-Enhancement in entrepreneurial competences. Although over the threshold of 80%, results and feedback show that this component lacked in diversity of best-practice applications, the prevalent delivering formats being interactive lectures and seminars.
It has been demonstrated that AHP is dominantly applied in higher learning institutions for measuring quality education with many benefits [29]. Within our research, AHP was used to prioritize the components within the designed framework and eliminate any bias which might have occurred during the process up to that point. Thus, overall efficiency was calculated with high accuracy. The overall efficiencies for the secondary (E OS -88.41%) and tertiary (E OT -85.57%) programs registered values over the desired threshold, but with almost 15% possibility of improvement. Improvement of overall efficiency of the framework targets a feedback adapted redesign of the structural development for both STEM intensive programs. Identification of specific components to be improved was undertaken using VA, in accordance with LD Miles adapted method [30,31]. The method was chosen due to its long-time proven results on efficiency improvement and cost savings [50]. The objective of VA was to redesign and diminish costs or increase the value for the components corresponding to the points above the regression line. By changing those specific points and re-plotting the diagram, the slope of the regression line modifies, and a new situation of VA can be evaluated. The process is usually iterative, and it is undertaken until both the customer and manufacturer requirements are met. VA of the STEM intensive programs revealed common components which needed readdressing, either, in terms of value increase, either, in terms of cost reductions. The common identified components were C6, C8 and C13, all of them target content and delivery format structure and organization. Overspending on these three components influenced the regression angle for both analyses. In this case, the preferred approach to obtain a better fit onto the regression line, will be value increase through the following: pre-program target group assessment; use of advanced interactive devices during the learning experience; live feedback paired with on the spot adaptive delivery formats; improvement in participative learning techniques. The most unbalanced component within the STEM intensive secondary program is C10-Friendliness and involvement of speakers and trainers. This component received a very poor evaluation from the target group, probably since they were unfamiliar with the trainers and the higher education approach. For the STEM intensive tertiary program, the target group previously knew the trainers and even had different courses together. Future improvement of this component involves both reducing costs and improving value, by hiring high school professors who are already familiar with the target group. All other program components that are above the regression line of the secondary program analysis will be targeted with cost optimization tools [51]. Responsiveness and helpfulness of trainers and speakers with provided information-C9, was the component which registered the poorest value-cost ratio within the tertiary program analysis. With a difference of 3 points, this component is the most unbalanced due to low value attribution. In this regard, in accordance with other significant research [36,52] we conclude that all endeavors of technology integration, curriculum development and learning environment enrichens should take place as student-centered approaches, but only with consideration of trainers' background, competences and beliefs [53,54]. Additionally, in support of improving the efficiency and increasing the value of this component, an artificial intelligence assistant is intended to be developed and used for all implemented modules. This is a relatively new addressed avenue for sustainable curriculum planning [37,55]. Other components within the tertiary regression analysis will be addressed through cost reduction.
It is important to notice that the content specific components have the best fit onto the regression line, for both of the STEM intensive programs, showing that the value to cost ratio was designed appropriately. Another noticeable aspect is that the regression points compiled for the tertiary program have a better fit alongside the regression line, than those calculated for the secondary program. This shows that although the efficiency of the secondary program was higher, the cost-value distribution still needs improvement.
Implemented using a mixed educational approach between existing methods and tools [38,56,57], the currently proposed framework is expected to be a sustainable approach to STEM intensive programs for secondary and tertiary education, with high potential of reproducibility.
Furthermore, UNESCO highlights [1] that learning objectives play a key role in sustainable development goals, but their inclusion is still under exploited by education systems. To align with this finding, the sustainable framework based on a PDCA cycle for intensive educational programs (Figure 1) identifies the learning objectives, together with structure development tools and methods, early on within the first stage. Thus, results obtained by the current study with VA in C5 component, shows that the proposed framework was well above average (90.28%-secondary and 82.69%-tertiary) in delivering clear learning objective-based content and structure.
The added value of the research is showcased by its' innovative approach, whilst addressing the limitations of other studies identified in the introduction of this research. The innovation aspect is given primarily by the following: definition and calculation of a new efficiency indicator for STEM intensive educational programs; correlation between the efficiency analysis and budget expenditure for STEM programs; efficiency evaluation of an integrated curriculum approach, which considers multiple design variables; design of a STEM interdisciplinary approach which considers work experience exposure in correlation with future integration of participants into the workforce market; design of a scalable and reproductible sustainable framework; correlation between secondary and tertiary design frameworks in order to construct an integrated approach for curriculum design in STEM intensive programs; aggregation into a single curriculum structure of theoretical courses with practical workshops, aiming at a mix of competencies and abilities acquired by the participants in order to increase the employability. The conducted research is a direct response to the identified need of an integrated approach of STEM field subjects, focusing on alleviating early dropout rates due to limited practical work experience exposure.
Unforeseen additional implications based on the study results considered mainly aspects related to personal experience of target group participants and trainers. Some of the most important are: students were much more receptive to learning content, up to overstaying the established schedule and demanding additional activities; learning content retention rate was improved due to the continuity of the taught concepts compared to traditional approaches; for pupils, the participation in the program was an insight into an engineering education; trainers from different universities expressed the intent to apply the framework within their home institutions.
Limitations of the study include the quantitative focused analysis, to the detriment of a full comprehensive qualitative analysis. This was a conscious choice of the authors, providing that the current quantitative approach allows better reproducibility of the research results, consequently a very similar implementation of the proposed framework. Additionally, the research did not include a preliminary study of efficiency levels for the 20 components, as this would involve a highly complex evaluation matrix, since the proposed STEM intensive programs are inter-and multidisciplinary. Currently, based on the structure of the STEM intensive programs and on the obtained results of the present study, the authors are designing a framework to evaluate a compound efficiency of individual disciplines, as they are studied in the regular curriculum of secondary and tertiary education programs, such as: mathematics, physics, informatics, 3D printing, CAD etc. In the future, this will be the basis of a comparison evaluation in performance increase.
Another mentionable limitation is the fact that AHP is a labor-intensive tool and for a large amount of analyzed criteria (e.g., 20 components) requires special training of the expert in order to obtain a consistency ratio under 10%. This should be taken into consideration when replicating or scaling the proposed framework.
Further research involves the transition to an online and mixed teaching environment, by adapting the content and framework structure with the aid of smart learning environments. Further assessment is needed in order to address the changing needs of the educational system, currently going through major transformations, due to the SARS-CoV-2 epidemic. In the short term, the online adapted framework will be tested and evaluated during the second STEM intensive program of TecHUB 4.0, in the summer of 2021. Results of the adapted framework will be compared with the currently presented programs and feedback will be considered for improvement. In the medium term, authors consider applying and testing the proposed framework within STEM master's programs in the field of industrial engineering (http://www.imst.pub.ro/Upload/2020/Planuri_de_invatamant_ 2020-2021/LICENTA/IELicenta2020. Acknowledgments: Authors would like to acknowledge the contribution of University POLITEHNICA of Bucharest for the support given in the form of infrastructure and other resources not covered through the funded projects.

Conflicts of Interest:
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.
Appendix A Table A1. Overall Learning Objectives of intensive programs for secondary and tertiary education (OLO).

OVERALL LEARNING OBJECTIVES
OLO 1. Students will be able to distinguish and evaluate the information structure of teaching activities and support materials, during each program module.
OLO 2. Students will be able to describe three main attractiveness characteristics teaching activities and support materials. OLO 3. Students will be able to illustrate one real-life scenario in which they use the support materials provided through the program. OLO 4. Students will be able to apply at least one interactive tool during each program module.

OVERALL LEARNING OBJECTIVES
OLO 5. Students will be able to identify at least one learning objective per deployed activity, during each program module. OLO 6. Students will be able to cover all topics within the program content, in a relaxed timeframe and logical information flow. OLO 7. Students will be able to execute all given tasks within the activities' given timeframe. OLO 8. Students will be able to evaluate specific characteristics of speakers and trainers' performance in relation to module content. OLO 9. Students will be able to defend the main advantages of undertaking the program in relation to their current and future activities and work. OLO 10. Students will be able to rate the usefulness of the practical lectures and workshops implemented throughout the summer school, by using on a 1 to 5 scale. OLO 11. Students will be able to assess the appropriateness of the program for other peers. OLO 12. Students will be able to evaluate the general quality of the event organization by the end of the program. Table A2. Specific Learning Objectives of the intensive program for secondary education (LO S ).

SPECIFIC LEARNING OBJECTIVES
LO S 1. High school students will be able to remember information already taught in high school regarding the mathematics curricula.
LO S 2. High school students will be able to select the most appropriate solving method for a specific math problem type, during one session of mathematics activities. LO S 3. High school students will be able to solve problems from the following mathematics topics: geometry and trigonometry, algebra, and mathematical analysis. LO S 4. High school students will be able to demonstrate working hypothesis and principles of math problems within the following topics: geometry and trigonometry, algebra, and mathematical analysis. LO S 5. High school students will be able to explain the working principles applied in: Mechanics, Thermodynamics, Production and use of direct current and Optics. LO S 6: High school students will be able to identify a specific type of solving method for given problems within specified physics areas. LO S 7. High school students will be able to solve problems and demonstrate hypothesis and principles from the following areas: Mechanics, Thermodynamics, Production and use of direct current and Optics. LO S 8. High school students will be able to understand the logical structure of specific problem type, during the initial program sessions. LO S 9. High school students will be able to correlate between given problems and the most appropriate software structure or a combination of them, during the informatics activities. LO S 10. High school students will be able to build software applications using conditionals, loops and other data structures (if, for, case, while and else). LO S 11. High school students will be able to demonstrate the functionality of built software applications using conditionals, loops and other data structures. LO S 12. High school students will be able to understand the 3D modelling software environment of Solid Works. LO S 13. High school students will be able to identify the main volumes, sketches and 3D operations (Extrude, Revolve, Sweep, Loft) which constitute a virtual model. LO S 14. High school students will be able to design at least two 3D models of real-life products using Solid Works software application. LO S 15. High school students will be able to generate 2D drawings for designed parts; LO S 16. High school students will be able to generate at least two photorealistic renderings of products, using Solid Works software application.
LO S 17. High school students will be able to identify appropriate 3D printing principles for given applications. LO S 18. High school students will be able to select material and equipment for at least two 3D printing applications.

SPECIFIC LEARNING OBJECTIVES
LO S 19. High school students will be able to validate build plate layout and 3D printing parameters in correspondence with product function. LO S 20: High school students will be able to optimize build plate layout and process parameters for given application. LO S 21. High school students will be able to 3D Print at least two given applications. LO S 22. High school students will be able to perform post-processing steps and demonstrate functionality of their final products. LO S 23. High school students will be able to understand the basic working principles of electronics, during the first mechatronics activity.
LO S 24. High school students will be able to correlate between the specific Arduino code structures and given tasks. LO S 25. High school students will be able to design simple circuits using Arduino boards and three options of sensors. LO S 26. High school students will be able to design Arduino code structures, compile and test run them on own circuits. LO S 27. High school students will be able to identify the characteristics of an IR contrast sensor. LO S 28. High school students will be able to plan the main displacements of an IR equipped robot. LO S 29. High school students will be able to develop an obstacle racetrack for a preprogramed robot. LO S 30. High school students will be able to demonstrate the working principle of an IR contrast sensor. LO S 31. High school students will be able to apply gained knowledge and demonstrate functionality of the developed robotic product. Table A3. Specific Learning Objectives of the intensive program for tertiary education (LO T ).

SPECIFIC LEARNING OBJECTIVES
LO T 1. Students will be able to identify at least two career paths by the end of the intensive program.
LO T 2. Students will be able to remember the information already learnt about manufacturing technologies, during the first module of the intensive program. LO T 3. Students will be able to identify main AM technologies for business applications in a Start-up environment.
LO T 4. Students will be able to explain the advantages of 3D printing in creating an individual Start-up. LO T 5. Students will be able to generate at least 4 3D printed product concepts within a specific topic of a Start-up company. LO T 6. Students will be able to manufacture at least 4 prototypes using 3D printing technologies. LO T 7. Students will be able to showcase the particularities of a 3D printing Start-up through best products and services. LO T 8. Students will be able to define the mission of a 3D Printing Start-up. LO T 9. Students will be able to carry out needs' analysis for customers of a 3D Printing Start-up. LO T 10. Students will be able to build products specifications for a custom 3D printed product range. LO T 11. Students will be able to assess financial aspects related to the 3D Printing Strat-up. LO T 12. Students will be able to manage operational aspects regarding necessary resources for a 3D Printing Start-up. LO T 13. Students will be able to test the functionality of the manufactured prototypes.