Assessment of Curriculum Design by Learning Outcomes (LO)

Abstract

This paper analyzes the assessment experience as part of curriculum design by learning outcomes of the Master in User Experience Design of the Universidad Nacional Abierta y a Distancia—UNAD Colombia and the University of Lleida—UdL Spain. The article presents the assessment route, which allows for continuous improvement and is tailored to the self-assessment process. Conceptual references on curriculum design, competencies, purposes, constant improvement, and assessment are outlined for presentation. The theoretical line is based on international and national legal references. Likewise, the educational, pedagogical, and curricular implications of learning outcomes are presented, among them: change of paradigm (teaching vs. learning), coherence of curricular design, change of evaluation (qualification vs. assessment), decision-making, professor training, change of professor attitude, sustainability through assessment, and implementation routes, all of them with the aim of continuous improvement and to maintain the high quality of the program. One of the main conclusions indicates that curriculum design based on learning outcomes should be aligned and coherent at the macro-, meso-, and micro-curricular levels in order to meet the needs and requirements of the professional field.

1. Introduction

The design of a curriculum proposal based on learning outcomes (LO) goes beyond compliance with current regulations and the evaluation criteria of national or international program accreditation and certification agencies. Curriculum design must be framed within the institution’s commitment, interests, and particular needs to comply with its mission and vision. This task requires the committed work of all actors directly involved: professors, students, administrators, graduates, employers, and consulted experts. According to relevance and need, these actors are organized in networks or other forms of participation.

The approach to a curriculum design based on LO has implications of various types, which are rarely mentioned in shared work experiences or within academic communities. This article highlights these implications while sharing an evaluation experience resulting from the LO curriculum design process of the User Experience Design Master’s Degree, a joint degree program offered between the Universidad Nacional Abierta y a Distancia de Colombia UNAD and the University of Lleida in Spain. This study was conducted to support the construction and development of a program under the LO approach, taking into account national and international regulations and the progress made in other areas and programs that have implemented LO.

A study of the curriculum design to be applied to the Master’s Degree in User Experience Design in computer engineering was conducted, considering national and international references in curricula and LO, the regulations of the Ministry of Education of Colombia and the Ministry of Universities of Spain, and the accrediting agencies of the two countries, to uncover all the requirements to offer the master’s degree program in both countries.

The research aimed to identify the conceptual basis of the LO-based curriculum design approach so that this design could be applied in the design of the master’s degree program. The paper is structured as follows: methodology, background, LO-based curriculum design, program correspondence, assessment cycle for the master’s program, and conclusions.

2. Methodology

This study’s objective was to uncover the conceptual basis for designing master’s degree programs under the LO approach. From this perspective, a systematic mapping of the literature was proposed to identify the guidelines of Colombia and Spain in the construction of master’s degree programs. Therefore, the general question was: What are the guidelines and references for designing programs based on LO?

The study was designed using guidelines defined by the Colombian and Spanish governments for program design. In searching for the procedures of international organizations that accredit the different programs and the development of other studies and programs built from LO perspectives, a systematic mapping was made to carry out this study. Once all the concepts, guidelines, certifications, and different cases of contributions were obtained, the proposal of the assessment and implications of curriculum design by LO was proposed.

A systematic mapping of the literature [1] was carried out to search for national and international references through learning outcomes evaluation. The proposal was developed to evaluate master’s programs using learning outcomes. Following the Petersen methodology, the following phases were defined: definition of research questions, keywords, databases, inclusion and exclusion criteria, query search, and results.

The purpose of the systematic mapping was to identify the bases and references of the evaluation of curricula through learning outcomes. For this purpose, two questions were defined: Q1: How do we assess the LO-based curriculum? and Q2: What are the international referents in assessment of LO-based curricula?

Keywords of the systematic mapping are in English and Spanish: English: learning outcomes, curriculum design, assessment, and quality; Spanish: resultados de aprendizaje, diseño del curriculo, evaluación, and calidad.

According to the mapping of the systematic review, we defined four (4) databases to search for information because they were the most internationally recognized in engineering, computer science, and education. These databases are Dimensions, SCOPUS, Science Direct, and Google Scholar.

The systematic mapping inclusion and exclusion criteria of the review were defined according to the search of research questions.

The inclusion criteria were: (1) papers published between the years 2017–2022; (2) papers published in conferences, journals, and book chapters; (3) papers written in English and Spanish; (4) papers that were found in the databases with details; and (5) papers related to learning outcomes, curriculum design, assessment, and quality in English and Spanish. The exclusion criteria were: (1) papers not available for download; (2) papers in languages other than English and Spanish; (3) papers that were not focused on keywords, (4) gray literature; and (5) papers duplicated in the databases or the other databases.

A general query search was defined based on the general concepts that allowed answers to the research questions. Each database was reviewed regarding how to perform advanced searches and we created a query search for each one, allowing for more specific results according to the keywords raised. The results were the general queries as follows: (((“Learning Outcomes”) OR (“Curriculum Design”)) AND (“Quality”) AND (“Assessment”)) AND (publication year > 2017)).

The results were obtained through systematic mapping, searching papers in each database, and applying the inclusion and exclusion criteria. First, each paper’s title, abstract, and conclusion were reviewed to decide whether the paper was accepted or rejected. Then, once it was determined whether the papers were accepted, a detailed review was carried out to determine whether they should be included in the systematic mapping of papers.

A total of 2278 papers were obtained from the databases, of which 32 were accepted and 2246 were rejected. Expressed in in percentage, 1.4% were accepted and 98.6% were rejected after applying the inclusion and exclusion criteria and the final detailed review of the papers defined in the systematic mapping (see Figure 1).

The answers to the questions are presented as follows:

Q1:

How to assess an LO-based curriculum?

Papers propose curricular design and strategic planning as an educational innovation. In contrast, the LO is presented as an innovative strategy and a heuristic for evaluating program learning outcomes. An exciting paper that we found is a methodology for developing learning outcomes and a paper that relates the programs’ learning outcomes and learning objectives. Finally, a paper that presents the continuous improvement of the programs based on the assessment of learning outcomes. It can be concluded that there are different proposals on how to carry out evaluations based on LOs; in this sense, an evaluation process based on the LOs of the curriculum is proposed. LO can include different forms of blended learning, the application of IA to measure and improve, and considering that students are at the center of these learning processes; without them, it is impossible to carry out the evaluation.

Q2:

What are the international referents in the assessment of LO-based curriculum?

The papers on the national and international referents construction and evaluation of the LOs of the curricula were found. The Accreditation Board for Engineering and Technology; the Accreditation Agency Specialized in Engineering, Computer Science, Natural Sciences and Mathematics Programs; the European Accredited Engineer; the National Agency for Quality Assessment and Accreditation of Spain; the Ministry of National Education; and the National Council of Higher Education CESU of Colombia are private or public organizations that work on the guidelines and evaluations of LOs.

Once the systematic mapping was complete, references were found in the literature for guidelines of the governments of Colombia and Spain, accreditation agencies, and authors who have worked on the topic of program design using an LO approach.

3. Background

The background section is based on the papers found in the systematic mapping and the documents of the institutional and government guidelines of the program, the inclusion of the international agencies used to assess the LO of the programs, programs studies related to the curriculum and LO, and the different proposals that include LO in the curriculum. In this order of ideas, it is necessary to establish conceptual definitions to unify criteria and understand the paper.

3.1. Learning Outcomes (LO)

Different denominations refer to learning outcomes (LO) and even some international agencies such as ABET (Accreditation Board for Engineering and Technology) [2]. Furthermore, Germany’s ASIIN (Accreditation Agency Specialized in Programs of Engineering, Informatics, Natural Sciences, and Mathematics) [3] differs in their denomination: ABET calls them student outcomes, and ASIIN calls them LO.

For the European Accredited Engineer [4], the term program outcomes describe the knowledge, skills, and competencies that graduates of accredited engineering degree programs must demonstrate. For ANECA, National Agency for Quality Assessment and Accreditation of Spain, the LO are concretions of the competencies; they describe what the student can do at the end of the training process or subject. The program learning outcomes (PLO) constitute the graduates’ knowledge, competencies, and attitudes that are expected. In contrast, the course learning outcomes must contribute to the program learning outcomes and be aligned with them. Therefore, program or course outcomes are expressed in competencies [5]. However, regardless of their denomination and definition, the various conceptualizations agree that they are precise statements through which training programs declare what a student is expected to be able to learn at the end of their educational process.

In Colombia, in Decree 1330 of 2019, the Ministry of National Education defines LO as the express statements of what a student is expected to know and demonstrate when completing their academic program. Likewise, it places LOs as a factor of self-evaluation, stating that such results must be articulated with the program’s graduation profile and the training needs in the discipline [6].

Similarly, Agreement 02 of 2020 of the National Council of Higher Education CESU of Colombia establishes that the LOs are express statements of what a student is expected to know. Moreover, students must demonstrate at the time of completing his/her academic program that they are capable of improvement, which is a process allowing for knowing the degree of compliance of the student’s learning. Likewise, they must make adjustments to the curricular strategies since their formulation must be aligned with the trends of the disciplines, the training profile, the nature of the program, the training level, the modality, and international standards [7].

Subsequently, Resolution 021795 of 2020 establishes that the LOs refers to what the student will know, understand, and be able to do as an integral result of their formative process; therefore, they favor student-centered teaching and learning processes [8].

The different legal guidelines in Colombia outline a possible horizon of curricular construction and indicate that LOs are not an addition to the curriculum and are actually a fundamental part of its design; therefore, they require planning, articulation, coherence, relevance, and evaluation.

Therefore, UNAD in its Academic Statute defines LOs as: “The explicit and precise statements of the conceptual, procedural, technological, disciplinary and con-textual performances that students are expected to achieve at the end of their training process, both in the courses and in the program. They dynamize the alignment between the training intentions, contents, learning strategies and activities with the evaluation strategies, in coherence with the competencies and the graduate profile established in the curricular design by the institution” [9].

It is necessary to specify that within the institutional guidelines of UNAD, the formulation of LO must be related and coherent with the Integrating Problem Nucleus of the program; contribute to the construction of the competencies of the Problem Nucleus to which the course belongs; contribute to the achievement of what is formulated in the training purpose; be coherent with the described learning strategy; and be proven through the activities suggested in the course [10].

Based on the above, the Master in User Experience Design (MDUX) formulates its PLOs for each of the eight learning areas proposed by EUR-ACE, namely: Knowledge and Understanding, Engineering Analysis, Engineering Design, Research, Engineering Practice, Opinion Formulation, Communication and Teamwork, and Lifelong Learning [4].

The study of the Governance, Policy, and Strategy of Learning Outcomes Assessment in Higher Education proposed emphasizing the need for ongoing reviews in this field to clarify policy relevance and opportunities [11]. Studies such as [12] propose the inclusion of AI artificial intelligence in the development of students’ LOs; in this particular case focuses on a business program, where the results obtained by the students concerning the LOs applying AI are evaluated. Hall [13] proposes a three-part method that allows teaching, learning, and assessing LOs using traditional education, simulation, and authentic experiences; these three factors contribute to the proposed LOs, in this specific case, in construction. Finally, the literature review on Assessment of Student Learning Outcomes in higher education and academic libraries raises the starting point of LOs from the students’ perspectives and how they have become a fundamental part of the teaching and learning processes [14].

3.2. Competencies

For UNAD, competencies refer to articulated sets of knowledge, capacities, abilities, skills, attitudes, and aptitudes that the institution, through its formative programs, expects students to develop and strengthen. For example, to enable the understanding and analysis of problems or situations and to act ethically and socially responsibly in different contexts. The competencies will be manifested in work or study situations and professional and personal performance [10].

The competencies are formulated from the program and are based on the constitutive elements of the macro curricular design. The formative intentions at the institutional and school levels are considered—from the meso-curricular design—decisions are made based on international, national, and local references regarding professional training developments. It integrates the core problems, training profiles, competencies, and the curricular mesh the student must follow in his training process. Each course structure is defined in articulation and coherence with the macro and me-so-curricular levels from the micro-curricular design.

3.3. Training Purposes

The training program’s purposes are the projected intentions for the graduates to show after graduation and in their occupational performance fields. Therefore, the purposes are stated in terms of the expected and the long-term intention that materializes through the academic route. That the student follows this plan until graduation must be monitored to understand to extend the program’s actions to allow their achievement or not.

In the case of the syllabus, purposes are formulated by the professor thinking of the course work’s formative intentions. Thus, the objectives are related to the productive work of promoting learning, the didactic design, and the actions to achieve the purpose of student learning.

3.4. Curriculum Design

Understanding curriculum design, it is necessary to address the concept of the curriculum since there are as many definitions as there are authors. For this paper, the curriculum will be understood according to Lozano and Lara’s definition [15]. They define it as a field of research and critical analysis of reality, allowing for processes committed to generating more consistent and coherent relationships between knowledge and appropriate actions for its social use. Studying specific curricula means learning them in the context in which they are configured and through which their expressions of educational practices and results.

Given the above, based on LO, the curriculum must respond to an effective organization and coherence, allowing students to achieve effective learning and allowing the curriculum designers to make decisions in favor of continuous improvement towards quality educational processes. Therefore, each institution will select the best curricular organization option for its needs, whether a curriculum organized by subjects, training fields, problems, projects, cores, areas, modules, or units. Depending on this first decision, the curriculum design implies, among other aspects, keeping coherence and relevance among the different elements that constitute it since this coherence and relevance is what, to a large extent, will make its execution and sustainability possible.

Facilitate the approach to curriculum design: this process is divided into three main phases: planning, operation, and evaluation of the curriculum. Planning is a fundamental phase because the procedure depends on it, and evaluation can also be foreseen from theoretical planning [16].

Curriculum design is a curriculum dimension that reveals the methodology, actions, and results of the diagnosis, modeling, structuring, and organization of curriculum projects. It prescribes a specific educational conception that, when implemented, aims at solving problems, moving needs, and, in its evaluation, enables the improvement of the teaching-learning process [17].

The curricular design includes educational, pedagogical, didactic, human, and social conception. Its formative intentions, the training profiles, implementation forms, the LOs, the problems to which it will respond, the curriculum, the assessment mechanisms, and the possibilities of flexibilization according to the new demands that arise in its achievement.

As of Decree 1330 of 2020 issued by the Colombian MEN, all training programs, to obtain their qualified registration, must design, in the case of new programs, or redesign, in the case of programs in operation, their curricular proposals based on LO in alignment with the other aspects that make up the curriculum.

The design is the initial phase of the curriculum; therefore, its reality is represented in it, but there must be continuity, coherence, and consistency between design, management, and evaluation as processes that guarantee its full implementation [18].

3.5. Continuous Improvement

Continuous improvement is a term born in the business world articulated to the total quality processes [19,20], which many organizations implement to optimize their operations and increase productivity.

It is a process that describes the essence of quality very well and reflects what companies need to do if they want to be competitive over time. The importance of this management technique lies in its contribution to improving the organization’s weaknesses and strengthening its strengths. As a result, continuous improvement achieves greater productivity and competitiveness [21].

On the other hand, Seminario [22] states that continuous improvement in companies is the ongoing effort to improve products, services, and processes through minor incremental improvements. It is based on the belief that these gradual changes will add to significant improvements over time.

The term continuous improvement has been used in educational institutions to refer to all processes, tasks, strategies, actions, and action plans carried out within the institution to promote academic quality processes. These processes focus on the educational service and how it meets the different demands of legal, social, and institutional order.

Thinking about the continuous improvement of education implies, from the curricular design, establishing the bases of self-regulation and external evaluation that will allow the institution to fulfill its formative intentions and the promise of value offered to society in the training of the different professionals through their graduate profiles. To speak of continuous improvement of the quality of education as one of the assessment purposes implies total coherence of the curricular design in articulation with the will of the top management and all the actors involved in the educational process.

4. Proposed LO-Based Curriculum Design

LO based on curriculum design has educational, pedagogical, and curricular implications and affects all those involved in these processes. The proposed approach aims at improving the quality of education and, for this purpose, must consider continuous improvement. Considering the above, the elements that should be in the design of the curriculum based on LO: the change of attitude of all the roles involved, the evaluation and sustainability of the assessment, the implementation route, the curricular correspondence, the shift in paradigm, the coherence and relevance of the curriculum design, the change in evaluation, informed decision making, and the training of professors. All of these elements are related from the perspective of LO-based curriculum design. These implications are in Figure 2.

4.1. Paradigm Shift: From Teaching to Learning

One of the first implications is the paradigm change from a formative process centered on teaching to a strategy centered on learning. Then, some professors were more concerned about what and how to teach, focusing their planning, from preparing classes to selecting the teaching method, the materials, and resources. Today, thinking about the curriculum from LO implies concentrating on learning and not on teaching. The professor is a designer of the learning strategy that allows the student to achieve the proposed results. The professor identifies the student learned at what level and under what conditions.

This paradigm shift brings extraordinary challenges to higher education that must be faced by professors and the different levels of institutional decision-making. Thinking about learning-centered formative processes implies, among others, knowledge of learning styles, knowing how learning takes place, the role to be played by the student and the professor, the selection of educational resources, and evaluation strategies.

Knowing the learning styles of students and professors is a starting point for curriculum design, classroom planning, and decision-making regarding learning strategies, activities to be developed, and evaluation processes to enhance learning and implement corrective actions, all within the framework of improving educational quality [19].

Planning a course focused on learning implies defining the CLOs, the activities to achieve knowledge, the evaluation criteria for such understanding, the performance levels, and the evidence to identify what the student has learned.

4.2. Coherence and Relevance in Curriculum Design

Addressing the LO as an addition to the curricular design is one of the most frequent errors that have been evidenced since Decree 1330 of 2019. Since the academic programs, which had qualified registration or accreditation in force, formulated such results by adjusting them to their curricular proposals to comply with the regulation.

The LO must be the articulating axis of the curricular designs from their different levels: macro-, meso-, and micro-curricular.

The macro-level reflects the coherence between the teleological and axiological framework. The institutional level is formed with the human being, society, and the community. Consider the normative frameworks and the different contexts. The substantive functions expressed in the institutional guidelines guarantee the formation’s pertinence and globality.

The meso-curricular level represents the curricular bet of the academic program itself articulated with the macro curricular level, dynamizes the development of competencies, LO, and formation purposes.

At the micro-curricular level, educational courses are managed, and pedagogical and didactic actions are oriented for the development of training purposes; at this level, the forms of selection, organization, and distribution of the different disciplines, topics, problems, and training processes are defined [9].

A curricular design with LO as its axis starts at the macro curricular level, based on current national and international regulations and trends. Next, its axiological and teleological frameworks formulate the guidelines and directives at the institutional level. Finally, it defines its LO, formulation scheme, and the forms of implementation and evaluation at the meso- and micro-curricular levels.

At the meso-curricular level, the task focuses on recognizing the professional to be trained. The program analyzes its institutional scenario, the national and international trends in its disciplinary field, and the training needs. The demands of the context, the guidelines of the certifying agencies, and the options in the occupational area raise questions about building pertinent LOs in dialogue with the trends and needs identified, guaranteeing the articulation of the elements that make up the different stages of curricular design, execution, and evaluation. The meso-curricular design implies a total and strict coherence between the educational purposes, the graduate profile, the competencies, the LOs, the curricular model, and the curriculum.

Therefore, by achieving this articulation and coherence, the PLO will reflect the knowledge, competencies, attitudes, and knowledge expected to be developed in the graduate, becoming an articulating axis that allows the implementation of the curriculum and, of course, its evaluation for making decisions related to the program.

From the micro-curricular level, the design must be aligned with the PLO, guaranteeing that all the courses that make up the program’s curriculum contribute to fulfilling the proposed results. Otherwise, there will be a risk of having a disjointed proposal where each professor, from his/her good intentions, points to the horizon that he/she believes convenient. Therefore, the syllabus design of each course must be structured from the articulation and coherence of the identification of the PLO course will contribute.

The purpose of the course in alignment with that learning outcome, the learning outcomes of the CLO course, the contents to be developed, the learning strategy, the activities, the evaluation criteria, the performance levels, and the evidence that demonstrates that the student has or has not acquired the learning.

4.3. Change of Evaluation: From Rating to Appraisal

Evaluation as a process that seeks to collect information for decision-making has been distorted due to multiple factors such as the lack of pedagogical training of university professors, the high number of students, the high academic assignments, and the report of numerical grades, among others. Evaluating based on LO requires other views, conceptions, and ways of proceeding. The professor must report a grade on a numerical scale that only represents a pass or fails requirement but says little or nothing about the learning acquired by a student. For example, a grade of 3.0 has several interpretations, among them, that the student passed the course; that the student learned everything foreseen in a course at an acceptable level; that the student learned very well some of the foreseen contents and not others; that the student learned a subject sufficiently and thus passed the course; in short, this dilemma has represented an educational problem that has not allowed to have certainty about the learning acquired by the student.

An evaluation by LO leads to a paradigm shift in the evaluation concept because numerical grades will continue to exist for many years. The professor must provide accurate information about what the student learned, under what conditions he or she understood it, and at what level of performance he or she acquired such learning, a task that looks pretty complicated for a professor with a high academic assignment and a large number of students. Like the curriculum design based on LO, the assessment also requires planning, pertinence, and coherence. When designing the syllabus, the following must be selected for each learning outcome: the learning activity, the evidence with which the student will know if he/she achieved that outcome, the evaluation criteria regarding the form of presentation of the evidence, the conceptual management of the product to be delivered, the interaction in the development of the course, the procedures, methods or paths that the student used to reach the learning and finally, the performance levels on which the evidence will be evaluated; this is the key to an evaluation-assessment process that accurately accounts for the student’s learning.

4.4. Informed Decision Making

A benefit derived from curriculum design by LO is to have pertinent and timely information for decision-making at the different macro-, meso-, and micro-curricular levels. Accompanying and monitoring the fulfillment of the LOs at the micro-curricular level allows the professor to know which aspects of the course should be modified: learning strategies, the evidence provided, evaluation criteria and performance levels, reformulating the CLOs, analyzing whether the course is contributing to the fulfillment of the program’s learning outcomes, among others. This type of decision at the micro-curricular level is based on the results obtained in evaluating each learning outcome, thus providing concrete information on what needs to be modified, taking the necessary corrective measures, and allowing to know the strengths and enhance them.

At the meso-curricular level, decisions are made based on the assessment in articulation with the program self-evaluation processes. This decision-making level requires the application of indirect methods such as surveys, focus groups, interviews, and data analysis, among others, in which various stakeholders such as graduates, professors, students, managers, employers, and entrepreneurs participate to evaluate the PLO.

Decisions at this level have consequences for the program’s curricular design since evaluating LO, profiles, competencies, and formative purposes must be reviewed. The important thing is that the decisions taken are informed and supported by the evaluation processes.

4.5. Professor Training

Professor training is an aspect that is rarely mentioned when talking about a curriculum design based on LO, and this becomes a priority since most university professors have a robust disciplinary background, but some lack pedagogical training. This new work paradigm requires professor training in pedagogy and didactics that contribute to recognizing and applying new course planning forms. In addition, criteria for selecting and using various resources that promote better learning, the consolidation of academic communities, knowledge in curriculum design that emphasizes the construction of meaningful and collaborative learning environments, and new evaluation practices focused on assessing knowledge.

Higher education institutions have significantly tried to train their professors at the doctoral level in disciplinary fields. Undoubtedly, this type of training will provide elements for improving educational practices from the areas of activity with excellent quality. However, such training should complement continuing education programs in aspects mentioned above and topics related to learning styles and strategies, evaluation, and micro-curricular design.

4.6. Change of Attitude

A fundamental part of the success of any redesign, innovation, or change process is directly linked to the attitude to face the new ways of proceeding. In this context, the curriculum design focused on LO requires that professors do not think it will have more work, effort, and one more task to add to their daily activities. Therefore, they do not give it the importance it requires and continue with their old pedagogical practices.

Attitudes such as those described above are an obstacle to the curricular proposal’s success since there is a danger that things will remain on paper. Working from LO requires the professor’s understanding, commitment, dedication, and devotion, always understanding that this work is part of the functions and responsibilities inherent to his or her position.

How was it determined what was learned? How was it learned? Moreover, at what level does the student learn something? It is the very object of his or her role as a professor, and he or she must assume it from that perspective, always seeking the best classroom practices for the success of his or her students.

4.7. Evaluation and Sustainability: Assessment

For ABET [2], assessment is the process developed to identify, collect and prepare the data used to evaluate the achievement of the “Student Outcomes” defined in the certification model. According to the “Student Outcome” assessment, the process uses direct, indirect, qualitative, and quantitative approaches. This concept, although not mentioned by other agencies, is a requirement to be evaluated in the accreditation processes; this is how EUR-ACE [4], within its criteria and guidelines, inquiries about “Do the assessment methods and criteria provide evidence of the ability of the same to verify the effective achievement of the LOs of the unit/module of the subject by the students and ensure that the level of achievement of the students is assessed credibly?”, Are the LOs of the units/modules of the subject adequately assessed?”, and “Are the LOs of the units/modules of the subject adequately assessed?”.

Given the above, assessment is defined as an organized, planned, intentional, and systematic process of gathering information on the achievement of the PLO and the course; this information is used for decision making, which is framed in the process of continuous improvement, allows high-quality standards to be obtained. It is essential to have evidence for timely intervention to improve the program in this process.

Assessment should not be confused with learning assessment focusing on different ways of evaluating the classroom [23]. They are two different processes that complement each other, providing helpful information to propose adjustments, changes, and improvements in the formative process. Similarly, it should not be confused with program impact assessment, which is summative and performed at the end of an intervention to determine the expected results [24]. This type of evaluation uses instruments or strategies to obtain information from the external context. Therefore, its purpose is not focused exclusively on the LOs or improving the teaching-learning process, as with assessment.

Assessment allows professors and school administrators to know the status of each learning outcome. With this information, it is possible to make decisions about teaching strategies, activities, evidence, the formulation of these outcomes, performance levels, and evaluation criteria for each outcome. In general terms, it allows them to know what students are learning and at what level.

For students, it allows them to know their learning achievements. For directors to know in which aspects of the formative process, decisions must be made to strengthen, correct, or develop a continuous improvement process for student success and enhance the culture of quality in all substantive responsibilities based on reliable data collected and analyzed during the process.

The assessment is part of the curricular design in its evaluation phase. It allows having first source information to make decisions regarding LOs, strategies, resources, evaluation criteria, professor training, training profiles, competencies, and contents, among others, becoming a fundamental tool to guarantee continuous improvement processes with the sole objective of achieving high-quality standards in the educational processes.

4.8. Implementation Route

In curricular designs from LO, no defined route serves as a model for other institutions since the diversity, the particularities of each program, the approaches, and the institutional philosophy, among others, make each institution a particular world with unique features and characteristics. For these reasons, its internal practices and needs demand specific practices similarly. Thus, each institution must define its route for designing, implementing, and evaluating its curricular proposals, making decisions that range from the simplest to the most complex aspects.

Regardless of the route defined by each institution, the minimum elements to be considered are the actors, the curricular approach, the formulation guidelines, implementation, the PLO and course evaluation, the critical control points, the criteria for curricular coherence and relevance, the forms and levels at which decisions will be made, the self-evaluation processes, the guidelines for formulating and implementing improvement plans, and the professor training plan.

Designing an implementation route based on LO involves all institutional levels, such as the academy, the administrative and financial areas, the academic support areas, the services area, and senior management.

5. Correspondence into Competencies, Learning Outcomes, Formative Purposes of the Program, and the Course

The formative purposes of the program belong to the meso-curricular level and state the formative intentions in a professional field or discipline. These purposes respond to the needs of the domain at the regional, national, and international levels and become the roadmap. Furthermore, the promise of value to which the university and the program are committed. On the other hand, the purposes of the formative course belong to the micro curricular level, and they are expressed in the syllabus. They are statements that in articulation with the guidelines and constructive proposal of the program, determine what the professor expects to work on during the formative process to facilitate learning in his course.

Therefore, the purposes of a course are the professors since they manifest their formative intentions based on the expected learning. At the same time, the LOs refers to what the student will know once the course or program is finished, i.e., these results are focused on the student and not on the professor. Furthermore, PLOs are stated at the meso-curricular level, while CLOs are formulated at the micro-curricular level.

On the other hand, competencies obey broad senses of training and describe the integration of the possibilities that a student must achieve successful performance in his/her professional practice by using the knowledge and procedures of the program he/she develops. While the LOs, on the other hand, show the specific learning that allows the achievement of competencies [9].

In summary, the evidence program formulated the PLO is evidenced in the learning acquired by the students at the end of each course and at the time of graduation; therefore, they are vital for improvement and educational quality. Furthermore, they are formulated at different levels of difficulty and mastery as the student progresses in his or her formative process [25].

The competencies are also defined by the program at the meso-curricular level based on the context analysis. However, they are evidenced in the student once he/she graduates and enters the work field, which does not mean that throughout the career in the courses, practices, and internships, there are no indications and indicators of the fulfillment of these competencies. Therefore, it can be said that the competencies evidenced in a graduate reflect the LO acquired during the career, a situation that leads some countries, such as Spain, to speak different competencies and LO.

6. Assessment Cycle for the Master in User Experience Design—MDUX

The assessment cycle presented in Figure 3 responds to the curriculum design of the master’s degree from LO, which has coherence among the elements that compose it and meets institutional and legal criteria and guidelines. This cycle evidences all the elements and implications that have been discussed so far regarding the curriculum design based on LO, allowing the intentional and systematic collection of information for decision making in the process of continuous improvement to achieve high standards of the educational quality of the program [26].

The starting point for this assessment is the curricular design articulated in LO and the collaborative work of Academic Networks (Curricular and Course), understood as “social organization devices, founded on relationships of equity among members of the same academic stratum or group with common objectives and goals” [27,28].

The elements of the evaluation cycle are the external sector related to the program, the collaborative work of curricular networks and program courses, consultation with stakeholders or interested parties, and the seven proposed phases:

Phase 1 Weighing of PLO program learning outcomes; Phase 2 General Mapping of PLO vs. curriculum and CLO course learning outcomes; Phase 3 Preparation of syllabus, activity guide and evaluation rubric; Phase 4 Recording of grades based on CLO course learning outcome; Phase 5 Performance measurement by evaluative moments of the CLO course learning outcomes; Phase 6 Overall measurement of POL program learning outcomes by the curriculum networks and the curriculum activities; and Phase 7 Improvement plant. All these are framed in the continuous improvement of the program’s curriculum.

Phase 1: 

Weighting of PLO program learning outcomes

The first phase of the assessment cycle is the assignment of weighting to the PLOs by the Professors’ Network based on the definition of the relative weight or importance each one has for the program. The total value of the weighting will be 100. This value is distributed by assigning relative weights among all the PLOs and noting a justification valuation given to each one. Therefore, it is necessary to keep in mind that the weighting is an exercise of the effect analysis that each PLO has on the program. The program director must participate with the professors who accompany the program design. For this analysis, the program approach, the program’s training needs, and the training purposes, among others, must be taken as a starting point. It should be noted that before this phase, the PLOs had already been defined.

Phase 2: 

General mapping of PLO, competencies, problem areas, integrative problem core, training profile, curriculum, and CLO course learning outcomes.

Before this phase, in the curricular design by problem nuclei and around LO, the articulations and coherences of the design established in terms of the PLO, the competencies, the problem nuclei, the problem integrator nucleus, the training profile, the training purposes, and the curriculum. In this phase of the assessment, a general mapping curricular design to determine which courses and CLOs will contribute to the achievement of the PLO. It is important to note that the same course may contribute to achieving one or more PLOs.

This general mapping defines the courses that will be the object of the entire assessment cycle (e.g., CDIO strategy courses, internships, laboratories, research courses, and practical component courses, which, as courses where students must put their knowledge into practice, are considered of vital importance for the student’s education). The courses selected for the entire assessment must contribute to the different PLOs, making it possible to evaluate them based on these chosen courses. The other courses of the study plan that were not selected for the whole assessment cycle will make the process within their Course Networks.

Phase 3: 

Preparation of syllabus, activity guide, and evaluation rubric

Before proceeding to the development of each syllabus and its respective activity guide and evaluation rubric, it is necessary to define for each CLO the contents, activities, evidence, and evaluation criteria and the performance levels with which the CLOs will be evaluated. When filling out the instrument recommended for this phase, it should be kept in mind that all the elements must be consistent. The activities, evidence, and evaluation criteria respond to the CLO and the PLO.

Phase 4: 

Recording of grades based on CLO course learning outcomes

To develop the assessment of the CLOs, it is necessary that the professors that make up the Course Network record their qualifications so that the report can show the achievements of each CLO. In addition, this phase must be consistent with the information recorded in phase 3 and have the necessary evidence and well-defined evaluation and performance criteria.

Phase 5: 

Performance measurement by evaluative moments of the CLO course learning outcomes

In this phase, we will work with the grade records of phase 4; for this, the course director and the Tutor Network must apply the VD performance vector [28] as follows: (# students in high, medium, or low performance x 00/# total number of students evaluated). The scores for the high-, medium-, and low-performance levels will be defined in the evaluation rubric for each moment.

Having the performance vector for each of the high, media, and low levels, the percentage (%) of achievement of each CLO will be obtained by adding the percentages or performance vectors obtained for the high and medium levels. To determine that the achievement of a learning outcome is as expected by the program, it must be equal to or higher than 75%. Achievement percentages equal to or less than 74% will be subject to review by the Tutor Network and their course leader. In this phase, the professor indicates the reasons for this percentage in each CLO and the maintenance or improvement actions that will improve the result if required; this information is the input for the analysis performed by the Tutor Network.

The CLOs that were not selected as a sample for the complete assessment will go to phase 7, formulating the improvement plan at the level of the Course Networks. The courses chosen for the complete evaluation will continue to the next phase.

Phase 6: 

Overall measurement of PLOs program learning outcomes by the Curriculum Networks and the Curriculum Committee

This phase is completely developed in three control moments throughout the course, and its purpose is to follow up on the compliance status of the PLOs. First, for its completion, the information obtained in the previous phase percentage of achievement of each of the CLOs is used; each course director prepares, as appropriate, the follow-up report on the accomplishment of the LO. Finally, a simple average of the values obtained is made to obtain the total performance percentage for each CLO.

Then, the average of the total percentage of the performance of each CLO is taken, indicating the level of performance achieved and making the respective analysis to formulate the improvement actions that may be necessary and the verification date. High performance will be considered from 75 to 100, medium from 60 to 74, and low between 1 and 59. The program leader will manage the information and analyze it by the curriculum committee.

According to the decision level, this general measurement of the PLO will be made within the Curricular Networks, the Curricular Committee, the School Council, or the Superior Council. In this phase, each course director selected for the assessment carries out the follow-up of the LO to be presented and evaluated by the Curricular Networks or the instances that may be necessary.

Phase 7: 

Improvement plan

With the information analyzed in the previous phases, the actions to be followed for the PLO and CLO are defined, identifying improvement and sustainability actions; these actions are included in the improvement plan approved by the respective instances, setting the verification dates.

The assessment cycle will be complemented with the consultation with the different actors of the process by the most expeditious means decided; thus, there will be perceptions and evaluations of students, graduates, professors, managers, and employers articulated to the self-assessment process.

7. Conclusions and Future Work

The assessment cycle must be developed in full articulation and coherence with the LO-based curriculum design and complement the self-evaluation processes, allowing informed decisions to be made at different levels to promote continuous improvement of the program.

The curriculum design based on LO has several implications for the teaching practice, and its success depends on the coherence and articulation of the different elements that comprise it.

It is necessary to have an implementation route for LO, which should be part of the curricular design and integrate, among others, the actors, decision making, control and evaluation points, and participation mechanisms. In addition, continuous improvement based on the assessment should involve the different actors involved in the process, including senior management.

Working for LO is a task that changes the traditional educational paradigms centered on teaching and the professor towards new educational possibilities centered on the student and learning. The mission now focuses on creating learning environments and strategies that allow knowing firsthand what, how, and at what level a student achieves his/her learning. Professor training, educational and assessment practices changes, different participation mechanisms, and new ways of making decisions that affect the curriculum. There is no single form of curriculum design, not only a single route implementation of LO. It depends on the organizational culture, structure, intentions, philosophy, and axiological and teleological framework.

Students become the center of curriculum development, and LO assessment is no different; from this perspective, students must be considered the center of curriculum development. Therefore, it is important to consider including education, teaching, and learning processes from any form for these designs and evaluations.

The study’s primary objective was met from the perspective of knowing the references of the proposed proposal, i.e., the guidelines of the governments of Colombia and Spain. These authors proposed a curriculum design based on LO. These international accreditation agencies use LO, which allowed the master’s degree program creation in user experience design between the National Open and Distance University of Colombia and the University of Lleida in Spain.

It reached the two governments’ requirements and obtained the records to offer the program in the two countries. In other words, the program is currently operating in its first country and is on offer to new students.

Future work should include ongoing evaluation of the LO curriculum design by the regulations of the two countries and the possibility of accreditation evaluation with international agencies. The coherence of the curriculum should be reviewed according to the activities, courses, and the program. Evaluate the efficiency and effectiveness of the evaluation by learning outcomes of the program. Tools should be implemented that allow the educational management of the LOs and consequently the application of learning analytics in the learning processes of the students of the master’s program. All this is to improve the LOs. Finally, new technologies should be included to support these designs and developments, such as U-Learning and AI applied in the LOs design program and course.