Enhancing Competency and Industry Integration: A Case Study of Collaborative Systems Engineering Education for Future Success

Abstract

This case study examines the effectiveness and industry relevance of a collaborative systems engineering master’s program in Kongsberg, Norway. Through close collaboration with industry partners, students gain practical experience and tackle real engineering challenges. The authors used statistical data, meeting notes, and an alumni survey to assess the program’s impact. The results indicate a high success rate of 87%, with alumni holding desirable positions in various engineering disciplines. The alumni expressed satisfaction with flexibility and teacher quality but desired more focus on leadership and soft skills. Strategic inputs highlight digitalization, sustainability, security, and progress in technology as critical topics for the industry, shaping the program’s evolution for continued relevance.

1. Introduction and Context

In the previous century, industrial practice and academia have grown into separate worlds in Western Europe. Within academia, research and education are drifting apart; for many academics, research is the holy grail, while education is a burden or a derivative of research (so-called research-based education). In this paper, we describe how we can bring research, education, and practice closer together to bring mutual value. We argue that good mutual relationships between them are beneficial for developing the competency and the practice of Systems Engineering (SE).

The basis for this paper is the development and evolution of the Master of Science (MSc) in systems engineering program at the University of South-Eastern Norway (USN) in Kongsberg, Norway. The development of this program started in 2005. The MSc program is a 120 European Credit Point (ECTS) master’s program (120 ECTS is the equivalent of 2 years full-time study load that we distribute over three years of elapsed time), which is the so-called Industry master’s (IM) program. We run a variant of this master’s program for experienced employees, an Executive master’s (EM) program, which is 90 ECTS. Since 2009, 222 candidates have graduated from the Industry master’s program and 75 candidates from the Executive master’s program.

The core of the Industry master’s program is collaborative education. We define collaborative education as “Collaborative education is an integrated approach where concurrent work in practice, courses with illustrative examples and cases, and assignments, and master project research are interconnecting education, practice, and research to improve competency for individuals, involved organizations, and education providers”. Students in the program must work at one of the industry partners while studying. The master’s program design supports this model of concurrent study and work [1]. The master’s thesis, at the end of the program, harvests experience from the first 2 ½ years of study and work by doing research on applying systems engineering in industrial practice [2]. The result is that 50 of the 210 candidates that started before 2020 published their master’s paper, mostly at the International Council of Systems Engineering (INCOSE) International symposium. Confidentiality limited publication of some papers.

One of the core features of our master’s program is the close collaboration with industry partners. Through this collaboration, students gain practical experience, solve real engineering challenges, and develop a deeper understanding of systems engineering within an industrial context. The strong bond with industry partners also ensures that the program’s curriculum remains aligned with the current and emerging needs of the high-tech industry.

As part of our commitment to collaborative education, we actively seek opportunities to partner with other Norwegian universities, such as the Norwegian University of Science and Technology (NTNU), the University of Oslo (UiO), and the Swedish Chalmers University, and international institutions such as the Stevens Institute of Technology from the USA and the Dutch TNO ESI. By leveraging the expertise and resources of multiple institutions, we can offer a comprehensive and diverse educational experience for our students. Collaborative efforts allow us to share best practices, exchange knowledge, and explore interdisciplinary approaches to systems engineering education.

Our collaborative approach extends beyond the confines of the classroom and extends to the larger systems engineering community, including our esteemed alumni. Alumni play a crucial role in providing valuable insights and feedback to help us continuously improve our master’s program. We actively engage with alumni through networking events, such as the Systems Engineering Study Group (SESG) (https://gaudisite.nl/SESG.html (accessed on 30 August 2023)) and Kongsberg Systems Engineering Event (KSEE) (https://www.usn.no/ksee (accessed on 30 August 2023)). Their contributions, in the form of an examiner or assessor, enrich our program and further strengthen the bond between academia and industry. Alumni involvement fosters a sense of community and creates a supportive ecosystem where past graduates contribute to the growth and success of current students.

The faculty staff are actively working to keep this program effective, attractive, and relevant for future systems engineers. The research questions guiding this paper are as follows:

• What data do we have, and what data do we need to research program effectiveness?
• How have the alumni perceived the program?
• How should the program evolve to stay relevant for the Norwegian high-tech industry?

This paper describes the Norwegian context; the literature on experiential learning; the design, structure, and evolution of the Industry master’s program; and its results. We discuss the research questions and answer them as far as possible.

1.1. The Norwegian Context

Norway has about 5.5 million inhabitants with a growth rate of 0.8% (https://en.wikipedia.org/wiki/Demographics_of_Norway (accessed on 30 August 2023)). It is a large, lowly populated country in Europe. Most people live in the Oslo area or elsewhere along the coast. Figure 1 shows Norway and the distribution of the people. The economy depends heavily on oil and gas production. The fishing industry is significant. The maritime, subsea processing, and offshore industry are well developed, serving the Norwegian and world market.

Norway has ten universities, of which NTNU is internationally recognized for engineering. Several of the ten are former university colleges that, after merging, became a full university. USN is a young university that resulted from the merger of Buskerud, Vestfold, and Telemark university colleges. USN has eight campuses in the southeastern part of Norway. It is the fourth largest university in Norway, with about 17,000 students and about 1900 employees (https://www.usn.no/english/about/ (accessed on 30 August 2023)).

King Christian IV of Denmark and Norway in 1624 founded the town of Kongsberg around the silver mines. When the mining operations decreased, the government established the weapon factory in Kongsberg in 1814. The Kongsberg Weapons Factory transformed into today’s Kongsberg Group (https://www.kongsberg.com/ (accessed on 30 August 2023)), which consists of several business units, such as Kongsberg Defense and Aerospace and Kongsberg Maritime. A crisis in 1987 triggered the sales of several other businesses (current owners in parentheses), such as the oil and gas subsea system (TechnipFMC), production of airplane engine parts (GKN Aerospace), automotive parts (Kongsberg Automotive), and gas turbines (Siemens Energy).

1.2. History of the Industry Master Program

In 2005, the Kongsberg industry identified systems engineering as one of its core competencies. Together with the national and regional governments, it started the Norwegian Center of Expertise of Systems Engineering (NCE-SE) [3]. One of the NCE-SE projects was to initiate a master’s program in systems engineering. Initially, NCE and the university college bootstrapped the master’s program by “buying” the entire master’s program from Stevens Institute of Technology (SIT) in Hoboken, NJ [3]. The first students received a master’s degree from SIT.

A unique prerequisite for students was that they must have a 3-year part-time position with one of the industry partners. The idea is that people should learn systems engineering mostly through experience. The program used the SIT course format, where students follow a one-week intensive course and follow that with a 10-week homework project. This format fits well with working part-time, since students are unavailable for their companies for a few weeks per year only, while the students can plan their homework flexibly. The teachers provide supervision during the homework, e.g., in the form of feedback, and assess the outcome of the homework project.

The university college in Kongsberg together with NCE-SE designed a Norwegian version of the systems engineering master’s program. It used the structure and content of the SIT master’s program as a starting point. This original master’s program had four mandatory courses as the foundation: systems engineering fundamentals, architecture and design, project management for complex systems, and systems integration. After these four courses, students choose five elective courses and then finish the program with a master’s thesis project. This program followed the ideas that later formed the Graduate Reference Curriculum for Systems Engineering [4]. The vision of the program staff is that research, education, and practice mutually reinforce each other when they are well connected; see Figure 2.

Educators need practice as a source of knowledge and inspiration for teaching; they help to develop new competent professionals. Important means for education are reflective practice, case-based (from practice) education, industry as the classroom (immerse students in practice), and industry in the classroom (bring practice examples and problems as questions and discussion items in the classroom).

Educators must stay up to date through research. The result is research-based education. Educators can bring new methodologies into practice and research their effectiveness, and vice versa, observing how and what students learn or struggle with provides insight into methodologies. The classroom can serve as a laboratory with various companies, domains, systems, and projects; the classroom allows a rich sample of this variety.

Researchers need practice for inspiration (since most competency is in practitioners) and validation (since the proof of the pudding is in the eating) purposes. Research may result in improvements of practice through new or improved methodologies. Industry as a laboratory [5] serves research in understanding the problem and challenges and at the same time provides a context for validation. The industrial Ph.D. is an effective way to strengthen the relationship between research and practice; a practitioner has time and mental space to increase fundamental insight into the way of working. This type of research is problem-, need-, and goal-driven instead of just a technology push.

Examples of industrial Ph.D. studies are the theses of Kjørstad [6] and Engen [7]. In their research, they worked with Industry master’s students carrying out their master’s project. Guntveit [8] and Sjøkvist [9] applied what they learned about methods for eliciting stakeholder needs and evaluating them. Haugland [10] evaluated the use of A3 Architecture Overviews. These examples show how we connect education, research, and practice,

In the transformation into the Norwegian master’s program, the staff added more provisions to support experiential learning [11] based on the vision in Figure 2. Specifically, the course Reflective Practice [12,13], based on Schon’s [14] and Kolb’s [15] work, helps students in connecting theory and practice. The staff developed the master’s thesis into a master’s project where students apply systems engineering in practice and evaluate the application in an academic way [2]. The students use a form of action research [16] for their master’s projects. The final deliverable of the master’s project takes the form of an academic paper of about 15 pages. Students with a grade of A or B (on the Norwegian grading scale from A (Excellent) to F (Failed)) try to publish their paper at a conference (All published master project papers are at https://gaudisite.nl/MasterProjectPapers.html (accessed on 30 August 2023)). The Norwegian accreditation organization, NOKUT (https://www.nokut.no/en/ (accessed on 30 August 2023)), approved the Industry master’s program in 2009.

Experiential learning emerges as a natural approach for systems engineering owing to its deeply experiential nature. A significant challenge lies in the considerable disparity between the current industrial practices and the theoretical instruction offered in educational settings [12]. Within this paper, we explore the ways for collaborative systems engineering education, facilitating the interconnection of research, practice, and education, which mutually reinforce each other [1], as Figure 2 illustrates.

In 2018, the staff described the objectives of the industry master’s program as follows [1]:

Prime business objectives are: (1) to develop a more competent systems engineering industry in Norway compared to international standards; (2) to decrease the time it takes to develop a systems engineer of high quality, from 15–25 years to half of that time; and (3) to support the industry in its endeavor to innovate, remain competitive, and manage risks pertaining to the businesses.

The key academic objective is to integrate the systems engineering industry with academia to ensure the highest learning outcome for the students within the field.

The master program has three boards that represent the formal interaction of the USN staff with industry partners:

• Strategic: the Industry Advisory Board (decision makers in industry);
• Tactical: the Reference Group (subject matter experts);
• Operational: the Human Resource Forum (HR staff).

The program staff organizes with the support of the NCE-SE the yearly KSEE in June. The KSEE is a networking event, especially for industrial practitioners with an interest in systems engineering. It spans from mid afternoon to the next day until 12:00 PM. The number of participants varies between 80 and 100.

The staff also organizes the SESG, typically in February and November. In this way, KSEE and SESG are regularly spaced networking events. The SESG is a half-day event, with a few very brief presentations followed by breakout sessions to stimulate participant contributions. The main intention of these boards and events is to connect well with and remain relevant to the industry.

Historically, the industrial cluster (NCE-SE) promoted the exploration and research of popular variants of the traditional systems engineering approach, such as LEAN [17] and design thinking [18]. It also funded research in electric mobility [19]. This helped in bootstrapping the research. Lately, interaction with industry partners and practitioners identified digitalization, sustainability, and security as the main areas where the curriculum may have to evolve.

2. Background Literature

The triangle in Figure 2 inspired this literature review. Specifically, we researched reflective practice and case-based education with industry and in the classroom. We sought data and methods to research program effectiveness and what pedagogic principles make a good systems engineering program. At the end, there is a brief review of industrial needs: digitalization and sustainability. These are the main areas of the curriculum that may have to evolve for the program to stay relevant and attractive.

2.1. Experience-Based Learning and Reflective Practice

Muller and Bonnema [20] asserted that the traditional, mono-disciplinary (e.g., mechanical, electrical, civil engineering) way of teaching does not suit systems engineering. Active learning plays a crucial role in understanding the concept(s) and exploring further. In teaching systems engineering, students/participants need to be active and learn the steps by participating. They proposed, based on Bonwell and Eison [21], a model for teaching systems engineering to graduate and undergraduate students, emphasizing active learning to make them aware of the broadness of systems.

Reflective practice, based on Donald Schon’s competency development approach [14] and Kolb’s learning cycle [15], has shown a significant impact on student learning in the systems engineering master’s program at USN. Muller [12] conducted a survey among USN systems engineering (USN-SE) master’s program alumni, where 46.2% of respondents acknowledged a significant impact on their learning through reflective practice. Ferrero et al. [22] applied experiential learning through roleplaying to enhance stakeholder collaboration in water safety plans. The study exposed participants to the four phases of Kolb’s experiential learning cycle: concrete experience, reflective observation, conceptualization, and active experimentation.

Experiential learning approaches in higher education institutions in the United States (U.S.) have increased over the last three to four decades, according to Buzeeli and Asfo-Adjei [23]. Yet, medicine and engineering fields have long promoted the benefits of applied learning. Only during the recent decades has there been a push for nearly all degrees and disciplines in the U.S. to include experiential learning requirements such as internships and field research, according to a survey by the U.S. National Survey of Student Engagement [24,25].

Research indicates the necessity of experience and reflection in engineering education. Frank [26] wrote about the competencies of engineers with high capacity for engineering systems thinking (CEST): “From an analysis of the results, it would appear that engineers with certain personal characteristics can acquire or enhance their CEST in a gradual and long-term process, by appropriate training, job rotation, systems work roles, job design, and development programs”. Hutchinson et al. [27] discussed three different forces, and they mentioned that experience is the greatest force to become a proficient system engineer. Beard and Wilson [11] emphasized the importance of reflection, stating that “In the third stage, we make links and connections to our previous experience and knowledge. Without these links, the experience may have little value in learning”.

2.2. Digitalization and MBSE

The Department of Defense (DoD) [28] defines digital engineering (DE) as “an Integrated digital approach that uses authoritative sources of data and models as a continuum across disciplines to support life cycle activities from concept through disposal”. Giachetti [29] stated, “The crux of digital engineering is the creation of computer-readable models to represent all aspects of the system and to support all the activities for the design, development, manufacture, and operation of the system throughout its lifecycle”. Shared digital engineering solutions maximize access to and enhance the use of information by all project participants during all phases, including “smart operations” [30].

As long as systems engineering activities are document-centric, “digital engineering” can be a transformation. Bone et al. [31] proposed a roadmap to realize the benefits of DE transformation to form the view of DoD acquisition systems. The roadmap addressed the following key points: technologies and their evolution; how people interact through digitally enabled technologies and new needed competencies; how methodologies enabled by technologies change and subsume processes; how acquisition organizations and industry operate in a DE environment throughout the phases of the life cycle (including operations and sustainment); and governance within this new digital and continually adapting environment.

A few questions arise: “How can digital systems engineering activities be carried out”, and “what actions are needed to support digital engineering practice?”. Huang et al. [32] presented their vision of digital systems engineering. The authors identified the challenges and enabling digital technologies in transitioning from traditional to digital engineering. Among others, they defined digitalization, unique identification, digitalized artifacts, and digital augmentation as core concepts to be included in the digital transformation journey. The paper also presented and discussed the big picture of digital systems engineering on four levels: vision, strategy, action, and foundation.

Dahmann et al. [33] provided an approach for applying DE to a large system of systems mission. They represented a system of systems mission with a top-level, lightweight end-to-end model while integrating systems and connectivity with mission threads, activities, metrics, and the context. They explored specific modeling (or interface) issues from the end-to-end models by extracting model “snippets”, i.e., to provide the base for the “deep dive” analysis of the issue at hand.

Over the past decade, the systems engineering community has been transitioning from traditional practices to more model-based practices, i.e., model-based systems engineering (MBSE). The INCOSE vision 2035 [30] claims, “The future of systems engineering is model-based, leveraging next generation modeling, simulation, and visualization environments powered by the global digital transformation, to specify, analyze, design, and verify systems. High fidelity models, advanced visualization, and highly integrated, multi-disciplinary simulations will allow systems engineers to evaluate and assess an order of magnitude more alternative designs more quickly and thoroughly than can be done on a single design today”. However, one should also note that modeling ecosystems are often rudimentary and incomplete.

Henderson and Salado [34] presented the benefits and value of MBSE through empirical research based on the literature. Their research results indicated that perceived evidence only supports two-thirds of claimed MBSE benefits, while only two papers reported measured evidence. They also concluded, “There is no empirical evidence present in the publicly available literature today that supports the hypothesis that MBSE is beneficial for the development of engineered systems”. Campo et al. [35] conducted another literature study with an MBSE viewpoint on “evaluating perceived value and metrics”. In their study, the authors categorized positive attributes (e.g., verification and validation capability, reasoning, etc.) and negative attributes (e.g., acceptability, familiarity, etc.) from 2900 MBSE claims while identifying that the most-stated benefits of MBSE were reduced time/costs, better communication, and better analysis capability. On the other side, drawback claims included increased time/cost/effort and worsened capability. Finally, they concluded that a large share of claims (47%) about MBSE were based on author opinions. Most claims (86%) were not substantiated by a metric.

2.3. Sustainability

The United Nations has defined a framework with targets for sustainability [36] with 17 Sustainable Development Goals (SDGs) (https://sdgs.un.org/goals (accessed on 30 August 2023)). All these goals are mutually related [37,38], forming a complex and rather wicked problem. Tackling these problems requires a systems approach, e.g., systems thinking and systems engineering with holistic and interdisciplinary approaches [30,39].

The scope of system developers for defense, oil, and gas and maritime is much smaller than the global sustainability challenge. Legislation enforces organizations to report their footprint based on the U taxonomy [40]. Companies are primarily in need of processes, methods, and tools to know and report their footprint, such as greenhouse gas emission reporting for Scopes 1, 2, and 3 [41]. Once the regulations or cost incentives put pressure on reducing the footprint, then life-cycle assessment [42] is the required type of method. Roadmapping is another useful method. Rob Phaal collected a great deal of knowledge about roadmapping (https://www.cambridgeroadmapping.net/ (accessed on 30 August 2023)), and defined a framework for it [43].

3. Research Method

The data supporting this research consist of statistical data related to the industry master’s program, notes from meetings related to the program, and a survey answered by alumni. The statistical data include the number of students, what domain of work they were doing during their study, and how much time they used to finish. The current research focuses on the industry master’s program during the period from 2006 to 2023.

We analyzed notes and minutes from formal meetings with the industrial partners, network meetings, and internal staff strategy meetings. The analysis of the results of the meetings with various stakeholder groups helps in identifying future competency needs. The results of these meetings and their minutes show who discussed what topics and how deeply. The analysis used the minutes of meetings and related presentations to determine the main topics of the meeting and the specific takeaways for the areas where changes in the curriculum are likely, e.g., digitalization, sustainability, and security. We reconstructed the timeline of the growing insight.

Alumni answered a survey exploring the students’ experience, program satisfaction, work–study balance, how the program helped students become competent systems engineers, and suggestions for improvement. We sent the online, anonymous survey to all 222 alumni. In addition to statistical analysis, we manually analyzed the current positions of the 74 alumni who started in the first 6 years of the program.

We designed questions to measure student satisfaction, the quality of teachers, and the adequacy of facilities. The survey utilized a Likert scale [44] consisting of five choices, allowing respondents to indicate their level of agreement or disagreement with the given statements. The choices ranged from “strongly agree” to “strongly disagree” or from “poor” to “excellent”. We varied the specific wording of the Likert scale choices depending on the survey design and the nature of the questions. We classified the Likert scale responses into three categories—promoters, detractors, and neutrals—to calculate the net promoter score (NPS) [45]. We considered respondents who selected the highest score of 5 as promoters, while we classified scores of 1 to 3 as detractors. We considered a score of 4 or its equivalent, indicating satisfaction or agreement, as neutral for promotion. Calculating the NPS involved subtracting the percentage of detractors from the percentage of promoters. The resulting NPS can range from −100% to +100%, with a positive NPS indicating more promoters than detractors and a negative NPS suggesting the opposite.

The use of the Likert scale and NPS in the survey allows for quantification and comparison of responses, providing a concise metric to evaluate the level of satisfaction and identify areas for improvement within the systems engineering master’s program. As a measure of achievement, we examined the positions the alumni have currently in their company, e.g., positions like systems engineer or project manager. We used the same classification of positions for the survey and the manual analysis. In addition to the Likert-scale questions, the survey also had a few open questions. This text gave more context to the Likert-scale questions.

SIT used the Likert scale in this way to evaluate individual courses using a standardized survey with eight questions about the course, eight questions about the teacher, and three open questions. SIT used two main questions about the satisfaction of the course and the teacher for a quality assurance check. A negative NPS score on one of these two questions triggers a talk with the course owner. USN continues this practice. Consequently, students and staff are familiar with the Likert scale and its use.

The survey has limitations due to the limited number of respondents (64 of 222 for the survey, 60 known positions out of 74 alumni). The Likert scale is used to avoid obtaining a too-positive set of responses. NPS provides a criterion for understanding the survey; however, this is still an interpretation.

4. Results

The results section presents four main datasets, that is, firstly, the program population data; secondly, the findings of the survey conducted among alumni of the systems engineering master’s program; thirdly, the analysis of positions of the first cohorts of the program; and, finally, the analysis of gathering strategic inputs.

4.1. Program Population Data

Figure 3 shows the number of starting students per year. For each year, it shows the number of students for the various domains. The graph clearly shows a fluctuation in the number of students. The number of students depends on the number of industry positions that the industry partners have available and the number of student candidates that apply to the program. A challenge is that when the economic climate is positive, the industry has many positions; however, the industry also recruits many employees directly. When the economic climate is poor, the industry partners tend to have few positions, while the employment situation triggers many students to apply. In other words, when the economic climate is good, then we often have to disappoint industry partners, and when the economic climate is poor, we often have to disappoint applying students.

Figure 4 shows that the oil and gas domain dominates in recruiting IM students. Defense, maritime, and providers are significant recruiters, too. Providers are companies providing information technologies (IT), consultancy, or engineering services, which often have the oil and gas, defense, and maritime domains as customers. The histogram also shows that Executive master’s students often come from the defense and maritime domains.

The IM program has a high success rate, where most students finish the program with their MSc degree (222 of the 254 students that started between 2006 and 2020 finished successfully). Figure 5 shows that all the students that started the program before 2009 completed their master’s. The number of students that complete the program is gradually decreasing. The average completion for all years is 87%. Of these, 82% finished within the normal time frame and 5% with a delay of a maximum of two years.

Executive master’s students may stop their studies prematurely, as the diagram shows. Of the 129 EM students who started before 2020, 75 finished the study successfully, accounting for 58%. Several of the Executive master’s students already have a master’s degree. They want to update their knowledge within the field of systems engineering. As such, it is not surprising that the number of completed master’s degrees is lower than that of IM program. In addition, this is dependent on what the companies are paying. The oil and gas companies paid their employees to obtain further education prior to 2016. Then, the companies stopped paying in 2017; and thus, many students did not want to complete at their own cost.

The IM program has a planned duration of 3 years. Figure 6 shows that 94% of successful IM students finish the program according to plan or faster. The study duration of Executive master’s students varies much more. Given a full-time workload of Executive master’s, the observed 3- to 5-year study duration looks quite reasonable.

4.2. Survey Results

The survey (Appendix A) aimed to gather insights on various aspects of the program, including the students’ experience, program satisfaction, work–study balance, how the program helped students become competent systems engineers, and suggestions for improvement.

A total of 64 alumni responded to the survey, with 81% of respondents identifying as male. The distribution of respondents’ graduation years showed that 47% graduated 5–10 years ago, while 25% graduated between 2–5 years ago (see Figure 7). In terms of current positions, 20% of alumni work as systems engineers, while the majority (33%) are employed in other engineering disciplines. Additionally, 17% of the respondents hold executive-level management positions, and 11% of the respondents work as project managers (see Figure 8).

4.2.1. Satisfaction and Flexibility

A total of 59% of the respondents strongly agreed that they had enough freedom and flexibility to maintain a work–study balance, while only 14% of the respondents encountered difficulties in this regard. This resulted in an NPS of 45.3%. Figure 9 shows the distribution of responses.

High satisfaction reported in the Likert-scale question regarding program flexibility aligns with the open-ended responses. Alumni expressed appreciation for the program’s hands-on approach, practical examples from real high-tech projects, and the ability to balance work and study. The suggestion for more industry-related subjects and examples further supports the positive sentiment towards program flexibility.

4.2.2. Interest in the Program

Although 36 individuals (57%) gave a score of 4, indicating that the program was interesting and enjoyable, the results yielded a slightly negative NPS of −4.8% (See Figure 9). The open-ended responses provided insights into this result. Some alumni expressed a desire for more practical use cases, industry-specific knowledge, and relevant resources to enhance the program’s interestingness. The suggestion for more guest lectures, industry speakers, and real-world engineering cases aligns with the need for increased engagement and relevance to improve enjoyment levels.

4.2.3. Teacher and Facility Qualities

To assess teacher qualities, 50% of respondents scored 4, indicating positive feedback, while 20 respondents gave a score of 5, and 12 respondents scored 3 or lower. The resulting NPS for teacher qualities was a positive figure of 12.8%. In contrast, facility qualities received a more negative response, with the majority of responses falling into category 3 (42%), resulting in an NPS of −25%.

The open-ended responses shed light on this outcome, with some alumni appreciating the quality of teachers and their experience, while others emphasized the need for more guidance and support during the master’s thesis phase. Similarly, the negative sentiment towards facility qualities resonates with alumni expressing a desire for improvements in resources, accessibility to relevant data, and simplified ways to find information.

4.2.4. Development of Competencies and Career Advancement

We asked participants to rate their experiences to evaluate the program’s impact on alumni’s competency as systems engineers, career advancement, and leadership skills. Although the majority gave a score of 4 to all of the questions in this segment, indicating moderate satisfaction, a significant number of respondents scored 1–3, resulting in negative NPS values for all three measures. The NPS for leadership aspects was −32.8%, reflecting the program’s limited focus on leadership. The NPS for alumni becoming competent systems engineers was −26.6%, indicating a need for further review. Lastly, the NPS for the program’s support in career advancement was −21.9%, suggesting room for improvement.

4.2.5. Improvement Suggestions

When asked for suggestions on how to improve the systems engineering master’s program, alumni provided various recommendations. These suggestions included organizing workshops after work hours to facilitate knowledge exchange and networking, forming partnerships with other Norwegian universities, providing simplified ways to access relevant data and resources, offering different study paths within systems engineering, and incorporating more industry-related examples and guest lectures. Alumni also highlighted the importance of soft skills development, maintaining a balance between presentations and understanding of the subject matter, and creating alumni networks for continued support and growth.

Furthermore, alumni suggested addressing specific industries and challenges, such as construction and oil and gas, by incorporating industry-specific domain knowledge and case studies. They also recommended preparing students for work realities, where systems engineering practices may not be recognized or valued, and providing more practical application examples.

4.3. Analysis of Alumni Positions of the Early Cohorts

The manual analysis of the cohort from 2006 to 2011 provided data on the current position and domain of 60 of the 74 students in these cohorts. Figure 10 shows that alumni in oil and gas, maritime, and manufacturing changed to other domains. Defense benefitted most from these changes; over the last decade, there has been a high demand for engineers in this domain. The large number of miscellaneous domains shows a diversification of domains. The underlying data show that the miscellaneous classification comprises quite a wide variety of domains.

Figure 11 shows the positions of alumni as of July 2023. It compares the position of the more experienced alumni from the first six cohorts (2006 to 2011) with the positions alumni reported in the survey. The response rate of the more-experienced alumni on the survey was quite low; hence, the diagram practically compares less- and more-experienced alumni. The biggest difference is that we see more executive managers with more experience (which is as expected), mostly at the cost of project managers.

4.4. Results from Gathering Strategic Inputs

The interaction between industry and the systems engineering program staff takes place in many meetings and events.

Table 1. Overview of meetings and events to shape the systems engineering research and education strategy of USN-SE.

Event or Meeting	Goal	Participants	Characteristics
Industrial Advisory Board (IAB)	Providing strategic direction	Industrial decision makers, academic leaders for SE	Meeting, once or twice per year
Reference Group (RG)	Subject matter discussions to monitor status and to anticipate the future	Line managers, leaders in SE, practitioners of SE	Meeting, twice per year
Kongsberg Systems Engineering Event (KSEE)	Networking between practitioners from the entire ecosystem	Practitioners in or with interest in SE	1-day event, once per year
Systems Engineering Study Group (SESG)	Networking between practitioners from the entire ecosystem	Practitioners in or with interest in SE	3-hour active workshop, twice per year
USN-SE strategy	Discussing research and education strategy for SE	USN-SE staff and guest experts	1.5-day workshop, once per year

shows the meetings and events with their characteristics.

Figure 12 shows the activities involved in shaping the strategy. Rectangular blocks relate to digitalization, while blocks with rounded corners relate to sustainability. Partners and staff have discussed digitalization for quite some time as an ongoing transition with a broad impact. Governments and universities have underscored sustainability as a vital field for research and education. The figure shows that we are adding sustainability slowly onto the agenda in the dialogue with industry representatives. Despite comprehending its significance, companies point to legislation and economics as the primary driving forces. The momentum behind sustainability is steadily growing, fueled in part by evolving legislation on reporting.

During the program’s strategy sessions, we concluded that digitalization is broad and pervasive. The deliberations around the strategy and the exploration of an educational roadmap break down its impact on the program into several facets:

• Evolving the existing courses to align with the digital paradigm;
• Developing specialized courses that are in the systems engineering realm, such as digital requirements management [46], and model-based integration, verification, validation;
• Partnering with other disciplines like computer science and data science, as the course “Introduction to Autonomy” exemplifies;
• Identifying external education opportunities that students can choose as elective courses.

Both sustainability and digitalization are broad areas. A succession of sessions helped to make the digitalization topic more digestible. Networking sessions conducted with SESG and KSEE served as a starting point in 2018. Subsequently, in 2019, the IAB set the direction, which we followed up with multiple sessions in the reference group and internal strategy sessions aimed at “charting the landscape”. The outcome is the digitalization cloud, as shown in Figure 13. We used an initial version of this map to brief reference group members in mid-2019, allowing us to elicit needs in 2020. In the 2020 session, we asked the industry representatives to list their needs. The bold items in the figure represent the listed needs that were already on the initial map. The items in bold and italic were new additions to the map. In parallel, amid the COVID-19 pandemic, we ran a series of online networking events on topics that are more specific. These events were a combined effort involving ESI in the Netherlands and USN-SE. During the 2023 strategy workshop, the USN staff with support of three ESI employees elaborated the digital workflow and explored the role of MBSE.

The map in the form of a digitalization cloud shows a classification of buzzwords that pop up in digitalization discussions. A challenge for making sense out of all these buzzwords is to transform individual words into a coherent line of reasoning. A line of reasoning starts with a customer and business value proposition. Such a proposition requires functionality that behaves with specific qualities. The functionality with the desired qualities uses technologies as a solution. For example, digital twins can support condition-based maintenance, health monitoring, and smart diagnostics. This requires sensor, communication, storage, and data analytics functionality with the right sensitivity and specificity, trustworthiness, and integrity. Many technologies may contribute in realizing this, e.g., data lakes, data mining, data analytics, artificial intelligence (AI), deep or machine learning, Internet of Things (IoT), 5G, micro services, the cloud, etc.

5. Discussion

Practitioners are the main carriers of systems engineering knowledge, and competency. INCOSE was founded in 1990, where most early members were practitioners. A quick sample of the INCOSE technical reviewers in 2004 [47] shows that 78 out of 83 reviewers were practitioners, 2 did not indicate an affiliation, and 2 were full-fledged academics. Gradually, the academic world is building up competency in systems engineering, with the INCOSE academic council (https://www.incose.org/incose-member-resources/academic-council (accessed on 30 August 2023)) as a guide. Many academic staff members nowadays have significant practical experience. Building up systems engineering capabilities and becoming competent requires practice (see Eraut [48] for the differentiation of competency and capability).

The industry master’s program in systems engineering in Kongsberg, Norway, has been running as a collaborative education since 2006. All the students in the program work at least 50% in a regular engineering job as a part of the study. To strengthen the experiential learning experience, courses and especially the master’s thesis integrate engineering and academic work. A major focus has been to deliver effective education for the industry and for the students. To stay effective, relevant, and attractive, the program arranges a number of formal and informal meeting points with the industrial companies employing the students. This paper is based on experiences from running the program, statistical data, meeting notes, and a survey of alumni.

The program has delivered 15 to 25 students per year consistently to the Norwegian industry, especially to oil and gas, defense, and maritime industries (Figure 3 and Figure 4). Defense is rather mature regarding applying systems engineering, while the oil and gas domain only recently identified systems engineering as necessary [49,50]. Defense and aerospace are the main origins of systems engineering, e.g., the majority of Honour’s [51] data sources are from these domains. The oil and gas sector is acknowledging that systems engineering is essential and has sought to adopt its principles for enhanced operations [49]. To this end, the oil and gas members of INCOSE started an INCOSE oil and gas working group [50] to promote the use of systems engineering in this domain.

The Norwegian maritime industry has strong ties to the oil and gas sector and the offshore industry in general, although it also serves fishing, cargo, and cruise ships. Hence, the industry master’s program serves a diverse range of companies at various stages of systems engineering maturity. In the current positions and companies, we see that defense employs many alumni. The mobility between domains, which we see in the domain analysis, is desirable for Norway since Norway has to transition away from oil and gas as a major employer.

5.1. What Data Do We Have, and What Data Do We Need to Research Program Effectiveness?

Based on the graph in Figure 5, the average completion for all years is 87%. Of these, 82% finished within the norm and 5% with a delay of a maximum of two years. For comparison, in Norway (2018–2022), 50% of master’s students finished within the norm, and 18% finished with a delay of maximum two years (National statistics of successful graduates from masters’ programs with a duration 1 ½ to 2 years (180 ECTS), https://www.ssb.no/utdanning/hoyere-utdanning/statistikk/gjennomforing-ved-universiteter-og-hogskoler (accessed on 30 August 2023)). In 2009–2017, the national numbers were approximately 51% and 19%, respectively. We argue that this high success rate indicates that the program supports learning systems engineering effectively.

Longer-term data on the current positions of the alumni show that they achieve desirable positions, e.g., 28–32% engineer, 18–20% systems engineer, 16 to 25% executive manager, and 5 to 11% project manager. Again, we argue that these numbers are in favor of an effective program.

The survey results indicate that the program has positively affected the Norwegian high-tech industry. Alumni reported working in various engineering disciplines and holding positions in executive-level management and project management. This indicates that the program has produced professionals who are making valuable contributions to their respective companies.

However, all assertions miss a well-defined success criterion. The arguments will become more credible when we collect data on the actual current competency of alumni and when we can compare student achievements with other programs. The question regarding the results for the Norwegian high-tech industry requires data collection from the companies. This could take the form of a longitudinal study on the need for alumni and their appreciation.

5.2. How Have the Alumni Perceived the Program?

The alumni are satisfied with the freedom and flexibility of the program and the skill level of their teachers. They are uncertain about how far the study prepares them for becoming a systems engineer and how much it contributes to their career advancements. The most prominent feedback is that the program insufficiently contributes to leadership and soft skills. Muller et al. [52] argued that adding leadership to systems education is a major improvement.

5.3. How Should the Program Evolve to Stay Relevant for the Norwegian High-Tech Industry?

The alumni answering the survey provided valuable suggestions for program evolution. They recommended incorporating more industry-related subjects, guest lectures, and real-world engineering cases. They also highlighted the importance of soft skills development and maintaining a balance between theory and practice. These suggestions provide insights into the areas that need attention to ensure the program’s continued relevance and alignment with the needs of the Norwegian high-tech industry.

The analysis of the strategic inputs shows that hot topics for the industry are digitalization, sustainability, security, and progress in technology such as material science and additive manufacturing. Over a period of 5 years, we explored these areas together with the industrial partners and practitioners in our network. The challenge for a relatively small research group is to maintain sufficient focus. For that reason, we took the strategic decision to cope with security and progress in technology by looking for partners with that competency. The program structure, with its many electives, enables students to augment their program with courses from other universities.

A major challenge that we are facing is how to include digitalization, MBSE, and sustainability into the curriculum. A number of the master’s thesis students are researching these topics, via case studies carried out at the companies. For example, Sandberg [53] analyzed the workflow for mechanical component design and handovers with the supplier. This is a common specific problem where digitalization can streamline the workflow significantly. Similarly, Thygesen [54] researched handovers between engineering and production. He showed the effort and documentation overload caused by the versions and ways in which the process manages it. Sundet [55] analyzed variation management and the role of the underlying IT tools. Through these case studies, we sampled the digitalization field, obtaining specific options for deploying digitalization.

These and other case studies all show that digitalization affects the full life cycle, the entire supply network, nearly all related processes, and the operation and its operational context. This pervasiveness of digitalization matches the literature [28,29,30,31,32,33]. The digitalization buzzword cloud, Figure 13, is another way to represent the breadth and variety of digitalization. One conclusion is that digitalization needs dispersing into most courses. Then, we may need specific new courses, which we can develop in steps in order to maintain the consistency of the program. For example, the research project DSYNE [46] develops new modules on digital requirements engineering.

MBSE is a specific subset of digitalization. Everyone agrees that modeling of systems in a formalized way is a necessary facilitator for digitalization [30]. However, we are still discovering how to do this effectively, as the various literature reviews show [34,35]. We have learned that exploring MBSE in master’s project case studies is challenging. The cost and effort of applying MBSE [35] form a handicap for smaller-scale research projects. Companies that are less mature in systems engineering see a big threshold for introducing MBSE, and this is not a feasible master’s project. In defense, where companies are using MBSE, master’s projects have difficulties in achieving effective interventions.

The program has a course on MBSE as part of the architecture and design course. The staff explored the industrial needs for MBSE and ways to offer education in MBSE in the 2023 workshop on MBSE digital workflow (Figure 12). The conclusion is that the current course is a good foundation. However, students who want to develop MBSE capabilities need additional education, which could be in the form of a few new elective courses. The students need to learn to facilitate digital content yet not model just to model. They have to develop from knowing the representation (such as SysML) and the tools to making appropriate models that are fit for the purpose. This development will take a few years, and just taking a few courses is not enough.

Systems engineering is one of the disciplines that can facilitate the sustainability transition [39,56]. Kirkels et al. [57] discussed the potential contributions of systems engineering together with limitations. They pointed to transition studies and sustainability assessment as fields to assist systems engineering in addressing sustainability. The needs from industry as discussed in the reference group suggested that life-cycle assessment as a form of sustainability assessment may fit well as a method for engineering systems. Sustainability is a cluster of quality attributes of a system, e.g., energy efficiency, raw material use, emissions, etc. Most of these qualities stretch beyond the system boundary. The upstream (supply chain), the system of interest, and the downstream (the use and life cycle of the system) all contribute to the system footprint. Most sustainability quality attributes require a system of systems perspective. Transition studies increase the scope even further to the organizational or societal level and increase the time horizon to decades.

In our exploratory research, we used Phaal’s framework [43] to explore the use of roadmapping for sustainability [58]. Roadmapping increases the scope and time horizon, which makes it an interesting method to connect to transition studies. Elvebakk [59] created a roadmap for the sustainability of a Dutch community. Drilen [60] and Vrenne [61] applied conceptual modeling in energy storage, one of the essential technologies for the energy transition. Since 2020, we have offered the elective course New Energy with methods like roadmapping and conceptual modeling. In other words, we are making small steps toward introducing sustainability-oriented education in the program.

5.4. Generalizing the Findings

This paper discusses the specific situation of the systems engineering program in Norway. However, based on the literature, we argue that most answers to the research questions are generalizable to other countries. Pedagogics as far as human nature is involved will be universal. However, cultural aspects will require customization of the ideas to the local context.

Content-wise, digitalization, sustainability, security, and technology developments are global concerns. Other universities will have to determine their specific focus based on their own strengths and the needs of the local ecosystem.

5.5. Conclusions

For evaluation of the program’s effectiveness, we need more data from students and employers on the achievements of alumni. We also need criteria to judge these findings, which probably require comparison with other alumni from comparable disciplines. Our alumni are positive about the program, although they have several improvement suggestions. A major suggestion is more support for soft skills and leadership.

The future program needs the evolution of existing courses to better cover digitalization. The students also need options to select electives from partnering universities for areas like security, gaps in our own digitalization and sustainability offerings, and for specialized technology courses, e.g., in the areas of material science and additive manufacturing. We expect that most comparable systems engineering programs have similar needs; however, they have to finetune their program development to their own specific situations.

5.6. Future Research

Collecting better data on alumni achievements from employers and alumni and collecting data for comparison is necessary to evaluate the program’s effectiveness. The program evolution for digitalization and sustainability needs monitoring and evaluation to maintain focus and to be effective. Additionally, investigating the integration of leadership and soft skills development within the systems engineering curriculum could lead to improvements in producing well-rounded professionals who excel in various roles within organizations.