Improving Decision Quality in High-Tech System Design: An In-Depth Study Leveraging Industry Expertise

Abstract

Technological advancement is driven by the development of high-tech systems. However, growing demands on technology lead to more system complexity and to complexity in developing organizations. Therefore, developing these high-tech systems requires systems engineering and high-quality decision-making to balance high performance requirements and a short time to market. In this work, we explore how to improve decision quality within the context of a partner company from the semiconductor industry that deals with these challenges daily. We use triangulation of literature, a survey of 93 systems engineers from the company, and case studies of twelve decisions at the same company. The results highlight three main aspects pertaining to decision quality, namely (1) focusing on the preparation of the decision, (2) gathering the right knowledge and information, and (3) prioritizing (pre-decision) stakeholder alignment. These results will be used to develop a support for the preparation of high-tech systems design decisions, focusing on what knowledge is required and where that knowledge comes from. We found that these are the most important aspects for improving decision quality in high-tech systems development with growing complexity.

1. Introduction

High-tech systems development is characterized by high performance requirements and tight development timelines. A short time to market is required for the success of new products [1], but fast results need to be combined with thorough analysis and high-quality engineering practices. This is challenging and requires careful balancing of many aspects. As systems grow in size and complexity, so do the organizations developing these systems, which makes this balancing increasingly difficult. Furthermore, the consequences of decisions are generally far removed from their causes, and mistakes are therefore costly to repair. This is especially true for early decisions, as they can have a considerable impact on the success of system development; see the work of Tan et al. [2], for example. These decisions are also necessarily made with the least certainty. Wrong decisions can lead to lower competitiveness for the organization or lower system performance for its customers. Growing system capabilities therefore require the improvement of development practices and quality decision-making.

Decision-making, as a topic, has been extensively researched in different fields. However, increasing system complexity and, therefore, increasing organization complexity [3] lead to the state of the art seemingly reaching its limits. With ever more aspects to balance in decision-making, including an increasing number of interfaces, newly developed technologies, and new key drivers such as sustainability, current practices strongly depend on the expertise and experience of the decision-makers. In this work, we seek to explore what is important in systems design decision-making practice in order to improve it with a support.

Our research focuses on system design decision-making in the context of the semiconductor equipment industry. This fast-growing industry is expected to keep on growing at pace with the growing use of semiconductor devices as technology becomes more and more indispensable [4,5]. Figure 1 shows how technological advancement, enabled by increasingly complex semiconductors, requires progressively more advanced semiconductor manufacturing capabilities. The need for these improved capabilities drives complexity at companies that develop equipment, making decision-making more challenging and ultimately requiring high decision quality. This overview shows how complexity, performance requirements, uncertainty, and short development timelines all affect decision-making in this context.

The central results reported in this work are factors required for decision-making quality in high-tech system design. We collected expert opinions through an extensive survey of 93 participants from an industrial partner and compare these to decision-making practices in multiple case studies in the same organization. We compare these results to the state of the art from the literature, allowing us to summarize the main requirements for high-quality system design decisions and work towards improved decision-making through a support. Therefore, the following are our main research questions:

1

What are the main requirements for quality design decision-making in high-tech system development?

2

How can these requirements be used in a future decision-making support?

Beyond these research questions, our study has several goals:

• Understand the context of complex system development in the high-tech semiconductor industry;
• Gain a common understanding of decision-making challenges encountered by the industrial partner;
• Identify directions in which to improve decision-making quality for a to-be-developed support.

This research is part of the SPLASH project at the University of Twente, in which we collaborate with an industrial partner company that develops semiconductor manufacturing equipment. The partner company has a dedicated systems engineering department that leads all big systems design decisions. They have a high combined level of experience in decision-making but also run into the challenge of growing complexity. Through collaboration with an industrial partner, we can use this industry as a laboratory [7], ensuring that research directions match industrial practice. This results in findings that are simultaneously grounded in practice and relevant for industry.

The structure of this paper is outlined as follows: We start by exploring background knowledge and the research context in Section 2, focusing on complex system development and decision-making. Next, we explain our methodology and specific methods in Section 3, followed by our results in Section 4. In Section 5, we discuss the results and address what they mean for the development of future decision-making supports. We also address limitations and potential biases. We show our conclusions in Section 6 and look at future work in Section 7.

2. Background

2.1. Developing High-Tech Systems

Designing high-tech systems is a balancing act between technical capabilities, available resources, and stakeholder values. High-tech engineered systems commonly have large and diverse sets of stakeholders, contain many interfaces, and are developed under uncertainty. References such as the INCOSE Systems Engineering handbook [8] and the Systems Engineering Body of Knowledge (SEBoK) [9], as well as models such as CAFCR [10], show which aspects should be part of a system architecture. They can aid in the design process by provide support in viewing the problem and potential solutions.

There are many methods and processes for realizing systems, described in models such as the waterfall model [11], the vee model [12], and the spiral development model [13]. These all lead from initial ideas to a realized system or product (and to a disposed system if we take the complete life cycle into account). At many points during the design, decisions have to be made that anchor the design and form the steps towards its realization. A common standard (endorsed by INCOSE [14] and the SEBoK [15], for example) for developing complex systems is ISO/IEC/IEEE standard 15288, Systems and software engineering—System life cycle processes [16]. It defines, among others, technical and technical management processes for the development of systems. This standard addresses decision-making in the decision management process in clause 6.3.3 [16] (pp. 47–49). Related to the decision management process is the system analysis process, which can be used to provide input for the decisions to be made.

A challenge in systems engineering and systems architecting is that the most impactful decisions have to be made when there is the most uncertainty [17] (p. 24). When the uncertainty cannot be reduced without spending effort or resources that go towards realizing the system, design decisions have to be made based on experience and general good practices. Systems architecting then starts taking after an art more than a science [18].

2.2. Decision-Making

Decision-making in the context of systems engineering often refers to finding and implementing one alternative from a set of alternatives to try to improve a situation. This form of making decisions is called rational decision-making [19]. A decision will set or change a formally agreed-upon state of a system design or architecture (the baseline [20]). INCOSE’s Decision Analysis Working Group [21] listed ten activities of rational decision-making and related these activities to systems engineering processes [22]. The activities relate to the system decision process [23], which prescribes a comprehensive process for decision-making in a systems engineering context. The systems decision process is grounded in decision analysis [24,25], a field in operations research that approaches decision-making as a logical activity that can be studied and improved. According to decision analysis, a decision is an “irreversible allocation of resources” [26]. Whether this is money spent when parts are acquired or time spent on design, the resources are used, and changing a decision therefore always comes at a cost.

Improving decision-making requires a measure of decision quality. While it would be the most logical to compare decisions to outcomes, this is often not feasible in practice. The time between a decision and a measurable outcome might be too long, and complexity could obscure the relation. Furthermore, there is always uncertainty, which can lead to adverse outcomes, even if the quality of the decision was good (and vice versa). Judging outcomes is therefore advantageous for improving a learning organization [27] in the long term but not for judging individual decisions. Decision analysis therefore proposes six aspects of decision quality [28,29] that can be judged at the moment of the decision. The partner company in this project also uses the concepts of decision quality but simultaneously relies on several decades of experience of their experts. Decision quality in a practical systems engineering setting therefore likely requires a sharper definition.

Besides rational decision-making and the normative field of decision analysis, many studies focus on descriptive models of decision-making. Whereas prescriptive models specify how decisions should be made, descriptive models aim to model how decisions are actually made. Bounded rationality, for example, shows that decision-makers often do not pursue the optimal solution [30]. The field of naturalistic decision-making models expert decision-making and is used as a source of inspiration for knowledge management in organizations [31]. The dual process theory [32] splits decision-making explicitly into an intuitive and a thinking side. Turpin and Marais [33] show that experienced decision-makers in organizations also rely on this intuition in addition to rational decision-making. In organizational settings, however, complex decisions with considerable consequences are often made in formal or structured meetings with specific rationality and processes. These formal meetings ensure more rigorous decisions compared to informal meetings but might take more effort, making them slower than informal decision-making. In a high-speed environment where the time to market is short, this creates a tension where rigid processes might be traded off for faster results.

2.3. Research Context

Cooperation with a company facilitates research methods that support knowledge creation based on practice. The company involved in this work develops semiconductor manufacturing equipment that consists of complex high-tech hardware and software systems. It is a large organization that runs multiple development programs in parallel, which will not be named for confidentiality reasons. The industry developing semiconductor devices and the high-tech equipment required to produce them is characterized by high performance and short times to market [1,34]. Organizations developing manufacturing equipment for semiconductors create value by developing new capabilities and bringing them to market rapidly; the earlier new capabilities are available, the more value there is for customers. System development at the partner company is assisted by a network of suppliers that deliver parts or complete subsystems. The partner company has grown significantly in recent years and has more than doubled its number of employees in five years (by its own account). Due to the large growth and increasing complexity of the company’s systems, knowledge within the company is becoming more spread out and fragmented. At the same time, design decisions increasingly comprise more aspects to balance than before and are guided by more key drivers. Examples of such newly introduced key drivers are the energy efficiency of systems and the reuse of materials.

The technical side of system development at the partner company is led by a dedicated systems engineering department. This development is guided by a product creation process [35] (Section 1.3). High-impact decisions at the system level are made within the context of stages of development through a formal structured decision meeting process. These meetings are planned for all system- and subsystem-level decisions and are chaired by systems engineers. This way of working satisfies the decision management process as described in standard 15288. These systems engineers can take on eight different positions within the systems engineering department: (1) Systems Engineers with a (sub)function or subsystem focus, (2) Systems Engineers with a product (the whole system) focus who function as project leads, (3) Systems Engineers with a supersystem focus, (4) Systems Architects, (5) group leaders with a people focus, (6) systems engineering program managers, (7) Systems Engineers with a focus on improving the discipline itself, and (8) Systems Engineers with a focus on customer sites. Some people combine roles from different positions, similarly to what was described by Sheard [36]. Systems engineers in different positions likely view decision-making in different ways.

3. Methods

3.1. Overall Methodology

Our project aims to improve system design decisions in the high-tech industry; therefore, we work with industry as a laboratory [7]. We have access to the industrial setting described in Section 2.3 to learn and also immediately apply new knowledge. Our methods are qualitative, as insights and underlying relations are necessary to investigate the requirements for high-quality decision-making in systems engineering practice. To reduce the risk of bias, we apply methodological triangulation [37] (p. 295). Similar to determining an exact location on a map using multiple reference points, triangulation in research refers to studying phenomena using multiple sources obtained via different methods to find the common truth [37,38,39]. Figure 2 shows the methods we used in this study and how we combine them to form a coherent picture.

3.2. Literature

State-of-the-art approaches and knowledge from literature form the foundation for all methods. Background knowledge was found through structured and unstructured searches and snowballing. The findings from the literature can be seen throughout this work where relevant. Results obtained in this work also prompted new searches for related literature to place the results in context and compare them to other works. We used this to support our discussions.

3.3. Survey

We sent out a survey to ask systems engineering experts from the partner company their views on decision-making and decision quality. This survey resulted in 93 responses. It was designed and reviewed in a pilot with a core team of systems engineers that is involved in our project. Participants were found by sending an email with a participation request to the entire systems engineering department. Participation was voluntary, and besides being from the systems engineering department, no other selection criteria were used. This study was reviewed and approved by the ethics board of the Engineering Technology faculty of the University of Twente under application numbers 230432 and 240305.

An overview of respondents’ experience is shown in

Table 1. Number of respondents from the survey population with certain years of experience.

Years of Experience ...	<1 Year	1–3 Years	3–5 Years	5–10 Years	>10 Years
...at the company	0	0	3	17	73
...in systems engineering	16	15	17	15	30

. Most respondents (87 out of 93) had more than 10 years of work experience, and the remaining 6 had more than 5 years of experience. The amount of experience within the company is also high for most respondents (shown in the first row), while the amount of SE experience is more varied among the respondents (shown in the second row). The partner company treats systems engineering as technical leadership, which implies that all employees working in the systems engineering department are knowledgeable and have significant experience with their systems.

To understand their decision-making better, we focused on the company’s structured decision meetings in the survey. We did this by asking participants to indicate their opinion on their current decision-making meetings and what could be improved according to them. This was done in three parts: (1) several questions to understand their current way of work and their structured decision meetings, (2) an open question about what could be improved in their current structured decision meetings, and (3) an open question about what is needed for better decision-making in general. In this work, we replaced the company’s specific term for the meetings with “structured decision meetings” for confidentiality.

Table 2. Survey questions used to collect opinions on the current way of making important decisions and directions for improvement.

Part	#	Survey Questions
(1)	1.	I prepare all structured decision meetings I attend (as a stakeholder) well
	2.	When I am a stakeholder at a structured decision meeting, I usually gain the information and knowledge that I require
	3.	I always manage to get my perspective taken into account at structured decision meetings where I am a stakeholder
	4.	I would like to see more informal meetings instead of structured decision meetings
	5.	I would like to see more structured decision meetings instead of informal meetings
(2)	6.	Based on the answers in this section, do you have any suggestions for improving or changing the structured decision meetings? What impact would you expect if this change is made?
(3)	7.	What would improved decision-making look like to you?

shows the survey questions on the partner company’s current way of making decisions. Part (1), as described above, consists of questions 1 to 5, for which participants had to indicate their opinion on a 5-point Likert scale ranging from strongly disagree to strongly agree. Questions 6 and 7 are open questions and match parts (2) and (3), respectively.

The answers to the open questions of parts (2) and (3) required further processing. As shown in Figure 3, we used an inductive open coding protocol, and all codes were reviewed by a colleague familiar with the research and adjusted in consultation. Here, we present all the codes that captured the answers from at least two survey participants. Other codes can be found in the complete dataset underlying this work [40].

Another section of the survey focused on decision quality and what experts, based on their practice, would add to the existing knowledge on this topic. We previously published these results on factors for decision quality [41]. The most important factors according to the experts were knowledge about the system and pre-decision alignment with all internal and external stakeholders. In this work, we continue the analysis of these results based on participants’ company roles and experience levels.

Submitting a survey response required answering mostly mandatory questions to ensure responses contained valid data. To address the potential issue of insincere responses, mandatory closed survey questions offered the extra option of “Not applicable” or “I’d rather not say” when those questions might not apply to everyone or might be considered too personal. Open questions were non-mandatory and could be skipped. The group of 93 respondents represents approximately 36% of the systems engineering department, providing a representative view of knowledge within the department. A total of three respondents made use of the “Not applicable” option at least once, but all three also added suggestions in the open questions, which shows that they were sincere in their responses. Our analysis is therefore based on all 93 responses, with “Not applicable” answers left out where relevant.

Roles and Experience Levels of Survey Participants

We analyzed whether the decision quality factors and aspects are universally recognized or whether there are differences between different types of systems engineers. This was prompted by several noticeable results that were partly already discussed in our earlier work [41] (where the study population’s opinions were split on the importance of one specific quality factor). Finding differences between groups would also influence support development, as different groups might have different needs from the support.

The partner company uses its own terminology for its official systems engineering positions, and we therefore categorized the company-specific positions based on universally recognized systems engineering roles for generalization. This was done based on the work of Hutchison et al. [42], who, based on the work of Sheard [36], defined fifteen systems engineering roles and categorized them in three different focus areas. These categories are (1) roles with a system design focus, (2) roles with a process and organization focus, and (3) roles with a focus on the systems engineering teams. Combined with three levels of experience (junior, mid-level, and senior), we created nine groups out of the study population. Figure 4 shows the criteria for grouping the participants. Only one systems engineer fit the process and organization focus area and was therefore grouped in other/unknown. In this picture, the supersystem focus entails a position that is in charge of system roadmaps and gathers the needs of (potential) customers before development. The customer site focus provides a customer interface during system operation.

After categorization of participants, the experience level of participants was assessed by the main author by comparison with the participants’ own descriptions. As a result, we made 19 manual changes to the experience levels to account for other indicators of experience in the context of this study. The final distribution of focus area and experience levels is shown in Figure 5.

We analyzed the answers to all survey questions based on the grouping. This includes coded answers and rankings of decision quality factors from our previous work [41]. We used descriptive statistics and visual analysis to check for notable differences between groups. Notable results are presented in this work, and we do not elaborate on the results that show no differences compared to the analysis of the entire population. All data, including the results that show no differences, can be found in the dataset underlying this work [40].

3.4. Case Studies

We analyzed twelve past decisions in case studies from a representative project at the partner organization. The (cyber–physical) system under development in this project was an improvement of an existing product. It is highly complex, containing many parts and inter-relations; large in scale; and the result of many disciplines working together. Performance, cost, and time to market were important drivers, and decisions therefore had to be made while accepting uncertainty. A lot of knowledge from the prior system could be assumed to be present in the company. However, this knowledge was also scattered because of the time that had passed.

The decision cases were chosen together with a senior systems engineer (design focus, Figure 4). Cases had to fit four constraints: (1) the outcomes of the decisions had to be clear at the time of the study so that they could be analyzed, (2) the cases had to be recent enough so that inquiry was still possible and people could reliably recall details, (3) the set of decisions had to include a range from clearly positive outcomes to clearly negative outcomes, and (4) supporting data such as the proposal and meeting notes had to be accessible.

We analyzed each decision on the basis of three aspects: (1) What was decided, and what was the context?; (2) What was the eventual outcome?; and (3) What factors influenced the link between decision and outcome? The decisions were all made between 1.5 years and 3.5 years before the beginning of this study. Judgment on the outcomes was provided by the senior systems engineer in consultation with other systems engineers who had been involved. To ensure learning from all factors, this judgment included unintended consequences of the decision—not only whether or not the goal was achieved.

The analysis of the cases consisted of the following steps:

1

Selection of cases together with a senior systems engineer knowledgeable on the subject;

2

Analysis of the decision proposal and the decision meeting notes;

3

For each case, writing down of the context, the decision, and the outcome (where possible);

4

For each case, writing down of the recognized factors linking decision and outcome by the main author and judging of the factors for either a positive or negative influence on the outcome;

5

Checking the correctness of all descriptions with the senior systems engineer in a discussion;

6

Updating of the data based on the discussion and follow-up inquiries between the senior systems engineer and other decision stakeholders;

7

Finalizing the dataset of each case by removing confidential information;

8

Grouping similar factors.

3.5. Additional Research Activities

As part of this research, the main author was also involved in content discussions and several structured decision meetings. This provided opportunities to better understand the context and made it possible to better analyze the survey responses and the cases as described.

In addition, to ensure the generalizability of findings, we performed additional research activities within our project. We checked preliminary results with university students from a multidisciplinary student team in a design workshop focused on making design decisions for their specific challenges. We also held brainstorming sessions with systems engineers from INCOSE The Netherlands [43], looking for problems and solutions in their own context. Both sessions were used to share preliminary results and gather feedback while discussing cases from outside of the partner company.

4. Results

4.1. Survey

Figure 6 shows a summary of the responses to questions one to five of the survey. Answers to question one show that not all respondents come to the meetings well prepared. Next, answers to questions two and three show that systems engineers mostly agree that the current way of making decisions via their structured decision meetings is effective for sharing knowledge and perspectives. Lastly, answers to questions four and five show that respondents seem to value the structured decision meetings but also think that informal meetings are necessary. When looking at individual responses to questions four and five, “Disagree” on one question often matches “Agree” on the other, although a large portion of the respondents (32 out of 93) match a (dis)agree on one question with a “Neither agree nor disagree” on the other question. Six respondents answered “Disagree” on both questions (four and five). All raw data of the survey can be found in the dataset underlying this work [40].

Comparing the different groups, we see that senior systems engineers are slightly more likely to agree on the statements about gaining the knowledge they require and having their perspectives taken into account. For the other statements, the distributions show no clear differences between groups.

Coding of open question six (In what way should the structured decision meetings be improved or changed?) resulted in 27 separate codes. Eight codes captured the answers of more than one respondent and are shown in

Table 3. All (coded) answers to the question “Do you have any suggestions for improving or changing the structured decision meetings?”.

Do You Have Any Suggestions for Improving or Changing the Structured Decision Meetings?	Frequency
Align with stakeholders	9
Limit attendance	7
Include all stakeholders	5
Stick to the template for showing information and discussing	3
Limit them to critical decisions only	2
Improve follow-up of the sessions	2
Prioritize the business case	2
Transparently choose whether or not to have this type of session	2

. Coding of open question number seven (In what way should decision-making be improved in general?) resulted in 60 separate codes. Of these, the ten codes shown in

Table 4. All (coded) answers to the question “What would improved decision-making look like to you?”.

What Would Improved Decision-Making Look like to You?	Frequency
Align with stakeholders	9
Commit to action	6
Have clear what the baseline is	5
Document the decision	4
Include all stakeholders	4
Stick to the template of the structured decision meeting	3
Limit attendance	2
Have a clear trade-table	2
Transparently choose whether or not to have a structured decision meeting	2
Frame the problem correctly	2

captured the answers of more than one respondent. In both cases, alignment with stakeholders is the most frequently given answer.

Analyzing the codes based on focus area and experience level shows several codes that captured answers from specific groups relatively often. There were six mid-level and six senior systems engineers with a teams focus in the study population. Three out of the six answers that were captured by “Meaningful and reliable information” were given by mid-level systems engineers with a teams focus. Similarly, two out of the five answers captured by “Including all stakeholders” and all three answers captured by “Stick to the template of the structured decision meetings” were given by senior systems engineers with a teams focus. All levels of systems engineers with a teams focus, combined, provided three out of the five answers that were captured by “Commitment to action”. The codes that captured the highest number of answers were more often mentioned by senior systems engineers than by mid-level systems engineers, even though the latter group is bigger. The comparison between groups for these codes and all other codes that capture two or more answers can be found in the dataset underlying this work [40].

We found several factors that seem to show a difference between groups when analyzing the ranking of factors from our earlier work [41] based on focus area and experience level. The distribution of ranking results are shown in Figure 7 and Figure 8. For every factor, these figures show the distributions of the positions in the ranking results from the survey. For most factors, medians of the distributions based on focus area and experience level roughly match the median of the distribution of the entire group. In the design focus area, the factor “Knowing all system aspects” seems to become less important with seniority, while the factor “Knowing whom to consult for information/help” seems to become more important with seniority. “Knowing the company’s strategic goals” has a broad distribution (and was ranked first most often) and seems less important for a design focus area. Internal and external stakeholders seem more important for the design focus area. “Having more alternatives” ranks as more important with senior systems engineers compared to junior systems engineers.

4.2. Case Studies

We found 31 unique factors that were recognized by systems engineers of the partner company to have influenced the outcomes of the analyzed decisions.

Table 5. One of the twelve cases analyzed as an example, showing a description of the decision for context, the recognized outcomes, and the identified factors.

Decision: Develop a low-cost version of a subsystem ...
...for similar machine performance. This was done with a good business case, showing the value for the customers, and good knowledge of the system. The only problem was the lack of available time of the development team, resulting in a system with the desired performance and cost but a late delivery.
Identified factors	Recognized outcomes
Having a business case	Lower machine price
Good knowledge of the system	Desired machine performance
Lack of time resources	Delay in project completion

shows an example of the analyzed decisions. All data, consisting of descriptions, outcomes, and the 31 unique factors of the decisions, can be found in the dataset underlying this work [40].

Table 6. Unique factors that were recognized more than once when analyzing 12 decision cases, including a judgment as to whether the factors have a positive influence (P) or a negative influence (N) on the outcome.

Unique Factor	Cases	Judgment
Having a business case	8	P
Proper modeling	4	P
Alignment across subsystem development teams	3	P
Having reliable information	3	P
Engineering mistakes	2	N
Good knowledge of the system	2	P
Improper modeling	2	N
Lack of measurement data	2	N
Missing use cases	2	N

shows the list of factors that were recognized in more than one case and includes the judgment on whether the factors had a positive or a negative influence on the outcome. From the table, three themes can be recognized: (1) factors on framing the problem correctly (1 and 9); (2) factors on required knowledge and information (4 and 6)—and the actions to create this knowledge and information (2, 3, 7, and 8); and (3) an external factor (5—engineering mistakes are not something that can necessarily be prevented at the system level). Other factors, which were only found in one case, are mostly related to these themes. This list of unique factors also includes risk-related factors, resource-related factors, and special cases such as management (wrongly) overturning a decision (a one-off in our study).

4.3. Additional Research Activities

The workshop with the student team found that the students were supported by our knowledge on what to address when making design decisions. However, they still required assistance in how to proceed with their own decisions. This showed us that actual improvement requires more than only presenting knowledge on what factors influence decision quality, at least for junior engineers. Participants in the INCOSE NL workshop discussed many different decision-making challenges and potential solutions. Mentioned challenges ranged from stakeholders blocking decisions to several challenges related to making decisions under time pressure and uncertainty. Compared to the junior engineers, they knew better how to deal with missing knowledge. Examples of solutions that were given included organizing workshops to uncover all hidden information in the group, creating overviews of dependencies, and looking at the risks and value of the decisions.

5. Discussion

The discussion section addresses how the results contribute to the goals of this research. We combine multiple results from triangulation to gain insights based on different topics.

5.1. Decision-Making Practice at the Partner Company

Our survey shows that most systems engineers think that the partner company’s decision-making process is effective. Interestingly, not all participants come to their structured decision meetings well prepared. This is either undesirable or it means that these participants do not consider extensive preparation necessary for effective meetings. However, this last point is in tension with the general respondents’ emphasis on decision preparation as an important factor for decision quality. Any support that decreases preparation effort might therefore increase meeting effectiveness. Even though decision-making is considered effective now, it can and should be improved to account for growing system and organizational complexity. This is supported by the large number of suggestions for improving decision quality given in the survey results.

Opinions on current decision-making show preferences for both more informal and more formal meetings. A few participants did not want more of either and likely find the current balance sufficient or want fewer meetings in general. We think the reason for the differences in opinions is the effort and time that formal meetings take versus the perceived gain in decision quality. Generally, rapid decision-making is necessary in the semiconductor industry, and the context is therefore an important factor.

5.2. Group Differences in Results

Splitting the study population into groups based on focus area and experience level allowed us to analyze the expert opinions from the survey in more detail, i.e., to ask what is important for what type of systems engineer. Tofan et al. [44] show how senior systems engineers face different challenges in decision-making compared to junior systems engineers. Similarly, we expect daily practice and concerns to be different for different groups.

In questions six and seven, a few codes captured responses of systems engineers with a teams focus more often compared to other groups. However, systems engineers with a teams focus contributed relatively often to the non-mandatory open questions in general. Similarly, senior systems engineers also generally provided answers more often. This might be because engineers in senior positions are used to more often sharing their knowledge with others. Since these groups replied more often across all factors, it is difficult to say whether any factors are actually considered more important for a specific group.

In the survey ranking questions, the experience level, indeed, seems to be a strong differentiator for how systems engineers approach decision-making. Tofan et al. [44] show that seniors are more comfortable with uncertainty, which could explain why they find it less important to know all system aspects in our research. The fact that seniors are likely to know their systems better could also play a role. Senior engineers likely also rely more on the network they have built over their career, resulting in the importance of “knowing whom to consult”. The same paper also found a link between decision quality and having more alternatives. This also seems to be recognized by senior systems engineers in our study, albeit not very strongly. The design focus area gives less importance to strategic goals (“too generic to head specific decisions” according to several participants) and complete information (“sometimes [...] not possible”). This category finds stakeholders more important than other categories. Even though systems engineers with a teams focus more often directly deal with external stakeholders, this shows that design is also very much stakeholder-oriented.

Comparing medians does not guarantee statistical significance, and group differences therefore require further research (more on that in Section 5.7). Group medians, at most, differed by 2.5 ranking positions from the overall median and often less. Most concerns seem universal among focus areas and experience levels, and therefore, the overall ranking seems to be a good indicator.

5.3. Main Requirements for Design Decision-Making in High-Tech System Development and Future Support

We recognized three main themes in the triangulation: focus on preparation, knowledge and information gathering, and stakeholder alignment. These themes relate to each other and sometimes overlap: the preparation contains all actions needed before a decision is made. Knowledge and information gathering is one of the actions that need to be performed by the responsible systems engineer. It should, among others, be achieved through stakeholder alignment that assists in creating or transferring knowledge. Stakeholder alignment, however, also aims for shared understanding of the problem and potential solutions.

5.3.1. Focus on Preparation

Many responses to the survey fit the complete system development process related to technical design decisions, similar to our earlier results [41]. Respondents prioritize preparation and follow-up. This is also reflected in the findings from the cases, where many factors relate to decision preparation and ensuring the right information is present. Most factors are not exclusively related to the moment a decision is made, and this matches the “Make and manage decisions” activity from standard 15288 [16] (pp. 47–49).

Decisions require reliable information that is needed to create and judge potential solutions (the “alternatives”) with the right criteria [28]. Engineering problems are dynamic and can change with later insights and design effort during the exploration and preparation of decisions. This can change the initial problem description, potentially increasing the required effort. This effort should therefore be balanced with the desired level of quality, within constraints of time and budget.

5.3.2. Knowledge and Information Gathering

The information required to make good decisions consists of many parts and is context-dependent. For high-tech systems, stakeholder values and key drivers of the organization are important criteria for judging alternatives. This is also recognized by the frequent occurrence of “Having a business case” in our case studies. When developing the state of the art, it is highly uncertain whether designs are actually realizable due to either physical limitations or time and budget constraints. Reducing uncertainty requires knowledge creation, either through the use of tools such as simulation and system analysis or by learning from the experience of others. Knowledge and information are important for decision-making quality [28], and it is important to continually capture all knowledge within the organization to make better decisions [45,46]. In larger organizations, knowledge fragmentation can become a problem [47].

Recurring themes in our results are (pre-decision) alignment with stakeholders and the importance of the right knowledge and information. Both themes show that accessing the organizations’ knowledge is challenging. The respondents to the survey indicated that decisions should be made with a small group (“limit attendance” from survey question 6; see Table 3). However, the growing size and complexity of the organization will increase the difficulties in ensuring that a small group has all the right knowledge for the decision. A common solution for solving this challenge is relying on more experienced people. The field of naturalistic decision-making also recognizes the effectiveness of experts in decision-making and the need to gather the right knowledge. It argues that decisions should be made by experts on the subject [31].

5.3.3. Stakeholder Alignment

Systems engineers deal with complexity [48], and their decision-making is therefore also complex, with many inter-relations and the emergence of system behavior. This requires effort in untangling these relations and making sure everything (and everyone) is aligned. It requires a common understanding of the problem but also of solution directions and their consequences. Several tools and methods already exist that aim to improve shared understanding during system design, such as the A3AO [49], COMBOS [50], the ${\mathrm{N}}^2$ diagram, and the nine-window diagram [17,51]. These tools and methods address communication challenges but do not reduce the number of potential stakeholders that can be and often need to be consulted. This is a problem when complexity grows and the number of sources grows with it.

From our results, it is clear that improving decision-making requires the implementation or gathering of all the prerequisites of decisions. Existing methods, such as those listed in the work of Falessi et al. [52] and the previously mentioned systems decision process [23], do not always include clear instructions for preparation and often require expertise on the side of the decision-maker [53]. Standard 15288 on developing complex systems, for example, advises to “Involve relevant stakeholders in the decision-making to draw on experience and knowledge”. It does not prescribe how to find relevant knowledge and stakeholders or how one can judge their expertise. In this and future work, we explicitly aim to address these prerequisites.

We therefore think that a focus specifically on the generation (through stakeholder alignment), gathering, and use of knowledge is the most important to improve decision-making for higher quality and faster time to market.

5.4. Does Practice Fit Perception?

Comparing the survey to the case studies showed differences in focus. The factors from the cases are more content-focused, while the factors from the survey are more process-focused. Most of the factors found in the cases relate to knowledge and information. They seem to be most often related to the outcome of decisions in this context. The survey participants recognize the importance of knowledge and information, although there is more emphasis on it in our case studies. Similarly, the importance of having a clear view of what is decided upon, or framing the problem correctly, was found both in the survey and the case studies.

Several factors mentioned in the survey did not show up in the case studies. The most significant of these factors are “Limit attendance” and “Commit to action”. Participants want limited attendance at decision meetings to prevent endless discussions and lower efficiency but, at the same time, recognize the need for different perspectives. This creates a tension, and limiting attendance to decision meetings without other changes to the process might result in missing knowledge. We expect this factor to be hard to recognize in a post-decision study, although we did find several factors related to having or missing certain knowledge and information. Commitment to action is a code from the survey (and an element from the decision quality framework [29]) that captures many different answers. Participants emphasize the need for decisions to be actionable and to take this action. This is likely present in all studied cases but was not recognized explicitly.

5.5. Other Trials

Our workshop results support the focus on pre-decision alignment and finding the right stakeholders. Participants in both workshops showed enthusiasm and interest in the subject. The student team especially benefited from explicitly considering the knowledge they needed but did not know how to proceed with the decision afterwards. This shows that guidance on the acquisition of the missing knowledge is needed and that awareness alone is not sufficient for junior engineers. The professionals from INCOSE NL pointed out many decision-making challenges and supported our research in terms of knowledge and information gathering, among others.

5.6. Acknowledging Other Factors

When discussing decision quality, other aspects, such as organizational structure and culture, potentially also have a strong influence. They are often outside the control of the decision-makers but can have a real influence on the outcomes of decisions. In the open survey questions, several participants mentioned factors related to these aspects, but the answers were too diverse to be included in our results. While we found a clear direction in which system design decision-making can be improved, this shows that the influence of numerous other aspects should not be neglected.

5.7. Limitations

Our survey had a large group of respondents with varying levels of experience and systems engineering positions. Unfortunately, the opinions of less experienced systems engineers seem to be under-represented. Our choice of categorical data set the highest category at 10+ years. Most systems engineers had more experience, which made it difficult to differentiate higher levels of experience. The choice of focus areas fits the company, but it is also clear that there is overlap in tasks and daily practice. The results might therefore be sensitive to the exact division and should be interpreted with caution. Testing for statistically significant differences would require more equal group sizes and better differentiation. Furthermore, the survey was only distributed among systems engineers. The improvement of decision-making might require knowledge of what stakeholders of systems engineering find important in decision-making as well (see Section 7).

Several survey questions required participants to judge their own behavior, which might have led to participants giving socially acceptable or desirable answers. The large number of responses and the use of triangulation should have ensured that we also captured other perspectives. As our cases studied considered one representative product from one line of business at the partner company, the results might be more representative for similar products. Thus, other developments might have shown different results.

The need for confidentiality makes it more difficult to fully describe the context of this research. Technical project details and company-specific terminology had to be removed from the data, limiting future comparison with other contexts. We do think, however, that the descriptions in this work are sufficiently clear to allow for general comparisons. Comparisons with other industries would benefit the generalizability of our findings but are outside the scope of this study.

5.8. Reflection on Potential Bias

Because our research was conducted in collaboration with a single company, there is a risk of bias due to limited possibilities for extensive comparison. Problems and potential solutions from and for one specific context might not hold in other contexts.

Collaboration with one expert from the partner company might have introduced bias in the case studies. The number of times the factor “Having a business case” was found to be present is notable compared to the other factors. This might relate to the interests of the expert. We checked results with other systems engineers and looked at different cases to minimize bias, but the results are still dependent on the memories and interpretations of the people involved.

The differences in focus between the results of the survey and the results of the case studies are partly due to the way we collected data; the survey required general answers, while the case studies focused on content. We also think that mistakes (or things that went well) in the process might not be as evident as missing certain information or knowledge. As one systems engineer explained: “When things go right people don’t see it as following a process, but just how we have always done things”. This demonstrates that people are not always aware of processes.

Lastly, we must address the role of the authors. In qualitative research, the observations and notes of the researcher influence the results. As explained before, we aimed to minimize bias by using several sources of data in triangulation.

6. Conclusions

Our study’s aim was to find out what decision quality means in practice and how it can be addressed in actual cases at a high-tech firm. In the context of our study, we found that the generation and gathering of knowledge and information were seen as the most important. The emphasis is on aligning with and among stakeholders in order to find the required knowledge and create shared understanding of the problem and potential solutions. This can lead to higher quality systems and faster time to market in large (and growing) organizations developing complex cyber–physical systems. A support aimed at improving decision-making using this knowledge should provide most guidance to more junior systems engineers. All levels of experience can benefit from an explicit structure for decision-making. The most pressing issues to address seem universal, but the different groups of systems engineers found different aspects of decision-making more important for their practice.

In summary, our study shows that the biggest challenges in decision-making in growing and increasingly complex high-tech system design revolve around the generation, gathering, and use of knowledge and information. These challenges should be addressed in decision preparation through alignment with stakeholders.

7. Future Work

Firstly, our future work will focus on the development of a support for better decision-making driven by the results reported in this paper. In other work, we looked at the key factor of alignment for decisions to guide the development of the support [54]. Decreasing the effort required for alignment during decision preparation should lower the time spent on decisions (for similar quality) or increase decision quality (with similar time spent). We aim to achieve this by introducing a communication tool and a method that assists in finding stakeholders and knowledge holders, in addition to structuring alignment. Our future work consists of developing and validating this support at the partner company, as well as in the context of other industries. Secondly, systems engineers are the technical lead for many design decisions, and stakeholders outside the systems engineering department deal with most of the consequences. Part of our future work will therefore be to conduct the survey with these stakeholders as well. This should lead to a better understanding of decision quality by showing the stakeholders’ perspectives on high-quality design decisions. Thirdly, in addition to evaluating the support in different contexts, conducting the survey in other industries would enhance the generalizability of the findings. Lastly, a further analysis using a larger sample size to test for statistically significant differences between categories of systems engineers would enhance the understanding of what decision quality means to systems engineers with different levels of experience and different focus areas.