An Approach to Represent and Communicate Product or System Design Ideas at the Fuzzy-Front End of the Design Process

Abstract

The primary challenge underscored and dealt with was how to represent the product’s or system’s use environment and processes and to communicate ideas and envisaged use contexts effectively at the fuzzy-front early stages of the design process. The work focused specifically on complex products or systems with physical, software and/or cyber components, and the question was how to represent, e.g., the operations of the product or system and the interactions between the user and the product or system betimes in the period between when an opportunity for a new product or system is first considered, and when the idea is judged to be ready to enter formal development. Several approaches are currently being used to express and to communicate ideas at the conceptualization, embodiment, and detail design stages of the design process, but none of them address the challenge described above. We therefore adapted and extended the abstract prototyping concept to allow for total representation of ideas, as well as of use environments and processes early on. Extended abstract prototyping (Ext-AP) entails using combinations of low and high-fidelity prototyping techniques to create cognitive virtual representations, which represent and help designers to express ideas and use contexts—namely, what complex product or system would be like, and how its users would interact with it. Real-world product development case studies have been used to demonstrate how the Ext-AP technique can be put into practice. One of the main observations from the application case studies is that the Ext-AP technique enabled the subjects to express ideas and use contexts more effectively early on. In addition, the extended abstract prototypes (Ext-APs) offered a low cost, yet effective solution for expressing ideas, representing concepts and using contexts, and allowed the subjects to think divergently, make associations, easily and quickly construct, combine, and evaluate alternatives, and work together on multiple ideas simultaneously.

1. Introduction

The designers’ understanding of what the designed product or system should be like and how it should function and interact with the user and its environment is vital for solving the underlying engineering design problems, achieving quality, and for the success of the eventual product or system in the marketplace. It is broadly known that problem-solving in engineering design in particular requires collective rational analysis of the needs, requirements, and constraints, synthesis of solutions to the problems, and sensible evaluation of design concepts [1,2,3]. Matching the solution concepts with the needs and constraints typically requires creative abilities. Working collaboratively in teams and proper understanding of both the design concepts and the use environments is crucial, as this allows the designers to think divergently, make associations, easily and quickly construct, combine, and evaluate alternatives; and to work on multiple ideas simultaneously. Hence, there is a need for a vigorous and effective technique to support representation and expression of ideas and collaboration in order to achieve better understanding of the products or systems and use contexts. In the reported work, we specifically focused on complex products or systems with interacting and synergistically functioning electro-mechanical, software and/or cyber components or subsystems. The challenge we tackled was how to represent the ideas and the envisaged use processes and interactions with the end user—including the environment in which the product or system will operate, in the period between when an opportunity for a new product or system is first considered, and when the idea is judged to be ready to enter formal development.

The hypothesis was that an aggregated good representation of a concept together with its use environment and processes can be an effective vehicle for sharing the understanding of ideas and a means of vitalizing collaboration. The idea was that a good integrated representation is the key to the understanding of the extent of fulfilment of the needs and to uncovering and avoiding the problems that the users might encounter in interacting with the intended product or system. In traditional product or system development processes, virtual and/or physical models or prototypes are used to communicate ideas and to enhance the understanding of the envisaged products or systems. Physical models or prototypes are known to provide more information. There are numerous methods and tools that can be used to build virtual or physical models or prototypes. Apart from typically being used to test concepts or processes, these models or prototypes also act as objects to learned from or be imitated. However, one of the shortfalls of the existing product or system concepts testing approaches and practices, which typically involves using physical prototyping techniques (in which prototypes or models produced through processes such as stereolithography, selective laser sintering, fused deposition manufacturing, laminated object manufacturing, ballistic particle manufacturing, 3D Printing, or computerized numerical control (CNC) machining are used) or virtual prototyping techniques (in which virtual reality (VR) methods and tools—including immersive technologies such as head/helmet-mounted displays (HMDs) or cave automatic virtual environment (CAVE) systems are used) is that the models or prototypes built usually do not represent the use environments or how the product or system would interact with the end-user. Apart from this, building a virtual or a physical prototype or model often requires significant investment in time and some of the resources needed are expensive. Furthermore, the very fact that the existing virtual and physical prototyping approaches do not entail prototyping of the use environment or how the product or system would interact with the end user limits their effectiveness in terms of supporting the designer, e.g., in keeping an eye on the use environment and processes whilst making associations, constructing, combining, and evaluating alternatives, or when working on multiple ideas. This, as a result, affects the decisions made and the creativity of the designer—we consider creativity as an ability to produce a solution that is new, i.e., which is novel, has not been experienced previously, and is fitting or meeting the product’s or system’s needs and constraints—see e.g., [4,5].

What is needed is an approach that would allow designers to easily and quickly express their ideas and represent the product or system and its use environment and processes. The inherent challenges in the preliminary development stage such as how to handle complexity of products or systems with physical, software or cyber components, deal with the incompleteness and vagueness of the needs and requirements, deal with emerging requirements, or how to handle new knowledge and ideas need to be addressed. It is also important to point out that in most product or system development process models, prototyping is a step between detail design and concepts evaluation, and as a result, most of the present prototyping practices normally aims to support the designers in these latter design stages. Overall, most of the existing prototyping approaches, methods, and tools are not intended to support the designers at the early stages of the development process. Furthermore, although the prototypes built by using the existing prototyping techniques characteristically fully portray the appearance and functionality of the intended product or system, they however hardly capture the operations, the interactions between the product or system and its end-user, or use contexts—see, e.g., [6]. Capturing the use contexts is vital to assuring the acceptability of the eventual product or system and the challenge that needs to be addressed here, basically, is how to represent the use processes and how to subsequently gather and to organize use process information. So, in short, an effective technique for enabling the designers to garner good insights into the intended products or systems and their use contexts at the fuzzy-front end stage of the development process is needed. Such technique should allow the designers to comprehensively explore concepts, including how the product or system will be like and how it will eventually function and interact with the user and its environment.

In light of the above discussion, there is an apparent need to develop effective technique to support the designers to express and represent conceptual ideas of complex products or systems with physical, software and/or cyber components. In particular, the theoretical, methodological, and implementation challenges of creating effective all-inclusive representations which comprehensively express concepts and use contexts need to be tackled. We attempted to address the challenges described above. This paper is intended to facilitate discussion among researchers, practitioners and other stakeholders on how to address the underlying issues and to propose a possible way forward. The paper is structured as follows. We first briefly review the state of the art in the following Section and present the concept we proposed for representing product or system ideas in Section 3. We then present the concept we propose for expressing and representing product or system ideas together with their use contexts in Section 4. Afterward, we identify different types of existing tools and multimedia presentation technologies that can be used in building effective representations and discuss how they can be applied in practice. Next, we illustrate, by using various application case studies, the applicability of the proposed concepts. Finally, we discuss the results and give some conclusions.

2. State of the Art Review

Literature that specifically addresses the challenges of ideation and representation of concepts together with use contexts of complex products or systems with interacting and synergistically functioning electro-mechanical, software and/or cyber components or subsystems at the fuzzy-front end stage of the development processes and how the way a concept is represented affects decision making are very scarce. In fact, we have not come across any literature that specifically deals with the challenge of representation of concepts and use contexts together in a common integrated representation. However, as we hinted out in Section 1, there are several virtual and physical prototyping techniques that can be used to build various types of virtual and physical prototypes to test concepts and/or serve as objects to learn from or to be imitated. The focus of most of the existing prototyping practices is exclusively on representing product or system concepts only—after an idea has been adjudged to be ready to enter formal development. Overall, the existing techniques are used in building prototypes which typically provide: the look and the feel of what the product or system (or its key parts) would be like [7], means of exploration of various aspects of the product or system [8], and/or working or test model that implements certain aspects of the imagined product or system [9]. One of the cardinal limitations of these prototypes is that the use environment and processes are not embedded or portrayed by the prototypes. As a result, these prototypes do not sufficiently help the designers to communicate ideas and use contexts, and in a way hinders thoroughness of analysis since the associations that the designers make, the alternatives that they construct, or the ideas represented by using these prototypes exclude use processes and environments. Combined representations of product or system concepts together with the use environments and processes have great potential. Apart from engineering, it is also important to note that modelling of use processes is already a common practice in other fields, for instance, in business, where accurate capture of the processes of a business is usually a vital part of any analysis [10,11]. Software tools are used, for instance, to document the underlying processes, tie the activities together, and to keep track of inputs and outputs. These tools can help, e.g., to uncover disconnects, capture details, or to point out to process errors, discrepancies, or overlaps. They may also be used differently, e.g., to represent gathered information graphically (i.e., to create visual process diagrams) and to show activities and information flows within complex business processes.

There are different ways of achieving innovative solutions. One way, which is relevant to the designers—see, e.g., [12,13] is through collaborative exploration. Typically, in the real-world, to resolve complex design problems and to spur creation of novel solutions, the designers often work together in multi-disciplinary teams and collaborate within and beyond the organization—see, e.g., [12,13]. It is widely acknowledged from psychology and other studies that the advantage of collaboration arises from shared resources and inventories, and from divergent backgrounds and perspectives of the collaborators [14,15,16]. While shared resources help discover, e.g., constraints for problem-solving [17], shared inventories inspire new ways of solution synthesis [18]. When collaborating, the participating parties share and analyse alternative solutions generated by experts with a wide spectrum of knowledge, expertise, and skills. Visual representations are central to attaining effective collaboration and vital in achieving creative solutions because they are means of communication that most effectively encapsulates and conveys information to collaborating parties—see, e.g., [19]. However, it is important to note that when collaborating by using visual representations, there is always a chance that the meanings of the represented design solution may be swayed, say, due to the dynamics of the activities carried out, interpretations during design, or due to the abstractions or simplifications of information, and this may, in some instances, lead to misinterpretation of solutions—see, e.g., [12,20]. Apart from collaboration, it is also imperative to emphasize that practices such as thinking divergently, making associations, constructing and combining alternative solutions, or working on many ideas simultaneously, as well as personal aspects such as motivation, flexibility, tolerance for independence, technical skills, knowledge of a field, or even mental health also influence creativity—see, e.g., [1].

In short, software and cyber components are increasingly becoming important elements in most of today’s products or systems with physical constructions. As a result, products and systems are increasingly becoming more and more complex. Researchers and practitioners generally see the need for new meticulous methods and techniques for handling complexity in the processes of development of these sorts of products or systems, and research in this direction is attracting the attention of many systems engineering researchers—see, e.g., [21,22] and of many research funding agencies—see e.g., [23]. There is a need for a dedicated and effective technique to support the designers to externalize ideas about complex products or systems at the fuzzy-front end stages of the development processes and to spur generation of creative ideas and realization of innovative solutions. Such a technique should provide a systematic way of helping the designers and engineers to express and represent their ideas and to explore various quality aspects such as functionality and usability whilst keeping an eye on the use environment and contexts. We present the technique we propose for expressing and representing ideas in the following Section.

3. Expressing and Representing Ideas

The fuzzy-front early stage of the process of development of complex product or system with physical, software and/or cyber components typically consists of multiple interrelated preliminary activities such as (1) development or selection of appropriate engineering principles or theories/concepts; (2) creation of methods or ways to put the developed concepts or the identified theories or engineering principles to use; (3) development of algorithms or architectural layouts; and (4) pilot implementation of the envisioned product or system. The de facto fuzzy-front end development process can be described as follows. Having known the needs and the constraints on the solutions and having defined the requirements, the developers define features of the envisaged product or system and formulate or select appropriate theories/concepts or engineering principles for each feature or set of features. Based on the defined theories/concepts or engineering principles, the underlying methods or functional units are created. The algorithms or architectural layouts are then created and pilot prototypes built. For software products, for example, algorithms are developed, and codes for the key features of software are written and tested. The initial activities of formulation, identification, or realization of appropriate theories/concepts or engineering principles are rather highly informal and typically happen intuitively. The designers are often faced with the challenge of expressing and representing their ideas about the in-process deliverables at the fuzzy-front early development stage.

As an attempt to formalize the above-described instinctive procedure and to enable the designers to express and to represent their ideas to allow more reviews and tests to be carried out at the fuzzy-front end stage of the development process, we developed a technique called abstract prototyping (see Figure 1). This technique emulates the above-described de facto practice, and defines the stages of the preliminary development process as creation or selection of theories/concepts or engineering principles, formulation or identification of methods or functional units, realization of algorithms or architectural layout, and development or writing codes for a pilot prototype. Abstract prototyping allows faults to be traced back to the requirements, theories/concepts or engineering principles, methods or functional units, algorithms or architectural layouts, or pilot prototypes rather than exclusively to the requirements or to the final design of the product or system. According to this technique, abstract prototypes of different in-process deliverables are progressively built and reviews are subsequently conducted to eliminate faults before the in-process deliverables (i.e., theories/concepts or engineering principles, methods or functional units, algorithms or architectural layout, or pilot prototypes) are passed to the subsequent stages. Scores assigned by some selected stakeholders of the products or systems to express their opinions on how well the in-process deliverable fulfils the requirements can be processed and used as metrics for evaluation of the acceptability and for estimation of the extent to which these fuzzy-front end stage in-process deliverables meet various quality requirements. We describe the fundamentals of abstract prototyping in the following subsections.

Figure 1. A general scheme for upfront representation of concepts in the framework of the abstract prototyping technique, refer also to [24,25].

Figure 1. A general scheme for upfront representation of concepts in the framework of the abstract prototyping technique, refer also to [24,25].

3.1. The Basics of Abstract Prototyping

In the development of complex engineering products or systems with physical, software and/or cyber components, it is important to ensure that right theories/concepts or engineering principles, methods/functional units, or algorithms/designs are used. It is also imperative to detect and to eliminate flaws early on as far as possible, preferably prior to commencement of production or coding. This is because, if a solution concept is changed later on after engaging in production or coding, overly demanding modifications might be needed to accommodate the changes. For example, for software products, large segments of code may have to be rewritten. The abstract prototyping technique has primarily been developed to ensure that proven theories/concepts or engineering principles, methods or functional units, and algorithms or designs are adopted. It is also meant to encourage and to facilitate early discovery of faults. To this end, abstract prototyping serves as a prototyping and review technique for the in-process deliverables of the fuzzy-front end stage of the development process—see, e.g., [24,25,26,27,28]. That is, theories/concepts or engineering principles, methods or functional units, algorithms or architectural layouts, and pilot prototypes are treated as reviewable or testable implementations.

The proposed four-stage scheme shown in Figure 1 guides the prototyping and review or testing processes and aims to ensure that the in-process deliverables are of sufficient quality. In the framework of the abstract prototyping technique, reviews or tests are carried out time and again in different contexts, by involving various stakeholders as subjects. Theories/concepts, engineering principles, algorithms, or architectural layouts are highly technical in-process deliverables and can best be reviewed by involving the developers and experts as subjects. Methods, functional units, and pilot prototypes are somewhat less technical and easy to perceive, and non-expert end-users may therefore be involved as subjects in review them. Prototypes of the in-process deliverables (referred to as abstract prototypes in this paper) are built and provide the basis for discussing and clarifying the ideas put forward by the developers. A review set-up typically consists of a panel of the representatives of the stakeholders’ community, who give opinions, e.g., regarding the extent to which the in-process deliverables fulfil the requirements. The opinions are subsequently analysed, and in this way, flaws in the in-process deliverables can be spotted early on as they evolve. The in-process deliverable can then be improved to eliminate the spotted flaws. The reworked in-process deliverable can subsequently be reviewed again before being passed to the next stage. The goal here is to ensure that all flaws are identified and eliminated closer to their roots.

In a nutshell, the principal tenets of abstract prototyping can be summarized as follows: (a) the preliminary development process of complex engineering products or systems with physical, software and/or cyber components passes through some de facto stages at which many activities are performed, and, as a result, various distinctive in-process deliverables are produced; (b) theories/concepts or engineering principles, methods or functional units, algorithms or architectural layouts, and pilot prototypes are the main reviewable or testable in-process deliverables at the preliminary fuzzy-front phase of development process—we refer to the stages at which these in-process deliverables are realized as levels of abstraction; (c) requirements are clustered by considering which levels of abstraction they can effectively be tested, and are used as yardsticks during reviews or testing; and (d) as shown in Figure 1, the representatives of the stakeholders—broadly classified as users and developers—are interchangeably involved in reviewing the in-process deliverables as they evolve.

3.2. Concept Evolution in the Context of Abstract Prototyping

Figure 1 depicts various kinds of likely transformations between the acquisition of requirements and realization of the expected product or system in the context of abstract prototyping, and how the stakeholders are involved in testing or reviews. The main path is the theories/concepts or engineering principles—methods or functional units—algorithms or architectural layout—pilot prototypes—final product or system concept (1-2-3-4-5) path. Having started by developing or selecting the appropriate sets of theories/concepts or engineering principles for solving an engineering problem, the designer further develops and transforms the solution concepts, and the transformed solutions appear under different guises, namely as a system of methods or functional units, algorithms or architectural layout, pilot prototype, and finally as the expected product or system functionality. The 1-2-3-4-5 path spontaneously guides the preliminary development process through the basic product or system evolution stages. There are other possible transformation scenarios, for instance, starting straight away to write codes for the expected functionality or to make the final product or system concept based on knowledge of theories/concepts/engineering principles (1-5), theories/concepts or engineering principles and methods or functional units (1-2-5), or theories/concepts or engineering principles, methods or functional units and algorithms or architectural layout (1-2-3-5), or vice versa. There are also some short cuts such as writing codes for the expected functionality or making the final product or system based exclusively on the understanding of the methods/functional units (2-5), algorithms or architectural layout (3-5), theories/concepts or engineering principles and algorithms or architectural layout (1-3-5), theories/concepts or principles and pilot prototypes of the expected functionality or final product or system concept (1-4-5), or methods or functional units and pilot prototypes of the expected functionality or final product or system concept (2-4-5). Depending on the available knowledge, any of the above-mentioned paths may be taken.

4. Expressing and Representing Concepts Together with Use Context

We extended the concept of abstract prototyping that we introduced and discussed in the previous section as an attempt to address the challenges of representing the concepts together with the use environment and processes of complex products or systems with physical, software and/or cyber components. We describe the extended abstract prototyping (Ext-AP) concept and process in the following subsections.

4.1. The Concept of Ext-AP

A concept for expressing and representing ideas and use environment and processes at the fuzzy-front end stage of the product or system development process is proposed. The emphasis in Ext-AP is in supporting the designers to express and to represent solution concepts of the product or system when in operation. An extended abstract prototype (Ext-AP) includes features of the imagined product or system, features of the product’s or system’s use environment, and use process features—showing how the imagined product or system would eventually be used (see Figure 2), i.e.

$${}^{{}^{Ext}}{}^{AP}\rho ={\rho }_{prod}\cup {\rho }_{use}\cup {\rho }_{env}$$

(1)

Whereby ${\rho }_{prod}=\{p_1,p_2,\dots ,p_j\}$ are the product or system manifestation features, ${\rho }_{use}=\{u_1,u_2,\dots ,u_j\}$ are a product’s or a system’s use process features, ${\rho }_{env}=\{v_1,v_2,\dots ,v_k\}$ are the use environment features, and ${}^{{}^{Ext}}{}^{AP}\rho$ are features of the Ext-AP in a single representation.

Combining features in a single representation makes an Ext-AP become a more effective means of expressing and representing preliminary design concepts. The extended abstract prototypes (Ext-APs) are primarily intended: (1) to improve cognition and understanding of the designers’ ideas about the products or systems and use processes; and (2) to support the designers to express and externalize their ideas. In building an Ext-AP, the appropriate elements of ${\rho }_{prod}$, ${\rho }_{use}$, and ${\rho }_{env}$ are identified, aggregated, and physically or virtually embodied. This allows the designers to build tailored Ext-AP and to focus more on testing specific aspects of interest (e.g., usability or functionality of the product or system), although such selective coalescing of features may also compromise the realism of the product or system represented by the Ext-AP. Nonetheless, Ext-APs offer an economical way of prototyping and testing product or system ideas early on by taking into account use contexts. The Ext-APs can be built quickly by using cheaper resources, and this, in turn, makes it possible to carry out multiple concepts try-outs economically. Low-fidelity or high-fidelity prototyping techniques can be used to create Ext-APs that express and represent products’ or systems’ ideas and their use contexts (Figure 3). Ext-APs may appear under various guises such as sketches, paper prototypes, video footages, or graphical animations.

Figure 2. The Ext-AP process, , refer also to [28].

Figure 2. The Ext-AP process, , refer also to [28].

Figure 3. Extended upfront product and use environments building scheme, refer also to [28].

Figure 3. Extended upfront product and use environments building scheme, refer also to [28].

4.2. The Ext-AP Process

The designer must at least have some preliminary ideas or vision of how the intended product or system should be like and how it would be used prior to commencement of the execution of the Ext-AP activities listed in Figure 4. The ideation and concepts development processes should obviously be accomplished according to the traditional engineering practices. For instance, involvement of various stakeholders in concept development processes is essential. Stakeholders such as the end-users are the arbiters of the envisaged product or system. It is therefore imperative that they should be involved in deciding on which concepts should be carried over as the development process proceeds. Ext-AP is a systematic concepts representation and review process (see Figure 4). The activities involved in Ext-AP are: requirements engineering (phase Ι); defining the elements of the representation (phase II), defining and setting out use scenario (phase III); and building a representation of the product or system and its use context (phase IV).

In Phase I, the requirements for the product or system are formulated, and then the concepts on how the Ext-AP should be like are developed. The main activities in Phase I are: analysis of the intended product or system with a view to identifying the qualities that the Ext-AP should exhibit, defining the objectives of Ext-AP, and identifying the operations of the product or system. In Phase II, the contents of the Ext-AP are defined, and this involves identifying the human beings involved in the operations—this extends to specifying their roles, actions, and interactions, specify the product’s or system’s functional and structural elements, affordances and interactions, and defining the product’s or system’s use contexts (i.e., use process and the environment in which the product or system will operate or be used in). In Phase III, the complete product or system use scenario is set up. The operations (the states and transitions of the product or system), human-product or system interactions, and the environment-related aspects of the product or system are constructed. Tools such as state charts—see, e.g., [29]—can be used in Phase III to model scenarios and flows of processes. The focus in Phase IV is on realization of the actual Ext-AP. This entails an assembling of the elements of the Ext-AP built in phase III, i.e., the representations of the ideas together with the envisioned use processes and environments—that is, the narrations (i.e., in audio and/or textual form) and enactments (i.e., virtual and/or physical representation of the product or system and its use environment and processes)—are aggregated in the intended contexts, i.e., all elements of the Ext-AP are assembled together to make a single representation in which the images of the product or system concepts are overlaid onto the real-world.

Figure 4. A four-phase process steps of Ext-APs realization, refer also to [28].

Figure 4. A four-phase process steps of Ext-APs realization, refer also to [28].

Each of the above-mentioned phases has a specific set of key process or performance indicator (KPI) questions that provide a way to assess the process in a meaningful manner. The process of identifying these KPI questions involved defining the processes comprehensively and formulating questions that can help the developers to judge or measure the results quantitatively or qualitatively and to compare the results with set goals. For Phase I, the KPI questions are: I (a): Has the analysis of the intended product or system helped to identify the qualities that the Ext-AP should exhibit? I (b): Has the prototype developer asked probing questions to help identify true needs? I (c): Have the requirements been thoroughly documented and correctly worded? I (d): Have the objectives of Ext-AP been defined? I (e): Have the operations of the intended product or system been identified? I (f): Are the activities related to I (d) and I (e) conducted by involving the stakeholders? The KPI question for Phase II is: II (a): Have the contents of the Ext-AP (i.e., the human beings who would use the product or system and involved in its operations—including their roles, actions, and interactions) been defined? For Phase III, the KPI questions are: III (a): Have the use scenarios of the intended product or system been defined? III (b): Have the operations (i.e., the states and transitions of the product or system during use) and the product’s or system’s affordances, functional and structural elements, interactions, and use environment and contexts been specified? III (c): Have an appropriate tool (e.g., state charts, etc.) been selected and used to represent use scenarios? And for Phase IV, the KPI questions are: IV (a): Has the Ext-AP been built appropriately (i.e., does it meet the needs, requirements and objectives as specified in I (a), I (b), and I (c))? IV (b) Does the Ext-AP include representation of the underlying ideas together with the envisioned use processes and environments? IV (c) Are the narrations (i.e., spoken and/or written accounts) and enactments (i.e., virtual and/or physical representation of the product or system and its use environment and processes) aggregated properly and in the intended contexts? IV (d) Have all elements of the Ext-AP been combined together to make a single representation in which the images of the product or system concepts and the associated spoken and/or written accounts are appropriately integrated and overlaid onto the real-world? IV (e) Does the Ext-AP work the way it is intended? IV (f) Are there any subtle things wrong? These KPI questions enable the product or system designers to monitor, understand and/or to measure the extent to which the expectations are met as the process of building extended prototypes progresses.

Figure 5 shows the information flows in the process of realization Ext-APs. As can be seen, the basic process inputs are the: media, i.e., the multimedia presentation technologies; product or system-operation, i.e., the depiction of the product or system itself and how it could be used, builders, i.e., designers and other parties involved in the development of the Ext-AP (who typically have differing demands and perspectives); actors, i.e., the individuals involved in demonstrating how the product or system will operate or be used; and the environment into which the intended product or system will be used. Each of these inputs has multiple attributes. For instance, the attributes of the ‘actors’ can be persona, roles and actions; the attributes of the ‘media’, can be narrations, enactments, and affordances; and so forth. Ext-APs are built by following the steps indicated in Figure 4 and, as stated above, an array of Ext-AP making resources or inputs (I) needs to be specified or defined first, i.e.,

$$I=\left(\begin{array}{c}{a}_1\\ a_2\\ .\\ .\\ a_l\end{array}\begin{array}{c}{p}_1\\ p_2\\ .\\ .\\ p_m\end{array}\begin{array}{c}{b}_1\\ b_2\\ .\\ .\\ b_n\end{array}\begin{array}{c}{e}_1\\ e_2\\ .\\ .\\ e_o\end{array}\begin{array}{c}{m}_1\\ m_2\\ .\\ .\\ m_p\end{array}\right)$$

(2)

Whereby $\{a_1,a_2,\dots ,a_n\}$ are the attributes of “actors”, $\{p_1,p_2,\dots ,p_n\}$ are the attributes of “product or system-operations”, $\{b_1,b_2,\dots ,b_n\}$ are the attributes of “builders”, $\{e_1,e_2,\dots ,e_n\}$ are the attributes of the “environment”, and $\{m_1,m_2,\dots ,m_n\}$ are the attributes of the “media”.

Based on the above-mentioned attributes, a tailor made Ext-AP can be composed by mapping the requirements (R) for Ext-AP to the elements of I whilst taking into account the objectives of Ext-AP, i.e.,

$$R\stackrel{({\displaystyle \sum _{i=1}^{q}Ob{j}_i})}{\to }I$$

(3)

Whereby $R=\{r_1,r_2,\dots ,r_p\}$ are the requirements for the Ext-AP, Obj_i with I = 1, 2, …, q are the objectives or purposes that the envisioned extended abstract prototype is intended to serve, and I is an array of inputs (see Equation 2). Equation 3 is used for the mathematical object of representing the process of mapping the elements of “R” to the elements of “I”. In this case, an element of “R” might map to several elements of “I”.

Figure 5. Flows in an Ext-AP process, refer also to [28].

Figure 5. Flows in an Ext-AP process, refer also to [28].

5. Tools and Technologies for Building Extended Abstract Prototypes

Various kinds of physical tools or objects, software applications, and multimedia presentation technologies can be used to build an Ext-AP. These include basic implements or materials such as papers, markers, and scanners—which can be used, e.g., in sketching or in scanning objects. Computer-aided design (CAD) systems can also be used to model and to represent product or system ideas in three-dimensional (3D) forms, which can then be augmented with images taken in real-world scenes. Additionally, low-cost technologies including, for instance, screen capturing applications such as CamStudio or Webcams can be used to record video tracks that can subsequently be edited and compiled by using low-cost everyday applications such as Adobe Illustrator or Moviemaker to produce Ext-APs. In addition, some low-cost applications such as BuildAR can be used to build 3D augmented reality scenes (i.e., to overlay the representations that depict the designer’s ideas onto the real-world scenes, e.g., including ideas on how the product or system will be like or about how the product or system would be put to use).

It is imperative to note that due to the state of influx of various ready-made applications that can potentially be used to build Ext-APs, picking one application in preference to the others without carrying out an in-depth and systematic analysis may sometimes be disadvantageous. In short, a systematic method and clear guidelines or criteria must be used to ensure that suitable tools and technologies are objectively selected. The scheme for guiding the selection of tools and technologies we developed in our prior work [30,31] can be used in selecting tools and technologies for building Ext-APs.

It should be noted that the Ext-AP technique may also be used by teams of designers collaborating remotely online through the Internet or other data networks. This paves the way, e.g., for using the Internet and IP tools in the execution of some of the abstract prototyping activities. For instance, a video content representing, say, some initial product or system ideas (i.e., an Ext-AP) can be created and edited remotely, and shared (e.g., as a compressed digital video file over the Internet or other data networks—the advantage here obviously is that video compression lessens the amount of data, reduces bandwidth requirements, and decreases transmission costs, see, e.g., [32]. The above-mentioned tools and technologies can be used to support the Ext-AP process, particularly in performing some of the Phase III and IV activities (see Figure 4) e.g., in developing contents such as narrations, creating enactments, and merging multimedia data streams or files. Obviously human experts are crucial in the processes of building Ext-APs—the prototype building activities in phases I, II, and III (see Figure 4) in particular are highly reliant on human experts. For instance, a human expert has an obligation of deciding, e.g., on which prototypes building constructs, or which tools and multimedia technologies should be used.

6. Illustrations

The Ext-AP technique was used in several case-study development processes to explore its applicability. In total, thirty-nine case-study design assignments were used in the investigation. All these assignments involved developing products or systems with interacting and synergistically functioning electro-mechanical, software and/or cyber components or subsystems—refer also to the task descriptions of the case study assignments in Figure 6. The design assignments were executed by teams of two or three graduate industrial design engineering students, who served as the subjects of the investigation. Each team was tasked to design a particular complex product or system—i.e., to develop, evaluate, and to select concepts and to subsequently build an Ext-AP within a time span of not more than forty hours. Due to space limitations, four selected cases are briefly described (refer also to Figure 6). The subjects built the Ext-APs by following the steps shown in Figure 4. This, among other things, entailed defining arrays of inputs (I) required to build an Ext-AP i.e., specifying the attributes of “actors”, $\{a_1,a_2,\dots ,a_n\}$ ; “product or system-operations”, $\{p_1,p_2,\dots ,p_n\}$; “builders”, $\{b_1,b_2,\dots ,b_n\}$; the “environment”, $\{e_1,e_2,\dots ,e_n\}$; and the attributes of the “media”, $\{m_1,m_2,\dots ,m_n\}$; and mapping of the requirements for the Ext-AP (R) into the elements of I whilst paying attention to the objectives (Obj_i) of Ext-AP (i.e., $R\stackrel{({\displaystyle \sum _{i=1}^{q}Ob{j}_i)}}{\to }I$). Different types of authoring tools—refer to Section 5, were subsequently used to build the Ext-APs. Augmented Reality (AR) provided a way of interacting with the real-world and virtual objects at the same time. As indicated in Figure 6, most of the subjects used the BuildAR application to build the Ext-APs—i.e., to create augmented reality scenes. BuildAR provided them with tools that enabled them to overlay 3D computer graphics contents on the real world in a way that made the prototype appear to be part of the real environment and allowed them to realize the interaction between virtual and physical building blocks of the Ext-AP. All the subjects needed were a PC, a webcam, and some printed patterns.

Figure 6. Description of some selected case-study design assignments.

Figure 6. Description of some selected case-study design assignments.

We discuss in depth the “Case A: electric food mixer” design assignment to illustrate the applicability of the Ext-AP methodology and of the toolsets described in previous sections. The subjects were asked to develop an electric food mixer that can search and recommend recipes, guide its user and give feedback, interact with the user, communicate with other connected devices, monitor energy consumption, and that can troubleshoot problems. Based on the above-stated needs, concrete requirements $R=\{r_1,r_2,\dots ,r_p\}$ were formulated and several product concepts developed and evaluated. A low-cost concept of an electric food mixer system equipped with sensors and mechanisms that allow it to collect and transmit data and to connect to other devices with a view to meeting the aforementioned needs was selected for further development. For the electrical food mixer to function and to operate as intended, suitable readily available technologies were incorporated, e.g., to enable inter-device connectivity and communication. Also, several networked sensors, including thermal and image sensors were incorporated to detect and to convey the information that constitutes an image as well as heat, deformation, or movements data; and appropriate mechanisms designed to process the collected data to enable the mixer, e.g., to stop when a user’s hand is too close to rotating parts, or to alert the user when its parts become excessively hot or deformed, e.g., due to extreme dough hardness.

An Ext-AP of the electric food mixer in the form of a 3.57 min multimedia video footage was subsequently built. In building this Ext-AP, the attributes of the “actors”, $\{a_1,a_2,\dots ,a_n\}$; the “electric food mixer-operations”, $\{p_1,p_2,\dots ,p_n\}$; the “builders”, $\{b_1,b_2,\dots ,b_n\}$; the “environment”, $\{e_1,e_2,\dots ,e_n\}$; and of the “media”, $\{m_1,m_2,\dots ,m_n\}$ were defined and then mapped to the requirements, i.e., $R=\{r_1,r_2,\dots ,r_p\}$ for the Ext-AP while paying attention to the objectives (Obj_i) of Ext-AP (i.e., $R\stackrel{({\displaystyle \sum _{i=1}^{q}Ob{j}_i)}}{\to }I$). In this case, the requirements, $R=\{r_1,r_2,\dots ,r_p\}$ that describe each of the above-stated needs concretely (i.e., being able to search and recommend recipes, guide the user and give feedback, interact with the user, communicate with other devices, monitor energy consumption, and self trouble-shooting) were defined. The environment in which the mixer was to be used was kitchen. Therefore, elements of $\{e_1,e_2,\dots ,e_n\}$ included kitchen layout and things commonly found in kitchens such as kitchen table and electrical sockets; the elements of $\{p_1,p_2,\dots ,p_n\}$ included a complete 3D representation of the selected concept of electric food mixer and the description of how it will function; the elements of $\{b_1,b_2,\dots ,b_n\}$ included the description of the subjects’ (design team’s) global mental ideas on how the electric food mixer should be like; the elements of $\{a_1,a_2,\dots ,a_n\}$ included persona, i.e., a description of the role played by the person who will interact with the electric food mixer and actions that this person will undertake; and the elements of $\{m_1,m_2,\dots ,m_n\}$ included narrations, text, and virtual objects that describe the affordances of the electric food mixer. SolidWorks was used to model the electric food mixer in 3D, CamStudio was used to record screen shots, webcam was used to record real-world scenes, and BuildAR application was used to overlay the recorded contents on real-world scenes. Different segments of the Ext-AP were subsequently combined together by using Movie Maker application. The Ext-AP built represented innovative ideas about the electric food mixer and included a storyline in the form of a narration, text and animations which described what the intended product can do and how the end-users would interact with it. While narration and text in the Ext-AP provided explanations on how the mixer will be used, animations provided a rich visual presentation of the mixer and its use environment, and AR provided a means to mimic the use scenario and to record the user’s interactions with the electric food mixer.

All thirty-nine Ext-APs were then evaluated to determine their suitability. This involved assessing if an Ext-AP (i.e., the combined representation of product ideas and its use contexts) correctly represented the ideas and the use processes and environment as envisioned by the developers. The assessors studied the Ext-APs, and then formed opinions on the suitability of these prototypes in terms of representing the designers’ ideas. The assessors were members of the teaching staff, who in fact were also seasoned assessors of assignments in other engineering design courses. They assessed the Ext-APs independently and assigned one of the following scores: “very good”—i.e., the Ext-AP portrays features of the product or system and use contexts completely; “good”—i.e., the Ext-AP depicts most features of the product or system and use contexts; “satisfactory”—i.e., the Ext-AP adequately enables third party individuals to understand how the product or system will be like and how it will be used; or “failed”—i.e., the Ext-AP failed to portray the underlying ideas and use contexts of the intended product or system. The subjects were also interviewed to seek their opinions regarding the suitability and benefits of the Ext-AP technique and how the process unfolded.

Figure 7 shows the evaluation results. As can be seen, the assessors had the opinion that 15.4% of the Ext-APs were “very good”, 17.9% of the Ext-APs were considered to be “good”, 56.4% of the Ext-APs represented the ideas and use processes satisfactorily, whilst 10.3% of the Ext-APs were considered to have failed to represent the products or systems and their use processes sufficiently. These case-study design assignments helped to uncover how the proposed Ext-AP technique works in practical settings. Furthermore, these case studies have demonstrated that an Ext-AP can effectively be used to express and represent new product or system ideas and their use contexts—i.e., 89.7% of the Ext-APs were considered to have represented ideas and use contexts very well, well, or satisfactorily. It is important to note that the authoring and media presentation tools used may influence quality of the Ext-AP, i.e., the extent to which the Ext-AP represents the ideas and use contexts. Our future research will therefore partly focus on studying the influence of authoring and media presentation tools on the suitability and effectiveness of the Ext-AP.

Figure 7. Evaluation of case-study Ext-APs.

Figure 7. Evaluation of case-study Ext-APs.

In summary, the Ext-AP technique obviously supported the subjects to externalize their visions of products or systems. It provided a systematic approach that helped them to express and represent ideas and to explore quality aspects such as functionality, completeness, and usability whilst keeping an eye on use contexts. It was observed that the Ext-APs, among other things, facilitated making associations, constructing and combining alternative concepts, and working on multiple ideas simultaneously.

7. Discussion

Many development processes, including those of development of complex engineering products or systems with physical, software and/or cyber components, typically start with needs analysis and acquisition of requirements. These requirements can serve as the basis for building Ext-APs. However, it is widely understood that some new additional requirements often evolve and become clear only during the deployment of the product or system and its use. Uncovering of such requirements at the fuzzy-front end of the development process can obviously help to refactor the product or system in advance, but is, nonetheless, a much difficult challenge, especially if the latter stage life-cycle use processes and environments are unknown. We have therefore proposed the Ext-AP technique to support the designers to understand, outline, and to precisely represent concepts together with use contexts. The application case studies have illustrated that the Ext-AP technique can be applied to express and represent ideas and use contexts effectively. The Ext-APs have been used successfully to explore new product concepts. The aspects that were explored by using the Ext-AP technique included functionality, completeness, usability, and the product’s or system’s interaction with the end-user. One of the cardinal characteristics of the Ext-APs is that both the concepts and the use processes are depicted in a combined common representation. Such co-existence of concepts and use contexts in an integrated representation and shared environment allows the designers and other stakeholders to understand the interactions of the product or system with its users and environment better upfront, and to experience how the proposed concepts would eventually work in practice.

Ext-AP can be institutionalized as a networked online or offline upfront prototyping and testing technique by using the existing multi-media presentation technologies. It can be applied in design offices to fore-create and represent use contexts virtually and to demonstrate how the end-users would use the eventual product or system. As mentioned earlier, the primarily objective of the Ext-AP technique is to support the designers to externalize their visions of products or systems. Ext-APs allow the designers to achieve this objective by using low-cost virtual representations created by using everyday multimedia presentation technologies. Apart from the low-cost technologies used in the research described in this paper to demonstrate how to build an Ext-AP, other existing technologies, including advanced visualization technologies such as VR and volumetric 3D displays, may also be used to represent the intended product or system and its use context or scenario. Several studies suggest that VR technologies can effectively support the designers at the initial stages of the design process—see, e.g., [33]. It is therefore expected that the features or capabilities provided by VR technologies such as 3D graphics and speech or tactile/force interfaces would more effectively support the designers within the framework of Ex-AP to express and represent ideas and use contexts. The drawback, however, is that using VR technologies would generally be much more expensive than using the technologies we used to build the Ext-APs in the case-study design assignments described in Section 6.

The Ex-AP technique extends the notion of prototyping and requires the designers and engineers to create cognitive models with a view to exploring new product or system ideas as well as their use scenarios, and to demonstrating or to reviewing new concepts. The cognitive virtual representations enhance comprehension of concepts; facilitates communication, discussions, and understanding; and stimulates the awareness of taking into consideration the knowledge of use processes and environment during concepts development. The Ext-APs can serve many purposes; for instance, they can be used in proof-of-concepts or principles testing, in exploring products or systems to gain insights into various aspects, in exploring user experience (e.g., in investigating how the users would interact with the product or system), in verification of consumers’ interests, or as an evaluating prototype throughout the entire development process to demonstrate and discuss the design. The Ext-APs can also be applied in marketing of new product or system ideas, or be used by the designers to present new concepts to the management in order to facilitate cognition and support decision making. The principal incentive in using Ext-APs is economics. The Ext-APs can quickly be built, and this reduces the prototyping and testing duration. The Ext-APs also allow the designers to explore usability of the product or system and its use contexts exhaustively early on, which helps to guarantee user satisfaction and to reduce uncertainty about the applicability of the envisioned product or system.

As mentioned in the previous section, we carried out a total of thirty-nine case-study product development assignments. Apart from illustrating how the Ext-AP technique can be put to use, the idea was also to observe and to seek opinions of the subjects on how the conceptualization and prototype building processes unfolded. According to the subjects and to the observations, some of the benefits of using the Ext-AP technique include increased agility during ideation and increased vigilance during concept development. Furthermore, an Ext-AP can be built more cheaply and quickly, thus offering a cost effective alternative to prototyping and testing new concepts. As we pointed out earlier, one possible way of putting the Ext-AP technique to use could be to implement it as a web-based prototyping and testing tool, with shared web resources. This would support collaborative design problem solving more effectively, allow designers to think divergently and to work together on multiple ideas simultaneously, and would as a result inspire realization of innovative solutions. It is widely acknowledged that collaborative problem solving contributes to realization of creative solutions, and this can typically be achieved through co-design, reflections, and evaluation of product or system concepts by involving the stakeholders and other collaborators [13,34]. Even some studies in the field of psychology—see, e.g., [14,35]—indicate that shared resources inspire generation or discovery of novel solutions and encourages more engagement in tasks. It can be argued that since an Ext-AP comprehensively represents the intended product or system in a wider context (i.e., represents both concepts and use environment and processes), the designers can clearly perceive the emerging solutions and understand the intents, actions, and the ideas put forward by fellow members in the design team much better. This facilitates effective reflection and critiquing, and enhances knowledge integration in the design team, which as a result spurs creation of new innovative ideas that might otherwise be missed if the use processes and environments had not been integrated into a singular common representation.

It is important to emphasize, however, that this paper aimed specifically at introducing the theoretical and methodological fundamentals of Ext-AP and at illustrating its applicability. Future work will focus on investigating the potential and the role of Ext-AP as a vehicle for achieving innovative solutions in collaborative design problem solving processes, comparing Ext-AP with competing methods, and on exploring if Ext-AP can increase creativity. This might involve, for instance, setting up several experimental design assignments that replicate practical design tasks in the real-worlds and allowing subjects to collaborate to resolve given design problems and to propose alternative design concepts. The subjects of the investigation can then be interviewed to gather their views regarding the effectiveness of Ext-AP as a prototyping and testing technique and of using aggregate representations of design concepts and use processes and environments. The obtained datasets can then be analyzed rigorously to uncover trends and patterns.

8. Conclusions

The paper has addressed the challenge of representing ideas and envisaged use environments and processes of complex products or systems with interacting and synergistically functioning electro-mechanical, software and/or cyber components or subsystems at the fuzzy-front early stages of the development process. The Ext-AP technique has been created as an attempt to address this challenge. Low-cost cognitive virtual representations are used to express ideas and to represent use environments and how the product or system would interact with the end user. An all-inclusive prototype—i.e., a representation which portrays the sense of “both the intended product or system and its use environment being there together”, is the cornerstone of the Ext-AP technique. Such a prototype improves the understanding of the product or system on a practical level and also helps the designers to explore, e.g., usability or social and psychological relationships in advance.

An ExtAP may take any of the following three forms. It may be (1) a demonstrative model created based on the ideas of a multidisciplinary design team to portray what the product or system will be like; (2) a pilot prototype used as a basis for performing tests to check the fulfilment of the requirements prior to implementation; or (3) a kind of virtual clay that allows a multidisciplinary design team to physically sculpt or model concepts together. The narrations in an Ext-AP spur cognitive communication whilst the enactments facilitate perceptive communication. It has been demonstrated that both low and high-fidelity prototyping techniques can be used to create Ext-APs that effectively represent and express the product or system and its use environment and processes. Practical application case studies have also demonstrated the applicability of the Ext-AP technique. It has been observed that the Ext-AP technique provides a practical, low-cost, and faster way of expressing and representing concepts. This technique also helps the designers at the onset of the development processes to avoid spending time and resources on ideas that may later on be discarded.

With insights from the case studies, it can be said that a combined representation of both concepts and use contexts enables thorough examination of various aspects of the product or system. Incorporation of concepts together with use contexts in a common virtual environment help to achieve shared understanding of the solutions and can inspire creation of new unexpected creative solutions, or new ways of problem-solving or designing products or systems. Furthermore, case studies have demonstrated that the communication cues in Ext-APs facilitated co-exploration and allowed subjects to make associations, quickly and easily construct, combine, and evaluate alternatives, and work on multiple ideas simultaneously. Future research will partly focus on investigating how the Ext-AP technique can be used alongside the existing design process models.