Verifying the Test Methods and Certification Criteria for New Technological Convergence Products: Using Living Labs as a Methodology

Abstract

The recent industrial transformation driven by industry 4.0 and related emerging technologies, such as big data and Internet of Things (IoT), have exerted significant impacts on the key aspects of new product development. As an example, new smart products are being developed in various ways through technological convergence. The convergence-oriented development of new products, however, may increase the difficulty of getting tested and certified due to the incompatibility with the existing certification standards. Additionally, it may take several years to develop and verify new standards, possibly delaying the product launch. To deal with this trouble, this study proposes an interesting idea that living labs can be effectively used in verifying the test methods and certification criteria for new technological convergence products (NCPs). Specifically, this study develops a framework for analyzing the real-use environment of NCPs and deriving the primary living lab components that should be implemented for verification purposes. The presented framework is then applied to real NCP cases to check its validity. To the best of my knowledge, this study is among the first elaborating the idea of using living labs as the methodology for verifying the test methods and certification criteria for NCPs. The specific methods developed in this study can be used as important guidelines that practitioners can follow step by step. Theoretically, this study also has significant implications for living lab literature and context-related research.

1. Introduction

The global economy is paying great attention to the fourth industrial revolution to make breakthroughs throughout all industries [1]. Compared to other preceding revolutions initiated by a single technology (e.g., steam, electricity), industry 4.0 embraces a set of technological developments that encourage the creation of new types of products that are totally different and innovative, to meet new market requirements [2]. For example, recent products and services tend to be based on big data, storing a vast amount of data, self-processing, and communicating with other devices and systems [3,4]. The industry 4.0 paradigm also stimulates the expansion of existing products and the emergence of new smart products through combination and convergence with recent advanced technologies. Accordingly, companies today are pursing innovation as essentials more than ever, to cope with the growing market demands on product diversity and functionality [5].

The development of innovative products with a special focus on the integration and convergence of technologies, however, may face unexpected barriers just before entering the market. This is due in part to the incompatibility of the new products with the extant standards for certification. In many countries around the world, various certification schemes are in operation to ensure the quality and safety of new products. All the new products subject to the certification systems cannot be sold to the public or for particular purposes, unless they pass certain accredited tests. Nowadays, the incompatibility issues of new products are much more evident than before, as the technological entanglement of new product development has dramatically increased.

For example, ENERTALK (ENCORED Inc., https://encoredtech.com/) is an energy tracking device and service for households, which keeps track of the energy consumption of home appliances and gives useful information such as where the most power is consumed. The new product was made possible by the convergence of various state-of-the-art technologies, and the novel functionality can be attractive to users. However, ahead of the product launch, the company confronted a major obstacle. In terms of certification, ENERTALK belongs to a meter managed by legislation, which means that it is forced to be certified before selling. The problem was that the test methods and certification criteria for the new technological convergence product (NCP) had not been prepared in the existing certification system. In addition, compared to leading the technology development successfully, the company had paid relatively less attention to the fact that the product shall comply with the structure and form set by the statute (e.g., a meter should be of a form that prevents counterfeiting and tampering). This situation is actually not unusual for the companies developing new convergence products using the latest technologies.

In the extant certification systems, a variety of industrial standards are offered for relevant products to comply with. In accordance with the test methods and qualification requirements specified in each standard, companies can manage the quality and safety levels of their products to be accepted in the market. However, this does not directly mean that every new product can benefit from the institutional support. As seen in the aforementioned case, compared with the rate at which technologies are converged by enterprises for new product development, the rate at which new standards are developed or the existing ones are revised is relatively slow. Due to the time lag, companies often encounter the situation in which they painfully experience the difficulty in having their NCPs tested and certified, which in turn increases the probability of missing the best time of product launch [6].

Even if the test methods and certification criteria are crafted quickly for NCPs, the verification process should be accompanied, taking into account the real-world environment of the product use. Since most NCPs did not exist before, it is not easy to develop the test methods and related criteria that sufficiently reflect the actual use environments. In general, an accredited test for certification checks whether a product works normally in much worse conditions than the actual use environment [7]. Therefore, without an accurate understanding of the actual use situations, the appropriateness of the test methods and threshold values that are newly developed cannot be completely confirmed. As a result, the verification process for NCPs may take longer time than other existing products.

Given these challenges, this study focuses on finding ways to improve the verification process for NCPs in effective ways and reduce the time required efficiently. Specifically, this study presents an interesting and feasible idea that livings labs, which are real-life spaces or settings for innovative and participative research, can be effectively used in verifying the certification test methods and criteria for NCPs. Despite the growing interest in living labs, little is known about how to utilize living labs for the verification purposes.

This study fills this gap by proposing a conceptual framework that helps analyze the real-use environment of NCPs and derive significant design implications for living labs. Then, the effectiveness of the framework will be assessed by applying it to real NCP cases. To the best of my knowledge, this is among the first elaborating the idea of using living labs as the methodology for verifying the test methods and certification criteria for NCPs. The analytic tools developed in this study effectively link one of the core properties of living labs, i.e., realism, with the necessity in the area of the certification, i.e., understanding of real-use environment, for the new products such as NCPs. The work in this study is not only a useful tip for practitioners, but also has theoretical significance.

2. Literature Review

2.1. Challenges for New Technological Convergence Products (NCPs)

With the advent of industry 4.0, attention to a series of global mega trends, such as smartization, servitization, and platformization, has rapidly increased all over the world [8]. It is a global trend that products and services are merged with recent disruptive technologies to be smarter (i.e., smartization), and the transition of business models is accelerated to make the products converged with services (i.e., servitization) [9]. There are also more and more cases in which products and services are connected into a single platform, which creates new markets and even changes the rules of the game (i.e., platformization).

Intriguingly, all these global trends are based on technology convergence, encouraging the development of new kinds of products and services to meet evolving market requirements. Indeed, various emerging technologies, such as artificial intelligence, augmented reality, big data, and Internet of Things (IoT), are extensively combined with existing or other new technologies. Accordingly, many new technological convergence products (NCPs) are appearing in almost every industry, providing unprecedented values.

As mentioned earlier, however, NCPs may have difficulty getting relevant certifications that are often formally required before entering the market. Certification is an act of verifying that the object of evaluation (e.g., a product) conforms to the designated standards or technical regulations, and thus proving the legitimacy that it is appropriate and safe to use and ship the product [10,11]. Standards usually contain performance test methods and qualification criteria, both of which specify the technical specifications acceptable in the market. In general, standards are developed and finally enacted for the product items that multiple providers need to comply with the relevant technical specifications [7]. This indicates that standards are not made for every kind of new products, such as NCPs.

Compared to other existing products, NCPs are more often asked to prove the quality and safety levels through accredited tests. From a buyer’s point of view, confidence in the products that did not exist before is inevitably low, and it is quite difficult to predict the serious consequences of the new technological attempts, such as the merging of disparate technologies. Therefore, despite the technical excellence, whether the product is certified or not has a significant impact on purchase decisions. However, in many cases, the standards in the existing certification systems are not perfectly applicable to NCPs, due to the differences in product structure, applied technologies, and functions. Without certifications, there will be limitations or restrictions to product release due to the lack of legitimacy [12]. This delays time to market for NCPs and increases the probability of losing chances for market preoccupancy [6].

Moreover, it usually takes several years for a standard to be established [7,13]. When a draft is developed to meet market needs, discussions among important stakeholders including sellers, users, and regulators are actively followed, and finally, agreed content is confirmed as a standard to be published [14]. The process for the discussion and consensus is indeed important because if standards are not properly written, low quality products can be distributed on the market, negatively affecting the health or welfare of users and damaging the whole market.

This procedure for the establishment of new standards also suggests that it is highly unlikely that new certification standards for NCPs will be developed within a short period of time whenever requested. In the case of NCPs, the levels of product diversity are expected to be higher than other products because of the rapid change rate of technologies. Thus, the peculiarity of each NCP may delay the standardization process longer than usual, because standards should not be biased in favor of a particular group of stakeholders [15]. Ironically enough, the company then may have to wait some time until other similar products are developed by competitors. In extreme cases, the higher the innovation through convergence of a NCP, the more likely that it will not be certified quickly, due to the absence of suitable certification standards.

2.2. Certification of Compliance with New Products of Industrial Convergence

To address the problem aforementioned, the South Korean government has been operating a new kind of certification system called the certification of compliance with new products of industrial convergence (hereafter called certification of compliance), in accordance with the Industrial Convergence Promotion Act [16]. The certification of compliance was enacted to support the rapid time-to-market of NCPs, in case there are currently no applicable certification standards under the relevant acts or certification schemes even if they must be certified prior to product launch.

Specifically, once the application for the certification of compliance is filed by an enterprise, the head of the competent central administrative agency shall notify whether the NCP is subject to the examination for the certification of compliance. Then, if there is no specific safety problem of the NCP, and it is possible to establish the test methods and qualification criteria that are suitable or applicable to the NCP, he or she shall go through the process for the certification within six months, including the development of the test methods and criteria [16,17].

Interestingly, an important feature of the compliance certification system is that it quickly develops the test methods and threshold criteria that are in effect only to the applied product. Subsequently, if all the accredited tests are passed, it issues a certificate with the same legal effect as obtaining the certificate that is originally required for product launch in accordance with the relevant act. In this way, the certification of compliance can resolve the NCP’s difficulty of getting certified, stemming from the time lag between technology development and standard establishment. Moreover, to further expedite the entire certification procedure, a series of government-supported projects are also being operated, in which the investigations on reference standards to derive draft test methods and the laboratory experiments to find threshold values for each test are performed [18].

As a deregulation policy of the Korean government, the certification of compliance has gradually drawn much attention from a wide variety of companies developing convergence products. According to KNICC (Korea National Industrial Convergence Center), the number of NCPs applied for the certification has been steadily increasing since the compliance certification began in 2011 [19]. In addition, there are also increasing real cases that demonstrate its effectiveness. For example, a recent report shows that the companies that used it have successfully launched their NCPs to the market after getting certified, and also achieved business performance (e.g., sales) [19].

These positive results prove that the expectations for the high quality of convergence products, increased by the perceived pace of technology change, have been met by the legitimacy provided by the certification [12,20]. More importantly, each of the products certified through the certification of compliance is real proof that it’s time to start revising the existing standards to extend the coverage to those NCPs, or start working on the development of new standards.

2.3. Verification of Test Methods and Certification Criteria

Compared to the standardization process that usually takes several years from the development of an initial draft to the confirmation of the final draft, the six-month period, set by the law of the compliance certification, can be seen as a relative advantage to enterprises. However, it is a big burden for regulators on the other side. Because of other important processes such as product testing, the time available to develop new test methods and criteria is only two-three months. Although the development of certification test methods and criteria is limited solely to the applied product, it is never easy to complete the process (e.g., draft, discuss, revise, and finalize) in such a short time. Moreover, the process for verification that usually requires a multiple-staged test or experiment should also be followed to confirm the suitability of the test methods and thresholds developed [21]. Hence, a more efficient way for the whole process of the compliance certification has been constantly explored.

In light of this, the present study focuses on the verification process and how to improve it. As explained earlier, an accredited testing for certification should be able to guarantee that a product can operate normally, even under much worse conditions than the actual use environment [7]. However, newly-developed products such as NCPs have never been used in real life by individuals or organizations, since the development of those products is based on new technological initiatives or attempts. This indicates that the situations, conditions, or environments in which the new products will be used should be analyzed in advance, and that the implications from the analysis should be reflected in the verification process, to make sure that the test methods and certification criteria are valid enough to confirm there will be no quality and safety issues after sales.

With these issues in mind, this study proposes a new verification method that uses living labs as a way to consider the primary elements of real-life product use environments, such as the characteristics of users and their activities, other technological devices or systems that the product works with, and the natural or artificial environments (e.g., temperature, buildings, and social events). Below, the concept of living labs is first explained, followed by the framework of using living labs for the verification purposes.

2.4. Living Labs as a Methodology

Living labs, known as laboratories in everyday life, are a user-cocreation approach that integrates research and innovation processes in real life settings [22]. Living labs serve as a user-centered innovation platform in which actual users participate as a key actor for innovation activities [23]. In this regard, living labs can be understood as an innovation methodology for identifying and applying the needs and opinions of actual users in an innovation project, such as product development and technology commercialization [24].

Many research institutes, local governments, and organizations are also using living labs for various purposes [25]. A widely spreading form is that living labs are used for the purpose of resolving or minimizing social problems or issues, such as disaster safety, security, and environmental preservation. In these cases, living labs are an integrated model of science, society and field, in which a set of technologies as a potential solution is applied directly to the real-life environment where problems exist. This suggests that living labs are utilized as a way of research and solution development (R&SD) [26].

Living labs are also evolving in other different forms as a new methodology. Of those, the use of living labs, as a testbed or demonstration project, is of particular interest in this study [27,28,29,30]. For example, in the Bukchon area in Seoul, IoT-based venture companies are testing their technologies to demonstrate, such as a parking sharing service through vehicle number recognition and IoT sensors, beacon and gateway-based fire and environment monitoring service, and AR/VR-based tourist information service. The smart safety living lab at Dongtan area in Korea is now in operation to demonstrate the performance and safety of new products prior to product launch. The living lab is built with several physical spaces, similar to actual home, hotels, nursing homes, and kindergartens, as well as virtual experimental spaces with VR equipment. To monitor user behavior closely, user activity measuring devices, such as observation cameras and eye trackers, are also equipped. Moreover, environmental information monitoring systems will be added by the end of 2020 for various types of demonstration projects [31].

As this literature review shows, there are many examples of how to utilize living labs. In particular, this study intends to extend the model of living labs as a testbed or demonstration project into the context of verifying the test methods and certification criteria for NCPs. To this end, this study proposes a conceptual framework that can be used to analyze the real-use environments of NCPs and draw design implications for living labs in the verification process. The next section explains it in detail.

3. The Conceptual Framework Proposed

3.1. The Five-Step Process of Verification Using Living Labs

Prior studies have suggested that the key characteristics of living labs include methods, tools, and approaches, and that living labs are suitable for developing, validating, and testing technologies [32]. However, no special methods have been studied on how to use living labs to verify certification test methods and qualification criteria.

This study proposes that the use of living labs as a methodology for the verification purposes can be specified in the detailed procedures shown in

Table 1. The Five-Step Process of Verification Using Living Labs.

Steps		Contents
Recognition of the need for verification	Step 1	Selecting the product item and its test methods and certification criteria that need verification
		- Analyzing the features of the product in terms of the quality and safety concerns
		- Analyzing the appropriateness of specific test methods and criteria and the need for verification in the real-use environments
		- Analyzing the feasibility of the verification using living labs
Development of verification methodology using living labs	Step 2 (the focus of this study)	Analyzing the real-use environment of the product
		- Analyzing user context, technological context, and physical context of the product use
		- Eliciting the key components of living labs in terms of the three real-use environments of the product
		- Prioritizing the key components of living labs
	Step 3	Implementing living labs for verification purposes
		- Analyzing the implications elicited in step 2 and designing living labs accordingly
Verification and analysis	Step 4	Conducting tests or experiments for verification
	Step 5	Analyzing the results and concluding the verification process
		- Analyzing the results of the tests or experiments to see whether the test methods and criteria are valid or need to be improved
		- Discussing how to utilize the results to improve the current test methods and criteria, if necessary

. Specifically, the entire procedure consists of five steps: selecting a product item (NCP) and its test methods and criteria that need verification; analyzing the real-use environments of the product; implementing living labs based on the analysis; conducting tests or experiments for the verification in the living labs; and analyzing the results and discussing the implications.

Firstly, the product item for verification can be selected in terms of the following factors: whether the quality and safety concerns about the item are high due to the novelty and the level of technological convergence; whether some of the test methods and certification criteria are not perfect enough to reduce those concerns; whether the verification requires tests or experiments in the real-use environments of the product; finally, whether the verification using living labs is actually feasible.

Next, once the item and its verification targets (i.e., specific test methods and threshold criteria) are determined, the next step is to analyze the real-use environments of the product and to draw design implications for living labs. This step (Step 2) is one of the most important steps of the whole process, directly affecting the effectiveness of living labs for the verification. Since this step provides the conceptual layout of living labs, the analysis of the actual use environments should be mutually exclusive and collectively exhaustive (MECE). This is why this study focuses mainly on this step.

The framework presented in this study provides an effective way to analyze the real-world environment of the product use in three aspects (i.e., user context, technological context, and physical context). The framework also helps derive the primary components of living labs and related design implications. Moreover, it guides you through how to determine the priority among the key living lab components in order to actually implement in the next step.

After the living labs are configured based on the implications from the preceding analysis, planned tests or experiments are then conducted in the setting of living labs, followed by analyzing the test results. Finally, it can be determined whether the test methods and certification criteria are valid enough to assure the expected performance and safety of the product after sales. Further, directions for the revision or improvement of current test methods and criteria can be discussed if necessary.

3.2. Analyzing the Real-Use Environments and Implementing Living Labs

According to previous studies, realism is the key to implementing living labs. For example, CoreLabs [33] suggests that realism is part of the five core principles of living lab operations, along with continuity, openness, empowerment of users, and spontaneity. Bergvall-Kareborn et al. [34] also emphasize realism as an important characteristic of living labs in addition to openness, sustainability, value, and influence. The importance of realism highlighted in this literature suggests that the essential requirement for a living lab implementation is to analyze the real-world environment of an object (e.g., products) that will be installed or tested in the living lab.

Based on this, a conceptual framework for the analysis of the real-use environments of NCPs is presented in this study. As shown in Table 1, the framework consists of three sub-steps. The first stage is to perform a detailed context analysis of the product use to understand the real-use environments of a product, and in the second stage, the key components of living labs are derived from the results of the context analysis. For these two, three types of real-world environment of product use are defined, and used as the main categories for classifying the real-use environments: user context, technological context, and physical context [35,36]. The third stage is then to determine the priority among the derived living lab components. Because of the limited time and cost, it may be impossible to implement living labs in every way exactly the same as the real world, so priority is necessary. For a better understanding, a detailed description of each stage is provided below.

3.2.1. User Context

A user is an individual using a product and receiving the benefits from it. According to the literature, product experiences of users can be divided into two types: direct and indirect [37]. On one hand, people use products directly. For example, users wear audio sunglasses and VR equipment, ride a car and an electric kickboard, or carry their bags and smartphones. They also operate products that are usually placed in a designated spot, such as refrigerators, washing machines, and TVs. On the other hand, there are also many products that users normally do not use directly, but benefit from them. For example, automatic spray air fresheners, fire alarms, building air conditioning facilities, traffic lights, and manholes are typical examples.

These different ways products are used by individuals suggest that the physical, physiological, or mental state of a user acts as the usage environment of a product. As seen in Figure 1, if a product is worn on a user’s body (e.g., heating jumpers), specific body size and shape of the user (i.e., physical state) constitutes the real-use environment of the product. In other cases, if a product is attached to the body, the physiological state of the user such as skin types (e.g., oily or dry) can be understood as the actual use context of the product. In addition, if a product is carried by a user, or normally placed in a specific spot like laundry room and pantry, the mental state of a user such as preferences, habits, and life patterns is the key elements of the real-use context of the product.

On the other hand, in the case of the products that people use indirectly, such as automatic spray air fresheners, other situational elements such as activity of the users are particularly important. Those indirectly-used products are physically away from the users and are not usually operated by them. Therefore, a user’s situation at the moment when a product is operating is necessary to understand the real-use context of the product, such as what the users are doing and who they are being with. In much of the literature, user activity has been regarded as an important situation of a user [38].

Lastly, the framework in Figure 1 also explains that a user’s demographic characteristics, such as age, gender, education, occupation, and income, can also determine the real-use environment of a product. For example, the actual use environment of MacBook laptops is often composed of female college students who major in design. Indeed, the information about what age, gender, income, and education levels of people are using a product is indispensable to understand the real-use situation of the product, no matter what way the product is used.

3.2.2. Technological Context

The technological context of product use refers to the technical environment in which a product is used or operates. Recently, many new products such as NCPs operate in conjunction with other devices or systems to provide enhanced functionality, thanks to the advancement of network technology. It is thus more important than ever to understand the technological real-use environment of a product.

As seen in Figure 2, when a product is used in connection with other products, the product is either a finished product or a part. Certainly, there are many finished products that are used independently of other products, such as electric shavers, respirator masks, and curtains. On the other hand, more and more products are being connected with other products or devices, working dependently. For example, unlike in the past, Bluetooth speakers have spread worldwide. Home appliances such as washing machines and refrigerators are now connected to mobile devices through IoT technologies.

Using the framework in Figure 2, the key elements of the technological real-use environment of a product can be drawn. In the case of a product that requires compatibility with a specific technology, the compatible products are an important element of the real-use environment of the product. Consider the case that computers and beam projectors are connected by the USB-C to HDMI cable. Also, some products are only compatible with certain brands. Next, for the products with higher connection levels, such as IoT-based home appliances and sensor-based measuring devices, those products function like a single system with other products or devices. Thus, the technological real-use environments of those products often consist of sensors, network devices, and other peripherals. Lastly, the framework also suggests that the information about the specification of a product and its technical statues (e.g., working, defected) is also important to understand the technological real-use context of the product. Such information is a common requirement for the understanding, no matter whether the product operates dependently or independently.

3.2.3. Physical Context

The physical context of product use refers to the natural and artificial environment of the place where a product is used [35,39]. Mostly, products are used indoors, outdoors, or both. For example, heating curtains and directional speakers are indoor products, while reflective pavement markers and IoT-based manhole tops are installed on the road. The use environments of these two categories of products are very different in terms of the natural environmental factors such as weather (e.g., rainy or sunny), temperature, humidity, time (e.g., day or night, seasons), and natural objects (e.g., trees), and the artificial environmental factors such as noise, social events, artifacts (e.g., buildings, facilities).

The framework in Figure 3 shows how these natural and artificial environmental factors can have different significances according to whether or not a product is mainly used in a specific place or condition. For example, workers are supposed to wear respirator masks and welding helmets in a welding factory where various kinds of welding fumes are generated and float in the air, and the temperature is kept high. On the other hand, heating curtains are a new product in which carbon heating wire is inserted into the ordinary curtain to prevent cold air from outside the window and to warm the indoor air. Although both welding helmets and heating curtains are used indoors, in contrast to welding helmets, heating curtains are installed in ordinary houses or offices, where the temperature and air conditions remain normal, and artificial environments, such as indoor structure and furnishings, exist in various forms.

In this manner, using the framework, the key components of the physical real-use context of a product can be drawn according to whether a product is used indoors or outdoors, and whether the product is used in a special or normal environment. Moreover, the information about the specific locations of a product (i.e., where the product is installed or used) is inevitably used to understand the physical real-use environment of the product, because depending on the location, the natural and artificial environments can vary significantly.

3.2.4. Living Lab Components

After the three sets of real-use environments of a product are analyzed and their primary elements are identified, the next step is to embody the elements from the viewpoint of living lab design or configuration. This involves contemplating how each of the key elements that makes up a product’s real-use environments can be implemented in living labs. In other words, it is to materialize the components of living labs to reflect the real-use environments of the product. Below is a specific example explaining the process.

Suppose that a new test method has been developed for a floor noise level meter that is installed in an apartment to measure floor-level noise. The test method for the noise measurement performance of the new product is to use an omni-directional speaker that generates noise against the product and then check how well the product measures the noise. The reason for the verification of this test method is that it does not consider the characteristics of the real-use environments of the product. Since this product is installed on the floor or ceiling of an apartment, the actual use environment is different from that of a portable noise meter. Therefore, a test method considering the positions of the noise measurement sensors may be more appropriate. For example, when installed on the ceiling, the test method should measure the noise passing through the floor from upstairs, and related threshold values should be determined depending on the materials of the floor (e.g., concretes, hardwoods).

Given this background, the actual use environments of the product can be analyzed using the frameworks of this study. Firstly, according to the framework shown in Figure 1, the floor noise level meter is classified as a product that people indirectly use (see also Figure 4). So, the key elements of the real-use environment in terms of the user context include a user’s location, activity, and others around the user (See Figure 1). Then, at the stage of eliciting living lab components, those elements of the real-use context of the product can be embodied in the following concrete forms: living rooms or bedrooms (user location), resting or talking (user activity), friends or family (others around the user).

Next, in the aspect of the technological context of product use, multiple sensors, network devices and servers are the primary real-world components that should be considered in designing living labs (see Figure 2). This is because the noise meter is a product that works as a system with other products and peripheral devices.

Finally, the framework in Figure 3 suggests that the noise meter is used indoors, especially at an apartment that can be understood as a normal environment. Thus, living labs should be implemented similarly to an ordinary apartment. This makes it clear how to design the living labs in the aspects of artificial environments, such as indoor structure, furnishings, and social events. Importantly, since the sensors for noise measurement is attached or inserted into the floor or ceiling of the building, the actual materials and thickness of the floor should be considered when setting up an apartment-like space in living labs. Social events, like a family gathering, are another consideration to emulate the real-use context of the product. On the contrary, there is no particular need to consider special conditions in terms of the natural environmental aspects. In sum, by deriving the key real-world components of the product use in this way, it is possible to prepare, in advance, for the implementation of living labs, for the verification of the test methods and criteria.

3.2.5. Prioritization

Once the living lab components are identified, the next step is to prioritize them in the order that must be reflected in the living labs. As explained earlier, realism is one of the most important principles in using living labs. However, in some cases, it may not be easy to implement living labs perfectly, to be the same as reality, due to time and cost constraints. Moreover, not all products may have the three real-use environments with equal importance, depending on their characteristics.

For example, an automatic airbag for hip joint protection is a recent product for the elderly [40]. It is worn like a belt at the waist, and then when a user suddenly falls down, the airbag bursts out of the product to protect the hip joint area. According to the frameworks in Figure 1, Figure 2 and Figure 3, it is a product that people use directly by wearing both indoors and outdoors, and that does not work with other products or systems. Hence, the user context (e.g., age, gender, body size and type) and the physical context (e.g., terrain such as slope, indoor structure such as stairs, outside weather such as snow and rain) will have a higher priority than the technological context.

Then, the priority can be further refined according to the test method that needs verification. For example, if the verification concerns how to measure the airbag deployment time and the appropriate threshold criteria, it would be more important to implement the product’s physical use environment most closely to the reality in living labs. This is because in terms of the performance and user safety, the product should be able to guarantee that the air bag is always fired within a predetermined time in any conditions of the physical environment, including ordinary roads, slippery roads, slopes, and stairs.

As this example case suggests, prioritizing the living lab components is another pivotal process when using living labs for verification purposes. So, a more structured method for the prioritization is presented in this study. As shown in Figure 5, the relative importance of the three real-use contexts can be compared in terms of four dimensions: function, performance, optimization, and relevance.

The function perspective pertains to whether the major features of a product should be designed considering the characteristics of users, the compatibility or connectivity to other devices, or the natural and artificial environment. Next, the performance perspective is useful to compare whether the performance evaluation of the product necessitates the consideration of user characteristics, compatibility or connectivity, or the physical use environments. The third perspective for prioritization (i.e., optimization) puts a focus on which of the three real-use environments should be considered more in order to optimize the product. Finally, the relevance perspective allows you to check whether the test methods and criteria that require verification are closely related to the user characteristics, the connectivity to other devices, or the natural and artificial environment.

In sum, the four perspectives help prioritize the living lab components in a systematic manner. Using the questions corresponding to the four dimensions, the relative importance of the three actual use environments can be calculated. Then, the priority is determined in order of high score.

3.2.6. Implementation

When the priorities of living lab components are determined, the next step is to actually implement the living labs for the verification purposes. This step includes the physical configuration of the actual use environment of a product, based on the analyses from the previous steps. This is beyond the focus of this study, so no detailed explanation is provided here. Instead, this study provides an additional method for the cases, where some of the real-use environments cannot be physically implemented, despite higher priorities.

In some cases, the physical configuration of living labs may be easier with less complex considerations. For example, in order to verify the test method for the noise measurement performance of a floor noise level meter, the key to the physical implementation of a living lab is to provide an apartment-like space in a building, and attach or insert sensors of the product on the ceiling of the living room or other rooms. Then, various noise situations, such as children running upstairs (i.e., artificial elements of the physical context), can be produced without much difficulty. In other cases, however, the dynamics of the real-use environment are very high, so it may not be possible to actually implement it. For example, it is impossible to fire a 12-floor building with many people inside, just to verify the test method of whether a fire evacuation solution guides the escape direction well to the users. If a fake fire is used instead, it cannot be confirmed that the test method can ensure the same performance of the solution in a real fire situation.

Prior studies on fire-fighting training introduced the idea of interactive virtual reality simulations that enable fire-fighting trainees to practice their skills and tactics under realistic conditions. In the VR simulation, the properties of a building, such as building types (e.g., wooden unit) and the number of floors, are represented in a 3D model, and the actions of fire fighters, such as aiming a high pressure hose against the flame, are interactively performed [41]. Moreover, the dynamic fire situations such as fire power and spread are also simulated. For example, if the extinguishment agent thrown by a trainee is not enough to put out the fire, the fire power rises again [42].

The literature suggests that living labs can be implemented in a mixed form of physical spaces and virtual spaces if the dynamics of the real-use environment of a product cannot be all emulated physically in a living lab. For example, only minimal components are implemented in the physical space to interact with the virtual space, where more complex environments and dynamic situations are implemented. The smart evacuation solution by CONERS CO., Ltd. (http://www.corners.co.kr/) is a combination of products, software, and services to help quick evacuation in the event of indoor fire. When a fire is detected, the solution calculates the optimal escape route and then guides the escape directions (i.e., left, right, straight, no entry) through the emergency exit signages installed at each corner of the building.

Suppose that we need to verify the method of testing whether the escape directions displayed on the exit signages can be well identified by users even in a dynamic fire situation. Since it is impossible put a building on fire for verification, an alternate way is to virtually build the real-use environments (e.g., other users’ locations and activities; compatible devices such as wireless access points and built-in speakers; indoor structure and office furnishings) and then simulate the hypothetical events (e.g., vertical fire spread, violent flashover) in that virtual space. On the contrary, the degree to which the physical implementation is required is relatively low, since users can participate in the experiment, wearing head-mounted VR displays. According to whether users placed at different locations in the virtual space can escape the building within a predetermined time, the extent to the escape directions on the exit signages is well identifiable in such a dynamic fire situation can be finally confirmed. The VR simulations can be further improved by the application of semantic technologies [42].

4. Case Analysis: Validity Check for the Framework

The best way to validate a conceptual model is to see whether the framework fits well in empirical cases. In this section, case analyses are provided, in which the proposed framework is applied to various NCP cases. First, a detailed application process of the framework is described for one product, and then four additional products applied in a similar way are explained.

The NCP selected for the detailed case analysis is a fiber-optic LED pad, recently developed as a road delineation (hereinafter referred to as fiber-optic LED pad). The development of the product was based on the convergence of different technologies; specifically, it integrates solar power charging technologies, fiber-optic manufacturing technologies, and fiber-optic implantation technologies into a single product (see

Table 2. Sample Analyses of the Real-use Environment of new technological convergence products (NCPs).

Context	The Characteristics of NCPs			Air-ventilated welding helmet	Heating curtain	Floor noise level meter	Internet of Things (IoT)-based Manhole tops	Fiber-optic LED Pad
User context	Direct use	Wearing	Worn	√
			Attached
		Boarding	-
		Carrying	-
		Seldom moving	-		√
	Indirect use	-	-			√	√	√
Tech. context	Finished products	Work independently		√	√			√
		Work dependently	Work together
			Work as a system			√	√
	Parts of other finished products	-	-
Physical context	Indoor use	Used in a special environment	-	√
		Used in normal environments	-		√	√
	Outdoor use	Used in a special environment	-
		Used in normal environments	-				√	√
	In/outdoor use	-	-

). In particular, without the fiber-optic implantation technologies, the development of this product would have been impossible, because the optic fibers could not be inserted into the flexible pad and kept stable.

Compared to other road delineation products such as retroreflectors and road pavement markers, this product has at least two major differences. Unlike the retroreflectors, the new product emits its own light using the sunlight as an energy source. Due to this difference, the existing test method for the reflective performance is not suitable for the new product. Instead, the luminous intensity needs to be tested, which is a measure of the amount of light that a light source radiates in a particular direction (unit: candela, cd).

Another difference is the location where the new product is installed. For example, road pavement markers are installed on the ground along the yellow center lines of the roads, while the fiber-optic LED pads are attached to the road safety facilities set up at the edge of the roads. Because of this, it is not straightforward that the test methods and certification criteria suitable for the similar products can be perfectly applied to the new product. For example, compared to LED road pavement markers that also emit the light but are installed along the center lines, the product’s luminous intensity test may have to take into account a wider visual angle. A possible reason is that in a car running at high speed on the innermost lane of a road with many lanes, drivers must be able to accurately identify what this product on the roadside is displaying (i.e., sharp curve chevron). Otherwise, it would cause serious user safety problems. Therefore, the verification process should be followed to confirm that there are no problems in using the same test method and criteria as other products to certify this new product.

Using the framework proposed earlier, the real-use environments of the product are analyzed below (see also Table 2). First, the fiber-optic LED pads are categorized as a product that is used indirectly. Thus, as seen in Figure 1, the key elements constituting the user context of the product include the user’s demographics (e.g., age, gender), his or her activity and location (e.g., driving a car in the second lane, listening to the radio while driving), and the presence or absence of other users (e.g., driving in other lanes). Next, in terms of the technological context, this product is an independently operated finished product. So, no other special information is needed to understand the technological real-use environments of the product, except the basic contextual information such as technical specifications and status of the product (see Figure 2). Finally, the framework in Figure 3 suggests that this product is used outdoors and installed in a place where the natural environmental factors such as weather (e.g., sunny, rainy or snowy), temperature, and humidity normally change, and various natural objects (e.g., trees) and artifacts (e.g., buildings, road sound barriers) exist nearby. Based on these analyses, it can be inferred that the frameworks for the analysis of the actual use environments and derivation of the key components of living labs are well applied to this product as a whole.

Next, when it comes to the framework for prioritization, it can also be concluded that the framework is not only applicable, but also useful for the product. According to the method in Figure 5, the technological context of the product has the lowest priority, because the product does not work dependently on other products or systems. In contrast, the key elements of the other two real-use environments of the product have higher priorities when implementing in living labs for the verification process.

Specifically, the road delineation by light, i.e., the major function of the product, is something that should be designed in consideration of the user demographics, because all users should be able to recognize the light while driving a car. Especially, according to previous research, human visual angle changes with age [43]. This indicates that the test method and threshold value of the luminous intensity should be confirmed, necessarily taking into account the different visual angles with ages. In a similar vein, other studies claim that drivers must be able to pay enough attention to cyclists running on the roadside [44]. In addition to the user demographics, various user activities, such as driving lanes and speed, and other drivers’ situations, such as the number of cars in other lanes, are also important in designing, evaluating, and optimizing the product. For example, when evaluating the performance of the product for certification, the measured luminosity of the product must be high enough so that the sharp curve chevron can be recognized at the maximum speed limit of the road when driving in the innermost lane (i.e., user activity at the time of the product is used).

The physical real-use environments of the product also have a higher priority. Particularly in the aspect of the natural environment, the fiber-optic LED pads must be designed to function in a variety of outdoor conditions such as heavy rain and high temperature in summer. Moreover, in order to optimize the product to the real-use environments including dusty areas, the dust-proof properties of the product should be designed, by considering the fact that the product is likely to be used on the roads with dirt, mountains, or deserts around it (i.e., natural environment), or on the roads adjacent to factories or construction sites (i.e., artificial environment). Lastly, it is already clear from the preceding analyses that the test methods that need verification (e.g., the luminous intensity) are highly relevant to the physical use environments, as well as the user characteristics. Taken together, these analyses above provide evidence that the framework for the prioritization also fits the real sample case well, as a useful tool.

Next, the applicability of the proposed frameworks is further examined with four additional NCPs: air-ventilated welding helmets, heating curtains, IoT-based manhole tops, and floor noise level meters. The analysis results are summarized in Table 2 and

Table 3. Sample Analyses of Prioritization.

Context	Dimension	Questions	Scores
			NCP#1	NCP#2	NCP#3	NCP#4	NCP#5
User context	Function	When designing the major features of the product, it is necessary to take into account the characteristics of the users.	5	2	1	5	5
	Performance	When evaluating the performance of the product, it is necessary to take into account the characteristics of the users.	4	3	3	2	4
	Optimization	In order to optimize the product, the characteristics of the users must be considered.	4	3	2	2	3
	Relevance	The test methods and criteria that need verification are closely related to the characteristics of the users.	5	4	3	3	5
Tech. context	Function	When designing the major features of the product, it is necessary to take into account the compatibility or connectivity to other products, devices, or systems.	1	1	5	5	1
	Performance	When evaluating the performance of the product, it is necessary to take into account the compatibility or connectivity to other products, devices, or systems.	1	1	5	5	1
	Optimization	In order to optimize the product, other products, devices, or systems that are compatible or connected to the product should be considered.	1	1	5	4	1
	Relevance	The test methods and criteria that need verification are closely related to the compatibility or connectivity to other products, devices, or systems.	1	1	4	2	1
Physical context	Function	When designing the major features of the product, it is necessary to take into account the natural and artificial real-use environment of the product.	5	3	3	5	4
	Performance	When evaluating the performance of the product, it is necessary to take into account the natural and artificial real-use environment of the product.	5	3	2	4	3
	Optimization	In order to optimize the product, the natural and artificial real-use environment of the product must be considered.	4	2	2	3	3
	Relevance	The test methods and criteria that need verification are closely related to the natural and artificial real-use environment of the product.	4	3	4	3	4

※ NCP#1: Air-ventilated welding helmet, NCP#2: Heating curtain, NCP#3: Floor noise level meter, NCP#4: IoT-based Manhole tops, NCP#5: Fiber-optic LED Pad.

. For a better understanding, the key background information about the four products is provided below.

Air-ventilated welding helmets are a new convergence product designed to reduce the heat increases when worn, by ventilating the inside air with external air. Despite expectations for increased user satisfaction, there are concerns that various welding fumes floating in the welding place may flow into the product through the fan. Some experts may say that the main function of a welding helmet is to protect a user’s eyes from the welding lights, and that even if fumes flow inside the product, the user is protected by a dust mask. However, other experts may point out that it is necessary to develop and verify a test method and criteria to ensure that hazardous fumes do not get inside the welding helmet due to the different features (i.e., ventilation fan). In this regard, they weigh the possibility that the fumes can be inhaled by the human respiratory system, even if a dust mask is worn inside the product.

Next, as mentioned earlier in this study, heating curtains is also a new product to prevent cold air from outside the window and to warm the indoor air. For this, carbon heating wires are inserted into the ordinary curtain, and this feature increases the need for verification of new test methods, such as how to accurately test the durability and related safety of the carbon heating wire, by considering the real-use situations, such as a large number of opening and closing of the curtain under condition of hardening caused by the heat.

Third, IoT-based manhole tops are a product made of composite materials rather than cast iron. It has a built-in IoT sensor that allows the remote transmission of information about the situation inside the manhole, such as sewer water levels and harmful gas levels. Composites are known more effective for data transmission than cast iron; however, on the contrary, the manhole tops made of composites may be weaker than cast iron. Given that manhole tops are installed on the roads where vehicles such as dump trucks come and go, the test methods for the strength of the new product are critical, and it is not easy to determine threshold criteria suitable for the composite-based manhole tops. Lastly, the floor noise level meters are mentioned several times earlier in this study.

5. Concluding Remarks

This study aims to develop a conceptual framework that can be used to analyze the real-use environment of NCPs and derive design implications for living labs. This study provides significant implications in several ways. First, this study extends the preceding research on the living labs-as-a-methodology into the context of verification of test methods and certification criteria for NCPs. The process of developing, verifying, and finally enacting new standards in existing certification schemes usually takes several years, including a considerable amount of research and discussion about the scope, test methods, and threshold criteria of the standards. On the other side, due to the time lag between technology development and standard establishment, newly developed innovative products such as NCPs cannot be certified at the desired time, and in turn, the product launch is inevitably delayed.

This study provides valuable clues that the existing verification methods for test items and qualification criteria can be extended to use living labs as a new methodology. In particular, this study shows that living labs can be effectively used for those new products that have never existed before and thus the consideration of real-use environment is relatively important. The analytic tools developed in this study effectively link one of the core properties of living labs, i.e., realism, with the necessity in the area of the certification, i.e., understanding of real-use environment, for the new products such as NCPs.

Next, this study elaborates the entire procedure of using living labs for the verification, which can be an important overall picture for practitioners. Given the lack of existing knowledge, without a full picture of step-by-step procedures and methods, it will be very difficult for practitioners to actually use living labs as the methodology for the verification purposes. The detailed explanations of this study about the framework for analyzing the actual use environment, deriving the living lab components, and determining their priorities, are useful guidelines that practitioners can follow on their own.

This study also has theoretical implications. The three different contexts of product use have been conceptually developed from a different angle, compared to prior research. Previous studies mainly focused on a user-centric context [39]. Specifically, they subdivided the user context in various forms, such as what the user is doing (i.e., task context), when and where they are (i.e., spatial-temporal context), what their surroundings are (i.e., environmental context), and what their mental state is (mental context). In contrast, the contexts defined in this study are product-centric concepts, such as how and by whom the product is used (i.e., user context), with which devices or systems the product is used together (i.e., technological context), and in what natural and artificial environment the product is used (i.e., physical context).

The advantage of this new approach is that it improves a product-centered understanding of situations, conditions, or environments, which in this study, allows the analysis of real-use environment of a product. Users are still considered important, but classified as one of the real-use environments of a product, not as the main entity. Likewise, this study has expanded the existing context-related research into the product side. It is anticipated that the product-centric classification of the context can be further elaborated by future research.

Moreover, the four dimensions and corresponding questionnaire developed to determine the priority among the living lab components also have theoretical contributions. Specifically, the dimensions can be further developed as constructs and used in empirical studies. Future research can define the four dimensions (i.e., function, performance, optimization, and relevance) as constructs that are measured by the three corresponding questions as reflective indicators. Or, the three contexts can also be defined as constructs measured by the four corresponding questions as formative indicators [45]. It is not clear yet which of the two would be more theoretically solid, but it can be started from this study.

Future research may also extend this work to further improve the living lab methodology. This study has checked the validity of the proposed framework by applying it to several real NCP cases. However, due to the rapid advent of new convergence technologies such as big data technologies, a great number of new products are steadily developed in various ways across almost all industries [3,46]. So, the validity of the methodology proposed in this study needs to be checked from time to time with new products. For example, the technological real-use context of a product can be more thoroughly analyzed, taking into account a detailed set of peripheral devices. Similarly, the other two contexts of product use can also be improved in further research, to be more suitable frameworks for new types of products.

Lastly, although the study presents an interesting way to use living labs for verification needs, the relative effectiveness of living labs has not been directly compared to the existing methods for standards verification. Instead, this study focuses on the need to analyze the NCP’s actual use environments and the effective ways to use living labs as a possible solution for the verification. Whereas the existing verification methods are usually based on repeated laboratory tests, the use of living labs allows verifications that consider the actual product use situations. Thus, living labs are more suitable for the new products, such as NCPs that have never been used before. However, it is worth noting that the proposed methodology of this study for using living labs can also be applied to existing products if the consideration of the real-use environment is necessary for standards verification. Future research may examine whether the living lab methodology also has other relative advantages, or may extend this study by investigating how to effectively combine the laboratory tests with the living lab approach for a more solid verification.