IoT Security-Quality-Metrics Method and Its Conformity with Emerging Guidelines

This study proposes a security-quality-metrics method tailored for the Internet of things (IoT) and evaluates conformity of the proposed approach with pertinent cybersecurity regulations and guidelines for IoT. Cybersecurity incidents involving IoT devices have recently come to light; consequently, IoT security correspondence has become a necessity. The ISO 25000 series is used for software; however, the concept of security as a quality factor has not been applied to IoT devices. Because software vulnerabilities were not the device vendors’ responsibility as product liability, most vendors did not consider the security capability of IoT devices as part of their quality control. Furthermore, an appropriate IoT security-quality metric for vendors does not exist; instead, vendors have to set their security standards, which lack consistency and are difficult to justify by themselves. To address this problem, the authors propose a universal method for specifying IoT security-quality metrics on a globally accepted scale, inspired by the goal/question/metric (GQM) method. The method enables vendors to verify their products to conform to the requirements of existing baselines and certification programs and to help vendors to tailor their quality requirements to meet the given security requirements. The IoT users would also be able to use these metrics to verify the security quality of IoT devices.


Introduction
Security becomes more important with the proliferation of Internet of Things (IoT) devices. In fact, many security breaches on IoT devices have already been reported, hence the increased need for IoT security [1][2][3][4]. Such breaches involve, for example, the "Mirai" malware and its subspecies spreading across cyberspace targeting IoT devices, including IP/web/network cameras, digital video recorders, home routers, smart speakers, and network printers [5,6].
Researchers of IoT security have made significant progress in mitigating security threats and vulnerabilities, such as remote attacks via wireless connectivity such as Wi-Fi, Bluetooth, or Zigbee [7][8][9], and securing an architecture to meet security requirements [4,10]. However, because these functions and mitigation technologies are not self-developed by IoT vendors in most cases, but are externally procured components, IoT vendors are required to assess the security quality of the communication components they adopt. Having said that, in reality, IoT security researchers have not yet clarified standard initiatives that IoT vendors easily adopt to ensure the development of secured IoT devices.
Unlike legislation on safety and environmental issues, laws, regulations, and international standards for IoT security have not been established yet. ISO 27400 (guidelines on IoT security and privacy) [11] is still under development.
Many developers and researchers have duly addressed information security issues via ISO 27001 [12] or have discussed a new cybersecurity certification method [13]. Although ISO 27001 outlines the management and protection of information assets [14,15],

Background and Necessity for This Study
From the discussion above, the issues with regard to IoT security can be categorized into the following two questions.
2.1. Question 1: Does Any Existing Literature or Standard Covering Cybersecurity-Quality-Control Measures for IoT throughout the Product Lifecycle Exist?
Many security experts have addressed the guidelines for IoT security management from the viewpoint of the basic principles, approaches, threats, and countermeasures.
To assess the cybersecurity of systems, researchers have developed methods such as the Evaluation Assurance Level with Common Criteria certification based on ISO 15408 [22] and EDSA (Embedded Device Security Assurance) certification based on IEC 62443 [23]. However, these certifications are extremely professional: ISO 15408 focuses on quality assurance and does not specify what initiatives to take, whereas IEC 62443 is specific to critical infrastructure in the industrial control system, which generally does not apply to IoT. Furthermore, both approaches do not present a simple way of describing the quality of cybersecurity in IoT products (for vendors and/or general consumers with no knowledge of security).
Although benchmarks and assessment methods for information security have been proposed [24,25], both fall short from a web-specific and a lifecycle perspective when utilized for product security in IoT.
A template to consistently describe the service level of a cloud service has also been proposed [26]; it is, however, specific to cloud services rather than IoT.
Similarly, the importance of IoT security has been pointed out [14]; however, the article only mentions the certification scheme and does not cover the entire lifecycle, service, or system. IoT security has also been previously discussed [27,28]; unfortunately, the discussions are limited to the security of communication protocols.
The majority of the literature includes high-level guidelines and baseline requirements for IoT security for IoT vendors in 2020, discussion of security for IoT with AI (artificial intelligence) [29,30], or with the cloud [31], and the user's quality of experience (QoE) [32,33] in 2021. Our extensive literature search failed to produce any literature concerned with benchmarks or suggestions for secure development for IoT vendors.
Thus, no simple, standard way of describing the status of cybersecurity readiness from the perspective of product security in terms of the quality of IoT devices was found.

Question 2: Does Any Reason for Visualizing the Cybersecurity Control Measures Exist?
Most electronic-device vendors producing IoT devices are familiar with ISO 9001, the international quality assurance standard that clarifies the process of product development to standardize the quality throughout the life of a product. Vendors predominantly follow the defined production process and do not perform anything outside the process for costefficiency. To prevent noncompliance, it is common to define (and visualize) processes for designing safe products and selecting components with a low impact on the environment.
Similarly, the modalities for product security should be defined in existing processes. In addition, the Information-Technology Promotion Agency of Japan (IPA) reported that approximately half of the IoT vendors have specific policies; moreover, over 70% of them have no concrete standards for their security responses in product development [34]. This implies that the reason behind the lack of concrete action might be that IoT vendors have no clear understanding of who would be responsible for the security; moreover, they do not recognize security measures as their responsibility even if they know the significance thereof. Because it is difficult to add security countermeasures at the implementation stage of the development process, engineers need to devise and apply effective countermeasures at inception. Therefore, the confirmation of effective functioning of the countermeasures at the verification stage is essential. If a new vulnerability emerges, even after the product release, it must be fixed. Therefore, it is necessary to define actions to be carried out at each phase of the product lifecycle. Thus, it would not be possible to develop a secure IoT product unless the relevant processes and practices are clearly defined in the corresponding development process.
In this regard, this paper is a continuation of our previous study [35] in which we reviewed the relevant literature and clarified prospective items in the development of the security-quality metrics discussed herein.
We considered the need for a methodology that would allow IoT vendors to tailor security-quality metrics in addition to existing quality metrics for their products. This would, in turn, indicate to consumers the level of security quality of the IoT devices they develop. We attempted to derive quality metrics for IoT devices based on the literature and perspectives reviewed in our previous work. We referenced the goal/question/metric (GQM) methodology commonly used in the quality field.
The following section is an overview of the previous study [35].

Past Research on IoT Security Quality
This section summarizes our previous study conducted, in which we reviewed the relevant literature and clarified prospective items that could be used in the development of the security-quality metrics. We conducted the works in Steps 1 and 2 using the research method described in Figure 1; the results of the research on developing prospective items are discussed in our paper [36].
corresponding development process.
In this regard, this paper is a continuation of our previous study [35] in which we reviewed the relevant literature and clarified prospective items in the development of the security-quality metrics discussed herein.
We considered the need for a methodology that would allow IoT vendors to tailor security-quality metrics in addition to existing quality metrics for their products. This would, in turn, indicate to consumers the level of security quality of the IoT devices they develop. We attempted to derive quality metrics for IoT devices based on the literature and perspectives reviewed in our previous work. We referenced the goal/question/metric (GQM) methodology commonly used in the quality field.
The following section is an overview of the previous study [35].

Past Research on IoT Security Quality
This section summarizes our previous study conducted, in which we reviewed the relevant literature and clarified prospective items that could be used in the development of the security-quality metrics. We conducted the works in Steps 1 and 2 using the research method described in Figure 1; the results of the research on developing prospective items are discussed in our paper [36].

Research Method
We conducted a systematic literature review [36] using a combination of keywords such as "IoT", "security", and "quality metrics" to find related work. The systematic literature review, with respect to the Wohlin guidelines, was conducted in three steps: (1) Planning the literature review; (2) Conducting the review; and (3) Reporting the review.
In addition, the survey methodology adopted by The European Union Agency for Cybersecurity (ENISA) was adopted as the reference model [37]. This research method starts with a literature survey. The proposal then follows and is succeeded by proof of the effectiveness of the proposal, as shown in Figure 1. Many security guidelines are formulated on the basis of similar sequences: screening the literature in the relevant fields, selecting items that fulfill the objectives of the guideline(s) to be developed, and reviewing the draft(s)-by experts and the public-before finalizing the guideline(s). In fact, ENISA's Baseline Security Recommendation for IoT [37] includes items from the National Institute of Standards and Technology (NIST) Cybersecurity Framework v1.1 [38] and the GSMA IoT Security Guidelines [39]. For example, there is a section about threat analysis that is cited in many studies. Therefore, this research method involving a literature survey is well-suited to this study.

Research Method
We conducted a systematic literature review [36] using a combination of keywords such as "IoT", "security", and "quality metrics" to find related work. The systematic literature review, with respect to the Wohlin guidelines, was conducted in three steps: (1) Planning the literature review; (2) Conducting the review; and (3) Reporting the review.
In addition, the survey methodology adopted by The European Union Agency for Cybersecurity (ENISA) was adopted as the reference model [37]. This research method starts with a literature survey. The proposal then follows and is succeeded by proof of the effectiveness of the proposal, as shown in Figure 1.
Many security guidelines are formulated on the basis of similar sequences: screening the literature in the relevant fields, selecting items that fulfill the objectives of the guideline(s) to be developed, and reviewing the draft(s)-by experts and the public-before finalizing the guideline(s). In fact, ENISA's Baseline Security Recommendation for IoT [37] includes items from the National Institute of Standards and Technology (NIST) Cybersecurity Framework v1.1 [38] and the GSMA IoT Security Guidelines [39]. For example, there is a section about threat analysis that is cited in many studies. Therefore, this research method involving a literature survey is well-suited to this study.

Definition of Scope (Step 1)
To commence the research, we defined the scope, as illustrated in Step 1 in Figure 1 of the research method. An IoT system is complex and comprises many IoT devices and cloud services. Therefore, to simplify the discussion, we focused on IoT devices that are primarily intended for consumer usage. The cybersecurity of cloud services is handled in the ICT (software) industry; there is no such culture as far as hardware is concerned. Historically, most electronic-device vendors are familiar with the physical or electrical safety aspects of quality, but few have ever connected a device to the Internet, a cyberspace fraught with malicious attacks. Consequently, the cybersecurity of IoT devices requires urgent attention.

Results of Literature Review (Step 2)
As mentioned earlier, we conducted a systematic literature review, as described Step2 in Figure 1, to screen for other researchers with similar research interests. However, we could not find any studies or standards from the systematic review.
The International Standard for Software Quality (SQuaRE, ISO/IEC 25000 series) places "security" (one of the subcategories of functionality) as a quality category for system software. SQuaRE lists "security" as a major nonfunctional requirement in terms of system safety. However, these standards only highlight ideas at the conceptual level together with examples to be considered. Although some ideas and items can be used as references, none of the security-quality-control items are clearly elaborated.
In the security evaluation based on GQM, Abdulrazig et al. [40] discussed the misuse of Web applications. However, this should not be misconstrued as a discussion on the security of IoT devices. Further, Yahya et al. [41] discussed the security assessment of cloud storage. Thus, it is worthwhile to define IoT security-quality metrics based on GQM that IoT vendors could use.
In this study, potential security-quality metrics for IoT devices were selected from the literature for each phase of the previously studied product lifecycles. Quality-control practices were then defined to reflect the opinions of security and quality experts on the parameters that should be considered from the perspective of IoT device users.

Itemizing IoT Security-Quality Metrics
Our literature review found no specific work on IoT quality from a security perspective. Therefore, before discussing IoT security-quality metrics, in Step 3 in Figure 1, it is necessary to define the IoT security quality.

Definition of IoT Security
Because the IoT system consists of electronic devices, it is composed of a hardware device consisting of electronic circuits, sensors, and occasionally actuators, as well as software that controls the functions of the electronic device. Consequently, the performance of every product to verify the quality cannot be comprehensively evaluated. Therefore, it is common practice to guarantee the quality of all products by ensuring that all of the predefined development and production processes conform to the required standards; thus, an assessment of the performance of samples alone is sufficient. Essentially, the collective quality should comprise both process and product quality.
Thus, the quality of cybersecurity of an IoT device may be defined as a combination of the quality of the product-development process and that of the cybersecurity performance of the product.
To outline the security development process, items indicating how to design, build, and support the product should be listed. These items include the results of the process review and the maintenance program based on the development lifecycle. Further, to outline the cybersecurity performance of the product, the results of the security assessment should be listed. To demonstrate IoT cybersecurity performance (mostly in software), the said items should reflect the static and/or dynamic security testing of IoT devices.

Requirements of IoT Security Quality
Before defining the aforementioned items, it is necessary to clarify the goals and the aspects to screen for.
First, the items must describe the development process in security transparently (for instance, the security policy of an IoT vendor and the organization's standardized security development process).
To accurately describe the product quality, items properly describing the cybersecurity capability are also required. The results of the product security evaluation should be listed. Importantly, the items should include those responding to market needs as well as those complying with international standards and guidelines.
A crucial source of consumer feedback is aftersales support. The security support program should partly comprise product cybersecurity quality. Activities such as security monitoring, receiving vulnerability feedback, and rolling out updates should be listed as items.
Furthermore, items should be easily comprehensible from the consumer's perspective; this is important for gaining the user's trust.
Thus, the requirements of IoT security quality are summarized in Table 1.

Transparency Model of IoT Security Quality
To comprehensively identify items of IoT security quality, we defined a transparency model of IoT security quality that describes the nature of items as presented in Figure 2prior to Step 3-by integrating the definition of IoT security quality and its requirements. This definition of IoT security quality would satisfy the requirement R1, which ensures transparency of the entire product development process.
This model provides a framework for the IoT security-quality metrics. The model is derived by mapping the security development lifecycle, which was released by many organizations such as NIST [42], Microsoft [43], Synopsys [44], and PwC [45], onto the V-shaped product development process that many IoT vendors follow. Nevertheless, by clarifying the relationship between the V-shaped product development process and the security development lifecycle, each of the members involved in IoT product development will know which security-quality metrics they should be responsible for.
When considering IoT security-quality metric items, this novel model not only allows each metric to be assigned to the appropriate area of responsibility but also makes it easier to determine the areas efficient to implement in the future as new requirements emerge.
The "transparent" model for IoT security quality is as follows: 1. The "security by design" area comprises two parts, namely, the process quality and corresponding product quality [46]. Those in charge of product planning and those in charge of determining basic specifications are mainly responsible for this area.

2.
The "security assurance assessment" area involves the evaluation results. Those in charge of product development and those in charge of quality assurance are responsible for this area. 3.
The "security production" phase entails the items of security management during production. Those in charge of manufacturing the product are responsible for this area. 4.
The "security operation" phase encompasses aftersales security monitoring and response to incidents. Those in charge of customer support, maintenance, and product security incident response team (PSIRT) are responsible for this area. 5.
The "compliance with law, regulation, international standard" area implies that the public or industry requirements have been fulfilled. Compliance with industry standards and regulations is relevant to all areas. All members, not just the product manager, are responsible for this area.
Based on this model, perspectives that should be regarded as the state of IoT securityfrequently alluded to in the literature survey-are listed.
IoT 2021, 2, FOR PEER REVIEW 7 2. The "security assurance assessment" area involves the evaluation results. Those in charge of product development and those in charge of quality assurance are responsible for this area. 3. The "security production" phase entails the items of security management during production. Those in charge of manufacturing the product are responsible for this area. 4. The "security operation" phase encompasses aftersales security monitoring and response to incidents. Those in charge of customer support, maintenance, and product security incident response team (PSIRT) are responsible for this area. 5. The "compliance with law, regulation, international standard" area implies that the public or industry requirements have been fulfilled. Compliance with industry standards and regulations is relevant to all areas. All members, not just the product manager, are responsible for this area. Based on this model, perspectives that should be regarded as the state of IoT security-frequently alluded to in the literature survey-are listed.

Draft Proposal Development of Security-Quality Metrics
Based on the transparency model ( Figure 2), IoT security-quality metrics items were examined, and a draft proposal was subsequently developed (Step 3 in the research method).

Software Quality
Because the functionality and security are controlled by software, perspectives of the software quality were referenced [42][43][44][45].
Software-quality control has traditionally been a challenge because an established method for assessing software quality did not exist. Previously, attempts such as visualization by using a bug curve and the number of defects identified have been tried as a method for quantifying software quality. On the other hand, in terms of software reliabil-

Draft Proposal Development of Security-Quality Metrics
Based on the transparency model ( Figure 2), IoT security-quality metrics items were examined, and a draft proposal was subsequently developed (Step 3 in the research method).

Software Quality
Because the functionality and security are controlled by software, perspectives of the software quality were referenced [42][43][44][45].
Software-quality control has traditionally been a challenge because an established method for assessing software quality did not exist. Previously, attempts such as visualization by using a bug curve and the number of defects identified have been tried as a method for quantifying software quality. On the other hand, in terms of software reliability, some studies observed that consistency, availability, and maintainability (less downtime) resulted in improved quality. However, when the security perspective is considered, the quality of the product appears to depend on transparency.

GQM Method
The GQM paradigm [47] is a three-tier measurement framework and modeling method in software engineering; the first, second, and third tiers represent the goal, question, and metric, respectively.
Metrics are constructed with reference to the "GQM" method in terms of what to achieve (goal), what to evaluate to achieve the goal (question), and what to use as the evaluation method (metric).

Setting Goals for Each Area
Based on the IoT security-quality requirements discussed in Section 4.2, the goals for each area of the transparency model were set, as listed in Table 2. Table 2. Goals for each area of transparency model.

1-A. Security by Design (Corporate Policy & Development Process Standard)
G1A-1: To provide secure products which gain the trust of customers.
G1A-2: To define the corporate standard of secure development processes so that all products provided can be manufactured with security throughout the product lifecycle.

1-B. Security by Design (Security measures, Secure Development)
G1B: To develop secure products based on the defined development standard from the planning stage of the product lifecycle.
2. Security Assurance Assessment G2: To evaluate and confirm that secure products are developed as designed.

Security Production
G3-1: To carry out production with a secure production operating system to avoid containing security risks.

Security Operation
G4: To take prompt actions to minimize the damage to customers when a security risk becomes apparent in the provided product.

Compliance with Law, Regulation, and International Standard
G5: To provide products complying with laws, regulations, and international standards of the destination market.
We set high-level goals for each area/phase. Furthermore, we excluded productspecific indicators from the goals because these would vary for each product and industry.
The Security by Design area under Area 1 is subdivided into two main areas. The goal of 1-A is to establish a basic policy for providing a secure product that would earn the trust of consumers and define a basic process for implementing the policy. This allows users to trust in management's commitment to developing secure products. G1A-1 corresponds to the first aspect of R1 and R5 of Table 1. G1A-2 corresponds to the second aspect of R1.
The goal of 1-B, G1B, is to develop a product that considers security throughout the lifecycle of the product in accordance with the basic policy and process in 1-A.
The goal of Area 2, G2, is to ensure that the product developed in Area 1-B is secure, as designed.
Area 3 is a perspective specific to IoT and is absent from general software development. Because IoT products consist of both software and hardware, they are assembly-processed similar to software products. The production process entails such actions as physical assembly, serial number labeling, and the setting of device-specific IDs and passwords for security. In certain cases, the hardware components required for production may be procured externally and assembled manually. Thus, during production, after verifying the security of the product, supplementary actions are taken to finalize the product prior to shipping to the market. There are security risks involved in this process, and the goal is to eliminate or reduce those risks in this area. Each of G1B, G2, G3-1, and G3-2 correspond to the requirement of R2.
The objective of Area 4 is to provide a unique security response that is different from traditional quality assurance. Traditional quality assurance operates such that if a product performs to a certain standard, it is shipped. However, unless a product that does not meet the standard is found in the marketplace (i.e., unless the personnel are notified of a problem by users), the quality assurance personnel do not check and monitor the status of the products in the market themselves. On the other hand, in the world of security, even with the best efforts to develop a secure product, the level of security perceived to be secure is changing every day as attack techniques constantly evolve. Security risks will gradually increase from the time the product is shipped. It is therefore necessary to monitor changes in the circumstances surrounding the product even after it is shipped. Accordingly, the goal, G4, is to have a response system in place to check and correct any product-related security issues discovered and be ready to respond at any time. G4 corresponds to the requirement of R4.
In the traditional approach, if a quality issue occurs after shipment, the cause of the problem may be identified and addressed. In contemporary scenarios, however, a security problem is different from traditional quality assurance because these problems are manifested by a malicious attack and must be dealt with via nonconventional means.
The goal of Area 5, G5, is to comply with the IoT security laws and regulations with which increasing numbers of countries and regions have been demanding conformity in recent years. In some cases, product sector-specific guidelines are provided in some markets; they are required as industry standards to comply. Although this objective should naturally be considered at the design stage, its content is subdivided into different areas. This is because security-related laws and regulations have recently come into force, and the requirements are related to the entire product lifecycle. Significant regional differences also exist. G5 corresponds to the requirements of R3-1 and R3-2.

Setting Sample Questions and Metrics for Each Goals (Step 3-4)
Based on the GQM method mentioned above, the questions and metrics were formulated for each area from the perspectives clarified in the previous research on the aspects of "What do you want to know about IoT security quality?" and "How do you want to make sure?".
From the perspective of IoT consumers, the questions are intended to reveal what security measures are being taken and how secure the products being made are.
From the perspective of the IoT vendors, identifying what to do and when to do it in the development process is necessary.
The metrics were devised with the following points in mind: (a) Do the metrics make sense to IoT vendors? (b) What are the criteria for the metrics? (c) Will they interfere with the existing development process?
For (a), clarifying the reason for performing metrics makes it easy to understand. For (b), the metrics are formulated based on "what and when", whereas for (c), the metrics are clear and can be incorporated into existing design processes.
First, the primary questions were listed. The secondary questions were then used to set up a more specific perspective and provide supplementary confirmation.
The metrics at this stage are set as simple assessments, such as the presence or absence of documentation and whether assessments were performed.
The reason for a simple evaluation is that a clear basis or objective indicator to classify the content of each response does not exist. When the organization is sufficiently mature to implement advanced initiatives, these questions and metrics sets are likely to evolve into an advanced form of evaluation by establishing complex questions and metrics with approximately three levels, such as well done, partially done, and nothing done.
The quality and security experts then review and evaluate the validity of the draft questions and metrics in Step 4 of the research method to refine the list of questions and metrics.
The details of the approach to set the IoT security-quality metrics and the results of the examination of the questions and metrics for each area are presented in Appendix A. However, the results of this study are based on the elimination of field-specific product perspectives as much as possible. As such, these results should be considered an example of questions and metrics for IoT in general. If there is a field-specific item necessary to assess, it can be modified to be field-specific by adding such field-specific questions and metrics.

Evaluation of Various IoT Security Guidelines with the Sample Metrics
In this section, the characteristics of the requirements presented in IoT security regulations, guidelines, and certification programs are examined. This examination is based on the common security-quality metrics for IoT products reviewed by the quality and security experts as a part of Step 5 in the research method to examine the effectiveness of the metrics. The results are presented in the form of bar charts, respectively. Table A7 in Appendix B shows the source of references examined for evaluating the effectiveness of the IoT security-quality metrics of this study.

Regulations
The following four regulations are compared with the IoT security-quality metrics: Terminal Conformity Regulation under Telecommunications Business Law by Ministry of Internal Affairs and Communications of Japan [48], California Senate Bill No. 327 [49], Oregon House Bill 2395 [50], and the consultation on regulatory proposals on consumer IoT security of the UK [51]. Figure 3 illustrates the area of the transparency model under which each regulatory requirement falls. The percentages on the vertical scale indicate the ratio between the number of requirements of each regulation that correspond with the IoT security-quality metrics items and the total number of IoT security-quality metrics items in each area. The number on the horizontal axis is the total number of metrics set for each area. Thus, the percentage for each area is the ratio of the number of metrics matched to the total number of metrics. The same is the case for Figures 4 and 5, shown below.
The requirements of these regulations are minimal, as can be observed in Figure 3. It is also clear that Area 1-A of vendor attitude (such as security policy) or Area 3 of assessment (such as vulnerability assessment) are not required. Therefore, the IoT security-quality metrics cover the range of regulatory requirements sufficiently well to ensure compliance.
As observed in Figure 3, all these regulations focus on Area 1-B and Area 3. The UK regulation requires a security operation after product sales. The requirements of these regulations are minimal, as can be observed in Figure 3. It is also clear that Area 1-A of vendor attitude (such as security policy) or Area 3 of assessment (such as vulnerability assessment) are not required. Therefore, the IoT security-quality metrics cover the range of regulatory requirements sufficiently well to ensure compliance.
As observed in Figure 3, all these regulations focus on Area 1-B and Area 3. The UK regulation requires a security operation after product sales.

Baseline Guidance
The following four standards and guidelines from the United States and Europe that are presented as baselines are examined here. These are NISTIR 8259 [52] and 8259A [53] and C2 Consensus on IoT Device Security Baseline Capabilities [54] of the US, and Baseline by ENISA [37] and ETSI EN 303 645: Cyber Security for Consumer Internet of Things: Baseline Requirements [55]. Figure 4 describes the results of the area of the transparency model that each baseline requirement fits into. The vertical scale indicates the same units as that in Figure 3. Values over 100% indicate that there are a greater number of requirements than the total number of IoT security-quality metrics items in each area.
The distributions of the two standards from the US are similar, and the trend of the requirements can be considered to follow the same direction. Certain functional requirements for devices that were not set in the IoT security-quality metrics were found in these two US standards.
The approaches taken by the two European guidelines differ from that of the "baseline". The ENISA baseline offers broad coverage, and many requirements appeared in all areas. In particular, the security function requirements in the area of security-by-design are very extensive, and hence incomparable to the sample metrics. On the other hand, the distribution of ETSI requirements resembles that of the two from the US, and the approach to baselines is also considered close.

Baseline Guidance
The following four standards and guidelines from the United States and Europe that are presented as baselines are examined here. These are NISTIR 8259 [52] and 8259A [53] and C2 Consensus on IoT Device Security Baseline Capabilities [54] of the US, and Baseline by ENISA [37] and ETSI EN 303 645: Cyber Security for Consumer Internet of Things: Baseline Requirements [55]. Figure 4 describes the results of the area of the transparency model that each baseline requirement fits into. The vertical scale indicates the same units as that in Figure 3. Values over 100% indicate that there are a greater number of requirements than the total number of IoT security-quality metrics items in each area.
The distributions of the two standards from the US are similar, and the trend of the requirements can be considered to follow the same direction. Certain functional requirements for devices that were not set in the IoT security-quality metrics were found in these two US standards.
The approaches taken by the two European guidelines differ from that of the "baseline". The ENISA baseline offers broad coverage, and many requirements appeared in all areas. In particular, the security function requirements in the area of security-by-design are very extensive, and hence incomparable to the sample metrics. On the other hand, the distribution of ETSI requirements resembles that of the two from the US, and the approach to baselines is also considered close.

Certifications
Several private IoT security certification programs have been released in the market. We examined the following four sets of requirements. The first is from the certification program of CCDS [56] in Japan, and the second is from the ioXt alliance [57] in the US. Finally, we analyzed the two different grades (Bronze and Diamond) of the IoT Security Rating of UL [58], also in the US.

Certifications
Several private IoT security certification programs have been released in the market. We examined the following four sets of requirements. The first is from the certification program of CCDS [56] in Japan, and the second is from the ioXt alliance [57] in the US. Finally, we analyzed the two different grades (Bronze and Diamond) of the IoT Security Rating of UL [58], also in the US.
The result for the area of the transparency model to which each certification requirement belongs is described in Figure 5. The vertical scale represents similar concepts as those in Figures 3 and 4, and the meaning of the values that are greater 100% is also the same.
Except for the requirements of UL Diamond, the other programs have a similar number of requirements, and these are covered (i.e., they are below the 100% line) by the metrics.
We also found that the requirements in the security functions of Area 1-B of UL Diamond are quite strict compared to the same level of ENISA baseline requirements. This implies that the ENISA baseline requirements are a very high-level set of requirements, despite being baselines.  The result for the area of the transparency model to which each certification requirement belongs is described in Figure 5. The vertical scale represents similar concepts as those in Figures 3 and 4, and the meaning of the values that are greater 100% is also the same.
Except for the requirements of UL Diamond, the other programs have a similar number of requirements, and these are covered (i.e., they are below the 100% line) by the metrics.
We also found that the requirements in the security functions of Area 1-B of UL Diamond are quite strict compared to the same level of ENISA baseline requirements. This implies that the ENISA baseline requirements are a very high-level set of requirements, despite being baselines.

Evaluation of IoT Devices with the Proposed Method
We evaluated the IoT devices by the proposed method. We selected two commercial dashcam recorders (Product A and B) with almost the same functional product specifications as IoT devices. These are products provided by Original Design Manufacturing (ODM) vendors.

Evaluation of IoT Devices with the Proposed Method
We evaluated the IoT devices by the proposed method. We selected two commercial dashcam recorders (Product A and B) with almost the same functional product specifications as IoT devices. These are products provided by Original Design Manufacturing (ODM) vendors.

Target IoT Devices
The two products are similar in the following aspects:

•
They are consumer products that can be purchased online and in stores. Easy to install and start using by powering from a cigar socket. • Downloadable applications for smartphones and PCs that can be connected to and functionally linked with a dashcam.
As mentioned above, the two dashcams are very similar in terms of functionality. The only differences observed from the specification are the following points.

•
Product design: shape and color. • Price: Product A is cheaper than Product B.
Simply speaking, as they are almost the same in terms of functionality, most users will choose Product A because of the price difference, unless they prefer the design of Product B too much.
However, as a user, the following points not readily apparent from the functional specifications are of concern. The points are the policy for handling personal information such as recorded image information, GPS information, information about the user, and the access restriction function for connection functions.

Evaluation with the Proposed Method
As far as we were able to confirm through interviews with ODMs, we evaluated and compared the security perspective with the metrics of the proposed method.
The results of the evaluation are shown in Figure 6.

Evaluation with the Proposed Method
As far as we were able to confirm through interviews with ODMs, we evaluated and compared the security perspective with the metrics of the proposed method.
The results of the evaluation are shown in Figure 6. The comparison results show that Product B has more product security measures in all areas than Product A, and we can infer that the security quality of Product B is better. This difference is probably reflected in the price difference.
Both companies had policies for handling personal information. However, there was The comparison results show that Product B has more product security measures in all areas than Product A, and we can infer that the security quality of Product B is better. This difference is probably reflected in the price difference.
Both companies had policies for handling personal information. However, there was a difference in the authentication of the Wi-Fi connection: Product A had the factory default access point name as the product name and no password (blank). Product B, on the other hand, had the same factory setting with the product name as the access point name, but the password was set to be unique for each device. This perspective is critical, as the requirements affect compliance with California law. Although the specification that anyone can use the device immediately without a password is appealing, the default setting that only the purchaser of the device can access is safer. Even in Product B, we found that there are few efforts in Area 2 and Area 4. The security-conscious IoT vendor of Product B has yet to demonstrate security verification or postshipment support.

Evaluation Result
The proposed method demonstrated that it could illustrate the differences in the security quality of IoT devices. In addition, it could increase the transparency of the security quality of IoT devices.
It is not easy for general users at present to make this kind of comparative evaluation since they have only the product specifications released by IoT vendors to judge.
However, IoT vendors will want to appeal to users the security measures they have invested in during the development and maintenance of IoT devices. At that time, this method can be a tool to support improvement and raise the level of security measures, since it visualizes areas where security measures are lacking.

Discussion of Findings
As described in Section 3, the IoT security-quality metrics are examined from a product lifecycle perspective; quality items are articulated in a manner inspired by GQM methods common in the quality industry, and the metrics that were reviewed by quality and security experts are produced.
Originally, this methodology was designed to help IoT vendors to produce their own IoT security-quality metrics. However, it has also proved to be a useful tool for developing categories of guidelines and certification programs to understand which requirements are lacking or missing from the product lifecycle. In practice, international standards by themselves are insufficient for practical implementation; hence, it is necessary to tailor the contents of international standards to suit the development target, development process, organization, and environment. As discussed previously [59], GQM is used for this tailoring. In the future, as IoT vendors' product security efforts advance, improvements will be required; the validating GQM (V-GQM) is proposed as a method for reviewing or improving each element of GQM [60]. The review or improvement should occur as soon as the values of the metrics are collected. The use of such a method is expected to make it easier to implement reviews and improvements.
As mentioned in Section 4, all requirements are not distributed evenly throughout the product lifecycle. All regulations are focused on Areas 1-B and 3, whereas only the UK focuses on the maintenance phase of Area 4. Additionally, ENISA suggests incorporating the items in all areas (especially items on high demand) into the policy, process, and security functions at the design phase. Other baselines focus on security functions and operations rather than the level of ENISA. Most certifications focus not only on security functions but also on security assurances.
This approach was also useful in helping IoT vendors to understand how regulatory requirements, guidelines, and certification program requirements are distributed across the product lifecycle and where they are focused. The results of Section 5 reveal that not all sets of requirements are the same and that there are differences in requirements. This may help IoT vendors tailor their IoT security-quality metrics according to the particular requirements specified by consumers. If any deficiencies are found, IoT vendors can make improvements by eliminating them from the quality metrics to meet the securityquality objectives early in the lifecycle of the product being developed, thereby saving time and effort.
In addition, we believe that this method will also serve as an indicator of the product security standard for consumers. From the results of Section 6, we also verified that this method could illustrate that there is a clear difference in security quality, which is difficult to indicate in the difference in product features in functional specifications. To date, a way to communicate the quality of IoT security has not existed. Nevertheless, we foresee that this novel approach will become a quality communication tool between product vendors and consumers.
A limitation of the proposed method is that it is conceived from a framework that assumes a conventional V-shaped development model. Therefore, we have not been able to evaluate its applicability to recent development methods such as an agile development [61,62] and DevOps [63,64].

Future Directions
There are three related areas that we would like to pursue in future work. The first possibility is to categorize the metrics that show the countermeasure capabilities of IoT products and those that demonstrate the efforts of IoT vendors. The current metrics belong to either or both of these areas. We plan to examine how to categorize metrics to easily distinguish between the quality of security in IoT products and the quality of the IoT management process at a glance. The second area that may warrant further research involves investigating methods to visualize the coverage of metrics. Herein, the authors selected the bar chart for this purpose; however, comparatively simple methods for visualizing the coverage may be available. Thirdly, although we believe that IoT vendors can adopt this proposed method because of their knowledge of the product manufacturing culture of electronic-device vendors, we also consider it necessary to ask several IoT vendors to adopt this method to verify whether their IoT products are developed securely. In addition, when security support by IoT vendors becomes common practice, and security threats are evolving day by day, we will need to add and refine the basic set of metrics in detail and would need to consider its refinement cycle in the future.

Conclusions
This study proposes a method for tailoring security-quality metrics for IoT devices to ensure the quality of IoT security, and the method demonstrates the validity to evaluate the characteristics of the emerging requirements and suggestions of relevant laws, regulations, guidelines, and certification programs in IoT security based on the produced metrics. In addition, the proposed method demonstrates its capability to reveal the difference of security quality behind the product functional specification of IoT devices.
The authors developed the six areas of the transparency model of IoT security quality to outline the entire product lifecycle with reference to the GQM methodology. The draft was reviewed by quality and security experts who reflected upon the findings and incorporated them into the final set of metrics.
To validate the metrics, they were analyzed against the requirements of various IoT security regulations, guidelines, and certification programs. Most of the regulations, guidelines, and certification programs only describe what needs to be actioned without designating an entity to put into practice and without clarifying the purpose of action to perform; this results in ambiguity over the extent of duty and may lead to nothing being accomplished. With this metric, however, the rationale is clear from the GQM. Thus, it is easier for the entity or IoT vendors to self-assess the security-quality metrics items needed for their security goal.
Although many guidelines are available for the development of secure software, no practical framework follows the lifecycle of a hardware-oriented product that is easy for IoT 2021, 2 776 device vendors to understand. The presentation of the metrics for each area as a framework enables IoT vendors to easily incorporate security initiatives into their existing processes.
The proposed method and the metric for IoT security quality were examined and found to be useful for (1) developing categories of guidelines and a set of requirements to review the balance of items throughout the product lifecycle, (2) enabling IoT vendors to determine the focal areas of guidelines and a set of requirements to meet, and (3) enabling consumers to use these IoT product security-quality metrics to communicate the security risks to the product vendors.

Conflicts of Interest:
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A
The process of how the IoT security-quality metrics were set and the results of the examination of questions and metrics for each area are described.

Appendix A.1 Area 1: Security by Design
To satisfy the goals of Area 1, we consider what to clarify, and how to clarify it. Area 1 covers the product lifecycle from policy to development.
To achieve the goal of Area 1-A in Table 2, the questions for Area 1-A were set as basics to assess whether the IoT vendors consider security support important [38,65].
We set two secondary questions to make the question more specific. The first inquired whether there is a policy in place stating that the management's commitment to security response is considered important. Because security responses require monetary investments, many guidelines recommend that security responses should be publicly stated as a management policy. The other question sought to confirm whether a secure development process was defined, and the environment was ready for all products to be secured using the same process as opposed to an ill-conceived security response. The questions and metrics for Area 1-A are listed in Table A1. Table A1. Questions and metrics for Area 1-A.

Sub-Question Metrics
Does the company recognize the importance of handling product security?
Does the company have a product-security policy?
It is documented. = 1 There is no policy defined. = 0 Is the product-security-development process defined?
It is documented. = 1 There is no process defined. = 0 Neither the quality experts nor the security experts raised any specific objections for these two questions and metrics. The quality experts stated that the same was true for clarifying the product security response as well, because it was important for the management to present the policy to make it an enterprise-wide effort in promoting product safety response.
In Area 1-B, the questions and sub-questions were set up to check whether the basic actions to be performed in the security development process were included [39, 66,67].
The questions and metrics are set up to check which security response items should be performed at the appropriate timing. With respect to these items, it should be clear what to do, when to do it, and under what conditions. For example, if engineers design a security countermeasure without conducting a threat analysis, they end up having to perform threat analysis to justify the developed countermeasure in terms of need and priority, which results in unnecessary work. A set of questions relates to the threats that IoT products may face and the risks that may arise from those threats. As a basic security response, it is necessary to recognize what threats exist to the target to be protected and the risks that may arise from those threats. In many cases, IoT device vendors consider that they implement security measures by setting IDs and passwords without recognizing these threats. It is then necessary to check whether appropriate security measures are selected to eliminate the threats that cause risks that should not occur. It is also important to identify the list of threats not to take into account, as it is impossible to prevent all threats within the limited development time and cost. In addition, the policy and protection measures for the acquisition and use of personal information, which is of great interest to the general public, are also important aspects. The questions and evaluation criteria included clarifying the software configuration of the IoT product in preparation for postlaunch security monitoring.
Neither the quality nor the security experts expressed any specific objection to these questions and metrics. However, the quality experts had certain concerns regarding the challenges in designing the software coding protocols and integrating the components included in the software into the metrics, given that this is a novel undertaking. The questions and metrics for Area 1-B are listed in Table A2. Table A2. Questions and metrics for Area 1-B.

Sub-Question Metrics
Is security considered from the planning/design stage?
Is threat analysis performed? There is an analysis result. = 1 It is not performed, or no result. = 0 Is risk assessment based on threat analysis performed?
There is an assessment result. = 1 It is not performed, or no result. = 0 Are threats selected for countermeasures based on risk assessment and risk mitigation countermeasure design implemented?
There is a list of threats to be protected. = 1 There is no list of threats to be treated. = 0 There is a security countermeasure design document. = 1 There is no countermeasure design. = 0 Is the threat excluded from countermeasures clear?
There is a list of accepted threats. = 1 There is no list of accepted threats. = 0 Are the methods for reducing threats excluded from countermeasures and alerts described in manuals, etc.?
There is a document for users. = 1 There is no document. = 0 Is the handling of personal information taken into consideration?
There is a personal information list to handle.

Question Sub-Question Metrics
Is the adopted outsourced software clear?
Vendor Is there a deactivation function or a fallback operation function when the security maintenance service ends?
There is a function. = 1 There is no function. = 0 Is the IoT product designed with consideration of disposal?
Is there a function to delete user data for disposal?
There is a function. = 1 There is no function. = 0 Appendix A.2 Area 2: Security Assurance Assessment In this area, the questions and sub-questions were set to ensure that the development process was implemented properly. For example, the questions sought to establish whether the software on the IoT devices was free of vulnerabilities, whether communication ports and connectors that are not normally used for development had been removed, whether software developed outside the company had been inspected prior to acceptance, and to determine the security level of cloud services with which the IoT products are connected [68,69].
One security expert pointed out that recent attack methods tended to find vulnerabilities through hardware analysis; the JTAG and UART, which are connection ports left on boards by vendors for flaw analysis, are commonly targeted. Therefore, it is important to verify that these ports are eliminated during production. Even the quality experts understood the reason for the removal. However, they were hesitant to make the removal mandatory because these connection ports were necessary for error analysis. Eventually, we decided to establish a blockade that requires connection authentication instead of eliminating these ports.
The questions and metrics for Area 2 are listed in Table A3. As in Area 1-A, because various evaluation methods are available, the methods suited to individual IoT products are different. Therefore, it is not meaningful to include specific methods in the question list until a common understanding in the industry is fostered.

Question Sub-Question Metrics
Is the product evaluated to ensure it is secure as designed?
Does the source code violate secure coding rules?
There are assessment results that comply with the rules. There is a contract (SLA clause) in place and confirmed. = 1 There is no confirmation. = 0

Appendix A.3 Area 3: Security Production
Area 3 is part of the production-process check that is specific to IoT products. The peculiarity of this part of the IoT production process is that the responsibility for this part exists not with the development or quality assurance department, but with the factory. There is no appropriate reference found regarding this aspect.
Even if it is confirmed that the IoT product has been developed into a secure product through Area 1-B and Area 2, it will not ultimately become a secure product unless proper production controls are established during the production phase. For example, if you were to set different passwords for individual IoT devices but accidentally ship them with the same password, even if only one of the devices is attacked, the other devices will still be affected; but the attack on the other devices can be prevented. Therefore, even if the product is designed to be secure, it will not be produced as a secure product unless proper controls are applied.
In addition, factory production systems have been under attack recently. In many cases, the systems that manage and control production lines were attacked and forced to shut down the lines. From the perspective of product supply continuity, factory production systems were also included in the scope of the study. The security of a production system of a factory is not the IoT product itself, and therefore the questions and metrics on the factory system management are unique. However, consumers' trust in the vendors of IoT products would certainly increase if the products are produced in factories that are safe from cyberattacks.
There was no specific objection or concern raised by either the quality or security experts against the questions and metrics. The questions and metrics for Area 3 are listed in Table A4. Table A4. Questions and metrics for Area 3.

Question Sub-Question Metrics
Is the product produced in a secure manufacturing process?
Is the identity of the line manager verified for in-house production?
All employees are identified. = 1 Not all of the person in the factory are identified. = 0 There is a record of the access control to the production site. = 1 There is no record of access control. = 0 Has the ODM producer's manufacturing process been verified?
Company name and country of production are confirmed. = 1 It is hard to confirm who manufactures. = 0 The results of the production process audit are confirmed. = 1 There is no confirmation. = 0 Is production under control to be produced with genuine parts?
Certificates of authorized parts are verified. = 1 There is no confirmation, = 0 Is the production process capable of setting each device with unique IDs and passwords?
It is capable of setting unique IDs and passwords to each device. = 1 It is not capable. = 0 Is there security measure in place for the production system?
Is it possible to detect cyberattacks such as malware infiltration, virus infections and others on production systems?
It is capable of attack detection. = 1 It is not capable. = 0 Are security measures in place for production systems?
Security measures to the production system are in place. = 1 There is no security countermeasure on the production system. = 0 Is coordination in place with CSIRT for incident response?
CSIRT is cooperating for factory incident. = 1 There is no incident response readiness. = 0

Appendix A.4 Area 4: Security Operation
The questions and metrics pertaining to Areas 1-3 relate to confirmation of the effort involved before launching IoT products into the market. On the other hand, those of Area 4 relate to the postmarketing stage. These questions and metrics are intended to ensure that a system is in place to provide security support for the IoT products being utilized in the market. For example, the questions and metrics sought to establish whether the company monitors vulnerability information about software components in IoT products, whether it has a defined process and members to respond to security incidents upon discovery, whether it has an in-house information management system, and the procedure to carry out when security support ends. The confirmation of the implementation of functions that enable the security support required for IoT products was also included in this area.
In response to the draft questions and metrics, one security expert pointed out the following issue. When a security issue is uncovered, a thorough investigation into what happened needs to be conducted. Logging is an important part of the investigation, and the need to maintain a log of connections and an activity history was pointed out. There was also a suggestion that the IoT devices themselves need to be able to self-verify the need for software updates; this functionality was added to the pertinent items. The questions and metrics for Area 4 are listed in Table A5.

Sub-Question Metrics
Is there a product security response team for the products in the market?
Is there an operating system to monitor vulnerability information for products? SOC (security operation system) is in place. = 1 There is no system to monitor vulnerability. = 0 Is there an incident response system for products?
PSIRT (product security incident response team) is in place. = 1 There is no response system. = 0 Is the incident response process defined?
The incident response process is documented. = 1 There is no process defined. = 0 Is there a contact point for receiving vulnerability information?
The contact information is publicly available. = 1 There is no contact information. = 0 Is there a personal information handling policy and management system in place?
There are a policy and a management system. = 1 There is no policy and management system. = 0 Is there a system for the stable operation of IoT products? Is there a system monitoring the operational status of the cloud services which IoT products works with?
The cloud operator's contact information is clarified. = 1 There is no means to check the cloud operation. = 0 It is capable of checking the status of cloud operation. = 1 It is not capable of checking the cloud operation. = 0 Is it capable of managing customer information for service in use?
It is capable of managing customer information based on the management rules documented. = 1 It is not capable. = 0 Are restrictions on product security support clearly stated?
Is the warranty period and exemption for security service/maintenance provided?
Security service/maintenance that the company provide is clarified. = 1 It is not clarified. = 0 Appendix A.5 Area 5: Law Regulation and International Standard Area 5 essentially needs to be considered at the product planning stage, as discussed in the goals section. However, according to the literature review, it emerged that the regulations and/or guidelines that need to be complied with may relate to the entire lifecycle of the product. Therefore, Area 5 is set as a separate area.
Depending on the industry sector and destination of the IoT product, the laws and regulations that need to be addressed and the international standards and guidelines that need to be ratified are different; hence, they need to be checked carefully. In particular, laws, regulations, and guidelines for IoT security are still evolving and changing in terms of their content. Thus, it is necessary to stay updated to ensure compliance.
The quality or security experts did not have any specific objection or concern about the questions and metrics. However, the quality experts suggested that it would be easier to convince company management of security initiatives if these were generally accepted by third parties in the form of certification. The questions and metrics for Area 5 are listed in Table A6. Table A6. Questions and metrics for Area 5.

Sub-Question Metrics
Does the product comply with the laws and regulations about the product security of the region to be sold?
Does the product meet legal and regulatory requirements?
There are the evaluation results that meet the requirements. = 1 There is no evaluation result. = 0 Does the product have the required certifications or conformity statements, if necessary?
After confirming the necessity of certification/conformity certificate, the acquisition result can be confirmed. = 1 The need for a certification/conformity certificate has not been confirmed. = 0 Does the product comply with the required international standards?
Does the product have the required certifications or conformity statements, if necessary?
After confirming the necessity of certification/conformity certificate, the acquisition result can be confirmed. = 1 The need for a certification/conformity certificate has not been confirmed. = 0 Does the product comply with private security certification?
Has the product acquired the certification of conformity with the standard that is decided to be required or voluntarily acquired?
After confirming the necessity or voluntary acquiring of certification/conformity certificate, the acquisition result can be confirmed. = 1 The need for a certification/conformity certificate has not been decided. = 0

Appendix B
The sources of references examined for evaluating the effectiveness of the IoT securityquality metrics of this study are shown in Table A7.